Publications
Delivering Unemployment Assistance in Times of Crisis: Scalable Cloud Solutions Can Keep Essential Government Programs Running and Supporting Those in Need. Mintaka Angell, et al. (2020). Digital Government: Research and Practice.Abstract
The COVID-19 public health emergency caused widespread economic shutdown and unemployment. The resulting surge in Unemployment Insurance claims threatened to overwhelm the legacy systems state workforce agencies rely on to collect, process, and pay claims. In Rhode Island, we developed a scalable cloud solution to collect Pandemic Unemployment Assistance claims as part of a new program created under the Coronavirus Aid, Relief and Economic Security Act to extend unemployment benefits to independent contractors and gig-economy workers not covered by traditional Unemployment Insurance. Our new system was developed, tested, and deployed within 10 days following the passage of the Coronavirus Aid, Relief and Economic Security Act, making Rhode Island the first state in the nation to collect, validate, and pay Pandemic Unemployment Assistance claims. A cloud-enhanced interactive voice response system was deployed a week later to handle the corresponding surge in weekly certifications for continuing unemployment benefits. Cloud solutions can augment legacy systems by offloading processes that are more efficiently handled in modern scalable systems, reserving the limited resources of legacy systems for what they were originally designed. This agile use of combined technologies allowed Rhode Island to deliver timely Pandemic Unemployment Assistance benefits with an estimated cost savings of $502,000 (representing a 411% return on investment).
Pre-prints
Adaptive Randomization in Conjoint Survey Experiments. Jennah Gosciak, Daniel Molitor, Ian Lundberg. SocArXiv.Abstract
Human choices are often multi-dimensional. For example, a person deciding which of two immigrants is more worthy of admission to a country might weigh the prospective immigrants’ education, age, country of origin, and employment history. Conjoint experiments have rapidly generated new insight into these multidimensional choices. By independently randomizing the attributes of a pair of fictitious profiles, researchers summarize the average contribution that each attribute makes to an overall choice. But what if the effect of one attribute depends on the values of other attributes? We present a method that uses data-adaptive experimentation to search for heterogeneity in the effect of one focal attribute as a function of all other attributes. Our empirical application of this method shows that U.S. adults weigh the education of an immigrant much more heavily for certain immigrants than for others. By targeting the heterogeneous effects of a focal attribute, our approach complements conjoint designs that target the average effects of all attributes.
Abstract
Contemporary social mobility research often adopts an ostensibly descriptive goal: to document associations between parent and child socioeconomic outcomes and their variation over time and place. To complement descriptive research, we adopt a causal goal: to estimate the degree to which parent occupation causes child occupation. We formalize this causal goal in the potential outcomes framework to define precise counterfactuals. We highlight a difficulty connected to the positivity assumption for causal inference: when the treatment is parent occupation, many counterfactuals never happen in observed data. Parents without college degrees are never employed as physicians, for instance. We show how to select causal estimands involving only the counterfactuals that can be studied with data. We demonstrate our approach using the National Longitudinal Survey of Youth 1979. Our causal approach points to open questions about how specific aspects of family background, such as parent occupation, causally shape the life chances of children.
Abstract
The mismatch between the skills that employers seek and the skills that workers possess will increase substantially as demand for technically skilled workers accelerates. Skill mismatches disproportionately affect low-income workers and those within industries where relative demand growth for technical skills is strongest. As a result, much emphasis is placed on reskilling workers to ease transitions into new careers. However, utilization of training programs may be sub-optimal if workers are uncertain about the returns to their investment in training. While the U.S. spends billions of dollars annually on reskilling programs and unemployment insurance, there are few measures of program effectiveness that workers or government can use to guide training investment and ensure valuable reskilling outcomes. We demonstrate a causal machine learning method for estimating the value-added returns to training programs in Rhode Island, where enrollment increases future quarterly earnings by $605 on average, ranging from -$1,570 to $3,470 for individual programs. In a nationwide survey (N=2,014), workers prefer information on the value-added returns to earnings following training enrollment, establishing the importance of our estimates for guiding training decisions. For every 10% increase in expected earnings, workers are 17.4% more likely to express interest in training. State and local governments can provide this preferred information on value-added returns using our method and existing administrative data.
Works in Progress
Anytime-Valid Inference in Conjoint Experiments. Daniel Molitor.Abstract
Conjoint experiments have become increasingly popular for studying how multiple attributes influence decision-making. However, determining the optimal sample size required to achieve adequate statistical power in conjoint experiments is challenging; conventional power analysis requires many assumptions to hold simultaneously, and can easily under- or over-estimate the necessary sample size. To overcome these limitations, I propose an alternative approach grounded in recent advances in anytime-valid inference. Rather than relying on conventional power analysis, this approach introduces anytime-valid confidence sequences (CSs) and corresponding p-values for key conjoint estimands, including the AMCE, ACIE, and marginal means. These procedures are computationally simple—building on standard regression outputs, guarantee valid Type I error control at any stopping point, and enable practitioners to continuously monitor their empirical estimates and implement data-driven stopping rules once their estimates of interest achieve sufficient statistical power or precision. In simulations calibrated to real-world conjoint studies, I show that this approach preserves nominal coverage, achieves comparable power to standard fixed-n approaches, and yields average sample savings of 10–40% across a broad range of effect sizes, sample sizes, and attribute levels. This approach gives practitioners a principled, efficient way to determine when to stop data collection without relying on pre-specified power analyses.
Abstract
This paper introduces the Modified Mixture Adaptive Design (MADMod), an experimental design for conducting adaptive experiments with multiple treatment arms that addresses both efficiency and robust inference. Building on the Mixture Adaptive Design (MAD) framework, MADMod retains anytime-valid confidence sequences—allowing experiments to conclude dynamically—while systematically reallocating samples to ensure each arm achieves sufficient statistical power. By integrating importance weights that decay once an arm’s average treatment effect is detected as significant, MADMod prevents under-allocation to suboptimal arms without sacrificing the efficiency gains of multi-armed bandits. Simulation results demonstrate that, compared to standard adaptive designs, MADMod substantially reduces Type 2 error rates and offers more precise inference across all treatments. This allows researchers to harness the benefits of adaptive sampling and early stopping while maintaining well-powered evaluations of every treatment arm.