Facing an urgent need to determine whether new coronavirus vaccines were safe, the government relied on a time-honored, rigorous method accepted by scientists for decades: randomized clinical trials.
Oddly, however, the government has rarely used this approach for evaluating its own programs. But that is starting to change.
The government has begun experimenting with multiple randomized trials — involving hundreds of thousands of patients nationwide — of different ways of financing health care. The first of these experiments is already beginning to change national policy.
The value of randomized trials is clear. Consider the coronavirus vaccines. When some people are randomly selected to receive a vaccine and others are not, a simple comparison between the two groups provides unbiased estimates of the vaccine’s effects.
If the vaccine evaluation had taken another route — comparing people who chose whether to get inoculated with those who chose not to — it would have been hard to separate the effects of a vaccine from differences in underlying Covid risks among members of the two groups.
Unfortunately, the government has typically evaluated its own policies with this type of potentially misleading approach, comparing outcomes between those who choose whether or not to participate in a new program.
Occasionally, there have been instances of randomized evaluations embedded in policy design — like job training, unemployment insurance and welfare reform itself. While such randomized tests have not become standard practice, some contributed to major policy changes.
For example, a randomized test of rental assistance programs aimed at helping low-income families relocate from high- to low-poverty areas found that children benefited greatly from such moves by the time they were young adults. About two decades after that experiment began in 1994, the results prompted new policies to empower tens of thousands of low-income families to move to higher-opportunity neighborhoods.
Now, in a series of experiments that could herald a new era in policy evaluation, the government is applying randomized testing to health care financing.
This could have broad implications because the U.S. health care sector is unusually innovative, constantly pioneering new technologies and ways of paying for them. In 2016, almost one-third of spending in the major program known as traditional Medicare — which covers almost 40 million Americans at a taxpayer cost of over $500 billion a year — was based on payment mechanisms created within the last five years.
Such policies can affect what care patients receive and even whether they live or die. Yet the idea of evaluating Medicare payment policy with the clinical trial approach required of new drugs was often deemed impractical. How could the federal government randomly assign hospitals or physicians to different reimbursement rules? Many experts said it couldn’t be done.
Now we know that it can. The government’s Innovation Center — which Congress created in 2011 to develop and evaluate new ways of financing health care — has started five large-scale randomized trials of new Medicare payment rules. These randomized tests represent a minority of the more than two dozen policy evaluations the Innovation Center has underway, but their presence is encouraging.
All of these randomized trials are in progress. The earliest began in 2016; the most recent started in January.
Interim results from one trial indicate that financial incentives improved the quality of services delivered by home health care agencies, and generated savings as well. Based on these results, the government has said, it intends to offer the incentives nationwide.
Sometimes, the tests have shown that a new program doesn’t live up to expectations. In one trial, which I and others are studying, the Innovation Center randomly assigned hospitals in 67 cities — including New York and Pittsburgh — to receive a single Medicare payment covering a hip or knee replacement and three months of follow-up care. In 121 other randomly selected cities — including New Haven and Chicago — hospitals, physicians and rehabilitation centers have been reimbursed separately for any specific care they provide.
The interim findings indicate smaller effects than had been expected, based on previous, non-randomized evaluations. In those, the Innovation Center gave hospitals the option of choosing the new payment method, and then compared outcomes among hospitals that made different choices.
That evaluation method is reminiscent of the traditional approach to assessing new surgical techniques — which have sometimes been shown to be worthless or worse, after being subjected to more rigorous testing. Radical mastectomies, for example, were widely used for decades before randomized trials showed that much less extensive and disfiguring surgery followed by radiation was an equally effective treatment.
Of course, conducting randomized trials can pose more challenges for evaluating a new surgery than a new drug. A surgical technique may be harder to standardize enough to be able to test it on a broad population, and blinding the patient to which treatment she receives can be far more difficult.
But such feasibility issues do not apply to new payment methods, which are well-defined and standardized interventions, and where blinding medical providers to the payment rules is not desired.
Still, as in medicine, not all public policies can — or should — have randomized evaluations. One-of-a-kind government projects — like the “Big Dig” in Boston or the Superconducting Super Collider in Texas — have no natural comparison group, randomized or otherwise. In times of crisis, or when policy disagreements are more about ideology than impact, evaluation itself may be ill advised.
But when — as is often the case — there is an opportunity for prospective evaluation and the law requires it, the Innovation Center’s experience underscores the value and feasibility of randomized trials of social policy. They can often be done at the same speed and cost as any prospective study, and can yield more compelling results. Random assignment where the government chooses by lottery who can receive the program may also be the fairest way to allocate an intervention on a limited basis.
Randomized testing may not be the standard for government evaluation yet, but such things take time. For example, the Food and Drug Administration was empowered in 1962 to obtain “substantial evidence” of a new drug’s safety and efficacy, yet it took more than five years for the agency to embrace randomized trials as the appropriate standard.
Now that the Biden administration is, again, properly emphasizing that all federal agencies need to make “evidence-based decisions” based on the highest scientific standards, perhaps truly rigorous testing of social policy will become as commonplace as it is for new vaccines. That would help ensure that government services are delivered as effectively and efficiently as possible.
Amy Finkelstein is the John and Jennie S. MacDonald professor of economics at M.I.T. She co-directs J-PAL North America, a research center at M.I.T. that conducts randomized evaluations.
Source: Read Full Article