Making sound regulatory decisions demands information and analysis. Several Administrative Conference recommendations encourage agencies to gather data when making new rules and when reviewing existing rules.[1] These recommendations reinforce analytic demands imposed on agencies by legislation,[2] executive orders,[3] and judicial decisions.[4]
Agencies need information about the problems that new rules will address, such as the risks involved and their causes. But agencies also need information about potential solutions to these problems. What possible alternative rules or rule designs might help solve the problems? How effective are these alternatives likely to be in addressing the underlying problems? Are there constraints, barriers, or unanticipated consequences that arise in the use of these different alternatives? In terms of understanding possible alternatives and how well they might work in practice, agencies benefit from having information from experience with different solutions. Learning from experience is the focus of this recommendation.
Learning from Regulatory Experience
No uniform or tidy formula exists as to how agencies should generate, gather, and analyze the data necessary to support sound regulatory decisions. A variety of well-accepted and widely-used methods exist from which agencies may choose, with the appropriate choices often varying agency by agency and even from situation to situation. Practical considerations such as resource and data availability will affect the choices agencies make about the methods of learning used to support regulatory decisionmaking.[5] Still, it is possible to identify some of the main methods for learning that agencies should consider using at different stages of the rulemaking lifecycle. These methods, which are not necessarily mutually exclusive, can be used before or after a rule is adopted, and they may be considered on occasion as part of the final rule itself, which might be structured to facilitate future learning by agency officials.
Variation is the key to agency learning. In this context, “variation” can refer to differences among jurisdictions[6] or across time,[7] with some jurisdictions or time periods having in place a version of a rule and others having in place a different version of the rule (or no applicable rule at all). It can also refer to differences among regulated entities or people within the same jurisdiction, with some entities or people subject to a version of a rule and others subject to a different version of the rule (or no applicable rule at all).
An agency can learn from all of these kinds of variation. For example, a regulation that goes into effect in 2017 leaves the agency with two distinct time periods to compare: the years before 2017, and 2017 and beyond. A rule that applies in jurisdictions X and Y but not in jurisdictions A and B leaves the agency with the ability to compare outcomes in X and Y with those in A and B, assuming the jurisdictions are comparable or that differences can be statistically controlled. The agency can then learn whether outcomes are improved in those time periods or jurisdictions with the regulatory obligation. However, agencies must be careful not to assume automatically that any differences in outcomes that they observe have been caused by the intervention of the regulation. Other factors that correlate with the observed outcomes might also vary across the same time periods or jurisdictions.
Using Observational or Randomized Methods to Learn from Experience
To learn from experience, agencies should seek methods that allow them to draw valid inferences about whether a particular regulatory intervention causes (or will cause) improvements in the desired outcomes. Concern about the validity of such causal inferences generally takes two forms. The first of these—external validity—refers to the extent to which the inferences from a study situated within a particular time period or setting can apply to other time periods or settings. In other words, an agency should consider to what extent the results of a study focused on entities or individuals in one period or setting are generalizable to entities or individuals in other times or settings. The second type of validity—internal validity—refers to the extent to which the outcomes observed in a study can be said to have been caused by the intervention rather than by potential confounders.[8] In other words, an agency should consider whether what might appear to be a relationship between a regulation and changes in outcomes truly derives from the regulation. For example, if a study shows that accidents from a particular industrial process have declined following the adoption of a regulation intended to reduce those accidents, concern about internal validity would lead agency officials to consider the possibility that the observed decline might have arisen from market or technological factors that led to changes in the relevant industrial processes around the same time as the regulation but which came about for reasons entirely unrelated to the regulation. An agency may wish to learn whether the observed decline came from the regulation or from other factors so as to know whether to redesign the regulation if further improvements are warranted.
To isolate the true effects of a regulation on relevant outcomes, such as risk reduction, agencies can use randomized approaches or observational approaches. Both of these approaches have advantages and disadvantages, and choosing between them will depend on a variety of contextual factors.
Randomized approaches promise to generate results with a high level of internal validity because, by making a random assignment of individuals or entities subject to a regulatory intervention, any other factors that might lead to changes in the relevant outcomes should be distributed randomly between the group subject to the regulatory intervention and the comparison group. Of course, randomized methods can also have their limitations. There is always a question as to whether the results of a randomized experiment are externally valid. For example, a perfectly designed randomized experiment may indicate that exposure to an intervention generates particular outcomes in a laboratory setting but may not mean that those same outcomes will occur outside of the laboratory. In addition, the results of randomized methods may lack validity if individuals, knowing that their behaviors are part of a randomized experiment, behave differently from how they would otherwise act. Researchers try to limit this particular threat to validity by using double-blind, or even just single-blind, study designs.[9] However, it is possible that, in many regulatory contexts, regulated parties will know they are subject to a randomized study and may engage in strategic behavior that may skew the results of the study.
In addition to these methodological challenges, randomized study methods may present legal, policy, and ethical concerns. From a legal standpoint, subjecting similar parties to different rules may be thought to raise concerns under the Equal Protection Clause of the Constitution or the arbitrary-and-capricious standard of the Administrative Procedure Act.[10] Of course, an agency might present a legally valid argument that the rational basis, or non-arbitrary reason, for its action is to generate information necessary to make an informed decision.[11] From a policy standpoint, if some entities are subject to regulation and others are not, an agency may well risk artificially distorting a market, depending on what a rule requires or how the study is designed. From an ethical standpoint, if a rule specifically sets up an experiment with the idea that, after the experiment, the agency may change the rule, a concern may exist if some regulated entities will by then have invested heavily in capital-intensive equipment required by the rule. Another concern might be with varying levels of health or safety protection to different members of the public. In the absence of countervailing considerations, legal, policy, and ethical challenges such as these may mean that regulatory agencies should use randomized study methods only under limited circumstances.
If randomized study methods are either unavailable or inadvisable, agencies can use a broad range of opportunities to learn from observational studies. Sometimes these studies are called “natural experiments,” as they seek to draw inferences based on variation that naturally arises over time or across settings in the absence of randomization. For this reason, observational studies lack some of the methodological advantages of randomization. Internal validity is generally a more present concern with observational studies, as other factors may confound a study’s results. In other words, other factors may also vary naturally with the intervention under study and affect the observed outcomes. An example of a potential confounding factor is when an intervention is accepted voluntarily; those individuals or entities who voluntarily choose to adopt a new practice may be different from the individuals or entities to whom a mandatory requirement would apply.
The possibility of such confounding factors should be accounted for when conducting observational studies and can be effectively addressed by using various methods that attempt to mimic statistically what occurs with randomization.[12] Assuming the potential threats to internal validity can be addressed, observational studies may in some circumstances lead to results with stronger external validity than randomization. As a general matter, observational studies will also not raise the same legal, policy, or ethical concerns as randomization. With observational studies, the agency is either exploiting natural variation that would have arisen from the rule anyway or allowing for learning from other existing variation, such as state-by-state variation.
Opportunities for Learning from Experience Throughout the Rulemaking Lifecycle
Agencies have opportunities to learn from experience throughout the rulemaking lifecycle. For example, one stage of this cycle occurs before a rule is adopted, as agencies are focused on a problem to be addressed and are considering potential regulatory solutions. Learning from experience at this early stage can help inform an agency of how a rule should be designed. Another stage of the cycle lies with the design of the rule itself. At this stage, as an agency writes a rule, it may design it in a way that can facilitate the type of variation needed to promote learning. Finally, yet another stage arises after the agency has promulgated the rule. At this stage, agencies can consider actions, such as waivers, that can facilitate learning from experience.
Learning Before Adopting a Rule
Prior to adopting a rule, an agency should gather information using appropriate methods to help inform the regulatory action it plans to take. An agency may wish to consider randomized or observational methods.
Randomized Methods. Agencies can analyze existing peer-reviewed studies that incorporate a randomized design. They can also initiate or support new pilot programs that produce randomized study data. For example, if an agency were trying to determine whether a certain default rule related to saving for retirement should be required of all employers offering 401(k) plans, it might, if consistent with applicable law, seek the cooperation of some large employers to see whether they would assign randomly some of their employees to a company policy that requires them to opt into a retirement savings plan and other employees to a company policy that defaults employees into the plan but then allows them to opt out. Such action would be voluntary by the company but random (and effectively involuntary) by the individual. The agency might be able to learn better which default rule will yield greater savings and then use these results to inform a decision about a regulation that would apply to all companies.
Observational Methods. Agencies can also undertake observational studies prior to creating new rules. An agency might, for example, employ a cross-sectional research design by looking at variation in existing policies at the state level (or perhaps in other countries), taking to heart Justice Louis Brandeis’s observation that “a . . . state may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.”[13] In fact, Congress has, on numerous occasions, directed agencies to analyze state-by-state variation to help determine optimal policies.[14]
Designing a Rule to Facilitate Learning
An agency can write a rule to facilitate future learning or to enable it later to take advantage of variation that stems naturally from the rule.[15] Again, an agency may wish to consider randomized or observational methods.
Randomized Methods. When appropriate, an agency might consider structuring a rule to allow for learning through a randomized method.[16] This could entail writing a rule in such a way that some entities or people that fall within the agency’s regulatory scope are subject to one version of the rule and some are subject to another version of the rule or not subject to the rule at all. The agency’s decision as to who falls within each category could be made on a random basis. For example, Michael Abramowicz, Ian Ayres, and Yair Listokin use as an example a test of speed limits in which the posted limits on different roads are randomly increased or decreased.[17] Drivers on these roads are informed of the regulatory intervention (i.e., the speed limit on that road) without necessarily knowing that they are participating in a randomized experiment. Although this example falls outside the realm of federal rulemaking, agencies at the federal level may have similar ways to structure the timing or application of a rule using randomization. Assuming any potential methodological, legal, ethical, and policy concerns about randomization can be addressed, there may be some circumstances in which randomization will be an appropriate way for an agency to generate variation that will facilitate learning from experience.
Observational Methods. For the reasons discussed above, agencies will generally find it more feasible to use observational approaches than randomized ones. In any rulemaking, there will be variation from observing the world before the rule went into effect and comparing it to the world after the rule has taken effect. Further, in the case of a rule that an agency has rescinded, there will be variation in three conditions: the world before the rule went into effect; the world in which the rule was in effect; and the world after the rule was rescinded. Such variation can present rich opportunities for observational studies, especially when a satisfactory baseline or control group can be identified. Agencies may well decide, at the outset when promulgating a new rule, to commit to setting up a longitudinal study. In doing so, they would need to collect data from regulated parties before the rule goes into effect and then collect data once the rule has taken effect, keeping in mind potential confounders and using statistical techniques to control for them.[18]
Additionally, agencies may consider deliberately introducing or allowing for some non-random variation in response to a rule by allowing for flexibility by states in the implementation of the rule. For example, variation can occur if the agency sets a federal minimum standard and permits states to exceed that standard. Agencies then can commit to using the resulting state-by-state variation to compare firms separated by a very short distance in neighboring states that have adopted different rules. Using the statistical technique known as regression discontinuity, the agency may be able to approximate randomization (i.e., the “assignment” of firms to a state with one rule versus another would be effectively random).[19]
Learning After Promulgating a Rule
An agency can also use either randomized or observational methods to take advantage of variation once a rule has been put into place.
Randomized Methods. An agency might choose, only if appropriate, after taking into account all legal, ethical, practical, and fairness considerations, to vary the application of a rule on a randomized basis to learn from variation.[20]
Observational Methods. In addition to varying the application of a rule on a randomized basis, agencies can achieve variation once the rule is in place by considering conditional waivers and exemptions. For example, if a regulated entity can present some evidence to suggest that it can meet the purpose of the regulation using an alternative approach, the agency might grant a waiver to that entity with the condition that the entity uses that alternative approach.[21] After granting a certain number of waivers, the agency could then test the effectiveness of its rule by comparing entities that have selected different approaches. The agency would likely find it necessary to use statistical techniques to control for potential confounders. Over time, these kinds of studies may provide the agency with retrospective information that justifies amending an existing rule. Fairness, legal, and ethical concerns might be minimized when using conditional waivers if the agency permits all regulated entities to seek a waiver based on presentation of evidence and the agency widely publicizes its waiver availability.[22]
Table 1 summarizes the main methods of learning discussed in the preceding sections.
Table 1: Examples of Methods for Regulatory Learning
|
|
Randomized |
Observational |
|
Learning before adopting a rule |
● Randomized voluntary pilot programs ● Studies that rely on randomization |
● Pilot programs where intervention is not assigned randomly (such as with voluntary programs) ● Analysis of regulatory approaches in different jurisdictions, including countries |
|
Designing a rule to facilitate learning |
● Randomized assignment of different regulatory obligations |
● Rules that allow for state implementation and variation (e.g., cooperative federalism) ● Analysis of temporal differences (i.e., “before and after” comparisons) ● Creation of regulatory thresholds that will facilitate later comparisons of entities above/below a threshold
|
|
Learning after promulgating a rule |
● Randomized application of rules in appropriate circumstances |
● Granting of waivers or exemptions that allow for the adoption of alternative approaches that can be studied |
Common Issues in Learning from Experience
As noted, each stage of the rulemaking lifecycle allows agencies to learn from variation. Agencies can learn from both randomized and observational methods, keeping in mind the virtues and challenges of each. Whichever method an agency chooses, at least two additional issues should be considered: data collection and public input.
Data Collection
Collecting data is essential. Only with information can agencies hope to learn from analyzing regulations. When collecting data, though, agencies must be mindful of the Paperwork Reduction Act (PRA), which can constrain their ability to send a survey instrument to ten or more parties.[23] As part of agencies’ data collection efforts, it may be helpful for agencies to work closely with the Office of Information and Regulatory Affairs to ensure proper use of available flexibility in accordance with the PRA and the Office of Management and Budget’s implementing regulations.
Public Input
Best practices generally call for some opportunity for the public to learn about and comment on the design and results of studies an agency undertakes. For pre-rule learning, the notice-and-comment process provides the required minimum process by which agencies should engage the public, but there are other methods of public input that might be useful, even at the pre-rule stage, for public input beyond just notice and comment.[24] If an agency is planning to revise a rule, a subsequent notice-and-comment rulemaking will provide an additional opportunity for public input. If an initial rule provides for its expiration on a certain date, that may also help ensure that the public has the opportunity to offer input on a future notice-and-comment rulemaking to keep or modify the rule. Even rules not subject to notice-and-comment procedures can benefit from subsequent opportunities for public comment.[25]
But even in situations in which the agency does not undertake a new notice-and-comment rulemaking or otherwise leaves a rule “as is,” the agency may benefit from outside input on the systematic learning effort it has undertaken, whether through a peer review process, advisory committees, public hearings or meetings, or just a supplemental solicitation of comments. The decision as to which approach to use to solicit public input will turn on numerous factors, including resource constraints.[26]
RECOMMENDATION
1. Agencies should seek opportunities to collect data to learn the most effective way to design their rules and analyze the effects of their rules. They can learn from experience at one or more stages of the rulemaking process, from pre-rule analysis to retrospective review. Before adopting a rule, agencies can learn from pilot projects, demonstrations, and flexibility among states or regulated entities. After promulgating a rule, agencies may, where legally permissible, use waivers and exemptions to learn. As agencies seek out such learning opportunities, they should give due regard for legal, ethical, practical, and fairness considerations.
2. When agencies analyze variation to learn more about the effectiveness of policy options, they should make every effort to collect data and conduct reliable analysis. Only where appropriate, agencies should consider creating variation through a randomized control trial.
3. To inform the learning process, agencies should consider soliciting public input at various points in the rulemaking lifecycle. This can include input on the design and results of any learning process. In addition to the public input required under 5 U.S.C. § 553(c), agencies should consider, as time and resources permit, the use of supplemental requests for public comment, peer review, advisory committee deliberation, or public hearings or meetings.
4. When gathering data, agencies and the Office of Management and Budget (OMB) should seek to use flexibilities within the Paperwork Reduction Act and OMB’s implementing regulations (e.g., a streamlined comment period for collections associated with proposed rules) when permissible and appropriate.
5. Agencies, as appropriate, should seek legal authority from Congress to take advantage of this recommendation.
[1] See, e.g., Admin. Conf. of the U.S., Recommendation 2014-5, Retrospective Review of Agency Rules, 79 Fed. Reg. 75,114 (Dec. 17, 2014); Admin. Conf. of the U.S., Recommendation 85-2, Agency Procedures for Performing Regulatory Analysis of Rules, 50 Fed. Reg. 28,364 (July 12, 1985); Admin. Conf. of the U.S., Recommendation 79-4, Public Disclosure Concerning the Use of Cost-Benefit and Similar Analyses in Regulation, 44 Fed. Reg. 38,826 (June 8, 1979).
[2] See, e.g., Data Quality Act, Pub. L. No. 106-554, § 515, 114 Stat. 2763A-153 (2001).
[3] See, e.g., Exec. Order No. 12,866, § 5, 58 Fed. Reg. 51,735, 51,739 (Oct. 4, 1993) (“[T]o . . . improve the effectiveness of existing regulations . . . each . . . agency will periodically review its existing significant regulations to determine whether any such regulations should be modified or eliminated so as to make the agency’s regulatory program more effective in achieving the regulatory objectives.”); Exec. Order No. 13,563, § 6, 58 Fed. Reg. 3821, 3822 (Jan. 21, 2011) (requiring agencies to “consider how best to promote retrospective analysis of rules that may be outmoded, ineffective, insufficient, or excessively burdensome, and to modify, streamline, expand, or repeal them in accordance with what has been learned”); Exec. Order No. 13,771, § 2, 82 Fed. Reg. 9339 (Feb. 3, 2017) (requiring the repeal of two existing regulations for each new regulation proposed, and leaving in place prior analytical requirements); Exec. Order No. 13,777, § 3, 82 Fed. Reg. 12,285, 12,286 (Mar. 1, 2017) (requiring the establishment of regulatory reform task forces that “shall evaluate existing regulations . . . and make recommendations to the agency head regarding their repeal, replacement, or modification, consistent with applicable law”).
[4] See, e.g., Motor Vehicle Mfrs. Ass’n v. State Farm Mut. Auto. Ins. Co., 463 U.S. 29, 43, 52 (1983) (explaining that the agency must show that its action was the result of “reasoned decisionmaking” consistent with “the evidence before the agency”).
[5] A general discussion of factors to consider in choosing methods and measurements in regulatory learning can be found in Cary Coglianese, Measuring Regulatory Excellence, in Achieving Regulatory Excellence 291–305 (Cary Coglianese ed., 2017) [hereinafter Coglianese, Measuring Regulatory Excellence].
[6] Cross-sectional analysis means analysis of data collected across at least two groups or jurisdictions, with one that is subject to the intervention (such as regulation) and one that is not. See Cary Coglianese, Empirical Analysis and Administrative Law, 2002 U. Ill. L. Rev. 1111, 1117–19.
[7] Longitudinal analysis is a research design that involves repeated observations of the same subjects over a period, where variation in the intervention occurs over time (i.e., data before and after an intervention is introduced). See Cary Coglianese, Measuring Regulatory Performance: Evaluating the Impact of Regulation and Regulatory Policy, Organization for Econ. Co-Operation and Dev. [OECD] Expert Paper No. 1 39 (Aug. 2012) [hereinafter Coglianese, Measuring Regulatory Performance].
[8] In this context, “confounders” refer to changes in outcomes that may appear to have been caused by the regulation but are actually caused by other factors. See Coglianese, Measuring Regulatory Performance, supra note 7 and accompanying text.
[9] “Blindness” in this context means subjects are not aware of whether they are in the treatment or comparison group. “Double blindness” means neither the subjects nor the researchers know which subjects received the treatment, and which received the placebo. See Michael Abramowicz et al., Randomizing Law, 159 U. Pa. L. Rev. 929, 948–50 (2011).
[10] See 5 U.S.C. § 706(2)(A).
[11] See Abramowicz et al., supra note 9, at 968.
[12] Examples of such statistical methods include: difference-in-differences, propensity score matching, instrumental variables, and regression discontinuity. See Coglianese, Measuring Regulatory Performance, supra note 7, at 39–42.
[13] New State Ice Co. v. Liebmann, 285 U.S. 262, 311 (1932) (Brandeis, J., dissenting).
[14] See, e.g., Energy Policy Act of 2005, Pub. L. No. 109-58, § 139, 119 Stat. 594, 647 (2005) (“[T]he Secretary . . . shall conduct a study of State and regional policies that promote cost-effective programs to reduce energy consumption (including energy efficiency programs) that are carried out by utilities that are subject to State regulation.”).
[15] These features can facilitate retrospective review. See Admin. Conf. of the U.S., Recommendation 2014-5, Retrospective Review of Agency Rules, 79 Fed. Reg. 75,114 (Dec. 17, 2014).
[16] See generally Abramowicz et al., supra note 9.
[17] See id. at 951.
[18] See Admin. Conf. of the U.S., Recommendation 2014-5, ¶ 7, Retrospective Review of Agency Rules, 79 Fed. Reg. 75,114, 75,116–17 (Dec. 17, 2014).
[19] See Jonah B. Gelbach & Jonathan Klick, Empirical Law and Economics, in The Oxford Handbook of Law and Economics (Francisco Parisi ed., 2017).
[20] In 2004, the Securities and Exchange Commission (SEC) varied the application of its “Uptick Rule.” See Order Suspending the Operation of Short Sale Price Provisions for Designated Securities and Time Periods, Exchange Act Release No. 50,104, 69 Fed. Reg. 48,032 (Aug. 6, 2004). Market observers characterized the SEC’s conclusion to be that the rule did not substantially increase market efficiency. The SEC rescinded the rule. See Zachary Gubler, Regulatory Experimentation 42 (Nov. 17, 2017) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/report/regulatory-experimentation-final-report.
[21] See Admin. Conf. of the U.S., Recommendation 2017-7, Regulatory Waivers and Exemptions, 82 Fed. Reg. 61,742 (Dec. 29, 2017); see also Aaron Nielson, Waivers, Exemptions, and Prosecutorial Discretion: An Examination of Agency Non-Enforcement Practices 30 (Nov. 1, 2017) (report to the Admin. Conf. of the U.S.), https://www.acus.gov/report/regulatory-waivers-and-exemptions-final-report.
[22] See Admin. Conf. of the U.S., Recommendation 2017-7, Regulatory Waivers and Exemptions, 82 Fed. Reg. 61,742 (Dec. 29, 2017).
[23] See 44 U.S.C. § 3502(3)(A)(i).
[24] See, e.g., Admin. Conf. of the U.S., Recommendation 2017-2, Negotiated Rulemaking and Other Options for Public Engagement, 82 Fed. Reg. 31,039 (July 5, 2017); Admin. Conf. of the U.S., Recommendation 2013-5, Social Media in Rulemaking, 78 Fed. Reg. 76,269 (Dec. 17, 2013).
[25] Admin. Conf. of the U.S., Recommendation 95-4, Procedures for Noncontroversial and Expedited Rulemaking, 60 Fed. Reg. 43,110 (Nov. 8, 1995).
[26] See Gubler, supra note 20, at 54.
Recommended Citation: Admin. Conf. of the U.S., Recommendation 2017-6, Learning from Regulatory Experience, 82 Fed. Reg. 61,728 (Dec. 29, 2017).