The Institut des politiques publiques (IPP) brings together researchers using different methods to analyse and evaluate public policies, mainly on the basis of statistical data. The selection of the most appropriate evaluation methods from among the available techniques lies at the heart of the IPP’s expertise.
The challenges of evaluation
The purpose of evaluating public policies is to measure the impact of a policy (either already implemented or under consideration) in a variety of areas that reflect the goals of the decision makers or the public (including employment, poverty, level of education and the like).
One of the main challenges of evaluation is to identify the counterfactual, that is, the situation that would have prevailed without such a policy, or with an alternative one. The fundamental problem of evaluation is that in practice this counterfactual can never be seen.
In particular, the correlation of certain behaviour with the implementation of a policy is not sufficient to demonstrate that the observed changes were caused or created by that policy. For example, a noted increase in the number of hours worked by low-waged employees after the implementation of an earned-income tax credit is not enough evidence to assert that the reform is the reason for those increased hours. Other factors could explain the expansion in the employment rate of the target population, including an upturn in economic conditions at the moment when the reform was introduced, the entry into the labour market of a generation of workers with higher levels of education, or the simultaneous adoption of other public policies in support of low-skilled labour.
The other challenge of evaluation is to measure the potential social benefit of an ad hoc scheme, or where a policy is spread more widely or generalised. In particular, the impact of a measure may depend on the scale of implementation. For example, a new support scheme to help the unemployed to find work may be quite useful when targeted to a limited number of unemployed but have little effect on the unemployment rate when generalised. Such would be the case if, for example, the targeted unemployed were competing for a small number of jobs. Moreover, the benefits of any policy must also be assessed in the light of its cost.
A range of methods available
Several approaches can be used to meet the challenges of evaluation.
The most rigorous method for identifying the counterfactual is the so-called randomized control trial, which is possible when measurement of the impact of the policy was planned from the start and evaluation built into the policy design. When the policy was implemented in the past, one can either exploit “natural experiments” or adopt a structural approach.
Where the aim to understand the social benefit of a policy, two types of methods may be used.
Conversely, where the aim is to measure accurately the ways in which economic agents react to the implementation of a policy, by checking for context effects, experimental economics can be used.