FuelWatch evidence runs on empty
Professor Don Harding
First published in The Age on July 2, 2008
The decision to introduce a national FuelWatch scheme provides a timely case study of evidence-based policymaking in Australia.
The Government based its decision on econometric work by the Australian Competition and Consumer Commission, which refuses to release the data. It claims its analysis is robust because it has been subject to scrutiny within the ACCC and by Treasury.
Experience with evidence-based policymaking in Britain raises doubts about such claims. The British experience led to the term "policy-based evidence", to describe the end result where government agencies filtered out information that was inconsistent with government policy.
The data used by the ACCC can be digitised from a graph in its report. I then used the data to assess the robustness of the ACCC analysis and econometrics. Since the Government and its advisers did not realise that the data could be obtained in this way, it provides a "natural experiment" in which its claims can be evaluated and tested.
I find that the ACCC applied the wrong tests to the wrong variable. Specifically, it studies the nominal retail margin when economic theory suggests that analysis of anything but the real retail margin to producers creates a misspecified model inconsistent with the econometric assumptions used.
The econometrics in appendix S of the ACCC report on petrol is substandard in application of techniques and in reporting of what was done.
The ACCC findings are not robust. When I apply the correct version of the procedures used by the ACCC to the correct variable, I find that the data does not support the original ACCC finding. Specifically, it is not possible to conclude, as the ACCC did, that FuelWatch did not raise petrol prices in Western Australia.
FuelWatch of itself is unimportant. The important issues here relate to the integrity of the Government's evidence-based approach to policy. The British experience clearly shows that relying on the untested opinion of "experts" leads to fudging of the evidence or overstatement of the conclusions that can be supported by the data. This ultimately corrupts the evidence-based policy approach. The FuelWatch experience shows that these dangers are present for Australian policymakers.
A natural but incorrect response is to blame the econometricians who undertook the ACCC analysis. The public service has many y able econometricians and economists and I draw no adverse conclusion about those who undertook this work or those who reviewed it.
Econometrics is a difficult art. The quality of econometric work is almost always improved by public scrutiny and particularly by seminars and workshops involving academics. Thus, a significant cause of the error is the increased secrecy of governments and their failure to make adequate use of academics and public scrutiny.
By the same token, the quality of econometric work is almost always reduced by factors such as secrecy, the issues being politicised, ambitious junior ministers seeking to make a name for themselves and when the "evil trio" of the "manager's manager", lawyers and "econometric guns for hire" are involved. Several of these apply to the FuelWatch case.
Rather than blame econometricians for the errors, the FuelWatch case should be seen as reflecting on the process through which their work is incorporated into reports and used by government. The fact that four departments got the analysis correct in the leaked cabinet co-ordination comments confirms that there is more than adequate econometric expertise in the public service. The failure is in the inability of government to heed that advice. There is also a failure of government to give adequate assurances of protection to public servants who provide it with accurate but unpalatable advice.
The Treasurer's comment that his own department's advice, which ultimately proved to be correct, was "too academic" gives a clear indication of where one source of the problem is located. Ultimately, these are errors made by a government that has become distracted by stunts, when the nation faces major economic problems.
Finding solutions to these problems will require the Government to assemble a wide range of evidence and to convince Australians it has not been "economical with the truth" in forming and communicating that evidence, or in basing policy on that evidence.
The danger with stunts such as FuelWatch that go badly wrong is that they corrupt the whole fabric of evidence-based decision-making and erode public trust in governments.
The existing review procedures, such as Senate estimates hearings and Senate inquiries, are inadequate to the task of separating evidence-based policy from "policy-based evidence". There are several reasons for this. One reason is that the Senate committees are far too political in their approach. Another is there is no incentive for a public servant to make a career-limiting move by revealing the defects in a policy that is being championed by ambitious ministers.
Greater respect for the wisdom in cabinet co-ordinating comments, transparency by government, publication of data and analysis underpinning government decisions and independent review of econometric work are the main protections against the danger of policy-based evidence. Failure to implement these protections will invalidate claims about the evidence base of future policy decisions by the Government.
This article is based on my paper titled FoolWatch: A Case study of econometric analysis and evidenced-based policymaking in the Australian Government.
Don Harding is professor of economics at Latrobe University.