Introduction

The workshop ‘Measuring Outcome’, hosted by the Belgian Agency for the Safety of the Food Chain and co-hosted by the Hungarian National Food Chain Safety Office was one of the four workshops organized in 2013-2014 within the context of the Heads of Agencies’ aim to share good practices.

Representatives of food safety agencies, risk assessors and scientists of 14 Member States, Iceland, Norway, Swiss, and the European Commission exchanged views on the subject of ‘How to measure the outcome of food safety (management) (systems)?’. Measuring the outcome goes further than just measuring the output. It makes the results of our work more visible and gives an indication of the effectiveness of our activities, which is important in these times of budgetary restrictions. The choice of a good performance indicator is also of the utmost importance and can be used as communication tool for consumers.

During this two half-day workshop, a state of play was presented of different initiatives both at international as well at national level. Examples of measuring the general outcome and the outcome of more specific actions were shared. In smaller workgroups, the definition of outcome, the actual measuring, and the (need for) a common approach was discussed and the results were fed back in a plenary session and round table discussion.

MANCP Network Document on Performance Objectives & Indicators

Julia Williams of the Food Standards Agency (UK) gave an overview of the efforts of the Performance Objectives & Indicators Working Group to produce a non-binding reference document on objectives and indicators as ways of measuring the effectiveness of official controls. The official control regulation requires the Member States to provide MANCPs with strategic objectives and to report annually on the execution of those plans. Since agencies need to manage their control systems by setting clear objectives and monitor the progress towards those objectives, they also need tools for measuring outcomes and impacts.

The reference document is aimed to provide guidance and principles when developing objectives and indicators, by describing among others the characteristics for good indicators, as well as their intended use and purpose. This would ultimately lead to a more delivery-led MANCP and Annual Report, by focusing on outcomes (effectiveness) and not on inputs (processes).

Although still work in progress, the workshop participants welcomed this initiative and there was a keen interest in the final document.

Codex Alimentarius (CCFICS) proposal for new work on ‘monitoring performance of national food control systems’

Juha Junttila of the Food and Veterinary Office explained the work that was initiated within the CCFICS framework by the United States on monitoring performance of national food control systems (NFCS). The proposal for this work was presented during the CCFICS committee meeting in October 2011, before the approval of Codex Principles and guidelines for national food control systems (CAC/GL 82-2013). There was a low response of member countries to the questionnaire aiming to make an inventory of existing experience in this field. Moreover, a lot of countries raised their concern about possible trade barriers if the result of the work would be used to evaluate the NFCS of exporting countries. Therefore, during the CCFICS committee meeting in February 2013, it was decided to redefine the scope of the work by means of working groups. Two workshops were organized in Costa Rica and Belgium. In a constructive atmosphere, the participating countries came to a common understanding about the purpose of the new work. It should provide guidelines for the practical implementation of section 4.4 of CAC/GL 82/2013, with focus on self-assessment and using measurements as a self-improvement tool. The guidelines should not fix indicators but should help member countries to develop them. Juha Junttila further emphasized that measurements are necessary, even if they are incomplete, not only have to a good idea about the actual situation and to set goals for improvement, but also to prove the relevance of competent authorities’ activities. He asked Member States to coordinate their position between MANCP/Audit network experts and those in charge of Codex matters in order to ensure that our apparently constructive approach is also reflected at the next CCFICS meeting.

Food safety systems – Share and compare

Sebastian Hielm of the Finnish Ministry of Agriculture presented the results of a workshop held in Helsinki in 2011 on sharing and comparing food safety systems. This workshop was based on the food safety performance world ranking system developed by S. Charlebois (University of Guelph) and published in 2010. It was the first international meeting in which benchmarking metrics were discussed in order to rank countries based on food safety risk practices. The primary aim of the Helsinki workshop was to set food safety indicators for a global comparative analysis between countries. However during the workshop it became obvious that the ranking approach remained controversial amongst the participants of the different industrialized countries and that many of the proposed indicators needed further fundamental reflection. It was concluded that the benefit of ranking is that it creates external pressure on national governments to improve their food safety management systems.  The Helsinki meeting did allow the participants to better appreciate several food safety system perspectives from around the world but did not result in further concrete action up till now. Meanwhile S. Charlebois is apparently continuing his work in food safety performance world ranking.

Barometers for the safety of the food chain: approach & methodology

Wendie Claeys of the Belgian Food Safety Agency (FASFC) presented the food safety, the animal health and plant health (phytosanitary situation) barometers, which were developed by the Scientific Committee of the FASFC with the feedback of risk managers and stakeholders of the food chain. The barometers allow to communicate in a simple, intelligible, comprehensible and clear manner to consumers and stakeholders on the yearly evolution of the safety of the national food chain and the results of actions taken regarding food chain safety. 

Each barometer is based on a set of indicators, which are related to different aspects of the control of the food chain (preventive approach, control of processes, conformity of products, surveillance and public health) and cover different categories of food chain hazards and segments. For the selection of these indicators, a number of criteria were established. Given the different importance of each indicator on food chain safety, their relative weight to be considered in the final result of the barometers was determined after consultation of stakeholders, risk assessors and risk managers.

After 5 years of measuring, the tools are currently being evaluated to examine in which direction they could evolve further. One of the opportunities identified in a SWOT analysis, was the possibility to link the barometers to the MANCP cycles, and this with respect to defined policy objectives and a periodical evaluation of the indicators.

Indicator system for food chain safety strategy

Ákos Jóźwiak of the National Food Chain Safety Office (Hungary) summarized the methodology and experience gained during drafting the 'Food Chain Safety Strategy 2013-2022' of Hungary. It was emphasized, that the scope of this document goes far beyond the scope of Regulation 882/2004/EC, and deals with all the stages of the food chain, and tries to control all the risks (health risks and other than health risks as well). As the Strategy came into force at the end of last year, there is no long-time experience regarding measuring of the indicators yet.

This broad scope results in quite broad high level objectives, and a very complex system of different level objectives as well. From indicator perspective the Strategy makers were inspired by the work done in the sub-WG on Indicators of the MANCP Network and the different preparatory and draft documents of this WG. The main idea was to draw up logic models for all the programs of the strategy, where those models would show the connection between the low level objectives and the high level ones. This process helped a lot in understanding of the core messages of the Strategy's programs and links to the final outcomes as well. Beside this the process helps in the question that what is the appropriate level of measuring the performance of the Strategy: indicators which are easily measurable and give a possibility to accountability (usually linked more to the outputs than outcomes) and the indicators which have a closer connection to the final outcomes/impacts, but might be hardly measurable or there is no full control over them.

The final message of the presentation was to have a set of indicators, containing elements which make the measuring of outputs, outcomes and impacts possible as well. Beyond this an interesting research area is the elaboration of a methodology of measuring impacts and in the future possibly the food chain safety agencies will have to move from measuring effectiveness towards taking into consideration the cost-effectiveness as well.

Using a consumer survey to monitor delivery of strategy

Sian Thomas of the Food Safety Agency (UK) showed another approach regarding to measuring outcome, in particular the monitoring of food hygiene behavior and the follow-up of agency’s recommended practices towards consumers. A consumer survey (Food and You) is used to examine reported behaviors, attitudes and knowledge related to food safety. The methodology consists out of face-to-face interviews  of a random sample representative of the UK population. Already 2 waves of 3,000 interviews were performed and a third wave is in the field. An index of recommended practice (IRP) is created and consists out of 17 questions on food safety, covering cooking, chilling, cleaning, cross-contamination (4 C’s) and use-by dates. For each question a binary score is applied, whether or not the response is in-line with recommended practice (RP) and for each respondent a % of responses in line with the recommend practice (%RP) is calculated. Afterwards, an analysis of the %RP in combination with different socio-demographic and economic variables of these respondents is performed. An important conclusion from the two accomplished surveys is that differences in reported food safety practices are related to who you are (socio-demographic variables, as age and gender) rather than other variables such as income, level of education and working status. Results indicate that giving information about recommended practices not necessarily results in changing behavior, even if one got the message.

Measuring microbial food safety output and comparing self-checking systems of food business operators in Belgium

Liesbeth Jacxsens of the University of Ghent presented the results of a study in which the effect was investigated of introduction of a self-checking system in food business operators on food safety output. A web-based food safety management system diagnostic instrument was developed enabling food processing companies to self-assess their compliance with microbial risk control and legislation. The instrument was based on seven food safety performance indicators and on four different levels of core control and assurance activities. Eighty-two Belgian companies of which 50% were certified for self-checking and 90% were certified for commercial quality systems participated in the study. Their results were compared in an European context. The self assessment provided insight in the strong and weak points of current food safety management systems and supported food businesses in identifying points for improvement. It was concluded that Belgian food processing companies demonstrated a good performance of food safety output with a rather advanced level of food safety management systems.  The favorable impact of introduction of self-checking systems was more clearly visible in the distribution sector compared to the transformation sector due to the already high presence of voluntary standards and certification systems in the latter sector.

Evaluation & Impact Assessment by Dutch Food and Consumer Product Safety Authority (NVWA)

Wendy Verdonck from the Dutch Food and Consumer Product Safety Authority (NVWA) showed the current developments in the organization of the NVWA regarding to measuring outcome by incorporating important evaluation steps or impact assessment for specific decisions and actions. The new approach was illustrated by 2 cases. The methodology was shown in the impact assessment of enforcement in the slaughterhouses and the impact assessment of published inspection results for the hospitality industry. The examples showed clearly the process of input, throughput and output, which results in intermediate outcome, meaning compliance effects and final outcome (social effects). Independently of the level of outcome, external factors (political attention, calamities, …) and side effects (positive/negative) have to be taken into consideration.

Results of the discussion in workgroups and final conclusions

Defining the scope

It was felt that there is a need for further clarification and harmonized understanding of different key terms such as output, outcome and impact. Everybody agreed that ‘output’ is more related to numbers and the productivity of a control agency, e.g. the number of official controls. ‘Outcome’ is more related to changes, e.g. the effect of controls on the level of compliance of operators and products. Some participants were of the opinion that a change in behavior was also an example of outcome, whereas others felt this should be classified as an example of ‘impact’. There was however an agreement that ‘impact’ should be linked to high level indicators such as public health, and as such, competent authorities have no full control on it. The question was raised of this kind of – high level – indicators were measurable at all, and if so, what the cost will be for this measuring. The proposed MANCP network document needs further refinement, but once finalized could be used as a reference guidance.

Measuring food safety outcome was considered to be a ‘must’ and should have the objective to improve control systems. Outcome could also be used as a communication tool, e.g. to demonstrate stakeholders our accountability, and for confidence building. It would also help to measure outcome if the food safety strategic objectives would be clearly determined. The question was also raised of how much food safety we really want to get (zero-risk versus acceptable level of risk?).

Measuring outcome

(A set of) indicators should be clearly defined in function of the scope, e.g. performance measurement versus food safety status measurement, outcome versus impact measurement, to improve or to compare systems, for communication purposes, … Therefore, the number of indicators and the level at which we set indicators depend on what we want to measure.

It can be more meaningful to measure (a baseline and) trends (changes in time) instead of status.

It was referred to EU baseline studies through which basis information for measuring outcome is becoming available; also it was suggested that Eurostat could play a role in measuring outcome.

The following criteria for indicators were mentioned: robustness (not sensitive to manipulation), comparability between systems (harmonized), quantifiable/qualitative,…  

Is a common approach possible?

There was no consensus between participants in the need for standardization or harmonization of indicators or criteria.

At different levels (Codex, MANCP WG, country level) debate is going on ‘measuring performance’. It was argued that if measuring performance is aimed at one general level system refinement will be lost, and therefore measuring systems could be set up at both country and supra-national level. It was again referred to EU baseline studies and to Eurostat. The results of FVO audits should not be used for comparison. Some stated that since food safety systems will ultimately be compared – either by others (i.e. the world ranking food safety system) or by us – it would be better that we develop ourselves comparable systems. And that in the ideal situation indicators should be defined at European level (not bottom-up but top-down).

However, others stressed that there is a risk in (peer) reviewing and making up a ranking of performance. On an international level, comparing systems could lead to trade barriers. Learning and understanding from each other, e.g. by sharing (measuring) tools was given more weight than comparing competent authorities or countries. Instead of fixing common criteria, it was felt that guidance was needed for elaborating these tools and by sharing them, measuring outcome instead of measuring output would be encouraged.

Final conclusions

All participants agreed that this meeting was a good exercise in sharing views and practices. However, the need of further reflecting on the topic – in international (Codex CCFICS) and European (MANCP) working groups, and on national level – was recognized.

Overall, the current state of play is encouraging to develop tools for measuring outcome.

Presentations

MANCP Network Document on Performance Objectives & Indicators (by Julia Williams)

Codex Alimentarius (CCFICS) proposal for new work on “monitoring performance of national food control systems” (by Juha Junttila)

Food safety sytems – Share & compare (by Sebastian Hielm)

Barometers for the safety of the food chain – approach & methodology (by Wendie Claeys)

Indicator system for food chain safety strategy (by Ákos Jóźwiak)

Using a consumer survey to monitor delivery of strategy (by Sean Thomas)

Measuring microbial food safety output and comparing self-checking systems of food business operators in Belgium (by Liesbeth Jacxsens)

Evaluation & Impact Assessment by Dutch Food and Consumer Product Safety Authority (NVWA) (by Wendy Verdonck)