Evaluating Occupational Health Interventions and Initiatives

Types of intervention

Interventions we might want to evaluate can range from the provision of a whole occupational health service, through to the implementation of a particular training programme, to a change in a policy or way of working or to the introduction of a new item of equipment. Interventions can impact at any or all levels of an organisation from the individual right up to the organisational; some intervention may even have a societal impact (we can dream). Having a clear idea of what the intervention is and what the possible/proposed impacts will be at each level of the organisation is the first part of setting up an evaluation study. Working out how to measure the impact will involve defining the variables of interest.

Variables

Variables can be described as dependent or independent. An independent variable (IV) is one which will be manipulated as part of the intervention. An example of an independent variable might be the presence or absence of the new piece of equipment whose effect we want to evaluate. A dependent variable (DV) is something you might measure following the intervention – for example injury rate; time taken to carry out a task or user satisfaction.

In addition to IVs and DVs it is important to be aware of potential confounding variables. These are variables which may have nothing to do with the intervention but have an impact on the DVs, thereby confounding the results. For example, if we are interested in the sickness absence levels due to MSDs in companies which have Ergonomics services over those which do not, we may find that it is only the companies with ‘high risk’ work who have Ergonomics services. They are therefore likely to have higher sickness absence levels due to MSDs than companies where there is no Ergonomics service, though this is due to the high risk nature of the work, rather than the presence or absence of the Ergonomics service. ‘Type of work’ is therefore a confounding variable and should be controlled for when evaluating the impact of an Ergonomics service.

Types of data – Qualitative and Quantitative

Quantitative data results from measuring, and is numerical. It is useful for demonstrating any statistically significant effects of an intervention. Qualitative data is descriptive and is useful for understanding the ‘how’ and ‘why’ questions associated with interventions.

Types of evaluation

Evaluation can take a variety of forms from a simple, qualitative, ‘after’, non-experimental design (e.g. using a survey to ask staff whether an intervention worked after it has taken place) to a randomised control trial, where an intervention is compared to a control and the participants are randomly assigned to each group. The

type of design chosen will depend on the balance between the resources available and the strength of evidence required. The strongest evidence comes from experimental designs, however these are also the most resource intensive.

Which form of evaluation is right for me?

Choosing what type of evaluation to carry out and which type of data to use can be guided by the following points

  • What strength of evidence is required?
  • Are there any ethical or legal considerations? e.g. we should not assign participants to an intervention condition if we know it is likely to be more harmful than another condition
  • What resources are available, both in-house and externally?
  • How far through the intervention are we already?
  • What is the time frame for having an answer?
  • Is it possible to randomise participants?
  • Does the workplace set-up preclude a control group from being influenced by the intervention?
  • What statistical power is needed?

How do I set-up an evaluation study?

Organisational Phase

  • Define the problem and the intervention
  • Define the purpose, timescale and budget for the evaluation

Planning and Development phase

  •   Investigate the available measurement methods
  • Specify the evaluation design, study sample and measurement methods

Pre-intervention phase

  • Measure base-line values of variables – qualitative and quantitative

Post-intervention phase

  •   Measure values of variables after the intervention – qualitative and quantitative

Analysis phase

  •   Quantitative and qualitative data analysis
  • Interpreting results and drawing conclusions