Are you developing guidelines for
public health,
health systems
or health policy?
Evidence-to-decision frameworks help guideline panels make recommendations in a structured and transparent manner. They examine the health benefits and harms of an intervention but also take into consideration various other criteria.
The WHO-INTEGRATE framework is rooted in the norms and values of the World Health Organization, emphasises the idea of complex interventions in complex systems and adopts a whole-of-society perspective. It is particularly suitable for guidelines related to public health, health systems and health policy.
The WHO-INTEGRATE framework is appropriate for tackling global health challenges, where the effectiveness and applicability of interventions may vary across contexts, as well as planetary health challenges, where interventions involve multiple sectors.
What are the WHO-INTEGRATE criteria?
The WHO-INTEGRATE framework includes six substantive criteria. The seventh criterion, quality of evidence, is a meta-criterion that applies to each of the six substantive criteria.
Each criterion is clearly defined. To aid application, each criterion is accompanied by a set of sub-criteria and example questions.
Balance of health benefits and harms: Weighs the positive health outcomes of an intervention against any adverse effects or harms.
Human rights and sociocultural acceptability: Evaluates the alignment of the intervention with universal human rights standards and socio-cultural appropriateness.
Health equity, equality, and non-discrimination: Assesses how the intervention impacts disparities in access or outcomes.
Societal implications: Considers the broader social and environmental consequences of the intervention.
Financial and economic considerations: Reviews the costs of the intervention and its broader economic impact.
Feasibility and health system considerations: Examines the practicality of implementing the intervention within a health system.
Meta-Criterion Quality of evidence: Assesses the quality of the evidence backing to each of the six substantive criteria.
Why use the WHO-INTEGRATE framework?
01.
Aligned with WHO norms and values
The WHO-INTEGRATE framework was designed in accordance with the norms and values of the World Health Organization, ensuring credibility and global relevance.
02.
Reflective of the changing global health landscape
The WHO-INTEGRATE framework has a broad, interdisciplinary and intersectoral view of health and its determinants and is adaptable for various contexts.
03.
Built for complexity
The WHO-INTEGRATE framework draws on systems thinking and incorporates criteria specifically suited for evaluating complex interventions within complex health systems.
04.
Universally applicable
The WHO-INTEGRATE framework is suitable for all levels of decision-making and stages of guideline development and adaptation, making it a versatile choice for a range of health challenges.
When to use the WHO-INTEGRATE framework?
••••
Complex health issues
Opt for the WHO-INTEGRATE framework when the guideline or decision tackles intricate public health, health system or health policy problems.
•••
Population-level interventions
Apply it when the intervention under consideration impacts a large population and involves multiple sectors and disciplines.
••
System-level mechanisms
Use the WHO-INTEGRATE framework if your guideline or decision involves interventions that operate through broader healthcare or public health systems.
•
Wider societal implications
Choose it when the intervention has conequences that extend beyond the health sector, affecting broader societal norms or values.
Apply the WHO-INTEGRATE framework
Key considerations when applying the WHO-INTEGRATE framework
01.
Early implementation
Start using the WHO-INTEGRATE framework early in the process, such as during the guideline scoping and stakeholder engagement phases. Don’t wait until you formulate recommendations, so that this does not become a tick-off exercise.
02.
Selective approach
While all six criteria and the meta-criterion quality of evidence are essential to consider, not every sub-criterion will be relevant for each guideline or decision. Select the most important sub-criteria at the outset of guideline development.
03.
Evidence-based approach
Decide early on which of the selected sub-criteria should be supported by evidence to ensure a robust evidence retrieval and synthesis approach.
04.
Comprehensive recommendations
Consider all criteria in the framework when formulating final recommendations to ensure a comprehensive and balanced approach.
About the WHO-INTEGRATE framework
Evidence-to-decision (EtD) frameworks intend to ensure that all criteria of relevance to a health decision are systematically considered. The WHO-INTEGRATE framework is an EtD framework that was developed for and with the WHO. It is rooted in WHO norms and values and adopts a whole-of-society perspective, making it particularly suitable for public health, health system and health policy guidelines.
The value of this framework for decision-makers was assessed at global and national levels and adoption was facilitated through suggestions for prioritising criteria and evidence collection methods. The framework structure based on WHO standards and values was developed in an iterative, principles-based approach. Preliminary criteria were derived from key documents and supplemented with comprehensive subcriteria obtained through an overview of systematic reviews of criteria employed in health decision-making. The extent to which the framework can accommodate features of complexity was assessed, and key informant interviews were conducted among WHO guideline developers. Suggestions on methods were drawn from the literature and expert consultation.
The WHO-INTEGRATE framework is intended to facilitate a structured process of reflection and discussion in a problem-specific and context-specific manner from the start of a guideline development or other health decision-making process. For each criterion, the framework offers a definition, subcriteria and example questions; it also suggests relevant primary research and evidence synthesis methods and approaches to assessing quality of evidence.
Frequently asked questions
When should we use the WHO-INTEGRATE framework rather than one of the GRADE evidence-to-decision frameworks? The WHO-INTEGRATE framework is an evidence-to-decision (EtD) framework that is rooted in global health norms and values, is particularly well-suited for complex interventions in complex systems and adopts a broad societal perspective. If your guideline meets one of the following characteristics, using the WHO-INTEGRATE framework is probably a good choice. • Guideline places a strong emphasis on human rights, ethical or equity issues • Guideline is concerned with public health or health system interventions • Guideline addresses interventions at multiple levels (e.g. individual, health sector, health policy) •Guideline affects or involves sectors beyond the health sector With clinical guidelines, the GRADE EtD framework is clearly the better choice.
How does user experience feed into the advancement of the WHO-INTEGRATE framework? The WHO-INTEGRATE framework was developed through a robust process, which included key informant interviews with those developing WHO guidelines as well as focus group discussions with those using WHO guidelines at national level. It was deliberately published as a version 1.0 framework, as we envisage that experience with using the framework will lead to improvements. These improvements may relate to the framework and its criteria and sub-criteria (e.g. organisation of criteria, formulation of sub-criteria), or to the guidance supporting WHO-INTEGRATE framework implementation.
Do we have to use all criteria? How should we prioritise criteria and sub-criteria within the WHO-INTEGRATE framework? All six substantive criteria of the WHO-INTEGRATE framework – i.e. balance of health benefits and harms, human rights and socio-cultural acceptability, equity, equality and non-discrimination, societal impact, financial and economic considerations and feasibility and health system considerations – as well as the meta-criterion quality of evidence are relevant to all guidelines and should be used. The sub-criteria are intended to facilitate comprehensive (rather than tick-box-exercise-like) consideration of each criterion, but very rarely do all of them apply. Therefore, it is important to prioritise sub-criteria during guideline scoping and during the first guideline development group (GDG) meeting. Where a guideline scoping meeting does not take place, members of GDG can prioritise sub-criteria through a survey. This will ensure that all important sub-criteria are considered in formulating and grading recommendations and will ensure that the guideline development process remains feasible. Sub-criteria within the WHO-INTEGRATE framework may, as a result of this prioritisation, fall into one of the following four categories: not relevant – of minor relevance – of major relevance – game changing relevance.
Do we have to apply the WHO-INTEGRATE framework to every single guideline or PICO question? There is no simple answer to this question – it depends. You may find that most criteria are rated in the same way across multiple related guideline questions, including PICO questions in the guideline and that you can apply the criteria to a package of PICO questions. Usually, however, there will be relevant differences in some of the criteria (e.g. balance of health benefits and harms). Where your guideline covers recommendations at multiple levels (e.g. individual, health sector, health policy), criteria will most likely be rated differently at these different levels and therefore need to be applied separately at each of these levels.
What is needed if we want to use the WHO-INTEGRATE framework well? Ideally, the WHO-INTEGRATE framework is used throughout the guideline development process, i.e. to inform the scope of the guideline, to instruct evidence collection/synthesis and to guide recommendation development. It may also be used during adaptation and implementation of a guideline, although, at present, we are not aware of any examples. The WHO-INTEGRATE framework should be used from the very beginning of the guideline development process, i.e. during scoping of the guideline and during the first meeting of the guideline development group (GDG). Its use will broaden the guideline’s perspective and influence: • How a guideline is framed and how a guideline development group (GDG) is composed • Which questions are asked (guideline questions, including those using the PICO structure) • Which systematic reviews and other evidence products are developed • How recommendations are formulated and graded Evidence should be collected or synthesised for those WHO-INTEGRATE criteria that have a major impact on how a recommendation is formulated and/or graded (i.e. criteria of minor or major relevance). This may take place through the best possible (“gold standard”) approach and quality assessment or through a pragmatic approach. The WHO-INTEGRATE framework, as an evidence-to-decision-framework, should be used when formulating and grading recommendations. It may influence how a recommendation is formulated (e.g. with the wording of the recommendation making explicit reference to human rights or ecological impact), how a recommendation is graded (e.g. leading to a conditional recommendation where the overall benefit-harm-balance is heavily affected by variation due to financial and economic considerations) and how implementation considerations are formulated (e.g. where a recommendation requires or is positively influenced by aspects relating to feasibility and health system considerations).
Which evidence should we gather for which of the criteria of the WHO-INTEGRATE framework? In an ideal world, every criterion (and sub-criterion) would be backed by evidence. In reality, this is not feasible within the normal timeframe of guideline development and with limited human and financial resources. In fact, seeking to conduct an appropriate systematic review to back every criterion (and sub-criterion) may be an enormous waste of resources. It is thus important to prioritise criteria and to decide on how to collect or synthesise evidence regarding those criteria that have a major impact on how a recommendation is formulated and/or graded. For “game-changing” criteria, one should then implement the “gold standard” approach to gather the evidence, normally a systematic review (e.g. a qualitative evidence synthesis assessing the socio-cultural acceptability of an intervention from the perspectives of different stakeholder groups). For criteria of major relevance, one may opt for a more pragmatic approach (e.g. a sample of studies from across multiple continents examining socio-cultural acceptability, or a brief survey of sociocultural acceptability among different stakeholder groups). For those criteria considered important but not of major relevance, it may be sufficient to address these through deliberations by the guideline panel/ guideline development group (GDG).
Should we use the WHO-INTEGRATE framework when we do not have any evidence? Whether or not evidence is available or likely to be available should not influence the decision to use the WHO-INTEGRATE framework rather than one of the GRADE evidence-to-decision (EtD) frameworks. The choice of EtD framework should be guided by the characteristics of the guideline. Characteristics, such as the guideline placing a strong emphasis on human rights, ethical or equity issues and the guideline affecting or involving sectors beyond the health sector, suggest that the WHO-INTEGRATE framework is likely to be a good choice.
How does the use of the WHO-INTEGRATE framework influence recommendations? The application of the WHO-INTEGRATE framework in going from evidence to decisions can influence recommendations in multiple ways. They weighing of criteria against one another can: · Affect the way a recommendation is formulated · Influence the strength of a recommendation · Lead to the formulation of specific implementation considerations.
Can we use the WHO-INTEGRATE framework with the GRADEpro software? Yes, you can – a digital version of the WHO-INTEGRATE framework is available as part of the GRADEpro software. If you are using GRADEpro for the development of your guideline, you can select the WHO-INTEGRATE template as follows: In your project, (1) go to the project setup module in the left panel, (2) select project set up > ETD templates, and (3) under select base template for management questions pick WHO-INTEGRATE from the drop-down menu. The evidence-to-decision section will now be pre-populated with the WHO-INTEGRATE criteria.