Grade | Criteria for judging agreement | Decision rule |
---|---|---|
A** | Median rating is 9 for both expert panels | Include in round 2 |
A* | ≥ 70% of respondents in each panel rate an outcome domain ≥ 7 | Include in round 2 |
A | Median rating for an outcome domain is ≥ 7 for both expert panels | Include in round 2 if one of the panels achieves a median score of 9 OR qualitative evidence supports further consideration |
B | Median rating for an outcome domain is ≥ 7 for only one expert panel | Include in round 2 if this group achieves a median score of 9 OR qualitative evidence supports further consideration |
C | Median rating for the two panels combined is ≥ 4 and ≤ 6, and the median rating for no single panel is ≥ 7 | Exclude from round 2 (unless strong qualitative evidence supports further consideration) |
D | Median rating for the two panels combined is ≥ 1 and ≤ 3, and the median rating for no single group is ≥ 7 | Exclude from round 2 (unless strong qualitative evidence supports further consideration) |