FRCR exams: our commitment to standards

Our hearts go out to all doctors impacted by the results issues recently announced by the Federation of the Royal College of Physicians of the UK. Understandably this will be worrying to doctors who have recently sat, or are about to sit their specialty exams. We want to tell you about the quality assurance mechanisms we have in place to minimise the risk of this issue arising at The Royal College of Radiologists across our suite of exams. Ensuring fairness and accuracy is of the utmost importance to us when candidates sit their exams. 

Assurance

All our exams have been approved by the General Medical Council (GMC) and General Dental Council (GDC) as meeting the standards defined in Excellence By Design. Regulators looks for assurances on exam content, delivery and associated workflows, such as results production. The questions they ask include:

• How are all stages of the production of individual questions and complete exams quality controlled?  
• How do you check marking and moderation of exam for accuracy? 
• What safeguards make sure exams have been scored and reported accurately and fairly? 

Running exams and awarding pass/fail outcomes requires the skill of our large team of examiners. They are supported by our office team - staff with experience and expertise in the administration, operation and quality management of exams – as well as invigilators, exam platform suppliers and venues. Input from psychometricians and educationalists is also reflected at various points in the quality management process for activities such as examiner training and identifying the passing standard (standard setting). 

Processing results

Examiners and staff are collectively responsible for ensuring that exams provide an assessment that is demonstrably fair and reliable. Our examiners are trained in generating exam content, developing scoring schemes and marking, including calibration and safely recording marks awarded to candidates in real time. 

Software packages that process assessment data – including candidate scores and results - help to minimise administrative and processing errors. Regulators encourage awarding bodies to use technology to manage exams data for this reason. Our exam platforms house content, present exams to candidates and, for some exams, mark candidate answers automatically. Staff are trained in the use of these platforms, and workflows – such as programming the platforms with correct answers and generating data for results processing - are well documented with standard operating procedures. We have contracts and service level agreements in place with platform suppliers, and ‘product owners’ within the exams team work with them to understand – and be able to continually optimise - the platforms functionality and robustness. 

Candidate results are processed differently depending on exam format and whether questions are marked by software or examiners. Prior to results publication, staff and examiners will conduct a statistical review of the question paper performance and quality control checks. Poorly performing questions are scrutinised and removed if an error or ambiguity is found - this analysis allows us to ensure that the exam is reaching optimal levels of reliability.  

If results data is moved to an alternative location temporarily – such as an excel spreadsheet to undertake allied activities outside the software platform - the exams team perform multiple QA checks to minimise any risks around data transfer. In addition to human checks there are flags built into spreadsheets to highlight anomalies for further investigation. Any anomalous data is escalated to the senior exam team for additional review. As the results generation process moves closer to completion, a further final check is carried out by RCR staff before publication, which involves both the quality and operational teams cross-referencing scores achieved against results letters.

More details on the specific content, format and approaches to quality assurance are provided on the RCR website for individual exam components. 

Continuous improvement

We continually monitor our exams to maintain quality and relevance, to improve the candidate experience and to minimise costs. 

Exams sit within a wider governance framework and an overarching exam board reviews pass-rates for all exams bi-annually. We also report pass-rates across all exams and for subsets of candidates to the regulator. Pass rates are published on our website for all exams.

Exams operate within regulations and policies that support our compliance with regulatory requirements and reflects exam-specific guidance issued by the GMC or the Academy of Medical Royal Colleges. We are currently reviewing our policy framework to ensure it incorporates up to date best practice. More broadly, it’s important we work with other Colleges and awarding bodies to understand successes, issues and ‘near misses’ and to learn from issues experienced and ensure we share good practice with those operating in a similar environment.  

If you have any queries or concerns, please reach out to us at [email protected].