Too often, competency-based training asks one instructor, standing at the back of a cabin mock-up, to watch a crew perform a live scenario and to observe, record, classify and grade performance across seven competencies and dozens of observable behaviours, find the root cause of whatever went wrong, and often having to technically run the scenario. All at the same time.
Human attention does not work like that. Research suggests that asking Instructors to consider fewer dimensions of performance can improve what they notice, how well they distinguish these performances, and how consistently they asses them.
This article is about the limits of Instructor performance, what the research suggests happens when we ignore those limits, and design choices that bring assessment back within reach of what a human being can realistically achieve. The last proposed design choice moves the broader competence decision beyond a single assessment session altogether.
Consider the following example
A recurrent session. Four active trainees, a smoke-in-the-lavatory scenario, twenty to thirty minutes runtime. The instructor has an assessment form with the full competency model printed on it, the operator’s standard, and Instructor guidance. Whatever is written down will have to feed the training records, the next cycle’s design, and perhaps a feedback session with a trainee.
The purser coordinates, a cabin crew member fights the fire, another handles the cabin, and yet another is on the Interphone. Four people, seven competencies each, twenty to thirty minutes.
The limits
For half a century, the familiar figure for how much a person could hold in mind at once was Miller’s seven. Cowan’s 2001 review placed the capacity closer to four meaningful chunks under conditions that limit rehearsal and grouping. Expertise, familiarity and the way information is organised all matter. But the conclusion is clear: the person holding the grading sheet does not have unlimited processing capacity.
The question is what happens when the assessment task becomes too demanding. Medical education has been investigating this for years. Experiments by Tavares, Eva and colleagues, summarised in a 2019 review by Paravattil and Wilby, gave raters recorded performances and asked them to rate either two dimensions or seven. Those considering two identified more of the relevant behaviours, distinguished performance levels more clearly, and agreed with each other more. Similar findings emerged with both new and experienced assessors. Additional distracting tasks had little measurable effect on rating quality; narrowing the assessment focus did.
A later study distributed the dimensions across several assessors, each considering a smaller subset. Combining those assessments produced more and broader feedback with higher score reliability than asking individual assessors to consider all the dimensions simultaneously.
Two findings in the same review: shortening a checklist from forty items to twenty did not improve accuracy or agreement and a single assessor-training intervention also failed to improve accuracy afterwards.
This does not mean that assessor training is ineffective. Qualitative assessor training matters, but it cannot be the only answer to an overloaded assessment design.
What the CBTA principles actually ask
IATA’s guidance for assessing pilots, instructors and evaluators describes four steps: observe performance, record what was seen (by means of notes), classify the observations against observable behaviours, and then assess the performance.
The guidance also recognises that, depending on the training objective, some competencies may be irrelevant to assess or record. It does not turn every training event into an instruction to complete every line of the competency model. Yet it is easy to print the whole model on every asessment form, ask for a grade on every competency after every scenario, and reduce observation, interpretation and judgement into one number.
The overload is not an unavoidable result coming from the CBTA principles itself, but often the result of design choices.
The industry has recognised the problem. Assessment means learning to observe, justify judgements and investigate the reasons behind performance—not simply learning to operate a grading scale.
Technology can be part of the answer to reduce the workload of the Instructor and giving them more time and attention to carry out their responsibilities in a meaningful and feasible manner.
Why this could be part of the calibration problem
In a previous article, I discussed the reported finding that only 67.7 per cent of 2,600 instructors correctly identified when a pilot needed further training, and argued for effective concordance programmes.
Overload is one plausible contributor to Instructor disagreement. The raters in the Tavares studies described how they coped with demanding assessment tasks: they prioritised, simplified and selected what to attend to. Those choices were not necessarily the same from one assessor to another. A concordance programme can measure the resulting spread without addressing that part of the problem.
Six design choices
Target, do not spread: For each phase of a scenario, decide which observable behaviours the situation is intended to bring out, and under what conditions. Keep the planned observation task manageable. Link those behaviours to the competencies they inform, rather than asking for a broad competency grade on the spot. That focus guides the attention; it does not prohibit recording important behaviour outside the plan. Nor should it force the crew into a rigid script. Across the programme, provide suitable opportunities to cover the required scope. Within each activity, assess what was actually demonstrated.
Separate observation from the broader judgement: Observe and note behaviour during the event. Assess the relevant behaviour against the agreed criteria while the context is still fresh, using the notes to support that assessment. Was it effective? Partly effective? Ineffective? What happened, and what assistance was needed?That is already a professional judgement. It is not yet a conclusion about the person’s competence across different situations. Avoid forcing both decisions into the same instant.
Split or sequence the observer load: Where two instructors are present, divide the observation focus deliberately: by participant, relevant behaviours or phase of the activity.
Build a clear opportunity for the behaviour to appear: A scenario should create a realistic reason for the trainee to act. Define the objective, the trigger, the relevant behaviour and the criteria for assessing it. The trigger creates an opportunity; it does not guarantee a particular response. If the scenario develops differently and the opportunity never arises, record that honestly. A missing opportunity is not evidence of poor performance.
Give the instructor an instrument, not a form: Structure notes around the event, the participant and the relevant observable behaviour. Keep the conditions, assistance and consequences attached to the observation. Make the record useful for debriefing now and review later.
ObserV, the competency tool I am building, is designed around this distinction: assessing observable behaviour within exercises and scenarios, then bringing that evidence together to support the broader competency judgement. The principle holds whichever tool you use.
Move the broader competence decision beyond the single session: This is the one that changes the most, and it deserves its own section.
A panel for the competence decision
The next question is whether the broader competency judgement—the one used to inform readiness and further training—could be made by a designated reviewer or, preferably, a panel examining evidence from several exercises and scenarios, rather than being reconstructed from a competency grade after every activity.
Medical education has worked with this approach for more than a decade, and can therefore be useful as guidance. Under programmatic assessment, many observations contribute to an accumulating record, and higher-stakes decisions draw on that record rather than on one event.
A 2018 study involving 962 veterinary students and approximately 328,000 assessment data points found reliable longitudinal assessment scores and patterns of development across competencies. That supports the value of evidence collected over time. [Bok and colleagues, 2018]
A 2019 study of a paediatric competency committee found that structured discussions brought together perspectives on both problematic and excellent performance, while a shared understanding of the criteria developed over time. [Duitsman and colleagues, 2019]
Transposed to cabin crew training, the proposed model is this:
During an exercise or scenario, the instructor assesses what the trainee actually does. Was the relevant behaviour effective, partly effective or ineffective in those conditions? What happened, what assistance was needed, and what was the consequence? These observations are recorded against the framework. They do not need to become a grade for the whole competency at the end of every scenario.
Across the programme, the observations build a record of performance. The same behaviour may appear in different tasks, under different conditions and with different levels of independence. A successful action after help from the Instructor tells a different story from a successful action performed independently.
A small qualified panel then examines the record. Is there enough evidence? Does it cover the relevant demands? Is performance consistent? Have significant weaknesses been resolved? After all considerations, they resolve the competency grade. Its authority to recommend or make a final decision must be clearly defined.
The review is more than an average of scores. Several effective actions do not automatically cancel a significant unresolved failure. Equally, missing evidence does not justify inventing a negative judgement. Sometimes the correct conclusion is that another suitable opportunity is needed.
The instructor in the room remains responsible for observing, assessing behaviour, intervening when necessary and debriefing. The later review brings those observations together to answer the broader question of competence.
What about the rules?
Pilot EBT provides a useful comparison, but it is not automatic permission to replace cabin crew checking with a panel review.
For cabin crew, ORO.CC.115 requires checks covering the relevant training elements, performed by appropriately qualified personnel, and expressly distinguishes CRM training. A longitudinal competency review and a required practical check should therefore not be treated as interchangeable.
An operator considering this approach would need to establish how the review fits its training and checking programme: which observations may contribute, which demonstrations and checks remain required, who is authorised to assess and decide, how gaps and failures are addressed, and what approval or acceptance is necessary.
To conclude
An instructor cannot observe everything. Good assessment design starts by accepting that limit: focus attention, assess behaviour in context, and build the broader competency judgement on evidence collected across the programme. Filling every box on a form is not the same as seeing the full picture.