The Evidence Ladder: What Three Airworthiness Directives and a Preliminary NTSB Update Can Teach You About Not Jumping to Conclusions
There’s a moment in every forensic engineering case — and in every good accident investigation — where the temptation to know outweighs the discipline to wait. You have a failed part, a damaged aircraft, an injured party, and a client or a public that wants an answer now. The engineer’s job is to resist that pull long enough to build the case properly.
I call the discipline the evidence ladder. It has five rungs, and you don’t skip any of them:
Observation — what physically happened, documented without interpretation
Exclusion — what can be ruled out, and on what basis
Event sequence — the order in which things occurred, established from data, not narrative
Contributing factor — what made the event possible or worse
Final cause — the conclusion, reached only after the first four rungs are solid
This week’s aviation news handed me four live examples of engineers and investigators working this ladder in public, at different rungs, with different levels of caution. They’re worth walking through together, because the differences are instructive.
Rung 1–2: Preliminary data, held at arm’s length
The NTSB’s update on the September 6 Boeing 767 runway excursion at Miami — 21 Air Flight 7598 — is a textbook example of Rung 1 discipline. The Board released preliminary cockpit-voice-recorder and flight-data-recorder information, and it went out of its way to flag that the timing and descriptions remain preliminary and unsynchronized.
That caveat is not boilerplate. It’s an engineering statement. Raw CVR and FDR data before synchronization is a pile of timestamped observations, not yet an event sequence. You can see things happened. You cannot yet say in what order, and you certainly cannot say why. The NTSB is sitting on Rung 1, publishing what it has, and explicitly refusing to climb higher until the data lets it.
The NATS investigation into the UK air-traffic-control disruption is doing something similar at Rung 2. NATS said it did not believe a cyberattack was responsible — that’s an exclusion, not a conclusion. It narrows the causal tree (software failure, hardware failure, operational-resilience gap, external threat) without yet claiming to know which branch is correct. Good forensic practice: rule out the sensational hypothesis early if the evidence supports it, but don’t let ruling one thing out feel like ruling something else in.
Rung 3–4: Root-cause pathways, made operational
The FAA’s three new airworthiness directives this week are further up the ladder, because each one already encodes an established event sequence and contributing factor — and converts it into a control.
AD 2026-18-03 addresses tool-mark damage in A320-family main-landing-gear support-rib lugs. Contributing factor: a manufacturing/handling process introduced damage that wasn’t part of the original design intent.
AD 2026-17-09 addresses missing chamfers under critical A350-1041 pylon fasteners. Contributing factor: a tool-control or manufacturing-spec gap — a geometry detail that should have been present wasn’t.
AD 2026-18-01 adds repetitive operational tests for ATR 42-500 cargo-compartment ventilation shut-off valves. Contributing factor: maintenance-program drift — a component whose failure mode wasn’t being caught by the existing inspection interval.
What I want you to notice is that these are three different rungs of causal pathway — supplier process, design/manufacturing specification, and maintenance-program adequacy — arriving at the same kind of output: an inspection, a life limit, or a repetitive test. That’s the whole point of the evidence ladder. You don’t need every root cause to look alike. You need every root cause to be traceable back to evidence, and you need the fix to match the actual failure mode, not a generic one.
The Trap Above the Ladder: Correlation Is Not a Rung
There’s an old standard worth repeating until it’s boring: correlation is not the leading note in forensic work, cause is. It’s tempting to treat a pattern of similar-looking incidents as if the pattern itself were the finding. It isn’t. The pattern is a question, not an answer.
The same discipline applies to how you handle a single anecdote inside a big-picture trend review. Citing one dramatic incident while you’re surveying broad trends is a guideline violation, not a rhetorical flourish — it lets one vivid case quietly stand in for a statistical claim it can’t support.
The harder question sits underneath both of those: how do you know today’s issue is actually related to something that happened five weeks ago, or five months ago? That’s the trifecta of grouping — you’re implicitly claiming a shared cause across time, and you have to earn that claim with a grouping variable, not a gut feeling.
Take AD 2026-18-03, the A320-family main-landing-gear support-rib tool-mark damage, as the working example. The AD itself already names the fix — inspect for the damage, repair or replace as needed. But “tool marks” as a bare fact is not enough to pass judgment on what produced them, and the AD doesn’t have to settle that to do its job. The scope question — is this one aircraft, one operator, one manufacturing run, or the whole fleet — is a separate finding, and it depends entirely on what you pair the tool marks with. “Tool marks” is a single visible defect signature — but it can arise from at least three different processes, and each one clusters differently in time and space:
Manufacturing-induced marks cluster by production date and lot — same tooling, same shift, same supplier batch. If you group the affected aircraft by build date and the marks line up with a tooling change or a specific production run, that’s your causal grouping.
Repair-induced marks cluster by service event — same MRO facility, same mechanic crew, same repair procedure — regardless of when the aircraft was originally built. Build-date grouping would show nothing here; service-date grouping would show everything.
Fleet-wide procedural marks cluster by directive or procedure-issuance date across the entire fleet, independent of both build date and repair date, because the common cause is a maintenance instruction or ground-handling practice everyone started following at the same calendar moment.
Same defect, three different clocks. Pick the wrong clock to group by and you’ll either see no pattern where one exists, or manufacture a false pattern where none does.
That’s also where the “none found or 1000 found” problem comes from. Search a case history or a literature base too narrowly and you’ll get zero hits — not because nothing relevant exists, but because you grouped by the wrong variable and missed it. Search too broadly and you’ll get a thousand hits that share only surface keywords, none of which actually share a causal mechanism. Knowing the history — being able to state the relevant facts across a broad range of prior cases — is what lets you pick the grouping variable that produces a meaningful result count instead of an artifact of your search terms.
That search problem gets worse when the search itself is scoped too narrowly by source, not just by keyword. The official record — ADs, NTSB reports, shop work orders — is where most investigations start and, too often, where they stop. Two sources routinely left out are textbooks and community chat boards, and they fail in opposite directions. Textbooks encode the generalized, already-digested failure modes for a class of design — useful as a baseline of what’s known to happen, but slow to update and silent on anything too new or too narrow to have made it into a published edition. Community chat boards — mechanic forums, type-specific owner groups, shop-floor message boards — carry the opposite problem: they’re current and specific, often reporting a pairing fact (which shop, which environment, which batch) years before it reaches an official database, but they’re unfiltered, and treating a forum post as equivalent to a logged finding is its own kind of premature synthesis. Leave either source out and you bias your grouping toward whatever’s easiest to search — official databases — instead of toward what’s actually true.
The evidence ladder gets you to a defensible cause for a single event. Grouping discipline is what lets you defensibly say a cause is shared across events — and it’s every bit as easy to get wrong.
From Outlier to Emerging Trend: The Label Changes Before the Cause Does
Here’s the part that trips people up in practice: the first time a tool-mark case turns up as a root cause, it gets labeled an outlier against whatever large trend you’re already tracking. That’s the correct label at n=1 — but “outlier” is a label, not a fact. It’s a placeholder for “insufficiently paired,” not a verdict. Treating it as a settled fact is exactly the premature-synthesis error the evidence ladder is built to prevent; treating it as a fact-in-waiting, still needing to be paired, is what actually speeds the research up. An outlier paired with the right relevant fact from the start gets resolved fast, one way or the other. An outlier filed away as just “an outlier” sits there doing nothing until someone rediscovers it from scratch.
Finding the outlier is not the finish line — it’s the point where the research widens, not where it stops. Internally, that means checking your own case files, your own shop’s history, your own fleet records for anything that resembles it, paired or not. Externally, it means checking whether the same pattern shows up somewhere you don’t control: other operators, other shops, the manufacturer’s own service bulletins, the textbooks and chat boards mentioned above. Calling something an outlier and closing the file is how a real emerging trend stays invisible for years — the research obligation doesn’t end at the label, it starts there.
But the label has a shelf life, and the trigger that ends it isn’t the raw count — it’s the count paired with a relevant fact. Every finding gets stronger when it’s paired with something outside itself: the operator, the maintenance event, the manufacturer, the operating environment. A bare tool mark is a fact. A tool mark paired with who found it, when, and under what conditions is tribal knowledge. Three quick illustrations of what pairing looks like in practice:
Pairing with an operator’s procedure. Say Birdway Airlines runs annual heavy checks, and the same tool-mark pattern shows up at every annual on that fleet. That’s not three coincidences — it’s one finding paired with a maintenance-schedule clock. The pattern is telling you to look at Birdway’s annual-check procedure, not at the airframe design.
Pairing with logged history. Say an outlier repair-shop finding shows up once, gets dismissed, and then shows up again at the same shop eighteen months later. The mistake isn’t missing it the second time — it’s that the first occurrence got filed away instead of logged with the relevant facts: which shop, which mechanic crew, which procedure, which tooling. If that first outlier had been logged properly instead of closed out, the second occurrence would have been an immediate two-count trend instead of a fresh outlier that has to rediscover its own history. And don’t assume the second occurrence is the only other one, either — a repair-based root cause found at one shop is not guaranteed to be the first instance anywhere, or the only shop it’s happened at. It’s just the first one that got paired.
Pairing with environment. Say Fastair-manufactured business jets keep showing a short service life specifically when the aircraft is based in desert environments — but not elsewhere. That’s a manufacturing finding that only becomes visible when it’s paired with basing location. Judge it on manufacturing data alone and you’d conclude the whole fleet has a service-life problem. Pair it with environment and the finding narrows correctly: it’s a desert-basing interaction, not a universal defect.
These three cases group at very different speeds, and that speed difference is itself worth knowing going in. A fleet-wide or manufacturer-level issue — like the Fastair case — is already centralized: one production lot, one design, one directive, visible across every operator at once. It groups fast, almost by construction. A repair-based root cause is the opposite: it’s scattered across independent shops, crews, and procedures with no central log connecting them, so the same underlying cause can sit invisible for years, one unpaired outlier at a time, before enough of them get logged with the same relevant fact to group at all. Don’t expect a repair-based trend to announce itself the way a fleet or manufacturer trend does — it has to be found, shop by shop, on purpose.
None of these become an emerging trend on tool marks alone. They become one once you’ve paired three or more occurrences against the same relevant fact — same operator procedure, same shop history, same environmental exposure — not just three occurrences that superficially resemble each other. At that point “outlier” stops being the honest word, and the investigation posture has to shift: from “explain away this one case” to “find out how far this pattern already extends.”
The threshold matters because it’s the trigger for going back and re-examining every prior case you’d previously closed out as an unrelated outlier. If you logged the pairing facts the first time — which operator, which shop, which environment — that re-examination is fast and specific. If you filed the first outlier away without them, you’re stuck re-opening everything, because you don’t know which prior “outliers” belong to the same cluster and which genuinely don’t. That’s the real cost of treating an outlier as closed instead of as pending.
Why this matters if you’re not an accident investigator
If you’re an expert witness, a design engineer, or just someone trying to write clearly about a failure, the evidence ladder is a discipline against two very human failure modes:
Premature synthesis — reaching for “here’s what happened” before the event sequence is actually established. The NTSB’s Miami caveat is a model for resisting this. So is NATS declining to speculate about a cyberattack it hadn’t confirmed or ruled out with full confidence.
Padding — manufacturing a conclusion because a report is expected on schedule. This week’s digest deliberately excluded new forensic-engineering court cases and root-cause reports because nothing met the inclusion bar. That’s Rung 1 discipline applied to reporting itself: no new high-confidence evidence, no new conclusion, full stop.
The takeaway for your own case files
Next time you’re building a forensic report — or explaining one to a jury, a client, or a Substack reader — try writing it in five explicit sections, one per rung. Force yourself to write the Observation section using only what a camera or a sensor recorded. Force the Exclusion section to name what you ruled out and why. Only let yourself write “Final Cause” once the first four sections hold up on their own.
It’s slower. It’s also the only way an opinion survives cross-examination — or, for that matter, survives being right.
Sources: NTSB preliminary update on 21 Air Flight 7598 (Miami, Sept 6, 2026); FAA airworthiness directives 2026-18-03, 2026-17-09, 2026-18-01; NATS UK ATC system investigation, FlightGlobal.
Herbert Roberts, P.E. is a licensed professional engineer with 32 years in aviation research and development across three companies, and has spent eight years analyzing accidents under his P.E. license.