Loading the study.
Built with Claude, an AI coding assistant. Nothing on this page comes from Rocket Lab or describes its systems.
Each bar is the share of faults I planted into real test logs that a check caught: dark for this checker, mid blue for the plain checks a stand would otherwise run (a flatline over 100 samples and a timestamp-gap monitor), light grey for a range limit set from the stand's other runs, and dashed for flagging at random with the checker's number of false alarms.
Drag it. Left makes every threshold stricter, so the checks flag more healthy channels; right relaxes them. The plain checks scale with it; the range limit moves as tight as it can go without flagging more healthy sites than the checker. The middle is the setting fixed before the measured runs were checked.
Everything below is how this was measured and where it fails. The planted faults sit in copies of real runs; the real problems in the published logs are counted separately, in act two.
A simplified stand on the left; on the right, the rows it logs, with the three faults marked in red.
| time s | tank psi | chamber psi | temp C | valve cmd | valve out V |
|---|---|---|---|---|---|
| 0.010 | 702 | 12 | 21 | 1 | 12.0 |
| 0.020 | 698 | 140 | 21 | 1 | 12.0 |
| 0.060 | 681 | 388 | 21 | 1 | 12.0 |
| 0.070 | 677 | 402 | 21 | 1 | 12.0 |
| 0.080 | 672 | 411 | 21 | 0 | 12.0 |
An illustration, not data. Every value here is inside a plausible range, which is the point.
The data comes from three projects that publish raw test-stand logs under an open licence: a student liquid-engine controller on GitHub, a cryogenic feed-line rig at the German Aerospace Center (DLR), and an igniter bench at TU Delft. Several student teams publish richer logs without a licence; those are not used. Nothing on this page comes from Rocket Lab or says anything about its stands, sensors or software.
Why this mattersBefore any count means anything, you need to see what a pass and a fail look like on a real channel.
Click a run above. The charts and the failed lines of its report change with it.
Shaded in red: what the checker flagged. Shaded in purple: a real problem it missed. Thin grey lines: the sensor beside the one in question, where the stand has one.
One run shows the checker can read a log. The next question is how it does on every public run, including the places where it is wrong.
Why this mattersA checker that only finds faults I planted could still be useless on real logs, where problems are rarer and stranger.
I ran every check on every run and labelled each flag myself after looking at the raw data: a real problem in the published log, or a false alarm. Real problems the planned checker did not flag are listed as missed, with what the other checks made of them. The same finding in many runs of one day is one row.
Click a row to read why it got its label.
The real logs give a few dozen problems, too few to measure detection on. So the next act plants hundreds of faults of known size into real runs, and lets you plant one yourself.
Why this mattersA planted fault has a known place, time and size, so whether a check caught it is a fact rather than a judgement.
This is the 2019 hotfire log from the engine-controller project, the cleaner of its two logs. Pick a channel or valve, a kind of fault and its size. The checks run again in your browser on the changed log, and the three verdicts below update.
Drag the sliders. The grey line is the log as published, the dark line the log with your fault in it, and the shading is where the fault sits.
Each panel is one kind of fault, planted times at each size into the measured runs. Up is the share caught at the planned setting; across is the size of the fault. A dot on the floor means none of the 30 was caught.
Catching a fault is half the job. The other half is a table an engineer can review before the data is used, which is the next act.
Why this mattersThe output of a data check is not a chart; it is a pass or fail per channel that an engineer reviews, with the calibration records, before the data is used.
Pick any run. The table lists every record, channel and valve the channel map names, with the evidence for each verdict in plain words. It covers the integrity of the data only.
The coda says how all of this was built and what it cannot do.
The numbers above could have been typed by hand. So the page recomputes the headline counts from the per-site scores it ships, and re-checks both hotfire logs with its own JavaScript copy of the checker, then prints what it gets beside what the Python build got.
Running.
Nothing here is a trained model, but a threshold chosen by looking at data is still a model, so it gets the same card.
I looked at the raw data behind every flag and wrote down whether it is a real problem in the published log or a false alarm, with the reason, before the planted-fault results were counted. The reasons are in act two. I wrote the checker too, so these labels are not independent; each one states what in the data supports it, so you can disagree with a specific row.
It cannot see a sensor that freezes but keeps its noise. A blocked pressure port does this: the transducer keeps reading the trapped pressure plus its electrical noise. Catching it needs a second sensor at the same point to compare against, as the DLR rig has at each station, or a check of the pressure against the valve sequence. This checker has neither.
The planned checker missed the saturated dynamic sensors in the DLR tests. They sit flat at their top for , and the stuck check waits on fast channels. I saw a research note about these flat tops before fixing the thresholds and did not tune for them. A rail check for a channel flat at its own highest or lowest value, added after an outside review, catches all of them and the converter codes; it is reported apart from the planned numbers.
The freeze limits were fitted on short student runs. On a stand that holds a steady state, such as a chilled-down line or a pressurised hold, a low-resolution channel can keep one reading far longer and these limits would raise false alarms; the two thermocouples the plain flatline wrongly flags show the effect. A next version should scale each limit by the channel's resolution and noise floor, not by a stopwatch.
Calibration is out of scope. Pre-test zero against a known ambient reading (a vented tank should read zero gauge), post-test zero drift, and calibration traceability are checks I have not built; a reviewer would need them beside this table.
Gaps are inferred from timestamps. Where a logger writes a sample counter or an overflow flag, that is the better evidence, and the check should use it; none of these logs carries one.
Valve timing on the Delft bench is only as good as its valve log. The valve states carry no timestamps and no documented rate, and their record length disagrees with the thermocouple record by up to percent in one run. A valve that answers half a second late cannot be told apart from that drift.
Three stands is a small sample. The thresholds are per channel type, not per stand, and a new stand would need its own fit runs.
This is a post-test check in Python and JavaScript. It is not a real-time interlock, not PLC logic and not LabVIEW, and it reads files after the test rather than signals during it.
The range limit here is a stand-in. A real stand's limits come from its test plan. Mine come from the stand's other runs, which suits a range limit as well as any data-driven choice can, but it is still my choice.
Six, all listed with reasons in PLAN.md: planting faults only where no check already flags; starting a planted valve fault where the valve check starts looking; counting the range limit on channels only; and, after an outside review, adding the rail check, adding the plain checks as the named comparison, and counting a planted fault as caught only when it changes the verdict. None changed a planned threshold, and the planned checker's numbers are the same as before the review.
Changes to the data: the hotfire logs are used as published; the DLR workbooks were converted to arrays; the Delft pressures were converted from volts with the README's formula; and every chart is thinned for drawing while keeping each window's minimum and maximum. The checks always run on the full data.
python3 study/fetch.py downloads the three sets and checks them against their published
checksums; python3 study/run_study.py regenerates every number on this page; pytest tests
runs the unit tests. Code and plan: github.com/bolgacg/stand-check.