Experiments and negative results
Four times the obvious answer was wrong. Measuring first is the only reason that was discovered before shipping rather than after.
My contribution
These are decisions I made or contributed to, each one settled by measurement rather than by expectation. Three of the four ended in rejecting the approach that looked obviously correct beforehand.
Context
Performance work has a failure mode that is worse than being slow: being confidently wrong. A newer library, a bigger model, a smaller buffer: each is the kind of change that sounds right in a design review and gets adopted on that basis.
These four were adopted or rejected on measurement. Three of them contradicted the expectation going in.
The problem
What follows is a short record of hypotheses that were tested and, mostly, disproved, kept here because the reasoning is the transferable part rather than the outcomes.
Four hypotheses, tested
- Hypothesis
- Upgrading the integrated signal-processing library from a 2010-era version to a current one will be faster, since the current version uses AVX2 and the old one does not.
- Experiment
- Integrated the newer version into the backend and benchmarked it against the existing version on the same workloads.
- Benchmark
- The newer AVX2-enabled version measured worse than the version it was meant to replace.
- Decision
- Did not ship the upgrade. Instead implemented the specific hot functions — windowed row and column sums, and mean/standard deviation — directly using AVX2 intrinsics, and shared those implementations with the team that owned the calling code.
- Hypothesis
- A purpose-built ECAPA language identification model will be the right choice for the language identification check.
- Experiment
- Integrated the model and tested it for correctness and performance against the languages the product supports.
- Benchmark
- The model did not meet the requirements for this use case.
- Decision
- Rejected it and moved the work to Whisper, which became the basis of the shipped implementation.
- Hypothesis
- The largest available speech model will give the best language identification accuracy.
- Experiment
- Evaluated five configurations on the same 981-file corpus, holding preprocessing constant.
- Benchmark
- The 2.9 GB model scored 98.47%. The 138 MB model scored 99.18%, better accuracy at one twenty-first of the size.
- Decision
- Shipped the 138 MB model. Language identification is a narrower task than transcription, and the largest model is optimized for the harder problem.
- Hypothesis
- Shrinking the internal read buffer for interleaved MXF will reduce wasted reads and speed up analysis.
- Experiment
- Benchmarked the existing 256 KB buffer against 10 KB, 50 KB and 100 KB.
- Benchmark
- 50 KB was the best of the three alternatives, and every alternative was slower than the original 256 KB.
- Decision
- Kept 256 KB and changed the surrounding problem instead. Once chunk information came from the index table rather than from speculative reads, a 64 KB buffer became viable: the same change succeeded only after the reads around it became precise.
The hard part
Being willing to not ship the work
The library upgrade was already integrated and working when the numbers came back worse. Finishing it would have been easy, defensible, and wrong.
The cost of shipping a change that does not help is not zero: it is a slower system plus a dependency upgrade nobody will question again for years, because it is already done.
Three of these four hypotheses were wrong, and each was disproved by measurement before it reached production. The fourth was right only in a different order than expected.
Engineering takeaway
Benchmark before believing. The newer library, the bigger model, and the smaller buffer were all wrong, and only measurement caught it.