Reading less of the file

For MOV and MP4, the sample tables in the container describe exactly this: chunk offsets, sample sizes, the mapping from samples to chunks, and the edit list. I added a step that runs before the reader thread starts and constructs the chunk map directly from those tables, after which the tracks it covers can be dropped from the read list entirely.

For ProRes and DNxHD, that removes the essence read completely: the access points are built from the map, and the data itself is never touched at this stage.

AVC-Intra could not be handled the same way. Building access points for it needs the sequence and picture parameter sets, and those are properties of the bitstream, not the container, and no amount of metadata reading recovers them.

But the H.264 specification places both at the start of each sample. So instead of skipping the read, I bounded it: the reader fetches only the first 5000 bytes of each sample and zero-fills the remainder of the buffer. The parameter sets are always inside that window, and the rest of every sample stays where it is.

MXF stores its essence interleaved across partitions, which defeats the MOV strategy: even if you know exactly which bytes you want, reading around them pulls in the neighbouring tracks you were trying to avoid.

The way through is the index table, together with the random index pack that locates each partition. From a partition pack you can derive where essence begins, accounting for the header and index byte counts, the partition offset, and the alignment grid the format uses to pad KLV elements onto byte boundaries. From the index table you can then locate an individual picture item within an edit unit, whether edit units are fixed-size or variable, and derive its length by computing the offset of the next element and subtracting.

Each picture item and its trailing filler becomes one chunk. A format-specific step then skips the filler when populating track buffers, so the padding that makes the arithmetic clean never reaches the analysis code.

MXF analysis used a 256 KB internal read buffer. With interleaved data, a buffer that large reads neighbouring tracks by accident, so the obvious move was to shrink it.

I tried 10 KB, 50 KB and 100 KB. 50 KB was the best of the three and all of them were slower than the original 256 KB. The buffer was not the problem: while the pipeline still had to read essence speculatively to build chunk information, a large buffer was doing useful work by amortising round trips, and shrinking it just added round trips.

Only once chunk construction came from the index table did a small buffer make sense: with reads now precisely targeted, 64 KB at both the storage and format layers stopped pulling in data nothing would use. The ordering mattered: the same change that failed on its own succeeded once the reads around it became precise.