S3 read optimization
A file reader written for local disks behaves badly against object storage. Rebuilding it around parallel, chunked reads changed the access pattern rather than the code around it.
My contribution
I integrated the AWS S3 CRT library into the network file reader, used its multithreading capabilities, and implemented chunked reads for S3 data access, to improve remote media-file reading and overall processing performance.
Context
Media QC analysis reads very large files, routinely tens of gigabytes and sometimes more than a hundred. When those files moved from local and network-attached storage to S3, the analysis code did not change. The storage underneath it did.
The network file reader is the layer that hides that difference from everything above it, which means it is also the layer where the difference has to actually be handled.
The problem
Object storage is not a filesystem wearing a different hat. A local read is a syscall against a page cache; an S3 read is an HTTP request over a network with real latency. Code written against the first assumption issues access patterns that are merely suboptimal locally and genuinely pathological remotely: many small sequential reads, each paying a full round trip, none of them overlapping.
The reader was fundamentally serial against a backend whose throughput comes from doing many things at once.
Chunked, parallel reads
I integrated the AWS S3 CRT library into the network file reader and built the read path around its multithreading, with data access issued as chunked reads rather than a single serial stream.
The shift is from "read the next bytes and wait" to "have several ranges of this object in flight at once." Object storage rewards that: a single request is bounded by round-trip latency no matter how fast the link is, while several concurrent range requests fill the pipe. Chunking is what makes the parallelism expressible in the first place: you cannot overlap a read you have not divided.
Serial reads spend most of their time waiting on round trips. Chunked ranges in flight concurrently turn latency into something you overlap instead of something you pay.
Writing back is not symmetric
Related work in the same subsystem added a file-writer interface covering multiple content location types on both Windows and Linux. The two backends are deliberately not the same shape: network paths support seek-based editing, while the S3 writer supports sequential writing only, via multipart upload.
That asymmetry is honest rather than unfortunate. Object storage genuinely does not support rewriting the middle of an object, and an interface that pretended otherwise would push a failure that belongs at compile time into production instead.
The hard part
The access pattern is the API
The interface above the reader stayed the same, which is what made the change adoptable, because nothing upstream had to know. But keeping the interface while changing the access pattern underneath means the reader has to reconcile a sequential contract with a parallel implementation: buffering ranges that arrive out of order, and keeping enough in flight to be worth doing without holding more of a hundred-gigabyte file in memory than the machine has.
Remote media-file reading moved from serial requests to chunked, concurrent range reads through S3 CRT, improving remote file access and the analysis throughput that depends on it.
Described qualitatively: no measurement isolating this change from the other work in the same subsystem is published here.
Engineering takeaway
Object storage is not a filesystem. Code written against the wrong assumption is merely suboptimal locally and pathological remotely.