Distributed processing architecture
Taking a single-machine media analysis engine and splitting one large job across a fleet of workers: queue, locking, ordered aggregation, and failure handling.
My contribution
I designed and implemented this distributed task-processing architecture from scratch, including the coordinator, the Redis-backed job queue, the worker client, distributed locking, task splitting, result aggregation, and retry and error handling. I also wrote the design document.
Context
Automated quality control on a large media file is a long, CPU-bound job. On a single machine, the only lever is the number of cores in that machine, and past a point adding cores stops helping because the job is one indivisible unit of work.
The system needed to distribute a single large analysis task across many machines, and then reassemble the output as if one machine had produced it.
The problem
Splitting the work is the easy half. The hard half is that the output has to be identical to the single-machine result: a report assembled from twelve workers must read exactly as it would have if one process had produced it, in order, with no gaps, no duplicates, and no dependence on which worker happened to finish first.
On top of that: many workers and a coordinator all contend for the same queue, any worker can die mid-task, and intermediate results accumulate in shared storage faster than they are consumed.
Shape of the system
A coordinator process divides a task into sub-tasks, assigns each an identifier, and pushes them onto a job queue held in Redis alongside the metadata each sub-task needs. Worker clients pop from that queue, fetch the metadata for the sub-task they claimed, run the analysis engine over their slice, and write structured output back to the shared store.
A worker signals completion by setting a transfer-complete flag for its sub-task. The coordinator watches for those flags, pulls finished output, merges it into the final report, and deletes the intermediate data it has consumed.
One coordinator, one shared queue, N stateless workers. Workers hold no durable state, so a worker dying costs one sub-task rather than a partial job.
Contention and ordering
The queue is read by every worker and written by the coordinator, so claiming a job is a critical section. I used a distributed lock to serialise it: without one, two workers can pop the same identifier under the right interleaving, and the resulting duplicate work is silent: the report still assembles, it is just wrong.
Result aggregation is strictly sequential even though completion is not. Workers finish out of order, and the coordinator merges strictly in sequence: after sub-task five it waits for six, even if seven through ten are already sitting complete in the store. That constraint is what makes the distributed output byte-for-byte comparable to the single-machine output.
Failure and cleanup
Sub-tasks are retried when they fail, which is only safe because workers hold no durable state: everything a worker needs comes from the shared store when it claims the job, so a retried sub-task on a different machine is indistinguishable from the first attempt.
Cleanup is part of the protocol rather than a background chore. Once the coordinator has merged a sub-task’s output, it deletes that sub-task’s metadata and intermediate results. Without that, a long job accumulates every intermediate artefact it ever produced in memory-resident storage, and the system fails on its largest and most important inputs.
The hard part
Out-of-order completion, in-order output
Workers finish whenever they finish. The report has to be assembled in order. Holding those two facts together is the design.
The naive fix — have the coordinator wait for everything, then merge — means intermediate results for the entire job are live simultaneously, which is exactly what a large job cannot afford. Merging strictly in sequence as results arrive lets the coordinator free each sub-task’s data as soon as it is consumed, so peak storage tracks the window of outstanding work rather than the size of the whole job.
A single analysis task can be divided across a fleet of worker nodes and reassembled into one ordered report, with duplicate work prevented by distributed locking, failed sub-tasks retried on any available node, and intermediate storage released as results are merged.
- Scaling
- Throughput scales by adding worker nodes rather than by growing a single machine.
- Correctness
- Ordered aggregation makes distributed output equivalent to single-machine output.
- Fault tolerance
- Stateless workers make sub-task retry safe on any node in the fleet.
No published benchmark exists for this system.
Engineering takeaway
Distributing work is the easy half. Making distributed output indistinguishable from single-machine output is where the design actually lives.