Rust Queue Worker
In-memory job queue with a Tokio worker pool, exponential backoff + jitter retries, and a dead-letter list.
Problem
I wanted a small, readable model of a worker pool: bounded queue backpressure, retries that do not deadlock a full channel, and shutdown that does not drop jobs still being produced.
Architecture
- A CLI producer enqueues jobs into a bounded tokio::sync::mpsc queue. enqueue waits when full; try_enqueue returns Full.
- N worker tasks pull jobs. Failures are simulated with --fail-rate (no external I/O). On failure with attempts left, a detached task sleeps (exponential backoff with full jitter, capped) then re-enqueues. Sleeping inline on the same bounded channel can deadlock when the queue is full.
- When attempts are exhausted the job goes to a dead-letter list. Metrics count enqueued, processed (attempts), succeeded, failed, retries, and dead_lettered.
- Shutdown: pending counts unfinished work. Workers exit only after mark_producer_finished and pending is 0. Exiting on pending == 0 alone can drop later jobs if a fast worker finishes before the next enqueue.
Design decisions
Bounded queue for backpressure
Queue capacity is fixed. Producers block on enqueue or get Full from try_enqueue instead of growing without limit.
Trade-off: Protects memory; under sustained overload producers stall or drop depending on the call site.
Retries on detached tasks
Backoff sleep runs off the worker loop, then re-enqueues. Exhausted retries on a capacity-1 queue go to dead-letter without deadlocking.
Trade-off: Correct under a full channel, but the queue is not durable — a process restart loses in-flight jobs. There is no broker, ack, or reclaim.
Producer-aware shutdown
Workers wait until the producer is marked finished and pending is 0, so early exit cannot close the channel before later enqueues.
Trade-off: Shutdown is correct for this in-memory lab; it is not a production worker and cargo test does not measure latency.
Stack
What I'd improve next
- A durable broker (ack / reclaim) if persistence across restarts is needed.
- Latency measurements under a fixed fail-rate and queue capacity.
Results
- cargo test covering bounds, backoff, early-shutdown loss, capacity-1 exhausted retries
- 8/8