Mini-CDN
A CDN simulation with three edge cache nodes, round-robin routing, TTL/LRU caching and Prometheus/Grafana monitoring.
Problem
To understand how a CDN actually behaves (what happens on a miss, when an entry expires, when the cache is full, when the origin or an edge is down), I built a small one end to end. Each mechanism is a few hundred lines of Go that can be read and tested.
Architecture
- The router reverse-proxies each request to the next healthy edge (round-robin), probes every edge's /healthz every 2 seconds, and marks an edge unhealthy on a proxy error.
- Each edge has a disk cache with an in-memory index (size, cachedAt, lastAccess). Entries older than CACHE_TTL (5m in the compose file) are refetched from the origin.
- Purge is fanned out: POST /purge clears every edge, and DELETE /cache/<path> removes one file everywhere.
- Every service exposes Prometheus-format /metrics from a small hand-written registry. Grafana is provisioned with a dashboard.
Design decisions
Stream to the client while writing the cache
On a miss, the origin body is copied to the response and a temp file at the same time (io.MultiWriter). The temp file is then renamed into place.
Trade-off: No extra latency for the first client, and a partial download never becomes a cache entry. The response headers are sent before the copy finishes, though, so a mid-stream failure cannot become a clean error.
Serve stale when the origin is down
If the origin fetch fails and an expired copy exists, the edge serves it with X-Cache: STALE instead of a 502.
Trade-off: Higher availability, at the cost of possibly outdated content.
LRU bounded by bytes, with the new file protected
CACHE_MAX_BYTES (500 MB by default) caps each edge. Eviction removes the least recently accessed entries until the total fits, but never the file that was just written.
Trade-off: Simple and correct, but the victim search is a linear scan over all entries.
Path safety first
CleanRequestPath rejects '..' segments and SafeJoin keeps every disk path inside the cache root, both with unit tests.
Trade-off: A small amount of extra code that closes path-traversal bugs on both origin and edges.
Round-robin across all edges
Requests rotate over healthy edges regardless of the path.
Trade-off: Even load and an easy demo, but the same file ends up cached on all three edges (three misses per file).
Stack
What I'd improve next
- Single-flight concurrent misses for the same key, so a cold file hits the origin once per edge.
- Consistent hashing on the path so each file lives on one edge.
- An O(1) LRU (map + doubly linked list) instead of the linear scan.
- A repeatable load test that reports hit ratio and latency.
Results
- edge cache nodes behind a round-robin router
- 3