The bug that only shows up at 300 requests a second
Everything passed locally. Everything passed in staging. Then Monday morning happened.
A renewal window is the worst thing that can happen to an insurance platform. Traffic goes up maybe eight times for about four hours, and everything you got away with the rest of the year stops working at once. We had a page that took 400ms on a normal Tuesday and nineteen seconds on the first Monday of the month.
The page itself was fine. That's the part that takes people a while to accept. Nobody had written slow code — they'd written code that was only correct when it was alone.
Three things were happening at once
- A cache that stampeded. The entry expired, two hundred requests all missed, and all two hundred went to rebuild it. The database did the same expensive work two hundred times and nobody got an answer until it finished.
- A lock held across an await. Someone had taken a lock, called out to a service, and kept holding it. Under load that turns a 50ms call into a queue of everyone behind it.
- Connection pool exhaustion, which is what you see first and what tells you least. By the time the pool is empty, the real cause is already three layers down.
The fix for the stampede
Single-flight. The first request to miss the cache does the work; everyone else waits on the same task rather than starting their own. About fifteen lines.
private readonly ConcurrentDictionary<string, Lazy<Task<Quote>>> _inFlight = new();
public Task<Quote> GetAsync(string key, Func<Task<Quote>> load)
{
// Lazy<T> guarantees the factory runs once even if a hundred
// threads reach GetOrAdd in the same millisecond.
var lazy = _inFlight.GetOrAdd(key,
_ => new Lazy<Task<Quote>>(load, LazyThreadSafetyMode.ExecutionAndPublication));
var task = lazy.Value;
task.ContinueWith(_ => _inFlight.TryRemove(key, out _),
TaskScheduler.Default);
return task;
}The lock was easier to fix and more embarrassing: move the remote call outside the lock, and only take the lock to write the result. If you ever find yourself holding a lock across an await, you have a queue, not a cache.
What I'd do differently
I'd have load-tested the shape of the traffic, not the volume. We'd run ten thousand requests spread evenly and it was fine. Real traffic arrives in clumps, all asking for the same five things. A flat load test tells you almost nothing about contention.
Correct under one request is a different property from correct under three hundred. Most code is only tested for the first one.