Fintech Lab
Incident 65 decision steps
The webhook storm
Saturday 19:42. The payment provider's queue is replaying 72 hours of events into your /webhook in 90 minutes. Hope you indexed idempotency_key.

It's Saturday 19:42 WAT. You're the on-call engineer at a card-acquiring fintech. Your payment provider (Paystack, Flutterwave, Stripe, pick your poison) just made a routing change on their end. The unintended consequence: their webhook delivery queue, which normally trickles in real-time events to your /webhook endpoint, just started replaying the last 72 hours of deliveries in a 90-minute burst. Your endpoint is getting ~12,000 webhooks per minute, every one of which is a re-delivery of an event you already processed days ago. If your handler is idempotent, this is a load spike. If your handler is NOT idempotent, this is a double-booking catastrophe in real time. Walk it as the engineer who finds out which one you shipped.

Fintech Lab is a free, interactive lab for fintech engineers. Walk famous incidents from the engineer's seat at hour zero. See what else is here.

Search lessons

Type to find any of the 85 lessons. Press Enter to open.