The webhook storm

Saturday 19:42 WAT

Datadog fires: `/webhook` request rate jumped from a baseline of 8/min to 720/min in under 30 seconds and is climbing. Latency is still healthy. Your handler is processing every request and returning 200. The handler's logic is the simple version: parse the event JSON, look up the user, post a journal entry, return 200. You glance at the queue depth on the provider's status page: 'redelivering buffered events from 2025-09-15 onward.' That's 72 hours of buffered events.

What's the first move?

• ✓ Pull the last 200 webhook events from your DB and check whether each event_id has been processed multiple times today

  Right. The first question is: are we double-processing? A SELECT against your journal_entry table grouped by source_event_id tells you immediately. If you see counts > 1 in the last 5 minutes, idempotency isn't holding and you have a live correctness incident. If you see all 1s, you have a load spike to manage, not a correctness break.

• Reject every webhook with 503 until the storm passes

  Webhook providers retry on 5xx with backoff. Returning 503 for 90 minutes guarantees you receive the same events AGAIN later (the provider's retry queue stacks on top of the replay queue). Plus, you're throwing away every real event mixed in with the replays. Process them; just make sure you can do so safely.

• Tweet that the app is down

  The app isn't down. Customers are reading the app fine. Premature comms about an incident you haven't characterized yet creates the panic the actual situation doesn't justify.

• Restart the webhook service to drop the in-flight queue

  Restarting drops in-flight requests, which the provider will retry. You've made the queue depth worse and you still haven't answered the actual question: are we double-processing?

Saturday 19:51 WAT

Nine minutes in. The handler is sustaining 720 req/min with ~40ms p99 latency. Most of those requests are immediately failing on the INSERT (unique constraint on source_event_id, idempotency_key) and returning 200 without posting a journal entry, the at-least-once-safe path. But your error logs are exploding with `duplicate key value violates unique constraint "journal_entry_idem_uniq"` exceptions; the handler is logging each unique-violation as an ERROR. Datadog's error-rate dashboard is bright red.

What now?

• ✓ Lower the log level for unique_violation specifically; it's expected behaviour under replay, not an error

  Right. unique_violation IS the idempotency mechanism. It's not an error, it's the system working as designed. Logging it as INFO (or even DEBUG) makes your dashboards reflect actual problems and stops alert fatigue. The exception itself should still be CAUGHT and turned into a 200 ack; just don't shout about it.

• Drop the unique constraint to stop the exceptions

  Catastrophic. The unique constraint IS the thing protecting your ledger from double-booking. Dropping it during a storm means every replayed webhook posts a fresh entry. Your wallet balances triple before the storm ends.

• Add a layer of Redis dedup in front of the handler

  Architecturally appealing, operationally the wrong move right now. The constraint is already doing the deduplication correctly. Adding a Redis layer is a multi-day project, the current behaviour is correct, only the log noise is the problem. Fix the log level today; consider Redis later if profile shows the constraint is the actual bottleneck.

• Mute the alert and ride it out

  Muting alerts during an active incident is how the next real incident gets missed. Fix the underlying log level so the alert correctly distinguishes signal from noise.

Saturday 20:17 WAT

Thirty-five minutes in. The replay is past its peak and starting to taper. Out of the ~18,000 events processed so far, the metrics are: 17,994 deduped via the unique constraint (correct, expected), and 6 actually new events that happened during the storm window (customers actually using the app on a Saturday night, processed correctly into the ledger). One detail caught your eye in the dashboard: 3 of the deduped events had a different payload than the original. The provider's redelivery isn't bit-for-bit identical, the amount in the redelivered payload differs from what was originally booked by ₦1-₦4 per event.

What do you do with the 3 payload mismatches?

• ✓ Log each mismatch as a structured event for the reconciliation team to investigate post-storm; do not change the already-posted ledger entries

  Right. The original entries posted at the original amounts and have already been reflected in customer balances and statements. Reversing or rewriting them mid-storm risks corrupting customer state worse than the ₦1-₦4 variance. Capture the mismatch as evidence, investigate offline, and post correction entries through the normal reconciliation process if needed.

• Update the existing journal entries to match the redelivered payload

  Lesson 5 territory: journal entries are immutable. Updating in place corrupts the audit trail. Any correction must be a NEW entry, not a mutation. And you don't yet know which version of the payload is authoritative.

• Trust the redelivered payload and post adjustment entries immediately

  You don't know yet which version is authoritative, the original at receipt time or the redelivery. The provider's redelivery could be the corrupted version. Capture the discrepancy as evidence, investigate before any adjustment.

• Refund the affected customers ₦4 each as goodwill

  Premature. You haven't confirmed any customer was over-charged. The original payload might have been right; the redelivery might be wrong. Pay-without-investigating is how fintechs end up giving away money to people who weren't affected.

₦9 total provision (3 events × max ₦3 each) booked to Cost: Reconciliation Reversals against FBO. This is a tiny provision that holds the worst-case true-up amount until reconciliation completes; if the original payloads turn out to have been right, the provision reverses cleanly tomorrow. The customers are not credited yet, and the original entries are untouched.

Saturday 21:12 WAT

The storm ends. Inbound rate is back to baseline within ~5 minutes. Final tally: ~21,400 webhook events received during the 90-minute window, 21,394 deduped correctly, 6 new events processed correctly, 3 payload mismatches logged for follow-up. Customer balances are intact. Zero customer-facing impact: the app stayed up, withdrawals processed normally, the only people who knew this happened are you and the support team (who saw a brief uptick in 'is my balance correct?' tickets that went away when the answer was 'yes'). Comms team is asking if you want to post anything.

What goes out publicly?

• ✓ Nothing public; brief internal note in the engineering channel; ack to the provider's on-call that you weathered it

  Right. There was no customer impact. Posting about an incident that didn't affect anyone CREATES the concern instead of allaying it; customers read 'incident' and immediately distrust the very service that just proved its resilience. Save the public comms for events where customers felt something.

• Publish a long postmortem detailing the storm and your handling

  Public postmortems are valuable signal-of-trust AFTER customer-impacting incidents. For zero-impact incidents they read as engineering vanity at best, scare-the-customer at worst. The internal write-up matters; the public one doesn't.

• Tweet a victory lap thread about how your idempotency keys saved you

  Tempting but every fintech that does a victory lap publicly invites the next incident attention from anyone with grievances about the platform. Quiet competence beats loud competence.

• Email every customer to reassure them the incident is over

  Most customers didn't know there was an incident. An unsolicited 'don't worry, everything is fine' email IS the thing that makes them start worrying. Don't manufacture distress.

Monday 10:00 WAT

Monday morning post-mortem. Three engineering follow-ups are on the table. The reconciliation team came back with their finding on the 3 payload mismatches: in each case the ORIGINAL webhook payload was correct; the redelivered version had been corrupted by a serialization bug on the provider's side during their backlog flush. The ₦9 provision reverses cleanly today. One structural follow-up is going into the post-mortem doc.

Top engineering follow-up?

• ✓ Add a CI test that replays the last 24 hours of webhook events N times against a fresh ledger and asserts the final balances are identical to one replay; pages on regression

  Right. Idempotency is the kind of property that holds today and gets quietly broken in a refactor 8 months from now if no test enforces it. A replay test in CI catches regressions the day they ship instead of the day the next webhook storm fires. This is the structural fix that turns 'lucky we shipped lesson 12 last quarter' into 'we will keep shipping it next quarter.'

• Migrate the entire ledger to a different database that handles webhooks better

  Whatever you migrate to has the same at-least-once semantics on its webhook ingestion. The fix is at the application + schema layer, not the database choice. Migration is a multi-month project that doesn't address the actual failure mode.

• Rate-limit the provider to a max events-per-second on our side

  Rate-limiting the provider just delays the replay; it doesn't change the at-least-once semantics. The handler still has to be idempotent. Adding a rate-limit on top doesn't replace the fix; it just hides the problem under a queue.

• Move the webhook handler to a serverless function that scales automatically with load

  Auto-scaling is irrelevant to the correctness problem the incident exposed. The handler scaled fine on its existing infrastructure. The structural fix is making sure idempotency stays correct as the codebase evolves.