Your sponsor bank just went 503
Tuesday 10:14 WAT. The bank API is down. Withdrawals are stacking up. Customer support is two minutes from being on fire.
By Solomon Ajayi · Free to read, no signup
You're the on-call engineer at a Nigerian neobank. You ride on a sponsor bank for actual money movement, you hold customer funds For Benefit Of (FBO) at the sponsor, and every deposit / withdrawal / inter-wallet transfer eventually maps to a settlement call against their API. At 10:14 WAT on a Tuesday, the sponsor bank's API starts returning 503 Service Unavailable on every call. Their status page says nothing yet. The first 5,000 customer withdrawals of the morning are now stuck in an in-flight state, and traffic to your app is climbing toward the lunchtime peak. This incident isn't on a Wikipedia page; it's the kind of incident every neobank engineer lives through and never quite writes up. You're going to walk it now from inside, the way the actual call sheet runs.
The decisions, beat by beat
Tuesday 10:14 WAT
PagerDuty fires: 'sponsor_bank_api error rate 100% for 2 min.' Your dashboards confirm: every call to the bank's /transfers, /balances, /wires endpoint is returning 503. No 2xx in the last 90 seconds. The bank's status page still shows all systems operational. Customer-facing app traffic is normal. The 5,000 withdrawals already in your queue are stuck in 'submitted' state, the inbound webhook from the bank that normally confirms settlement never arrived.
What's the first move?
✓ Open a bridge: page the sponsor bank's on-call, page your own founding engineer, start a comms doc
Right. The first 5 minutes of an incident are about gathering ground truth and creating shared situational awareness. The bridge is the workspace for everything that comes next. Coding fixes alone in a Slack thread is how 30-minute outages become 4-hour ones.
Restart your services to clear the 503s
The 503s are coming from the BANK, not from your services. Restarting yours just drops in-flight requests and makes reconciliation harder when the bank recovers.
Reverse every stuck withdrawal back to the user's wallet immediately
Premature and dangerous. The bank API is down but the bank's BACKEND might still be processing in-flight requests. Reversing now risks paying users twice when the bank's queue eventually drains. You need to know what's actually happening at the bank first.
Tweet that your app is down
Your app isn't down, the sponsor bank is, and a vague 'we're down' tweet without specifics churns more customers than the actual outage will. Comms come after you know what to say.
Incident bridge opened; sponsor bank on-call paged; comms doc started
Three Slack huddles spin up: engineering, support, and a joint customer-comms thread. The sponsor bank's on-call confirms within 4 minutes: their primary database had a failover that didn't complete cleanly. ETA to recovery: unknown, but they're working it. You now have a number you can defend internally and the spine of a status-page update.
Tuesday 10:24 WAT
Ten minutes in. The 5,000 stuck withdrawals are aging. Your user-facing app currently shows the customer's spendable balance as if those withdrawals settled, your projection deducted the amount when the withdrawal was submitted. The most important call right now is what the ledger should reflect for that in-flight money. It is decisively NOT in the spendable wallet anymore (the user clicked withdraw), but it has also NOT yet left the sponsor bank (the API never confirmed settlement).
Post the journal entry that re-classifies the stuck withdrawals.
✓ User Wallet (Spendable) DOWN ₦125M; Wire In Flight (Held) UP ₦125M
Correct. The money has left the customer's spendable bucket (they pressed withdraw, app reflects that) but has NOT physically left the sponsor bank. The clean intermediate state is a Held bucket. Two-line entry, balanced; the FBO at the bank is unchanged because no money actually moved.
User Wallet DOWN ₦125M; FBO at Sponsor Bank DOWN ₦125M
You'd be claiming the money already left the sponsor bank. It didn't, that's exactly the problem the 503s caused. Recording it as settled is the lie that becomes a reconciliation break later.
User Wallet UP ₦125M; Wire In Flight UP ₦125M
Both sides UP on liabilities doesn't reverse the withdrawal click, it amplifies it. You'd be giving the user back their spendable balance AND claiming a held amount. Double counting.
No journal entry; wait for the bank to recover
Doing nothing means the spendable-wallet projection lies for the rest of the outage. The user sees balance reduced (the projection trusted the withdrawal submission), but the ledger says the money is still spendable. Reconciliation breaks the moment the bank comes back.
| Account | Debit | Credit |
|---|---|---|
| User Wallet (Spendable) (2000) | ₦125,000,000.00 | |
| Wire In Flight (Held) (2500) | ₦125,000,000.00 |
User Wallet (Spendable) DOWN ₦125M; Wire In Flight (Held) UP ₦125M. Two-line liability reshuffle. The sponsor-bank FBO is untouched because no money has physically left. The Held bucket is now the canonical source of truth for 'we owe this to the user OR to the bank, depending on who gets the eventual settlement confirmation.'
Tuesday 10:42 WAT
Twitter is starting. Three threads, ~120 mentions in 8 minutes, all saying variants of 'app is broken / money missing / can't withdraw.' Support's inbox just crossed 400 unresolved tickets. The sponsor bank's status page now shows yellow but with no ETA. You have enough information to put out a customer comms update; the question is what it says.
Post the customer status update.
✓ Name the sponsor-bank outage; promise no funds are at risk; commit to a status update every 30 min
Right. Specificity buys trust during an incident: 'our banking partner is having an outage, your funds are safe in the bank's FBO account, withdrawals will process when their API recovers, next update in 30 minutes.' That language calms the people threatening chargebacks because they can see you know what's happening.
Tweet 'we are investigating an issue'
Vague tweets during a 503 storm read as concealment. The mentions accelerate. You burn the trust that specific comms would have built.
Don't post anything yet, wait for the bank to recover so the story is clean
Silence during a public outage is read as panic. By the time the bank recovers, the narrative has set, and a 'we just had an issue, all resolved' post one hour after the fact lands as a cover-up. Talk now.
Blame the sponsor bank publicly by name and quote a banking executive at them
Naming the bank as the cause is fine and accurate, but quoting executives or escalating publicly burns the relationship you depend on for your entire banking license. You can be honest without being a hostile counterparty.
Customer comms posted: sponsor-bank outage named, funds safe, updates every 30 min
The status page goes live at 10:48. Twitter mentions plateau within 20 minutes. Support response macros shift from 'investigating' to a link to the status page. The thread of customers threatening chargebacks slows by ~80%, what remains is the small subset who chargeback regardless of what you say. That's a known cost.
Tuesday 12:31 WAT
Two hours and seventeen minutes after onset, the sponsor bank's API recovers. Status: green across all endpoints. Their team sends a reconciliation file: 4,612 of your 5,000 in-flight withdrawals were successfully processed by the bank during the outage (their backend queued them); 388 are confirmed as not-processed and need to be re-submitted or refunded. The hard part starts now.
How do you reconcile the recovered batch?
✓ For each in-flight withdrawal, look it up in the bank's reconciliation file; mark settled the ones the bank processed; refund the ones the bank didn't
Right. The bank's reconciliation file is the authoritative source for what actually moved. Per-row reconciliation is slow but correct: each Held entry resolves into either a Settled entry (Wire In Flight DOWN, FBO DOWN) or a Refund entry (Wire In Flight DOWN, User Wallet UP). Anything else risks double-charging or double-refunding.
Mark all 5,000 as settled and trust the bank's file is right
You'd be marking 388 withdrawals as settled that the bank did NOT process, the customers see their money gone but the bank holds it. Reconciliation must be per-row, not batch-trust.
Refund all 5,000 to the users; let them re-submit later
You'd be refunding 4,612 withdrawals that the BANK ALREADY PROCESSED. Customer gets the money in their wallet AND in their external bank account. That's the catastrophic outcome reversing prematurely was always going to risk.
Re-submit all 5,000 to the bank and let it dedupe
Sponsor bank APIs typically don't dedupe at this layer (your idempotency keys do, if you used them). Re-submitting risks the bank processing 4,612 of them a second time. Use the reconciliation file as ground truth.
| Account | Debit | Credit |
|---|---|---|
| Wire In Flight (Held) (2500) | ₦115,000,000.00 | |
| FBO at Sponsor Bank (1300) | ₦115,000,000.00 |
Of the ₦125M held, ₦115M is confirmed settled by the bank: Wire In Flight (Held) DOWN ₦115M; FBO at Sponsor Bank DOWN ₦115M. The remaining ₦10M (the 388 unprocessed withdrawals) gets refunded to user wallets in a separate per-row job, each entry carrying the original idempotency_key so a future user-initiated retry doesn't double-pay.
Wednesday 09:00 WAT
Post-mortem the next morning. Customer churn from the outage: about 0.3% of monthly actives (much lower than the team feared, thanks to the early specific comms). Per-row reconciliation finished at 23:00 the previous night, four customers had off-by-₦1 balance variances due to a fee-rounding bug that surfaced under the unusual flow, all fixed manually. One structural recommendation is going into the postmortem doc.
Top engineering follow-up?
✓ Ship a per-direction circuit breaker on the sponsor-bank API: fail fast after N consecutive 5xx so the queue doesn't stack and the user gets a clear error
Right. The root cause we couldn't fix today is that the sponsor bank had an outage. The root cause we CAN fix is that we let the queue silently stack for two minutes before we even knew, and once it was stacked the recovery cost an hour of per-row work. A circuit breaker bounds the damage on both axes.
Move all withdrawal traffic to a secondary sponsor bank immediately
Multi-sponsor is the correct longer-term architecture but it's a 6-12 month regulatory + compliance + integration project, not a one-week post-mortem fix. The circuit breaker ships this week and bounds the next event.
Add an auto-refund rule that returns held money after N minutes of bank API downtime
That's exactly the bug that would have caused the double-pay catastrophe today. The bank's backend queued requests during the outage; auto-refunding would have unwound 4,612 settlements that the bank later confirmed. The held bucket is correct; don't auto-resolve it.
Rebuild the entire wallet projection nightly to catch reconciliation drift
Nightly rebuilds are useful generally (covered in lesson 40) but they're an answer to a different question. The incident wasn't a projection bug; it was a downstream-dependency outage. The circuit breaker is closer to the actual root cause.
Postmortem locked: per-direction circuit breaker on sponsor-bank API
Engineering owner: founding engineer. Ship target: end of next sprint. Acceptance criteria: 5xx error rate above 30% for 60s opens the breaker for that endpoint, queued requests fail fast with a typed 'banking partner unavailable' response, the user-facing app shows a clear modal with a 'try again in a few minutes' message. The next sponsor-bank outage takes 2 minutes of customer pain instead of 2 hours.
What actually happened
The five beats you walked are roughly how this kind of outage actually unfolds. The most important engineering pattern is the one your step 2 entry made concrete: stuck withdrawals belong in a 'Wire In Flight' liability bucket, NOT credited back to the user's spendable wallet, because the sponsor bank's recovery might still execute them. Reversing prematurely creates a double-pay risk that's worse than the outage itself. Real sponsor-bank incidents like this typically resolve in 1-4 hours; the post-mortem patterns that survive are: (1) per-direction circuit breakers that fail withdrawals fast instead of stacking them, (2) a secondary sponsor bank for failover (regulatory permitting), (3) status-page automation so you don't manually communicate every five minutes, and (4) a 'reconcile after recovery' job that proves no customer was charged twice and no customer was credited twice. The customers who saw the clear comms are the ones who stayed; the ones who got 'system unavailable' with no detail churned. Operational incidents like this don't make headlines but they're the texture of running a fintech.
Play it from the engineer's seat
Reading the replay is one thing. Sit in the chair, make the calls live, and watch the consequences land in a real ledger. Free.