I run payments infrastructure, which means I spend a lot of time thinking about a problem product teams would rather not know exists: the providers we depend on go down. Not occasionally — routinely. A card gateway times out during a flash sale. A wallet provider's sandbox-grade uptime leaks into production. A transfer rail slows to a crawl at month-end.

If you operate in the Nigerian market — Paystack, Flutterwave, OPay, Paga — this is the weather, not a storm. Your users don't care. They picked "Card" on the funding screen and they expect money in their wallet.

I recently read Ibukun Olaifa's write-up of a self-healing wallet-funding service built on Django and Redis, and the design is sound: normalized provider wrappers, per-channel provider chains, a router with tunable retry budgets, and a Redis-backed circuit breaker with a clever single-probe recovery lock. I rebuilt it in Go to see how the design translates — and to interrogate its trade-offs from an operations seat.

This post walks through the Go implementation, then spends real time on the trade-offs, because that's where platform engineering actually lives.

The design in one paragraph

Every funding channel maps to an ordered chain of providers: card → Flutterwave, then Paystack; opay → OPay, then Flutterwave; NIP transfers → Paga alone. A router walks the chain. Each provider is guarded by a circuit breaker with three states: closed (healthy, gets its normal trial budget), open (tripped, skipped instantly), and half-open (cooldown elapsed; exactly one request gets to probe it). Breaker state lives in a shared store, so every service replica sees the same view of provider health. The user picks a channel; the backend picks the safest path.

The provider contract

Fallback only works if every provider speaks the same app-facing language. In Go that's one interface:

type Provider interface {
    Name() string
    InitializeCharge(ctx context.Context, req ChargeRequest) (ChargeResult, error)
    FetchStatus(ctx context.Context, providerRef string) (string, error)
    VerifyWebhook(ctx context.Context, signature string, body []byte) error
    ParseEvent(ctx context.Context, body []byte) (Event, error)
}

The funding service never sees a Paystack payload or a Flutterwave response shape. It sees ChargeRequest in, ChargeResult out. That normalization is the entire reason failover is a routing decision instead of a rewrite.

One Go-specific choice worth calling out: wrappers classify errors. Transport failures and 5xx responses come back wrapped in a sentinel:

var ErrProviderUnavailable = errors.New("provider unavailable")

The router counts those against the breaker. A declined card or a validation error is returned to the caller untouched — a user's bad CVV says nothing about Flutterwave's health, and letting business errors trip breakers is how you end up failing over for no reason.

The breaker: state lives in the store, not the struct

The most transferable idea in the original design is that the breaker object holds no state. Its state is derived from three TTL-bearing keys per (operation, provider) pair:

  • cb:funding:flutterwave:fails — failure count, expiring after the fail window
  • cb:funding:flutterwave:open — presence means open; the TTL is the cooldown
  • cb:funding:flutterwave:probe — the half-open probe lock
func (b *CircuitBreaker) State(ctx context.Context) (State, error) {
    open, err := b.store.Exists(ctx, b.openKey)
    if err != nil {
        return "", err
    }
    if open {
        return Open, nil
    }
    fails, ok, err := b.store.Get(ctx, b.failsKey)
    if err != nil {
        return "", err
    }
    if ok && fails >= b.cfg.Threshold {
        return HalfOpen, nil
    }
    return Closed, nil
}

Notice there's no explicit half-open flag anywhere. Half-open is an emergent state: the open key's TTL lapsed (cooldown over) but the failure count is still at threshold (fail window hasn't lapsed). The store's key expiry does the state machine's clock work. No timers, no background goroutines, no cleanup jobs.

In Go I put a small Store interface between the breaker and its backing:

type Store interface {
    Exists(ctx context.Context, key string) (bool, error)
    Get(ctx context.Context, key string) (int64, bool, error)
    SetNX(ctx context.Context, key string, value int64, ttl time.Duration) (bool, error)
    Set(ctx context.Context, key string, value int64, ttl time.Duration) error
    Increment(ctx context.Context, key string) (int64, error)
    Delete(ctx context.Context, keys ...string) error
}

Two implementations: an in-memory map behind a mutex, and Redis via go-redis. Same breaker code, swappable at startup. This turned out to matter more than I expected — more on that in the trade-offs.

The half-open probe lock

The subtlest part of the design. When a cooldown expires under load, dozens of in-flight funding requests can observe the provider as half-open simultaneously. Without a guard, they'd all "probe" it — a synchronized burst of traffic at the exact moment the provider is most fragile. That's a thundering herd wearing a recovery costume.

The fix is one atomic operation:

func (b *CircuitBreaker) TryAcquireProbe(ctx context.Context) (bool, error) {
    return b.store.SetNX(ctx, b.probeKey, 1, b.cfg.ProbeTTL)
}

SETNX semantics: first caller wins the probe slot, everyone else loses. And here's the important routing decision — losers don't wait, they fail over:

case breaker.HalfOpen:
    won, _ := cb.TryAcquireProbe(ctx)
    if !won {
        return provider.ChargeResult{}, fmt.Errorf("%w: %s probe slot taken", errSkipped, p.Name())
    }
    trials = 1 // one careful probe, not the full budget

A blocked probe is not a queue; it's a signal to take the fallback route. Users behind the losing requests never feel the recovery experiment.

The probe TTL is set to the provider HTTP timeout plus a buffer, so if the probing worker dies mid-request, the slot self-cleans. No lock janitor required.

I hit one real bug here that the original pseudocode doesn't cover: a failed probe leaves its lock behind. The probe fails, the breaker re-opens with a fresh cooldown — but if the cooldown is shorter than the probe TTL, the next half-open window arrives while the stale lock still exists, and nobody can probe. The provider gets stuck being skipped for longer than intended. The fix is to release the probe key whenever the breaker trips open. My concurrency test for this (50 goroutines racing for the slot, exactly one winner) caught the regression immediately; this is the kind of thing you want pinned by a test, not by an incident review.

The router: budgets, not loops

The router gives each provider a trial budget before moving on, resolved in priority order: provider override → channel override → global default. The reasoning is operational: a NIP transfer rail that's slow-but-reliable deserves more patience than a card-initialization call that should return in 300ms. Encoding that as configuration, not code, means tuning retry behavior is a config change, not a deploy.

One detail I added in the Go version: when a failure inside the retry loop trips the breaker, stop burning the remaining budget —

if fails >= r.cfg.Breaker.Threshold {
    break // breaker just tripped; don't waste the rest of the budget
}

— because retrying into a breaker you just opened is pure user-visible latency.

The last piece is bookkeeping that pays for itself later: the wallet transaction stores the provider that actually accepted the charge. With fallbacks, "card funding" no longer implies "Flutterwave." When the webhook arrives or a verification job runs, it must be routed to the wrapper for the provider on the transaction record, not the chain's preferred provider. Miss this and your reconciliation breaks in the exact scenario the system was built for.

Watching it heal

The repo ships a scripted demo. This is the actual output — Flutterwave is killed, the breaker opens, a probe fails, then recovery:

== healthy: card funding goes to the preferred provider ==
fund #1                      -> charged via flutterwave [flutterwave breaker: closed]

== flutterwave goes down ==
fund #3 (fw failing)         -> charged via paystack    [flutterwave breaker: closed]
fund #4 (breaker open)       -> charged via paystack    [flutterwave breaker: open]

== cooldown passes, provider still down: probe fails, breaker reopens ==
fund #6 (half-open probe)    -> charged via paystack    [flutterwave breaker: open]

== cooldown passes again, provider recovered: probe succeeds ==
fund #7 (half-open probe)    -> charged via flutterwave [flutterwave breaker: closed]

Every request succeeded. The user never saw Flutterwave's outage; the backend absorbed it, tested recovery carefully, and moved traffic home when it was safe.

The trade-offs

This is the part I actually wanted to write. The design is good; it is not free.

1. The breaker store is now on your payment path

Backing the breaker with Redis buys you the headline feature: shared health state. Replica A trips the Flutterwave breaker; replicas B through N skip Flutterwave instantly. With per-process in-memory breakers, every replica pays the full failure threshold independently — with 20 replicas and a threshold of 3, that's up to 60 failed user-facing requests before the fleet collectively learns what one Redis key would have told everyone.

The price: Redis is now a dependency of taking money. So you must decide, explicitly, what happens when the breaker's own infrastructure fails. My implementation fails open — if the store errors, the router pretends the breaker is closed and tries the provider anyway:

state, err := cb.State(ctx)
if err != nil {
    // Store outage: fail open. We'd rather try the provider than
    // refuse payments because the breaker's bookkeeping is down.
    state = breaker.Closed
}

That's a values judgment, not a technical one: degraded self-healing beats a self-inflicted payments outage. Fail-closed would mean a Redis blip halts all funding — the safety mechanism becoming the incident. But fail-open has its own cost: during a simultaneous Redis outage and provider outage, you're back to naive retry-everything behavior. Pick your failure mode on purpose and write it down.

2. The failure counter is only mostly atomic

RecordFailure is two operations: SETNX fails 0 EX window, then INCR fails. Between a key expiring and the increment, concurrent failures can race and the window's start time smears. In pathological interleavings you trip the breaker a failure early or late.

You could make this exact with a Lua script (or MULTI/EXEC). We didn't, and I'd argue you shouldn't: a circuit breaker is a heuristic, not a ledger. Whether Flutterwave gets marked unhealthy after failure 3 or failure 4 changes nothing material. Spend your atomicity budget where correctness is binary — the probe lock, which genuinely must have exactly one winner — and let the counters be approximately right. Knowing which of your primitives need to be exact is half the job.

3. Count-based thresholds are the bluntest instrument

This breaker trips on N consecutive-ish failures in a window. Mature breaker libraries (gobreaker, sony/gobreaker, resilience4j) usually offer error-rate thresholds over sliding windows: "open at 50% failures over the last 100 requests." Rate-based breakers behave much better under high traffic — at 1,000 rps, a fixed threshold of 3 trips on noise; at 0.1 rps, a 50%-rate breaker barely has samples.

Why accept count-based anyway? Because the state must live in shared Redis, and a sliding-window rate calculation across replicas means shipping request outcomes into Redis at request rate — meaningfully more load and complexity than three keys with TTLs. Count-based-with-TTL is what's cheaply expressible in shared state. That's a real architectural tension: the fancier your breaker math, the more it wants to live in process memory, which is exactly where multi-replica deployments can't leave it. Know which side of that trade you're on and tune threshold/fail_window for your actual traffic level.

4. Failover is a business event, not just a technical one

Routing card traffic from Flutterwave to Paystack is not a no-op that happens to succeed. The two providers have different fees (your unit economics just changed mid-incident), different settlement schedules (finance's reconciliation now has a fork in it), different webhook formats and retry behaviors, and different provider references. The design handles the mechanical half — storing the accepting provider on the transaction routes webhooks correctly. The organizational half is on you: reconciliation reports must group by accepting provider, and finance should probably know that Tuesday's card volume settled through the fallback.

There's also an idempotency seam. The app-side reference is your idempotency key, but if a charge times out ambiguously on provider A and you fail over to provider B, you can double-charge — provider A may have actually succeeded. A production system needs to distinguish "definitely failed" from "unknown outcome" and only fail over on the former (or reconcile the ambiguous attempt before retrying). The article's design doesn't cover this, and it's the sharpest edge in the whole pattern.

5. A chain of one is a breaker with nowhere to go

NIP transfers ride Paga alone. The breaker still earns its keep — when Paga is down, users get a fast, honest failure instead of hanging through timeout after timeout, and Paga isn't hammered while degraded. But nothing self-heals; there is no fallback to route to. The lesson generalizes: the breaker provides latency protection; the chain provides availability. You need both, and a dashboard that shows single-provider chains as standing risk items rather than as configurations that "work."

6. Config sprawl is the tax on tunability

Trial budgets at three levels, plus threshold, cooldown, fail window, and probe TTL — per operation. That flexibility is the point, and it's also a garden that grows weeds: six months in, nobody remembers why NIP gets 3 trials, and an unreviewed cooldown of 15 minutes quietly parks a healthy provider in the corner. Treat breaker parameters like the operational levers they are: version-controlled, code-reviewed, and — ideally — evaluated against replayed incident traffic rather than vibes.

What Go specifically bought us

Mostly ergonomics, but meaningful ones. context.Context gives every provider attempt a real deadline that propagates through the HTTP client and cancels cleanly — the per-attempt timeout isn't advisory. The Store interface plus errors.Is sentinel wrapping made both the memory/Redis split and the retryable/business error split feel native rather than bolted on. And the race-prone parts (the probe lock, the memory store) are pinned by a 50-goroutine contention test (and go test -race on machines with a C toolchain), which is a class of confidence that's hard to get cheaply elsewhere.

What I'd add before calling it production

  • Metrics on state transitions. A breaker opening is your earliest, cleanest signal that a provider is degrading — usually ahead of the provider's status page. Emit breaker_state_transitions_total{provider,from,to} and alert on it.
  • A manual override. During a declared provider incident you want force-open flutterwave for 2h, not a threshold negotiation. The key-based design makes this trivial: set the open key by hand with a long TTL.
  • Explicit handling for ambiguous charge outcomes before failing over (see trade-off 4).
  • Weighted or cost-aware routing as a later evolution — once you trust the health layer, "preferred provider" can become a function of fees and observed latency instead of a hardcoded order.

Closing

The phrase "self-healing" oversells slightly — nothing here heals the provider. What the system heals is your dependence on any single provider being healthy at a given moment. It converts a fragile single path into a switchboard with memory: it learns which routes are unsafe, redirects around them without waking anyone up, and tests recovery with exactly one careful request instead of a stampede.

The implementation is small — a breaker, a registry, a router, maybe five hundred lines of Go that matter. Every trade-off above, though, is a decision your team makes whether or not you make it consciously: what fails open, what must be atomic, who reconciles the fallback provider's settlements. The code is the easy part. Full implementation with tests, a chaos-injectable HTTP server, and the scripted demo: see the accompanying repository.

Credit to Ibukun Olaifa for the original design write-up this implementation is based on.