Our failover worked perfectly. That is how we missed a 100% failure rate.

We run two providers for airtime, data, electricity and TV. If one cannot deliver, the other does. That is the point of a second rail, and ours had been quietly doing its job for weeks.

Too well, as it turned out. Every single data purchase was failing on the first provider and completing on the second. Not some. Every one. For weeks. Nobody knew.

Here is how a system can fail completely and look healthy while doing it.

The evidence said everything was fine

Data purchases were succeeding. Customers got their bundles. Nothing alerted, no error rate moved, no support ticket arrived.

We had also just switched several services to the new rail, so we did what you do: we checked the transactions.

product   provider    status     count
data      incumbent   success    6
dstv      incumbent   success    3
airtime   new-rail    success    12

Airtime on the new rail, working. Data and TV on the incumbent, working. Reasonable, except for one detail: data and TV were configured to route to the new rail. They were supposed to be on it. There was not a single row showing the new rail even attempting a data purchase.

Not a low success rate. No rows at all.

Why the evidence was missing

When a provider fails and we fail over, we re-pin the transaction to whichever provider actually delivered. That is correct: the record should say who fulfilled the order.

It also means the first provider's attempt is erased. The row that said "we tried the new rail" is overwritten by the row that says "the incumbent succeeded". If you only look at completed transactions, a provider that fails 100% of the time is indistinguishable from one you never called.

So we went to the logs, and found this:

failover new-rail -> incumbent for ref=...

There it was, on every data purchase. And that line is all we got. Not why. Just that it happened.

The reason was thrown away three times

We had the provider's explanation the whole time. We discarded it, in three separate places, each for a defensible-looking reason.

It was never logged. Our adapter logged transport errors, which is where you expect trouble: timeouts, DNS, connection resets. But a provider that answers with HTTP 400 and a perfectly clear explanation is not a transport error. It is a normal response carrying bad news, and it returned in silence.

It was never saved. We had a database field for exactly this, and the central dispatch path never wrote to it. A webhook handler wrote it for one specific case: orders the provider accepted and later failed. But an order rejected outright at creation never produces an order, so it never produces a webhook. The one class of failure that means we sent a malformed request was the one class whose reason we never stored.

And then it was overwritten. Even the fields we did save were replaced by the successful provider's values when the failover re-pinned the row.

Three mechanisms, each sensible on its own, combining to guarantee we could not answer the only question that mattered.

The fix was one log line

Before failing over, say what the first provider said.

failover new-rail -> incumbent for ref=...:
  new-rail returned failed_unknown
  reason=Field 'bundle_code' is required

We deployed it. The very next purchase told us:

We were sending the wrong field name.

The provider wanted the data plan in a field called one thing. We were sending it under another. Every data purchase we had ever sent them was malformed, rejected on arrival, and rescued by the other rail before the customer noticed.

Two more problems were sitting behind it. TV needed a different field name again, so fixing data alone would have moved the failure one step along instead of curing it. And the plan code we were sending belonged to the other provider's catalogue entirely, so even a correctly-named field would have carried a value they had never heard of.

None of that was mysterious. All of it was documented. We had simply never been in a position to read the answer.

What we actually got wrong

Not the failover. It behaved exactly as designed: it detected failure, routed around it, and served every customer. If you judge it on customer outcomes it was flawless.

We built a system that could route around a broken component without ever mentioning it. Resilience and silence are not the same property, and we had accidentally coupled them. The stronger the fallback, the longer the primary can be dead before anyone notices, because the fallback is busy making sure nothing looks wrong.

There is a commercial edge to this too. We had signed with the new provider for better rates. On the services that mattered most, we were earning none of them, because those services were quietly being served by the provider we were trying to move away from. That is not an outage anyone escalates. It is just money not arriving, invisibly, for weeks.

Three things we changed

A failover names what it routed around. Provider, status code, and the provider's own words. The reason is also stored on the transaction in a field the fallback cannot overwrite, so the failure survives its own recovery.

A rejection is not a transport error. We log every non-success answer from a provider, not only the ones that arrive as exceptions. A clean HTTP 400 with a clear message is the most useful failure you can get. It should not be the quietest.

The success card names the rail. Every completed sale now shows which provider served it. If one goes quiet, we see it in the normal course of reading them, without going to look.

The rule we took from it

A fallback that hides the failure it absorbs is not resilience. It is a delayed outage with better manners.

If you have retries, failovers, or circuit breakers, ask what they look like when they are working constantly. If the answer is "the same as when they are not working at all", you have this bug too. You just have not been told yet.

That question, which provider is actually serving what, turned out to be hard to answer because every product spoke to every provider itself. We fixed it by moving all of that behind one service, so there is one place that knows which rail served a sale and one place to read it from.