When the Backup Fails: Fuel Degradation and Enterprise System Downtime

Most enterprise architecture diagrams stop at the server rack. Above it sit the applications everyone argues about: the ERP, production planning, warehouse and transport software, the integrations holding them together. Below it sits something that rarely appears on any diagram, which is the power those servers depend on.

When the grid fails, that power comes from two places in turn: first the UPS batteries, then the standby generator. Continuity plans usually treat the second step as a formality. The generator is serviced, tested monthly and signed off. It will start.

The part nobody signs off is the fuel. It can sit in a tank for years, and it changes while it waits. In our work on critical sites we put it this way: the risk is not that the fuel gets thrown away. The risk is that it is there, and it does not work.

How long can a data centre run on batteries alone?

Not long. A UPS bridges a gap. Battery autonomy in most data centres is sized in minutes, typically ten to fifteen. That window exists for one job: to keep the servers running until the generator starts, stabilises and takes the load.

If the generator does its job, users never notice the outage. If it does not, the batteries run down and the servers go with them, and systems can drop mid-transaction.

The whole resilience model rests on one handover, from battery to engine, inside a few minutes.

That makes the generator and its fuel an IT concern, not only a facilities matter.

What actually stops when the servers stop?

Data in flight can be lost or left inconsistent, and the penalties in hosting and customer contracts were written for exactly this. But the bigger cost sits one layer up. The infrastructure that goes dark is running the ERP. Order entry stops. Production planning stops, so the factory floor loses its schedule.

Warehouse systems cannot allocate stock or print labels, and transport cannot tell drivers where to go. A power failure that started in one building ends up stopping a supply chain.

Recovery takes longer than the outage. Databases must be checked, interfaces restarted in order, and half-completed transactions reconciled. When the power comes back, the real work starts.

What happens to diesel fuel while it waits?

It degrades. Fuel degradation is the loss of diesel quality inside a storage tank while the fuel sits unused. Standby fuel can wait years, and diesel is not an inert liquid. Four things happen in the tank.

It oxidises. Contact with air over long periods produces gums and fine sediment.

It takes on water. Tanks breathe as temperatures change, and the humid air they draw in condenses on the walls and settles under the fuel.

Microbes grow. Bacteria and fungi are present in fuel as delivered. Once free water collects at the bottom, they multiply at the boundary between water and fuel, where they have both water and food.

Sludge forms. The products of all three collect at the bottom as a dark, soft layer that blocks filters and damages injectors when the generator finally has to run.

None of this is visible from the tank hatch. The level gauge still reads full, but a share of what is in that tank is no longer something an engine can burn cleanly.

Why does the monthly generator test not catch it?

Because a monthly test and a real outage put very different demands on the fuel. Of everything here, this is the point most worth taking back to the continuity plan.

In a routine test the generator runs a few minutes, often with little or no load. The engine draws little fuel, and a partly contaminated supply gets through the filters without trouble. The test is recorded as a pass.

A real outage asks something different. The generator has to carry the full site load, for hours rather than minutes. Fuel flow rises sharply. Sediment that sat undisturbed through every test is pulled towards the suction line. Filters that coped with a short, light run start to clog, and injectors meeting contaminated fuel under full load stop atomising it properly.

What follows takes one of two forms. The generator does not start, or it starts, picks up the load and stops a few minutes later when the filters block. Either way, the UPS has already spent most of its minutes.

Where the consequences go beyond the balance sheet

In an ERP environment, a failed backup is a commercial problem. Elsewhere the same failure has consequences no service level agreement can price: hospital theatres and intensive care, airport runway lighting and radar, telecoms carrying emergency calls when the grid is already down, and military command and control. All share one feature with the enterprise data centre. The generator may go years without being needed, then has to work first time, at full load, for as long as the outage lasts.

What does fuel maintenance involve?

Two things: testing the stored fuel against the standards, and fuel polishing to keep it within them.

The standards give the benchmark. ASTM D975 in North America and EN 590 in Europe define what diesel has to meet. NFPA 110 treats the fuel supply as part of the power system and requires stored fuel to be tested at least once a year. The sample that matters is drawn from the bottom of the tank, where water and sludge collect; one from the middle of a contaminated tank can come back looking clean.

Testing tells you the condition. It does not change it. Sites use fuel polishing. The stored fuel is circulated through staged filtration and water separation, so water is removed before microbes can use it and particulates before they reach an engine. A one-off tank clean deals with the contamination present on the day. A system running on a schedule deals with the process that creates it, because the tank keeps breathing and the water keeps arriving.

Questions for the next continuity review

• When was the stored fuel last tested, and was the sample drawn from the tank bottom?
• Does any generator test reach full load, and does it run long enough to matter?
• How long can the UPS carry the load today, not when it was installed?
• Who owns the fuel: facilities, the generator service contractor, or nobody?

If nobody can answer the first one, start there.

The part of the chain nobody tests

Enterprise resilience is measured at the application layer: replication, failover, recovery time objectives. All of it assumes the power stays on, which during an outage means the generator takes over before the batteries run out.

The generator has a service record and the UPS has a battery report. The fuel often has nothing more than a delivery note from years ago. It degrades on its own while the routine tests keep passing, and the first real test it gets is the outage itself. Evergee designs and installs fuel polishing and fuel maintenance systems for critical facilities, including data centres, hospitals and airports. More at evergee.com.tr.