Two Datacentres, One Failure: The Hidden Problem with Redundancy

Share
Two Datacentres, One Failure: The Hidden Problem with Redundancy
Photo by Taylor Vick / Unsplash

Running infrastructure in two datacentres sounds reassuring. Different buildings, separate power systems, independent cooling and enough geographical distance to avoid a localised incident affecting both sites. On paper, that's a good start. But physical separation doesn't automatically translate into operational independence. It's entirely possible to build an expensive dual-site infrastructure platform that still contains a single point of failure. The difficulty is that the shared dependency often sits somewhere outside the servers themselves.

Follow the cables

Consider two datacentres connected by dedicated fibre circuits. Each site has redundant switches and multiple physical servers. The applications are replicated, and the architecture looks resilient. But what happens if both inter-site circuits follow the same physical route? A single cable incident could interrupt both connections. Alternatively, the circuits might use different providers but ultimately depend on the same underlying wholesale fibre infrastructure. Carrier diversity and physical route diversity are not necessarily the same thing.

Similar problems arise with shared management platforms, centralised authentication, DNS infrastructure and storage dependencies. If Site B requires a service in Site A to become operational, it isn't completely independent of Site A. That might be an acceptable design decision, but it needs to be understood, documented and tested rather than assumed away.

Replication introduces its own decisions

Keeping data synchronised between locations sounds straightforward until latency and consistency enter the picture. Synchronous replication generally requires writes to be acknowledged by the relevant replicas before completion. Across significant distances, network latency can become part of application write latency. Asynchronous replication reduces that coupling but introduces the possibility of losing the most recent changes if the primary site fails. Neither approach is inherently wrong. The choice depends on the application, the distance involved, the acceptable recovery point and the expected behaviour during network failure.

There's also a particularly unpleasant scenario known as split-brain, where separated systems both attempt to operate as the authoritative copy. Without suitable quorum, fencing and failover controls, a network failure can turn into a data integrity problem. It's one reason that complex active-active designs deserve considerably more scrutiny than a diagram full of reassuring arrows might suggest.

Design independence, not just duplication

For businesses considering multi-site private cloud, it's worth asking what each location can actually do on its own. Can administrators access the recovery environment if the primary site is unavailable? Are DNS and authentication still functional? Can applications start without reaching services in the failed location? And, perhaps most importantly, has anyone tested the failover process under realistic conditions?

At KORE, multi-site infrastructure design involves much more than placing equipment into separate racks. Networking, routing, storage replication, management access and recovery procedures all contribute to the result. There are situations where a simpler primary-and-recovery architecture makes more sense than a complicated active-active deployment. What matters is having a design that behaves predictably under failure. Two datacentres provide an opportunity for resilience; making them genuinely resilient is the engineering work.

Read more

Beyond the LLM: Engineering Secure, Reliable AI Agents for Enterprise Infrastructure

Beyond the LLM: Engineering Secure, Reliable AI Agents for Enterprise Infrastructure

Building a capable artificial intelligence model is one challenge. Integrating that intelligence into business-critical infrastructure, while maintaining security, reliability and operational control, is an entirely different engineering problem. The rapid advancement of large language models has created considerable enthusiasm around the possibility of intelligent automation. Modern models can interpret

By KORE Pulse