Architecting for resilience: validator HA without slashing risk
Architecting for resilience: validator HA without slashing risk
Part 2 of 3 in our blockchain infrastructure series
For most infrastructure, resilience means staying up. For a validator, staying up the wrong way can cost more than going down. That tension shapes every design choice that follows.
Part 1 made the case for dedicated hardware beneath blockchain workloads. We ended with a promise: the hardest part of any migration isn't deciding to move. It's proving the move works.
We work every day with validator operators, rollup teams, custody providers, and blockchain architects. The same questions come up again and again. This three-part series answers them with concrete figures and architecture patterns drawn from that work:
- Part 1, Building for Scale: why dedicated hardware fits blockchain workloads.
- Part 2, Architecting for Resilience: multi-region high availability and validator HA that avoids double-signing.
- Part 3, Beyond the Node: confidential compute, GPU workloads, and AI-native blockchain applications.
In this blog, we give you three things for your next architecture review: a redundancy map, a slashing-safe failover pattern, and a migration playbook.
Two failure modes, very different price tags
Validators fail in two ways, and the costs aren't close:
- Downtime. Your validator goes offline and misses attestations. Missed attestations cost penalties roughly equal to the rewards you’d have earned, so an hour offline takes about an hour to earn back.
- Slashing. Two active signers use the same key and sign conflicting messages, such as two different attestations for the same slot. The protocol applies an immediate penalty, ejects the validator for good, and locks the remaining balance for about 36 days, with penalties building up the whole time. Midway through, it adds a correlation penalty that grows with how much stake was slashed in the same window. One isolated slashing costs a small slice of your stake. A failover bug that slashes every key in your fleet at once is what the correlation penalty is built to punish.
Slashing rarely comes from an attack. Most often, it comes from failover or migration done wrong. The HA reflex that protects a web app, promoting the standby automatically, is the same reflex that slashes a validator.
Here's how it happens. The primary validator goes quiet after a network partition or a hung process. Failover logic promotes the standby, with the same keys loaded. But the primary was only unreachable. When the network recovers, both clients sign conflicting messages for the same duty, and the validator is slashed.
The failover logic worked perfectly. The real problem is where the key lives. If the signing key sits inside the validator client, any HA design that can start a second client can double-sign.
Redundant everywhere the keys aren't
Start with one question at each layer of the stack: does it hold a signing key?
- RPC and API endpoints. They serve queries and hold no keys. Run as many as you need behind a load balancer with session affinity or head-lag checks.
- Execution clients (Geth®, Reth, Nethermind). They hold chain state and the transaction pool, but no validator keys. Active-active across regions is safe, as long as each beacon client has its own dedicated execution client. The pair is the unit you duplicate.
- Beacon clients (Lighthouse, Prysm, Teku). They hold the consensus view, with no validator keys. Redundant beacons are recommended, and most validator clients accept multiple beacon endpoints natively.
- Validator client. It requests duties and signs attestations. Run one active instance per key while keys are local.
- Signing keys. They're what gets slashed, so move them out of the client.
Everything that doesn't sign can run in two places. Picture two OVHcloud® regions, US East (Vint Hill) and US West (Hillsboro), each with a full stack on dedicated bare metal. Splitting clients, with Geth and Lighthouse in one region and Reth and Teku in the other, helps reduce the risk of correlated failures from client-specific bugs. One validator client connects to both beacon endpoints and switches if one fails.
OVHcloud vRack, our private network, links the regions. Traffic between your own nodes, including snapshot copies, monitoring, and operational traffic moves privately at no additional transfer cost. On major hyperscalers, data transferred between regions is typically billed per gigabyte, separately from charges for data sent to the internet. With unmetered bandwidth, a second region costs the second server, not a second bill. That makes HA a budget line you can defend.
Remote signing: move the key out of the client
The fix for the dangerous layer is architectural. With remote signing, validator clients no longer hold keys. They send signing requests to a dedicated remote signer, which holds the keys and a slashing-protection database.
That database is what makes failover safe, not remote signing on its own. If an active and a standby client request conflicting signatures for the same duty, the database refuses whichever arrives second. A signer without slashing protection configured is not a safety net.
The database records every signature the signer produces. It refuses anything that conflicts with that history, such as a double block proposal, double vote, or a surround vote. EIP-3076 (Ethereum® Improvement Proposal 3076) defines a standard format to export that history and import it elsewhere, so the protection moves with you.
Open-source implementations include Web3Signer, from Consensys, and Dirk. Evaluate either against your own security requirements. On OVHcloud®, the signer runs on its own hardened bare-metal node, reachable from both regions over vRack.
The complete picture
Here's everything in one reference architecture: two client-diverse regions syncing over vRack, one validator client connected to both beacons, and one remote signer.
Every box above the signer can fail, fail over, or run twice. The signer, with slashing protection configured, is the single authority that keeps any of that from becoming a slashing event.
Beyond the single signer
Two more patterns round out the picture:
- Distributed validator technology (DVT). DVT splits the validator key across several independent nodes using threshold cryptography. A subset, say three of four, must agree before any signature is produced. Institutional staking operations are the early adopters, while remote signing covers the HA need for most operators today. Implementations include Obol Network and SSV Network.
- Sequencer HA. Most rollup sequencers today don't face slashing, but their state is hot, and a bad failover can delay or reorder transactions. The pattern that works is a single active leader, continuous state replication to a standby over vRack, and promotion only after the old leader is provably fenced off.
Put each workload where it fits
A hybrid design puts each workload where it fits. Validators, the sequencer leader, the signer, and full nodes are timing-sensitive and always on. They belong on dedicated bare metal at a flat monthly cost.
RPC surge capacity, indexing, analytics, testnets, and continuous integration (CI) are spiky and stateless. They can burst to OVHcloud® Public Cloud, where you pay for what runs.
The rule of thumb is simple. If it holds keys or signs, it lives on dedicated hardware. Both layers share the same vRack, account, and support, so your architecture stays one system.
The migration playbook
Here's the three-phase plan we promised in Part 1:
- Run in parallel (weeks 1 to 2). Stand up your OVHcloud stack alongside your current provider. Sync full nodes from scratch on dedicated NVMe and up to 5 Gbps networking. No production traffic moves.
- Validate everything (weeks 2 to 4). Compare attestation timing, peer counts, sync health, and alerting side by side. Shift read-only RPC traffic first, since it's reversible in minutes. Rehearse failover drills, beacon switches, and restore tests.
- Cut over the signer (a planned hour). Stop the old signer and its validator clients, and make sure they can’t restart. Then export your slashing history with EIP-3076. Wait at least two epochs, about 13 minutes on Ethereum®, then import the history and start the new signer.
Stopping first keeps the history complete, and the wait adds a safety margin against the two signers overlapping. The cost is a few missed attestations in a window you choose. For once, the asymmetry works in your favor.
What's next in the series
Part 3, Beyond the Node, asks a new question. You've built infrastructure you can trust. What if you didn't have to trust it at all? We'll explore confidential compute with Intel® SGX and AMD® SEV-SNP, which is designed to keep data protected even from the host and the infrastructure provider .
Geth® is a registered trademark of Stiftung Ethereum (Foundation Ethereum). OVHcloud® is a registered trademark of OVH SAS. Ethereum® is a registered trademark of Stiftung Ethereum (Foundation Ethereum). CONSENSYS® is a registered trademark of ConsenSys AG), used under license by Consensys Software Inc. SSV™ and SSV.NETWORK™ are trademarks of SSV Foundation. Intel® is a registered trademark of Intel Corporation. AMD® is a registered trademark of Advanced Micro Devices, Inc. All other trademarks are the property of their respective owners.
Get started today
Phase 1 of the playbook is a proof of concept (POC). Stand up the stack on dedicated bare metal and measure your own sync times and attestation timing. It runs alongside your current provider, with no production commitment required to start.
Prefer a conversation first? Bring your current topology to an architecture review. We'll map it against the redundancy layers above and show you where your keys live. Contact OVHcloud Sales to scope either option.