Buying immutable storage is a purchasing decision. Recovering from it is a skill, and you only find out whether you have that skill by trying it under realistic conditions. Restoring one file or one VM into a healthy network, with every admin account working, proves very little about a day when the network is gone.
Here’s how we’d run a drill that does prove something.
Set the scenario honestly
Write the scenario down before anyone touches a system. Assume the attacker has held domain admin for weeks, production is encrypted, and email, chat and ticketing are unavailable. Assume they have also had your backup console credentials, because those are an obvious early target.
Then ask what each control really gives you in that situation.
Immutable snapshots can’t be changed during their lock period, if the lock truly holds. Check whether an administrator can shorten retention, delete the whole volume, or change the clock the storage system trusts. Remember too that snapshots faithfully preserve whatever was there when they were taken, including the attacker’s tools.
Air-gapped copies survive almost anything, and they are the slowest to bring back. Find out how long a copy takes to reach somewhere useful, and whether the hardware to restore onto actually exists.
Integrity verification tells you a backup hasn’t changed since it was written. It can’t tell you that what was written was clean. A backup of already encrypted files will pass a checksum.
Run the drill
Don’t restore into production, and don’t start with everything. Pick one important business service and build an isolated recovery environment with no route to your live network. Then work through it with the people who would do it for real.
Follow the order you would need on the day. Identity, DNS and time come first, then whatever the application depends on, and the application last. You may discover at this point that nobody has written that order down, or that the restore tool itself needs a directory service you haven’t restored yet.
Insist on the awkward conditions. Treat normal admin accounts as burned, so the team uses break-glass credentials kept offline. Keep runbooks on paper or on a device off the network. Have people coordinate on a channel that doesn’t depend on your own email.
Choose the restore point on evidence. How do you know this snapshot predates the intrusion? Check restored systems for unexpected accounts, scheduled tasks and other signs of persistence before you call them clean.
Finish with the business owner, because a server that boots doesn’t count. The test is whether someone can log in and complete a real transaction, using data they accept as correct.
Score it and repeat
Time each stage, and compare the total with the recovery time you’ve promised the business. Record how much data you lost against the recovery point you promised. Note every moment someone said “we’d need to ask”, or reached for something that wouldn’t have been there. Those moments are the useful output of the day, so fix them and update the runbook.
Full drills are expensive, and you won’t run many. We’d mix tabletop exercises through the year with one or two real restores of your most critical services, and rotate who leads, so recovery doesn’t depend on one person being reachable. For a small estate, one well-run restore of your core system each year beats an ambitious schedule you’ll abandon.
Put a date in the calendar today. Choose one service, name the person who will lead, and agree that the backup administrator’s everyday login won’t be used on the day.