KeyControl Backup and Restore

Understanding the Admin Key

All KeyControl data (policy information, encryption keys, and so on) are held in an encrypted object store that is clustered across all KeyControl nodes in the cluster. Each KeyControl node has its own storage, which does not need to be identical in size to other nodes.

When you first login to KeyControl and fill in user information for secroot, you will either be emailed an “Admin Key” part, or if you checked the Disable E-mail notifications box, your key part will be visible as an Alert. You must keep this key safe, as it will be used if you restore from backup or if you change any hardware configuration on KeyControl.

You can also download the Admin Key on demand, for storage on your local computer. To download the Admin Key, click the Settings Icon, which will bring up the following screen:

Click Download Key to securely download the Admin Key onto your computer. You can click Clear Key to remove the Admin Key from the KeyControl, for security purposes. Note that if you clear the Admin Key, you will have to generate another Admin Key before you can download it.

To generate another Admin Key, click Admin Key Parts (on the same screen) and then Generate New Key.

Each time a Security Admin is added or removed, a new Admin Key is generated and split amongst the all of the “M” Security Admins. It is important that you keep the Admin Keys and take note of the date on which they were generated. If a restore takes place you will need to enter “N of M” Admin Keys in order for the restore to succeed. The value of N is “1” by default, but this can be changed at any time by clicking Admin Key Parts from the Settings Icon and changing the value of Minimum Key Parts.

The following figure highlights four events and shows how the Admin Keys are related to backup and restore.

Time	Action	Admin Keys generated	Admin Keys needed on restore from backup
t0	Installed system	Key 1
t1	Backup taken		Key 1
t2	Security Admin created	Keys 2
t3	Backup taken		Keys 2

A restore from any backup taken before the new Security Admin is created will require “Key 1”. A restore from any backup taken after the new Security Admin is taken requires M of “Keys 2”. It is highly recommended that all sets of Admin Keys are stored securely by each Security Admin. For example, after time t3, if there was a need to restore from the backup taken at time t1, “Key 1” would be required. In addition to a restore from backup, you will be required to enter “N of M” Admin Keys if the hardware inside KeyControl changes.

For details on downloading the Admin Key to your local computer, see Account Settings

Storing the Admin Key on an External KMIP Server or HSM

KeyControl also supports the ability to store the Admin Key on an external KMIP server, as the following figure shows. Note that the KMIP server can be one of your KeyControl servers, set up for this purpose. KeyControl can also store the Admin Key on a Hardware Security Module (HSM), discussed elsewhere in this guide.

If this option is chosen, there is no need to enter the Admin Key parts in the event that a restore takes place or a hardware change occurs on KeyControl. If either of these conditions occurs, KeyControl will talk to the KMIP server and fetch the Admin Key as necessary.

The KeyControl backup image should be safely escrowed and can be brought back at any time to restore KeyControl to the state it was in at the time the backup was taken.

For further information on using KMIP for system recovery, see: System Recovery From the External KMIP Server

KeyControl Virtual Machine Snapshots

Another common backup methodology is to take a virtual machine snapshot or to use a 3rd-party backup application that can operate on snapshots. Restoring from a snapshot will work without issues. Be warned though that if any part of the VM changes (IP address excepted), you will be required to go through Admin Key recovery

Virtual Machine Backup

For VMs containing the HyTrust DataControl Policy Agent, there is metadata associated with each disk that allows the disk to be self identifying, as the following figure shows:

On Linux we support multiple encrypted partitions per single disk. There is currently a limit of one encryption partition per disk for Windows.

In a private area on the disk, we store globally unique identifiers (GUIDs) which are references to the keys that are used. During a rekey operation, we retain both the old and new GUIDs and once the operation completes, only the new GUID is retained. Note however that we do not delete old keys from KeyControl following a rekey operation; keys are kept until they expire and are shredded. They are also retained following expiration, if you set them to read-only.

To back up, you need to ensure that the whole disk is backed up (e.g. VMDK file on a VMware vSphere environment). This ensures that the GUIDs representing the keys are also backed up. Thus any time a VM is restored, we will be able to identify which keys are needed based on the GUIDs stored on disk. A re-auth may be needed if the VM has exceeded the grace period, but the keys will be accessible.

Please note that if you put an expiration date on a key (keys don’t expire by default) and mark it to be shredded, the key will actually be deleted when the expiration date is reached. A restore of a backed up VM using that key will no longer be possible; it is suggested that key expiration is set to match your organization’s data retention policy. Note that shredding keys only deletes the key from KeyControl. It does not involved wiping data on the actual disk of the VM.

Backup and Restore: Step by Step

The object store on each KeyControl node is identical and therefore backup only needs to take place from a single KeyControl node. A restore operation must be performed on a single node. Only after restore can additional nodes be added to reconstruct the cluster.

Keys, policies and other information within each KeyControl node is encrypted. At the top of the KeyControl key hierarchy is the Admin Key, which must be carefully protected. To protect the Admin Key, we provide an "n of m" Admin Key backup model. The Admin Key is split into "m" pieces where "m" is the number of Security Administrators. Each administrator is given part of the key and to reinstate the Admin Key, "n" of the "m" administrators must be present to enter their part of the key. Note that the admins are not given part of the actual key itself but a mathematical representation of part of the key.

Consider the following figure:

In this example there are three Security Administrators and each is given a key part. We choose the value of "n" such that 1 <= n <= m. In this example, the value of "n" is "2". When restoring the Admin Key, any two of the admins need to be present. This is covered by one of the following combinations:

Security Admin A and Security Admin B
Security Admin A and Security Admin C
Security Admin B and Security Admin C

Admin Key parts are generated under the following conditions:

During installation of the first KeyControl node. In this case, the secroot admin gets the only part.
A Security Admin is added or deleted. After one of these operations, key parts are generated again.
You decide to explicitly generate new key parts.

If you wish to generate a new Admin Key, from the Settings Icon, click Admin Key Parts.

You then see the Default Settings dialog box, open to the Admin Key Parts tab.

Enter the Minimum Key Parts (the value of "n") and click Generate New Key. Each Security Administrator will receive a key part as an Alert, and in email (if you enabled email). If you wish to change the value of "n" but still retain the existing Admin Key, in the window above, change the value and then click Save, but do not click Generate New Key.

If you disabled email access on first GUI login, Admin Key parts will be sent as Alerts only. Each Security Administrator will be able to locate their key part by clicking the Alerts Icon, at the top of the screen. Select the Alert that begins “Here is your Admin Key...”

Note: HyTrust recommends that you use email attachments for key parts. If you need to, go back and re-enable email, as described in "Adding Email Support Back", discussed elsewhere in this guide.

It is vital that these key parts are stored securely. If "n" pieces are not available, restore of a KeyControl node will not be possible. Note however, that a new Admin Key can be generated at any time up to the point of KeyControl backup/restore.

NOTE: Admin Keys are generated automatically any time a new Security Administrator is created, broken into an appropriate number of key parts and distributed one per Security Administrator.

Important: Be sure that your Security Admins keep and protect all of their Admin Key Parts, because they might need an Admin Key Part from an earlier time, as described above.

Backing Up the KeyControl Cluster

When you back up, the backup image is encrypted and protected by the Object Store Key (which in turn is ultimately protected by the Admin Key). Backup can be done via the WebGUI or via NFS.

Backing Up Using the WebGUI

To create a backup on your command, click the Cluster Icon, and then the Actions button. The KeyControl Backup dialog box appears, showing the date and size of the most recent backup. Click Perform Backup to generate an updated backup. To download the backup image, click Download.

Backing Up Using NFS

KeyControl backups are made available via NFS. All backup data is securely encrypted.

It is possible to restrict access to the backup images to one or more hosts. The list of allowed clients can be modified in the Backup Hosts field of the Clusters page :

Enter a list of hostnames / IP addresses, separated by commas or spaces, for nodes that you want to be able to access the backups. Note that you can add 0.0.0.0, which allows access to the encrypted backup images from any client.

From one of the specified Backup Hosts, you can now mount the exported filesystem as follows:

# mount -t nfs 192.168.140.135:/hcs/backup /backup
# ls -l /backup
total 506
-rw-r--r--  2 root  wheel  129536 Aug 18 17:28 htkc.bu

Note that the backup images are available from any KeyControl node in the cluster.

Restoring from a KeyControl Backup Image

Restoring from a KeyControl backup will only need to be performed if there is a catastrophic failure in the KeyControl cluster. If one KeyControl node becomes unusable, for example due to hardware failures, simply remove the node from the cluster and add a new node.

WARNING: Restore is a destructive process. The data that was on that node is wiped out and a "clean" KeyControl node is put in its place.

Note: To restore, install as before, and then choose Restore.

When restoring a KeyControl node there are two possible scenarios:

You are restoring to an node that was previously used in the cluster (that is, it has the same hardware ID and no hardware components have changed). In this case, you will not need to perform Admin Key recovery.
You are restoring to new hardware, which will involve Admin Key recovery.

In either case, you will follow the same procedure for restore as you did for installation.

First of all, click OK to install the software.

Once the install is complete and the system reboots, you will then have the option to restore a KeyControl node from backup as the following screen shows:

Backup KeyControl Install Post Boot Menu

You enter the root password, set up networking, and then you are prompted to supply the KeyControl backup image from which you want to restore:

Mount the exported filesystem and copy in the KeyControl backup image as follows:

# mount -t nfs 192.168.140.138:/hcs/restore /restore
# cp htkc.bu /restore

# umount /restore

At this point, in the installation menu, enter the name of the file you copied in above and press Enter. The restore process now starts.

KeyControl Restore can be performed from the WebGUI as well. To restore a KeyControl node, click the Cluster Icon and then click the Actions button:

Click KeyControl Restore to open the KeyControl Restore dialog box. Click Browse to select the image to be restored.

Once selected, click Verify Image to validate the backup image. Once verified, click the Restore Image button to start the restore process.

Whether you select the image through the webGUI or the console, the length of time needed to load the image will be dependent upon the size of the image being restored. Once complete, you will be able to proceed to log on through the WebGUI.

At this point there are two things that can happen:

You have restored to a system with identical hardware. In case we detect that the hardware has not changed and you will be able to log on as you usually do.
The hardware has changed and you will be prompted to choose an option:
- Recovery using Keypart upload
- Recovery from External key server
- Decommission

You must now enter the appropriate number of Admin Key parts that were generated. For details on Admin Key parts, see Understanding the Admin Key, elsewhere in this topic.

Admin Key recovery can also be done from a KMIP server. For details, see Enabling the KMIP server,

Once you have entered all the appropriate key parts, the system will restart services. You will see a message as the system restarts. You should then see the login screen appear.