This chapter covers adding new nodes to the KeyControl cluster and upgrading nodes in the cluster.
KeyControl nodes can operate singly, but it is recommended that they operate as part of an active-active cluster. There is no theoretical limit to the number of KeyControl nodes in the cluster. HyTrust QA tests up to four nodes. Most customers deploy at least two nodes that are often in different geographic environments.
One question that comes up often is “What happens if there is a network disruption between the nodes?” The cluster has a "cluster operation timeout value." If communication between the nodes is lost after the timeout, the cluster goes into “degraded mode.” You can review cluster configuration and timeout setting by clicking the Cluster icon, and then clicking on the cluster you want to check. The health of the KeyControl cluster shows on the Status line and on the Cluster Icon, as shown below:
Here you can see that the cluster Status is “Healthy.” The Cluster icon shows a green heart with the number "2" inside of it, to indicate that there are two nodes in the cluster.
If KeyControl nodes cannot talk to each other, the Status switches to “Degraded.” When the cluster is in degraded mode, you can still view everything, keys can be delivered, but changes cannot be made. You will also get an email Alert to let you know that the cluster needs attention. If the cluster were in degraded mode, there would be a red "X" across the Cluster Icon. Here is a cluster in degraded mode:
Network latency can be managed by modifying the cluster operation timeout value. By default it is set to 5 seconds, which is a pretty high value. Typically, network latency is measured in milliseconds. Based on the clusters we have deployed at Amazon, the network latency from N. California to Oregon was under ~100ms, North Virginia to Oregon was around ~200ms. and N. California to Ireland was over ~280ms. The values varied based on time of day and the day of the week but the values are still considerably less than 5 seconds. If delays are consistently above 5 seconds, there is a network problem somewhere or a node is down.
To change the timeout value, click on the Cloud icon and then click Cluster Operation Timeout. Deploying multiple nodes in a cluster is very simple. For customers who are concerned about network latency, launch another KeyControl VM and monitor to see if the cluster goes into degraded mode.
For more information, see Managing Clusters.
You may need to remove a KeyControl Node from a cluster, using the webGUI. Note that you cannot remove the current node. To begin open the Cluster Icon, and highlight the node or nodes you want to remove. Click the Actions button and then click Remove:
You will be prompted to confirm removal of the node. Click Proceed.
To rejoin the cluster, follow the instructions in Installing Additional KeyControl Nodes. However, you will start by logging into the console menus.
The process for installing an additional KeyControl node or reinstalling a KeyControl node is similar to installing the first KeyControl node but has an additional authentication step.
Initial installation is described Installing KeyControl Nodes.
Once installation has completed and the system has rebooted, you will be prompted to select the system type as follows:
Select option 2, confirm your selection, and press Enter:
Next, you set up the console menu password and set up networking following the procedures you used in setting up the first KeyControl node.
At this point, the new KeyControl node needs to be authenticated with the KeyControl cluster. The following screen appears informing you of the need for authentication. You will need the IP address of any other KeyControl node in the cluster.
The first thing to enter is a description for this node. This allows you to specify information about the new KeyControl node being installed. For example, it could be location information such as "This is the KeyControl in the Miami data center." When you go to the webGUI to authenticate this node, this description is displayed.
The next piece of information to enter is the IP address of any KeyControl node in the existing cluster:
The final piece of information required is the passphrase itself. We require a minimum of 16 characters. If you knowingly type a passphrase incorrectly, you can hit CTRL+C and select the Re-authenticate This System option in the console menu.
The node must now be authenticated through the webGUI, as the following message indicates:
At this point you need to log on to the webGUI with HTDC Domain Administration privileges. The new KeyControl node will automatically appear as an unauthenticated node in the KeyControl cluster, as shown below:
To authenticate this new node, click the Actions button and then click Authenticate. This will take you to the authentication screen, where you are prompted to enter the Authentication Passphrase.
On the new KeyControl’s console, you will see a succession of status messages, as shown here:
This is followed on the new KeyControl's console by the standard console menu.
Once authentication completes, the KeyControl node is listed as Authenticated but Unreachable until cluster synchronization completes and the cluster is ready for use.
This should take no more than a minute or two. Then the node will show as Authenticated and Online.
Once the KeyControl node is available, the status will automatically move to Online and the cluster status will change back to Healthy.
At this point, the new cluster/node is ready to use.
All KeyControl nodes must operate at the same software version. Prior to upgrade, we recommend that you back up the KeyControl cluster. For further information, see KeyControl Backup and Restore.
We only support upgrade of a single KeyControl node at this time. Also, you can only upgrade between successive versions.
Upgrade to v 3.4 is only allowed from v 3.3. However, you can upgrade from 2.7 to 2.7.1 to 3.0 to 3.1 to 3.2 to 3.3, and from there to v 3.4.
Note on upgrading for AWS users: Upgrades for AWS users must be done using the webGUI, shown below. Other users can use the webGUI or the ISO upgrade. For more information, see Upgrading Using an ISO Image.
You should follow the steps listed below for upgrading your KeyControl nodes and clusters:
To remove a KeyControl node from the cluster, select the Cluster icon, select the Servers tab, and choose one of the KeyControl nodes to remove. Note that the node you are logged in on appears with a red-shaded background, which helps you to know which ones are safe to remove. Then click the Action button and click Remove.
| Note: | If you are using the Policy Agent, you need to make sure that you manage the KeyControl list on each agent. If you are upgrading an N-Node cluster and will return to the same N-Node cluster after upgrade, there is nothing that you need to do. However if you are changing IP addresses or changing the number of KeyControl nodes, please update the KeyControl list accordingly. Please refer to of the Policy Agent chapter for further information. |
Upgrading a KeyControl node is a very simple process that should take only a few minutes to complete. Deploying, of course, depends upon your environment. However, if your data connection is very slow and it takes longer than five minutes to upload the upgrade bundle, a timeout will prevent completion. In that case, upgrade using an ISO image, as shown in Upgrading Using an ISO Image.
To upgrade using the webGUI, click the Settings icon and then click Upgrade.
The System Upgrade dialog box presents you with a information about the upgrade process, and prompts you to browse to your upgrade file. When you have found the file, click Upload file.
When the upgrade finishes, you are prompted to reboot to finalize the upgrade.
The process of reverting to the previous version follows the same process described above.
Click the Revert Upgrade button. A message appears that a Revert is Pending. It is only possible to revert to the previous version, nothing further back. As with Upgrade, a reboot is required to finalize the reversion.
Warning: Any changes to the KeyControl node will be lost if you revert. This includes any keys or other objects created since upgrade.
Upgrading a KeyControl node is a very simple process, whether done through the webGUI, shown above, or through using an ISO image, shown here.
Begin by shutting down the KeyControl node that you are going to upgrade.
The CD/DVD containing the ISO image must be inserted into the drive (or attached to the VM) and the system should be booted from the CD/DVD. Once the system comes up, you will see:
Select option 1 and press Enter. The installer will check to make sure that the system can be upgraded and then perform the upgrade, which should not take more than a minute or so. Prior to actually performing the upgrade you will see a screen similar to the following:
| Note: | You should back up your KeyControl nodes and remove all nodes from the cluster through the GUI with the exception of the node being upgraded. |
If another node still exists in the cluster you will see the following warning screen. Do not ignore the warning and do not override upgrade. Overriding this warning is NOT recommended, and is likely to cause problems. The nodes cannot communicate at different software levels.
If you do have a single node in the cluster and you select OK to upgrade then the upgrade process will start. Once the upgrade has completed, you will see the following screen.
At this point, eject the ISO disk and press Enter to reboot. For this KeyControl node, log back into the webGUI, click the Settings icon and then Upgrade. You should now see the new version number. You also see the previous version, and can click Revert Upgrade to return to it.
To add back the nodes that you removed, go through the installation as a fresh install. Then add each node into the cluster.
You can also revert to the previous release using an ISO. Boot from the ISO file, as described elsewhere in this topic, and you will be presented with the following screen. If you wish to revert, choose 3.
| Warning: | Any changes to the KeyControl node will be lost if you revert. This includes any keys or other objects created since upgrade. |