Manually Updating the CA Certificate on a Linux Boot Drive Encrypted VM

When you install a new SSL certificate on KeyControl, KeyControl automatically updates the associated CA certificate on all registered VMs. If a Linux boot-drive-encrypted VM was inaccessible during this process, that VM may not be able to boot because the CA certificate the VM is using can no longer verify the KeyControl SSL certificate. This means that the VM cannot retrieve the proper keys from KeyControl because it cannot verify the communication coming from KeyControl.

To fix this issue you need to manually update the CA certificate on the VM so that it can verify the SSL certificate KeyControl is currently using. This allows the VM to verify KeyControl's identity and to retrieve the appropriate keys.

Important: Manually updating the certificate requires ssh access to the VM. If you did not enable the HyTrust Debug Console when you ran the htroot encrypt command on the boot drive, you need to contact HyTrust Support at support@hytrust.com.

The following procedure is for Linux VMs with an encrypted boot drive. For other types of VMs, see Manually Updating the CA Certificate on a Windows Boot Drive Encrypted VM or Manually Updating the CA Certificate on a Data Encrypted VM.

Procedure 

  1. If you need a copy of the CA certificate that can verify the SSL certificate that KeyControl is currently using: 

    1. Log into the KeyControl webGUI on any node in the cluster using an account with Cloud Admin privileges.
    2. In the top menu bar, click Cloud.
    3. Select Actions > Download CA Certificate.

      KeyControl downloads a pem file to your browser's default download location.

    Note: If you are using an externally signed SSL certificate for KeyControl, make sure that you use the CA certificate you download from KeyControl on all registered VMs. Do not use the CA certificate that you received from the external certificate authority.
  2. Download the Bootloader SSH key (also called the id_rsa key) for the VM so that you can open an SSH session and copy the new certificate file to the Bootloader:

    1. Log into the KeyControl webGUI on any node in the cluster using an account with Cloud Admin privileges.
    2. Navigate to the Cloud > VMs tab and select the VM whose certificate you want to update.
    3. Select Actions > Download Bootloader SSH Key. KeyControl downloads the RSA key to file called server-name.key in your browser's default download location. For example, if the server name is winsvr68, the RSA file would be called winsvr68.key.
  3. Open an ssh session by entering the command ssh -i id_rsa root@vm_name, where id_rsa is the name of the id_rsa file and vm_name is the IP address or hostname. For example:

    # ssh -i ~/Downloads/winsvr68.key root@192.168.140.133
    Warning: Permanently added '192.168.140.133' (RSA) to the list of known hosts.
    BusyBox v1.20.2 (Ubuntu 1:1.20.0-8.1ubuntu1) built-in shell (ash)
    Enter 'help' for a list of built-in commands.
    
    HyTrust Debug Console
    
    1. Show HT encryption log file
    2. Authenticate
    3. Show Network info
    4. Restart Network
    5. Advanced access
    6. logout
    
    Action:  
  4. Select Advanced access. DataControl displays a secure shell prompt. For example:

    Action: 5
    sh-4.1# 
  5. Copy the CA certificate file to the VM by entering the command scp -i id_rsa cert.pem root@vm_name, where id_rsa is the name of the id_rsa file, cert.pem is the fully qualified path to the pem file, and vm_name is the IP address or hostname. For example:

    sh-4.1# scp -i ~/Downloads/winsvr68.key ~/Downloads/mycert.pem root@192.168.140.133
  6. Update the certificate by entering the command hcl update_ca -f cert.pem, where cert.pem is the fully qualified path to the pem file. For example: 

    sh-4.1# hcl update_ca -f ./mycert.pem
    
    Updating using cert file at: ./mycert.pem
    Updated CA certificate
    sh-4.1#
  7. Enter the command hcl heartbeat to prompt the VM to contact KeyControl. This updates the status information for the VM.
  8. Enter the command hcl status to confirm that the last heartbeat between the VM and KeyControl was successful. For example:

    sh-4.1# hcl heartbeat
    sh-4.1# hcl status
    
    Summary
    --------------------------------------------------------------------------------
    KeyControl: 10.238.65.65:443
    KeyControl list: 10.238.65.65:443 10.238.65.66:443
    KeyControl Mapping: kc41-nodes
    Status: Reauth needed (Virtual Machine not authenticated)
    Last heartbeat: Tue Oct 24 22:19:32 2017 (failed)
    AES_NI: enabled
    
  9. If the hcl status command says that the VM needs to be reauthorized, enter the command hcl auth -a [-u user [-s password]], where:

    If you do not provide a user name and password, you will be prompted for one. For example:

    sh-4.1# hcl auth -a
    Please provide the KeyControl login details
    username: secroot
    password: 
    Completing authentication on KeyControl node
    
    Authentication complete, machine ready to use
    sh-4.1#
    
  10. Enter the command hcl heartbeat to prompt the VM to contact KeyControl. This updates the status information for the VM.
  11. If the heartbeat is successful, the VM should automatically continue booting from the encrypted root drive. If it does not, enter the command exit to leave the secure shell, select logout from the HyTrust Debug Console main menu, and reboot the VM.