Troubleshooting the HTCrypt Driver

When you upgrade the KeyControl Policy Agent or the kernel files on a Linux VM, the Policy Agent automatically rebuilds the HTCrypt Driver using DKMS. If the required packages are not available at this time, the Policy Agent reverts to using offline encryption and will no longer encrypt, decrypt, or rekey devices using Online Encryption.

The following hcl status output indicates that there are issues with the HTCrypt Driver.

# hcl status

Summary
--------------------------------------------------------------------------------
KeyControl: None
Status: Not registered
AES_NI: enabled
HTCRYPT: unavailableERROR !!!!! htcrypt is installed but not accessible, run "htdrv status" for details

If this happens, you can manually update and rebuild the HTCrypt Driver. If rebuilding it does not work, you can also reinstall the driver.

Rebuilding the HTCrypt Driver

  1. Log into the VM on which you want to re-enable the HTCrypt Driver as root.

  2. Display the status of the HTCrypt Driver by entering the htdrv status command. If some packages are out of date, this command will show which packages need to be updated.

    The following example shows the results of the htdrv status command before the HTCrypt Driver has been installed:

    # htdrv status
HyTrust online encryption driver status
--------------------------------------------------------------------
package htcrypt is not installed

>>> htcrypt driver is not installed on this VM

ERROR !!!!! Missing kernel headers for the current kernel version: 3.10.0-514.el7.x86_64 or higher
ERROR !!!!! Missing kernel development package for the current kernel version: 3.10.0-514.el7.x86_64
ERROR !!!!! gcc missing -- required for htcrypt driver
ERROR !!!!! dkms missing -- required for htcrypt driver

Here are the Linux kernel and related packages installed on this system
------------------------------------------------------------------------
Installed Packages
kernel.x86_64                   3.10.0-514.el7                    @anaconda
kernel-tools.x86_64             3.10.0-514.el7                    @anaconda
kernel-tools-libs.x86_64        3.10.0-514.el7                    @anaconda
  3. Install the required supporting packages by entering the htdrv prepare command.
  4. Recompile the HTCrypt Driver by entering the htdrv rebuild command.

  5. To verify that the rebuild worked, use the hcl status command and make sure the HTCrypt Driver status is enabled.

    # hcl status

Summary
--------------------------------------------------------------------------------
KeyControl: None
Status: Not registered
AES_NI: enabled
HTCRYPT: enabled

    If the rebuild worked and the HTCrypt Driver is now available, proceed to the next step and finish the process by rebooting the VM. If the rebuild did not work, you should reinstall the HTCrypt Driver as described below.

  6. If the VM has its root devices encrypted before the installation of the online encryption driver, then in order to extend the online encryption driver support to the root devices, the admin should run the following commands: 

    # htroot update 
# reboot

  7. If the root or swap disk is encrypted on this VM, you need to reboot the VM to complete the installation process.

    If only data disks are encrypted on the VM, you can either reboot the VM or detach and then reattach all attached data disks to start the HTCrypt Driver on those disks. To detach and reattach the disks, use the hcl detach -a and hcl attach -a commands. For example:

    # hcl detach -a
Encrypted device sdi7 detached; encrypted contents no longer visible
Encrypted device sdi1 detached; encrypted contents no longer visible
# hcl attach -a
Encrypted device sdi7 (/dev/sdi7) attached; encrypted contents visible at /dev/mapper/clear_sdi7
Encrypted device sdi1 (/dev/sdi1) attached; encrypted contents visible at /dev/mapper/clear_sdi1

Reinstalling the HTCrypt Driver

If rebuilding the HTCrypt Driver does not work, you can uninstall the driver, clean up any old installation files, and then reinstall the driver from scratch.

  1. Uninstall the HTCrypt Driver. For RHEL run the command rpm -e htcrypt, and for Ubuntu run the command dpkg -r htcrypt-dkms.
  2. Clean up all HTCrypt Driver files by entering the rm -rf /var/lib/dkms/htcrypt command.

  3. Reinstall the HTCrypt Driver.

    • For RHEL, reinstall the HTCrypt Driver RPM by entering the command rpm -ivh /opt/hcs/drivers/htcrypt-10.3.1-buildnum.noarch.rpm, where buildnum is the build number that you are installing. For example:

      # rpm -ivh htcrypt-10.3.1-12345M.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:htcrypt-10.3.1-12345M               ################################# [100%]
Loading new htcrypt-10.3.1 DKMS files...
Building for 3.10.0-693.17.1.el7.x86_64
Building initial module for 3.10.0-693.17.1.el7.x86_64
Done.

htcrypt:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.10.0-693.17.1.el7.x86_64/extra/

depmod....

Backing up initramfs-3.10.0-693.17.1.el7.x86_64.img to /boot/initramfs-3.10.0-693.17.1.el7.x86_64.img.old-dkms
Making new initramfs-3.10.0-693.17.1.el7.x86_64.img
(If next boot fails, revert to initramfs-3.10.0-693.17.1.el7.x86_64.img.old-dkms image)
dracut...................

DKMS: install completed.

    • For Ubuntu, reinstall the HTCrypt Driver DEB by entering the command dpkg -i /opt/hcs/drivers/htcrypt-dkms_<buildver>_all.deb, where buildver is the build version that you are installing. For example:

      # dpkg -i /opt/hcs/drivers/htcrypt-dkms_5.40012345_all.deb 
Selecting previously unselected package htcrypt-dkms.
(Reading database ... 169211 files and directories currently installed.)
Preparing to unpack .../htcrypt-dkms_5.40000655_all.deb ...
Unpacking htcrypt-dkms (5.40012345) ...
Setting up htcrypt-dkms (5.40012345) ...
Loading new htcrypt-5.40012345 DKMS files...
Building for 5.4.0-64-generic
Building for architecture x86_64
Building initial module for 5.4.0-42-generic
Done.

htcrypt.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-42-generic/updates/dkms/

depmod.....

Backing up initrd.img-5.4.0-42-generic to /boot/initrd.img-5.4.0-42-generic.old-dkms
Making new initrd.img-5.4.0-42-generic
(If next boot fails, revert to initrd.img-5.4.0-42-generic.old-dkms image)
update-initramfs................

DKMS: install completed.
Building initial module for 5.8.0-38-generic
Done.

htcrypt.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.8.0-38-generic/updates/dkms/

depmod......

Backing up initrd.img-5.8.0-38-generic to /boot/initrd.img-5.8.0-38-generic.old-dkms
Making new initrd.img-5.8.0-38-generic
(If next boot fails, revert to initrd.img-5.8.0-38-generic.old-dkms image)
update-initramfs.................

DKMS: install completed.

  4. To verify that the installation succeeded, enter the hcl status command. For example:

    # hcl status

Summary
--------------------------------------------------------------------------------
KeyControl: 10.238.66.235:443
KeyControl list: 10.238.66.235:443
Status: Connected
Last heartbeat: Wed Mar 21 12:48:19 2019 (successful)
AES_NI: enabled
Certificate Expiration: Sep 11 22:16:13 2020 GMT
HTCRYPT: enabled

    At this point, the HTCrypt Driver is installed but is not yet running on any disks that are currently attached.

  5. If the VM has its root devices encrypted before the installation of the online encryption driver, then in order to extend the online encryption driver support to the root devices, the admin should run the following commands: 

    # htroot update 
# reboot

  6. If the root or swap disk is encrypted on this VM, you need to reboot the VM to complete the installation process.

    If only data disks are encrypted on the VM, you can either reboot the VM or detach and then reattach all attached data disks to start the HTCrypt Driver on those disks. To detach and reattach the disks, use the hcl detach -a and hcl attach -a commands. For example:

    # hcl detach -a
Encrypted device sdi7 detached; encrypted contents no longer visible
Encrypted device sdi1 detached; encrypted contents no longer visible
# hcl attach -a
Encrypted device sdi7 (/dev/sdi7) attached; encrypted contents visible at /dev/mapper/clear_sdi7
Encrypted device sdi1 (/dev/sdi1) attached; encrypted contents visible at /dev/mapper/clear_sdi1