Linux File and Folder Encryption

Introduction

To this point we have covered encryption of devices in Linux, both system (root and swap) and data disks. This mechanism allows you to encrypt all data within the VM to make sure that it’s not visible to virtualization and storage admins, a technique that is particularly useful in public cloud environments. There are however times when users want to encrypt specific directories. Such examples may include user's home directories or NFS client mounts.

This chapter shows you how to encrypt directories and how to share encryption keys across multiple NFS clients. First of all, let’s consider where the encryption is taking place as the following figure shows:

The HyTrust filesystem filter driver, based on the Linux eCryptfs filesystem, runs across all versions of Linux and sits just above the base filesystem. Thus any type of filesystem is supported (both local, NFS and SAMBA). There are no limits to the number of folders that can be encrypted per VM and encryption / decryption is transparent to all applications that are running over the encrypted folder. We encrypt not just the contents of the files but also the filenames. NOTE: eCryptfs is not supported in RHEL 7.x or CentOS 7.x.

Directory Restrictions

Unlike disk encryption, you cannot encrypt all directories in a Linux VM. The HyTrust Policy Agent starts up once all major services are available (networking etc). Be careful not to choose system directories for which clear-text files need to be available during bootstrap. The easiest way to understand how to use HyTrust file/folder encryption is that the specified directory must be empty prior to calling “hcl addfs”.

Filesystem IDs (FSIDs)

An FSID (filesystem ID) is a reference to an AES encryption key. Unlike disk encryption where no key is specified, we need a reference to a key since we need to be able to refer to the key from multiple clients (NFS client case).

NOTE: The Linux DataControl Policy Agent supports filesystem level encryption for some Linux distributions. However is does not support NFS on RHEL 6.x and above or CentOS 6.x and above.

To create and manage FSIDs, simply call one of the hcl options shown below:

  fsid <-c fsid_to_create [-s] [-a <cipher>] [-d description]
		[-e days_to_expire [-o "NO USE"|"SHRED"]]>
		<-r fsid_to_remove [-f]>
		<-u fsid_to_update [-d description]>
		<-l>

Let’s create a couple of FSIDs. Note that FSIDs are associated with the Cloud VM Set that the VM is a member of. Note that FSIDs can be named anything you like. We suggest naming them in a way that makes sense to you, and add a description that makes its intended use clear to you and to others who may use them. In the example that follows, my_local_key and my_nfs_key are not functionally any different, other than the way you are using them.

# hcl fsid -c my_local_key -d "this is a key just for this VM"
# hcl fsid -c my_nfs_key -d "this key is for NFS"
# hcl fsid -l
FSID                     Algorithm      Description
-----                    ---------      -----------
my_local_key             AES-256        this is a key just for this VM
my_nfs_key               AES-256        this key is for NFS

Note that you can change the default cipher which is AES-256, as well as the description, the number of days before the FSID expires (no expiration by default) and an action to be taken when the FSID expires.

The list of available ciphers that the HyTrust Policy Agent supports can be seen using the following command:

The list of available ciphers for folder encryption are:

# hcl ciphers
List of available ciphers:
	AES-XTS-512 (default)
	AES-256
	AES-XTS-256
	AES-128

The list of available ciphers for folder encryption are:

       AES-256 (default)
       AES-128

We can view the FSIDs in the GUI by selecting a Cloud VM Set and then selecting the FSIDs tab as follows:

Actions that can be taken on an FSID, is to revoke access or grant access if previously revoked. We shall discussion FSID revocation later in this chapter.

Adding a Folder for Encryption

Now that we have created an FSID, we can apply it to a directory that we wish to encrypt. Note that the directory must be empty. To map an FSID to a directory and to mount it for encryption, you need to use “addfs” option:

# hcl addfs my_local_key /local
Adding directory "/local" with fsid "my_local_key"
Authentication tokens are added to kernel keyring
Preparing to Mount on /local

Encrypted filesystem mounted on /local

You can also manually mount and unmount a folder, like this:

# hcl mount /local
# hcl unmount /local

You can see the encrypted directory in the GUI by selecting the VM and clicking the Folders tab:

You will also be able to see the mounted directory by running “mount” inside the VM:

# mount
...
/local on /local type ecryptfs (rw,relatime,ecryptfs_fnek_sig=ab350dc7f911f6bc,ecryptfs_sig=2e8d4a4956c09957,\
   ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_unlink_sigs)

The directory will be automatically mounted after every reboot.

Now that you have an encrypted directory that is mounted, just use it like any other directory.

# cp /lib/* /local 2>/dev/null
# ls /local
klibc-P2s_k-gf23VtrGgO2_4pGkQgwMY.so*    libip4tc.so.0.1.0   libiptc.so.0.0.0
libcryptsetup.so.4		       	  libip6tc.so.0	      libxtables.so.10
libcryptsetup.so.4.5.0		 	  libip6tc.so.0.1.0   libxtables.so.10.0.0
libip4tc.so.0		        	  libiptc.so.0

If you unmount the encrypted directory and look at the contents inside you will see files with encrypted filenames so you won’t be able to determine which file is which. And of course the contents are encrypted so unreadable.

# hcl unmount /local
# ls /local
ECRYPTFS_FNEK_ENCRYPTED.FWafBEr5yF5qj-aGe.x4Xla6MtnW2tMu1oNH9jQo0Dxk9TyncevY4yYaxk--
ECRYPTFS_FNEK_ENCRYPTED.FWafBEr5yF5qj-aGe.x4Xla6MtnW2tMu1oNHBB9lTjQqPPeRf1sQGaQUzE--
...

Note that although you can call “umount /local” we recommend that you use “hcl unmount” since it will remove key references from the Linux kernel keychain ring.

NFS Client Encryption

If we are running multiple NFS clients and wish to encrypt on each client, we need to ensure that each client is using the same symmetric key. In the following figure, we can see that there are two NFS clients accessing the same directory from the NFS Server. They are both going to share the FSID “NFS_KEY”.

Here are the steps that must be followed in order to for both VMs to access the same NFS mountpoint:

Mount the NFS-exported directory on each VM
Create an FSID on one of the client VMs
Enable sharing of the FSID in the GUI (allows other VMs to access it)
Run hcl addfs on both of the VMs

So from VM-A shown in the figure above, run the following:

# hcl fsid -c "NFS_KEY" -d "This is a shared key"
# fsid -l
FSID                     Algorithm      Description
-----                    ---------      -----------
NFS_KEY                  AES-256        This is a shared key
my_local_key             AES-256        this is a key just for this VM
my_nfs_key               AES-256        NFS key
# hcl addfs NFS_KEY /secret

Note that the exported filesystem must have already been mounted on /secret. Now if you run the following on VM-B:

# hcl fsid -l
FSID                     Algorithm      Description
----                     ---------      -----------

you can see that no FSIDs are listed even though this VM is in the same Cloud VM Set.

At this point, we go to the GUI, select the appropriate Cloud VM Set, select the FSIDs tab and select the appropriate FSID, in this case “NFS_KEY”.

Change “Shared” to “Yes” as shown and click save. Now if you go back to VM-B, you will be able to see the FSID:

# hcl fsid -l
FSID                     Algorithm      Description
----                     ---------      -----------
NFS_KEY                  AES-256        This is a shared key

and we can therefore access the contents as follows:

 #  hcl addfs NFS_KEY /nfs_mtpt

And now let’s create some files and access them:

VM-A	       # echo hello > /secret/a-file
VM-B		# cat /nfs_mtpt/a-file 
		hello
VM-B		# echo "how are you?" > /nfs_mtpt/b-file
VM-A		# cat /secret/b-file
		how are you?

and if we look on the NFS server, we only see encrypted filenames:

# ls /nfs
ECRYPTFS_FNEK_ENCRYPTED.FWZJQXtCp3pjBkaFaEsRCsBcjOxB13Rf0xJu66WoxAVbkwXf5f5x.WUYJk--
ECRYPTFS_FNEK_ENCRYPTED.FWZJQXtCp3pjBkaFaEsRCsBcjOxB13Rf0xJukdu2eF4grFOJ-oZGmxcqak--

and if you look at the contents of a file, you will see that it is encrypted.

Note that we do nothing to provide synchronization between files accessed by multiple NFS clients.

Backup, Restore and Migration of Encrypted Directory Hierarchies

As you’ve seen from examples so far, when you create and access files/directories through the encrypted mount point, they look like any other Linux directory. This is exactly what we want to achieve for application transparency but what about backup and restore? We have also shown the when the encrypted filesystem is unmounted, the encrypted file hierarchy with encrypted files and file names is then visible. Let’s look at the two states that an encrypted directory hierarchy can be in:

On the left, we have the encrypted directory mounted. The HyTrust filesystem driver is in place (“hcl mount”) so every time a file is read from the base file system, it will be decrypted. Similarly, when files are created / written to, they will be encrypted before being passed to the base file system. On the right, the filesystem is unmounted (“hcl unmount”). In this case, any access is to the encrypted files.

Obviously you do not want applications to generally access encrypted files but two exceptions to this rule are:

The need to backup encrypted files
An ability to migrate a directory hierarchy of encrypted files/directories from one VM to another.

HyTrust maintains meta-data at the top level of each encrypted directory. You will see a directory as follows:

# ls -a /secret
./  ../  a-file  b-file  .hytrustconfig/

You will see a directory called .hytrustconfig. You should not remove this directory or modify any files that reside in there. The configuration inside this directory allows the HyTrust DataControl Policy Agent to map an encrypted hierarchy back to an FSID. You can find the FSID in a specific directory as follows:

# hcl identify /secret
/secret encrypted with fsid "NFS_KEY"
/secret encrypted in CVM Set "AWS"

If you are going to backup an encrypted directory, it must be done without the HyTrust filesystem driver in place (i.e. unmounted). If this is not possible there are two other options:

Backup the clear-text files/directories and ensure that your backup application is able to encrypt its own archives.
Perform VM-level backup in which case the whole VM’s virtual disk can be recovered.

Let’s cover an example of how to migrate an encrypted directory. Using the same configuration we had before (VM-A and VM-B), perform the following operations:

# mkdir /share
# hcl fsid -c "migrate_key" -d "Shared key for migration"
# hcl addfs migrate_key /share
# hcl mount /share
Authentication tokens are added to kernel keyring

Preparing to Mount on /share

Encrypted filesystem mounted on /share
# echo "can I see this on another VM?" > /share/myfile
# hcl unmount /share
# cd /
# tar cvfz share.tgz share
share/
share/ECRYPTFS_FNEK_ENCRYPTED.FWZGnwaRQSIucUZzm9Gj-ll9zK5N.wo8tW3jP8aPUItIJERy2ZgvjsPiHE--
share/.hytrustconfig/
share/.hytrustconfig/fsid
share/.hytrustconfig/cvmset
share/.hytrustconfig/cookie

What we have done here is the following:

Remove the FSID “migrate_key” which is associated with the directory “/share” which is then added/mounted.
We create a file under this directory.
We unmount the HyTrust filter driver.
We tar up the directory. You can see the one encrypted file and the HyTrust configuration directory.

All good so far. Now let’s copy over the tar archive to a different VM and unpack it.

# pwd
/root
# tar xvfz share.tgz
share/
share/ECRYPTFS_FNEK_ENCRYPTED.FWZGnwaRQSIucUZzm9Gj-ll9zK5N.wo8tW3jP8aPUItIJERy2ZgvjsPiHE--
share/.hytrustconfig/
share/.hytrustconfig/fsid
share/.hytrustconfig/cvmset
share/.hytrustconfig/cookie

We can use the hcl identify option to determine which FSID was used in this directory hierarchy:

# hcl identify share
/root/share encrypted with fsid "migrate_key"
/root/share encrypted in CVM Set "AWS"

Next, let’s make sure that, through the GUI, you share the “migrate_key” FSID. Now let’s make sure that it’s available on the VM-B:

# hcl fsid -l
FSID                     Algorithm      Description
----                     ---------      -----------
migrate_key              AES-256        Shared key for migration

So now we can add the filesystem and mount it:

# hcl addfs migrate_key /root/share
Adding directory "/root/share" with fsid "migrate_key"
# hcl mount /root/share
Authentication tokens are added to kernel keyring

Preparing to Mount on /root/share

Encrypted filesystem mounted on /root/share
# cat /root/share/myfile 
can I see this on another VM?

So, we can now see the file/contents that we originally created on VM-A.

Revoking FSIDs

If you revoke an FSID, you will temporarily be unable to access any data on the VM. For example, a call to hcl mount will fail. To revoke access, within the GUI, select the appropriate FSID and choose Revoke FSID Access from the Actions menu as the screenshot below shows:

If you then attempt to mount the directory, you will see the following error:

# hcl mount /local
Error on fsid getkey my_local_key: FS not active

If you wish to grant access back to the FSID, simply select it in the GUI and choose Activate FSID Access from the Actions menu.

Removing an Encrypted Filesystem

To remove an encrypted filesystem, you must first unmount it and then remove it from HyTrust control as follows:

# hcl rmfs /root/share
Removing this filesystem will cause any data stored on them to become inaccessible
Use “hcl addfs” to access the data again

Do you want to proceed? (y/n) y

Encrypted filesystem on /root/share removed.
WARNING: If FSID <myfsid> is removed, the data will be permanently lost

This removes the filesystem from the HyTrust configuration database but doesn’t actually remove the FSID. Thus it would be possible to run “hcl addfs” to add it back.

To remove an FSID, no filesystem should be currently mounted or held within control by HyTrust. For example, let’s try and remove the “migrate_key” FSID that we used in our sharing example above. At this point we have run hcl rmfs on VM-B and now try to remove the FSID.

# hcl fsid -r migrate_key
WARNING: Removal of fsid will result in permanent failure to decrypt anything using that fsid.
Do you want to proceed? (y/n) y

Error removing fsid migrate_key : fsid migrate_key busy

We fail to remove the FSID because it is still in use on VM-A. To actually remove it, we must first unmount the filesystem on VM-A and also run hcl rmfs on VM-A too.

Note that once the FSID is removed, you will no longer be able to access the data again, nor will you be able to restore from its previous encrypted backup.

Clashing on Old FSIDs

When removing filesystems, be careful to remove all files and directories before the directory can be used again. For example:

# pwd
/root
# ls share
ECRYPTFS_FNEK_ENCRYPTED.FWZGnwaRQSIucUZzm9Gj-ll9zK5N.wo8tW3jP8aPUItIJERy2ZgvjsPiHE--
# rm -rf share/*
# hcl fsid -c new_key -d "another FSID"
# hcl addfs new_key /root/share
/root/share directory is encrypted with different fsid "migrate_key"

Here we remove everything under /root/share, create a new FSID and try and add the /root/share directory again. This time it fails and tells us that the directory is associated with a different FSID, in this case migrate_key. How can this be? Well, although we thought we remove all files, the HyTrust meta-data directory (share/.hytrustconfig) still exists. To reuse the directory you should remove this directory too.