Move /home on SELinux systems

Here’s a cautionary tale: taking shortcuts can lead to disaster! I wanted to move /home to a new partition on an SELinux protected system. I ended up locking myself out of my VM.

Moving /home to a new partition

My /home directory was eating all the space on my primary partition so I added a new disk to my machine with the intention of mounting the /home directory to it. This is a fairly standard sysadmin operation but a little risky as it requires moving all the files belonging to user accounts in /home. Here’s what I wanted to do:

# Find  new attached storage
df -h
lsblk # Shows that 50GB sdb is attached but not mounted

# Create, format and mount the new volume
sudo mkfs -t xfs /dev/sdb
sudo mkdir /home_new
sudo mount /dev/sdb /home_new    # Mount the new volume to a temporary location
df -h	# Yes! it's there

# Move /home to new partition
sudo cp -a /home/* /home_new/    # Copy data to the new volume
ls -lah /home_new/opc/	# Yes! My user dir is there
sudo mv /home /home.bak    # Move the old /home directory out of the way
sudo mkdir /home    # Create a directory to mount the new volume
sudo umount /dev/sdb    # Unmount the new volume...
sudo mount /dev/sdb /home    # ...and then mount it as /home
ls -lah /home/opc/		# Yes! My user dir is there
df -h				# /home is on /dev/svdb

At this point I’ve checked that /home is mounted on my new volume and that it contains my user account data. So I logged out and then logged in again just to check my account was found.

ssh [email protected]
[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Oh dear! I’ve just locked myself out of my account.

What went wrong?

First, some sloppy trade-craft. If you’re going to test your account is still available, don’t log out first! If your account is broken and your SSH key is lost, you can still use your existing session to recover. In the steps above, I’ve kept a copy of the original /home directory in /home.bak so I should be able to recover – so long as I have a session to work with. In this case recovering the files was easy. Getting into the system do do the recovery was extremely difficult!

Second, why was my SSH key rejected when my files were all present in the new /home directory? There are two important clues here. First, I was working on an Oracle Linux 8 system which has SELinux installed by default and running in enforcing mode. Second, I didn’t use cp - a to copy my files as noted above. I used rsync -a.

sudo rsync -a /home/ /home_new/

rsync is intended for copying files from one machine to another. However, you can use it to copy files between locations on the same machine. It’s often preferable as it can recover if it’s interrupted. I’ve been told that it can be faster but your mileage may vary.

Be aware though that rsync -a is not the same as cp -a, specifically with respect to how it handles SELinux file context.

How to copy files on SELinux protected systems

SELinux associates a context with each file. You can view the context attributes with the ls --context or -Z switch:

[opc@selinux-demo ~]$ ls -Z /home/opc/.ssh/
system_u:object_r:ssh_home_t:s0 authorized_keys

The format is <user>:<role>:<type>:<level>. So my authorized_keys file is labelled with type ssh_home_t for the system_u user. The ssh_home_t context type is marks a file as belonging to a user’s SSH configuration.

When we copy the file with cp -a, the context is preserved:

[opc@selinux-demo ~]$ sudo cp -a /home /home_cp-a
[opc@selinux-demo ~]$ ls -Z /home_cp-a/opc/.ssh/
system_u:object_r:ssh_home_t:s0 authorized_keys

When we use rsync -a however, the SELinux context is lost:

[opc@selinux-demo ~]$ sudo rsync -a /home/ /home_rsync-a
[opc@selinux-demo ~]$ ls -Z /home_rsync-a/opc/.ssh/
unconfined_u:object_r:default_t:s0 authorized_keys

If SELinux is in enforcing mode, it will deny SSH from accessing the file. If SELinux is in permissive mode, it will allow the SSH login but raise an alert which can be viewed using the sealert or audit2allow commands:

$ sudo audit2allow -w -a
type=AVC msg=audit(1548909218.552:1037): avc:  denied  { read } for  pid=13996 comm="sshd" name="authorized_keys" dev="dm-0" ino=4663556 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:admin_home_t:s0 tclass=file
        Was caused by:
                Missing type enforcement (TE) allow rule.

Do I have SELinux?

SELinux can be installed on many (all?) Linux distros but be aware of distros that have it installed and enforcing by default:

• Red Hat Enterprise Linux (RHEL)
• Fedora
• CentOS
• Oracle Linux

I’ve found it’s installed but running in permissive mode on the default Amazon Linux image provided by AWS. That means that it will report policy violations but not block. Many distros such as Debian and Ubuntu do not have it installed by default but it can be added so best to check if you’re not sure.