tier1.jp

Debian read only root filesystem on LVM-over-LUKS (2nd.)

STATUS: Frozen (2019-11-17)

No more updates for this guide.

Please refer to the TOC page.

Now we have "almost plain non-ro filesystem Debian".

In this section, we make it "Read-Only filesystem Debian".

Attention!

In this "read-only filesystem" is NOT complete read only filesystem such as live CD ones (with unionfs, etc.)

We let /var, /home are mounted as ordinal "read-write" mode.

Rev4

Add fstab failure related comments.

Add buster apparmor cache dir.

First boot to tweak filesystem

Boot with the encryption password.

  1. "Please unlock disk sdX_crypt": (enter the passphrase of LUKS.)
  2. Login as root then enter rescue.target.
root# systemctl isolate rescue.target
(follow the message and re-login)
  1. During this section, never abort unless explicit reboot written.

Check /var/mail owner and permission

/var/mail seems not to be configured right if you separate it.

root# ls -ld /var/mail
drwxrwxr-x 3 root root ... (if this is shown, then,)
root# chown root:mail /var/mail
root# chmod 2775 /var/mail
root# ls -ld /var/mail
drwxrwsr-x 3 root mail (this is correct.)

Setup tmpfs /tmp and APT

We use strict volatile /tmp and set APT for it.

root# rm /tmp/* -fr # Be careful!
root# cp /etc/fstab . # making a backup is always good idea.
root# nano /etc/fstab # add a line below
tmpfs /tmp tmpfs noatime,nodev,nosuid,noexec 0 0
root# findmnt --verify # check fstab
root# mount /tmp
root# findmnt # confirm /tmp is tmpfs now, and its mount options.

APT assumes executable /tmp; hence we need to provide evasion for APT.

root# mkdir -p /srv/apt/temp
root# nano /etc/apt/apt.conf.d/01-tempdir
APT::ExtractTemplates::TempDir "/srv/apt/temp";
DPkg {
    Pre-Invoke  { "mount -t tmpfs tmpfs /srv/apt/temp" };
    Post-Invoke { "umount /srv/apt/temp" };
};

Commit interval shifting (optional)

Commit interval shifting and data=writeback seems to provide very slight I/O performance improvement.

Add commit=N and data=writeback options if you want.

Basically, stay on 5.

Use small, different prime numbers if you change those commit values.

root# nano /etc/fstab # and edit options
root# findmnt --verify # check fstab
MOUNTPOINT ext4 commit sample values
/ NEVER CHANGE data OPTION; it fails.
/home data=writeback # NEVER INCREASE COMMIT VALUE
/var/cache commit=17,data=writeback
/var/mail commit=11
/var/spool commit=7
/var/tmp commit=23,data=writeback

What "man ext4" says about "data=writeback"

It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.

So, it is not so much dangerous to apply it on /home, provided you stay on commit=5 (initial value.)

Warning

Long commit interval could be potential risk of data loss on power failure, etc.

In case you made mistake on /etc/fstab

People make mistakes. How to recover matters.

There are 2 basic ways even for LVM-over-LUKS system.

  1. At Boot splash, hit "e" and edit "ro" into "rescue", then hit F10.
    • In many cases it provides isolated rescue terminal.
    • If it does not help you, try "emergency" and mount/remount on your demand.
    • (buster) "mount -n --options-mode ignore -o remount,rw /" may help.
  2. Use Installer ISO image, boot, "advanced option", then "rescue mode".
    • In case you made mistake around "/" and cannot remount as "rw".
    • Boot from installer env using target system; /target is your /.
    • Edit /target/etc/fstab, then reboot.

You should at least use both method once.

SSD/NVMe TRIM (discard) over LVM over dm-crypt

root# nano /etc/crypttab # add option "discard"
yourdevice_crypt UUID=~ none luks,discard

root# nano /etc/lvm/lvm.conf # find (Ctrl-W) and change the value.
issue_discards = 1

root# update-initramfs -u # NEVER FORGET THIS!

Caution!

Every time you edit those settings, you must do update-initramfs -u.

NEVER FORGET /boot IS ALSO READ-ONLY.

We need mount -o remount,rw /boot, too.

Second boot to check

At first, let us check some settings above work.

root# reboot -n

Note

It boot up as emergency mode if there are any mistake. In that case, follow the information displayed and fix them.

Confirm mount points and their options

root# findmnt # confirm /tmp is tmpfs

Confirm TRIM

Note

Do this TRIM manually sometimes (per month or after upgrades.)

root# fstrim -av # check all mountpoints' TRIM status.
/boot: X MiB (xxx bytes) trimmed
....

Making Read-Only filesystem

Note that you'll need to mount -o remount,rw / every time you do something after this settings.

debian login: root
Password: (enter)
root# mount -o remount,rw / # and /boot
root# (do something)
root# mount -o remount,ro /

apt can be done without those explicit remount commands, since we setup APT remount hooks.

Prepare LVs for AppArmor and /media

When we make / read-only, some features needs to be patched.

  1. AppArmor cache directory under /etc/apparmor.d/cache
  2. /media for (GUI) USB device convenience by udisks2.

buster apparmor

buster uses /var/cache/apparmor; no need to prepare the above.

These two are least; some apps needs some more "writable area" outside "/var" and/or "/srv", unfortunately.

What we need is simple; some separated mountpoints; amounts are not so much required.

First, prepare some small LVs for AppArmor cache and /media.

root# systemctl isolate rescue.target # and re-login
root# lvcreate --name /dev/VG/aacache --size 512M /dev/VG
  Logical volume "aacache" created.
root# lvcreate --name /dev/VG/media -size 128M /dev/VG
  Logical volume "media" created.
root# mkfs.ext4 /dev/mapper/VG-aacache # do it one-by-one.
root# mkfs.ext4 /dev/mapper/VG-media # NEVER do VG-{aacache,media} globing.

Second, move/clean-up contents of target mountpoint.

root# mount /dev/VG/media /mnt
root# mv /media/* /mnt
root# umount /mnt

Then, mount them.

root# nano /etc/fstab # add those mountpoints
/dev/mapper-VG-aacache /etc/apparmor.d/cache ext4 noatime,nodev,nosuid,noexec 0 2
/dev/mapper-VG-media /media ext4 noatime,nodev,nosuid,noexec 0 2
root# findmnt --verify
root# mkdir -p /etc/apparmor.d/cache # since we don't have this yet.
root# mount /etc/apparmor.d/cache
root# mount /media

Other mountpoint issue and workaround

If you need some extra because of read-only-root "/", basically what you need to do is same.

  1. Isolate into rescue.target.
  2. lvcreate and mkfs.ext4 it.
  3. clean-up/move the mount point content.
  4. Edit /etc/fstab and mount them.

Note

This breaks FHS rules a bit, and Debian installer does not allow to separate /media.

That's why we are doing after the installation.

APT remount tweaks

Even in Read-Only file system, APT can work without manual remounts, with this setting,

/boot/efi depends on the installation. If you have it (findmnt), then add those lines.

root# nano /etc/apt/apt.conf.d/01-remount
DPkg {
    Pre-Invoke  {
        "mount -o remount,rw /";
        "mount -o remount,rw /usr";
        "mount -o remount,rw /usr/share";
        "mount -o remount,rw /boot";
        "mount -o remount,rw /boot/efi";
    };
    Post-Invoke {
        "mount -o remount,ro /";
        "mount -o remount,ro /usr";
        "mount -o remount,ro /usr/share";
        "mount -o remount,ro /boot";
        "mount -o remount,ro /boot/efi";
    };
};

We assume UEFI system, that's why there is /boot/efi.

Note

APT sometimes fails to remount /usr especially.

Simple workaround for it is "prepare a scheduled maintenance and reboot time."

Make Actual read-only system

root# nano /etc/fstab # add "ro" mount options for,
/dev/mapper/VG-rootfs /            ext4 ro,...
...                   /usr         ext4 ro,...
...                   /usr/share   ext4 ro,...
...                   /boot        ext4 ro,,...
...                   /boot/efi    vfat ro,...
root# findmnt --verify

Then confirm them by,

root# mount -o remount,ro /
root# mount -o remount,ro /boot
root# mount -o remount,ro /boot/efi
root# mount -o remount,ro /usr
root# mount -o remount,ro /usr/share
root# findmnt # are those mount points now "ro"?

If it seems okay, reboot.

Keep proceeding "Next", please.

published: MODIFIED: