tier1.jp

Removable and isolated Scan/Rescue Debian Device for emergency

Debian's installer ISO has rescue mode. We can use it to fix some even on LVM-over-LUKS system.

There are also other (GNU/Linux based) rescue and/or scanning ISO images.

However,

  1. Every time we need to setup some common configurations such as keymap.
  2. It has limited commands and sometimes we need some more commands/packages, such as clamscan, chkrootkit.
  3. If you want more tools they might not sufficient for your cases.

So, let us create our own external Debian device (we call this "edisk") to do those stuff.

Rev 2

Tweaks around noatime and/or relatime, network setting on installation, locale, and some minor expressions.

Introduction

Typically, it is a Debian bootable external USB SSD/NVMe for,

  1. Rescue broken fstab system and you cannot remount "/" as "rw".
  2. Scan (compromised) machine without booting it with its filesystem.

Warning

Do not attempt and/or expect fixing broken LVM-over-LUKS. It is very complicated and you cannot be sure whether it is fixed or not.

Just backup important data in multiple storage devices. We do not recommend backup LUKS header and attempt to restore.

Consider a situation like LUKS header broken; it could also has damaged LVM metadata, etc.

Again, do backup periodically in external devices.

What this article covers

This article,

  1. does not show detail steps about creating edisk itself.
  2. shows how to mount the target filesystem with edisk, including LVM-over-LUKS.
  3. shows an offline (not booting by the target filesystem) malware scanning.
  4. shows a pitfall about upgrading your edisk.

Prerequisites

  • An external USB 3.1 SSD/NVMe (recommended).
    • Never use USB 2.0 devices; very slow and frustrating for this purpose.
    • External HDDs are slow for this usage but useful for emergency storage.
    • Avoid USB memory device (non-SSD, simple flash memory); they are quite slow.
    • SSD/NVMe (and its external USB 3.1 cases) not so expensive now a days.
  • A machine without any storage device (to avoid /boot confusion).
    • You may remove those devices by BIOS feature (disabling them as SATA/NVMe ones).
    • Otherwise you should remove storage devices physically.
    • A Machine with removal bay without internal fixed storage is recommended.

Warning

Do not create and/or update edisk on your working Debian machine with its storage devices are online.

It may cause GRUB, kernel, initramfs confusions (and potential system information leakage).

Installing Debian buster on an external device

It's simple. All we have to do is just to install Debian on an external (USB) device and setup some nice tools for scanning, recovery, and more.

  1. Ordinal external USB 3.1 SSDs are almost same with SATA SSDs.
    • Provided you connect them directly on a USB 3.1 port on your motherboard.
      • Avoid connecting via USB HUBs/cables, etc.
    • Performance is relatively slow compared with native SATA SSDs.
      • Add noatime or relatime is recommended.
    • Debian buster installer detect it as an ordinal /dev/sdX block device.
    • You can use Debian stretch, but should be careful with the differences.
      • Especially cryptsetup version (LUKS1 and LUKS2, its commands, etc).
  2. Use standard buster netinst ISO, expert mode.
    • Partition table and UEFI could be problematic if you have various old/new machines.
      • Partition table for edisk: GPT or DOS, etc.
      • System partition: UEFI system partition or BIOS reserved partition or none.
    • Avoid using NLS support.
      • Stick with locale "en_US.UTF-8" (default) or "C".
      • Only set keymap (and timezone).
    • No normal user (root console only) or a normal user with sudo.
      • sudoers does not matter (ALL permission is fine) for edisk.
  3. Install the system.
    • Create workshop mount point: say /mnt/target.
    • To make it simple, just create one partition and format it with ext4.
    • No network setting; add DHCP and/or static interface after the installation.
    • No swap basically.
    • It is okay to use LVM-over-LUKS pattern here.
      • tier1.jp do it in that way, and make them work as a temporary recovering (encrypted) storage device, too.
      • In that case, keep in mind that LVM related names be unique on your system.

swap partition on edisk

It would be VERY slow performance with small DRAM amount machine.

It could also be potential data leakage through edisk's swap partition.

So, in case you need swap, keep an empty partition and create swap on demand.

Remember flushing the swap just before rebooting (swapoff and zero filling for example).

Setup noauto network

edisk should not connect to any network automatically to avoid extra risk on emergency situations.

We installed edisk without network settings; there should be no active network.

root@edisk# ip address
(shows no active IP address for your NIC)

You should bring up the edisk NIC manually on your demand (upgrading and virus pattern updating, etc.).

root@edisk# nano /etc/network/interfaces # add only DHCP line.
iface enpX inet dhcp
root@edisk# ifup enpX # on your demand

Warning

Keep "automatic" packages least and do not install unattended-upgrades.

Install some additional packages

  • cryptsetup-{bin,run} and lvm2 for LUKS and LVM handling.
    • Expert mode installation without LVM and/or LUKS does not install them.
  • chkrootkit and clamav for offline malware scanning.
    • Especially chkrootkit should be used from edisk.
      • Potential compromised system can bypass its scanning.
  • and some more if you like (say filesystem related tools, and editors).
root@edisk# apt install cryptsetup-{bin,run} lvm2 chkrootkit clamav
root@edisk# dpkg-reconfigure clamav-freshclam # or consider manual updates.

Handling the target filesystem without booting it

After installation, check your edisk can be bootable.

  1. Manually override boot priority/device on BIOS screen.
  2. Set BIOS boot priority to be boot from external SSD/NVMe (edisk).

How to mount the target filesystem depends on your purpose.

Case 1: When you want to edit some files in "ro" root filesystem

Most GNU/Linux system boot with "ro root filesystem" when something important during boot sequence went wrong.

In that cases, it probably cannot remount it "rw" with that system.

This is typically "a broken /etc/fstab" case; your system can boot but cannot remount "/" as "rw", hence your cannot edit the broken /etc/fstab.

Tip

Every time you modify /etc/fstab run findmnt -x to check whether it works or not.

First, installed Debian system and/or Debian installer ISO would help.

  1. At Boot splash, hit "e" and edit "ro" into "rescue", then hit F10.
    • In many cases it provides isolated rescue terminal.
    • If it does not help you, try "emergency" and mount/remount on your demand.
    • "mount -n --options-mode ignore -o remount,rw /" may help (buster and later; stretch does not support this).
  2. Use Installer ISO image; "advanced option" then "rescue mode".
    • In case you made mistake around "/" and cannot remount as "rw".
    • Boot from installer env using target system; /target is your /.
    • Edit /target/etc/fstab, then reboot.

If you need to use edisk,

root@edisk# mount /dev/TARGET /mnt/target # for each mount points.
root@edisk# nano /mnt/target/etc/fstab # for example.

Note

Debian installer sets /etc/fstab root entry with "errors=remount-ro" option.

You should not remove that.

However it makes things messy when something went wrong and you cannot remount root file system "rw" even in emergency mode (hence cannot rewrite /etc/fstab).

Case 2. Scanning the target file system without booting the target

Suppose you have a suspicious malware infected Debian system.

You cannot trust the system, including clamscan, chkrootkit, everything.

Especially rootkit scanning, we must not boot the target, since we cannot trust the target.

Our choice would be,

  1. Clean install and restore from intact backup (we recommend this).
    • Backup data for recovery must be scanned before deploying them.
    • Backup data for recovery must be in some external and/or removable devices.
  2. Try offline system scanning.
    • If your system is too complicated to reinstall, try this first.
    • edisk might help this.

This article aims #2 scenario.

Boot up from edisk

  1. Plug your edisk on suspicious system and boot up from it.
    • BIOS boot override would be required.
    • Some M/B does not recognize the external USB devices sometimes; workaround is simply rebooting.
    • If you have doubt on the BIOS, you should consider CMOS clear too.
  2. After boot up,
    • login as root.
    • check devices and partitions.

After that, check your target system devices and distinguish them from edisk device.

root@edisk# lsblk # shows hardware devices and partitions.
root@edisk# blkid # shows some more detail, such as LUKS partition.

Mounting the target system's partition

Ordinal (ext4) partitions are simple.

root@edisk# mount -o ro /dev/sdX /mnt/target # for each mount points.
(or you can only mount specific target such as LVM LV contains``/home``.)

Warning

DO NOT chroot into the target system.

Just mount the target filesystem for your purpose; "rw" when you want to fix some file, "ro" when you want scan it.

Caution!

NEVER USE TARGET "/etc/fstab" TO MOUNT THEM.

MOUNT THEM ONE BY ONE MANUALLY.

If the target machine is LVM-over-LUKS system

!DANGER!

BE EXTREME CAUTIOUS.

NEVER USE "cryptsetup erase" COMMAND.

You should do some extra commands to mount filesystem on LVM-over-LUKS.

root@edisk# cryptsetup open /dev/sdX crypt # "crypt" is temporary name.

Debian buster based edisk provides "cryptsetup open" command and automatically activates detected LVM LVs. lvscan would show ACTIVE LVM LVs after you open the LUKS container.

root@edisk# pvscan # shows the target LVM PV information.
root@edisk# vgscan # shows the target LVM VG information.
root@edisk# lvscan # shows the target LVM LV information (important).
(you can tell the LVM path on this output; assume it /dev/VG/LVX)

If LVM LVs are not ACTIVE (or when you use stretch based edisk),

root@edisk# lvchange --available y /dev/VG/LVX # if LVs are not ACTIVE.

Then, mount the target filesystem.

root@edisk# mount -o ro /dev/VG/LVX /mnt/target # and continue for every LVs.

stretch case

cryptsetup version differs. LUKS1 (not LUKS2), no "open" command.

You should "cryptsetup luksOpen" and must "lvchange --available y" for each LVs with stretch based edisk.

Scanning the target system offline (without booting it)

After mounting all the target mountpoints as ro under /mnt/target,

root@edisk# findmnt # check properly mounted.
root@edisk# chkrootkit -r /mnt/target | more # confirm every entry.
root@edisk# clamscan -i -r /mnt/target # assuming clamav updated.

Updating edisk with care

You must use "a machine with no storage device" for updating edisk.

  • If you use ordinal Debian bootable machine with separated /boot, the external Debian detect that and mount as its /boot.
  • Physically detach storage devices or disable them by BIOS.
root@edisk# lsblk # confirm ONLY edisk is shown.
root@edisk# ip address # would show no network connection.
root@edisk# ifup enpX # obtain network,
root@edisk# apt update && apt upgrade
root@edisk# systemctl restart clamav-freshclam # if you installed clamav.
root@edisk# sync && shutdown -h now

Attention!

AGAIN: NEVER USE A MACHINE WITH ITS STORAGE DEVICE INSTALLED.

Summary

  • You can/should have your own edisk.
    • It helps when you want do something without booting the target.
    • So, it works as a more secure scanning device.
    • You should use USB 3.1 external SATA SSD/NVMe (for performance).
  • Updating your edisk should be done on a machine WITHOUT storage.
    • Or you would have trouble around /boot.
    • Disable SATA/NVMe by BIOS or physically detach them.
    • Removable bay machine is useful (not only for this edisk).
  • You should mount the target system "ro" and not chroot into it.
    • Unless you explicitly want rewrite some files on the target system.
    • Even that case, you should not chroot.

Thank you for your reading. Have a nice day.

published: MODIFIED:
UP