The Empire Systems

Moving to heywoodlh.io

2023-08-22T00:00:00+00:00

I’m moving my blog to https://heywoodlh.io. I’m not going to keep posting here, but I will keep this domain alive and will try to keep all the posts up and in the state they currently are in.

Aside from this post, I’m keeping all of my posts at the new domain as well. And I will be refreshing the look of the site at my new domain as well.

I chose the-empire.systems as a domain name because I couldn’t think of anything better. Switching to heywoodlh.io makes a lot more sense as my handle everywhere is heywoodlh.

Hope to see people going there, but if not, I hope you’ve enjoyed reading at https://the-empire.systems so far. :)

Making the iPad Suck Less (for sysadmin, development, etc.)

2023-07-31T00:00:00+00:00

At Apple’s 2023 WWDC event I found myself having a metaphorical, angry seizure because for yet another year my iPad Pro was still going to be a giant iPhone – per the lack of features announced for iPadOS at the event.

I do a lot of Linux systems administration and development. The perceived shortcomings I project onto the iPad are likely very specific to me, so while the iPad might work extremely well for some, its featureset is limited for what I need and want.

This post is partly an opinion piece on the current state of the iPad for power users, but also an attempt to document some of the things I have been doing to make my iPad Pro actually useful to me.

P.S. I wrote this post from my iPad in Vim

Why do I own an iPad?

I bought an M1 iPad Pro in 2021 thinking it was a fairly safe bet that the iPad Pro would eventually become a viable option for me as a laptop replacement. My reasons in 2021 for thinking it would become a laptop replacement were the following:

Apple had recently announced iPadOS specifically for the iPad – separating it from iOS
The iPad Pro now shares the same processor as the Macs

Fast-forward to Apple’s 2023 WWDC event where Apple announced almost no meaningful iPadOS updates that would help me better fulfill my goal of being able to use it for my laptop workflows. I’m not the ideal iPad user in Apple’s approach because I use Linux often (for workstations and servers) and I want to do development on languages that are not Swift.

Here are the things I like about the iPad:

I really like the form factor, combined with the Apple Magic keyboard case
The battery life is good
Compared to a laptop, the time I can get to doing things after opening the lid is much faster

Here are the downsides for me:

Local development just isn’t possible or limited
iPadOS lacks extensibility/flexibility compared to other desktop operating systems (MacOS, Linux, Windows, etc.)
Unfairly, on iOS/iPadOS Safari is the only browser that supports extensions
Ultimately, iPadOS still feels pretty much just like iOS with some tweaks for iPad

So, let’s get into how I try to adapt to iPadOS to lessen the impact of these perceived shortcomings.

My assumptions

My assumptions for anyone finding this article useful are that you have an iPad Pro with an external keyboard of some sort. I prefer Apple’s Magic Keyboard case (despite it being ridiculously expensive).

Keyboard-driven workflows

On Linux and MacOS, I use a tiling window manager to automatically tile my opened applications. Pivotal to this workflow are keyboard shortcuts to make navigating the tiling windows and virtual workspaces possible.

Use Spotlight for switching app focus/workspace switching

I recently observed a coworker using Spotlight on MacOS to not only launch applications but also to switch focus to already open application windows. This workflow not only works on MacOS, but also on iPadOS. This may seem “duh” to everyone, but for me, it was a revelation.

Start using Spotlight (invoked with Cmd + Space) not only to launch apps but to switch between already opened instances of apps.

Use built-in app keyboard shortcuts

I have found that many apps on the iPad have a robust amount of keyboard shortcuts built into them. Whilst in any app on iPadOS, if you hold Cmd, an overlay will appear to show you all of the keyboard shortcuts built into the app.

Start using built-in keyboard shortcut for the apps you care about. I’ve found that most apps have the same keyboard shortcuts on the iPad as they do for their desktop apps. For example, the keyboard shortcuts for Slack in iPadOS mostly translate to comparable shortcuts on Slack in MacOS or Linux.

And again, if you’re in doubt on your iPad as to what the shortcuts are in the app you’re in, just hold Cmd.

Use Stage Manager

I won’t cover how to use Stage Manager, but it’s basically a more complex window manager for iPadOS and MacOS. Enabling Stage Manager allows a user to have rules for windows and how they open.

Using Stage Manager can further enable keyboard-driven productivity.

Safari

I don’t use other browsers on my iPad because Apple unfairly restricts them in ways that Safari is not restricted.

There are a couple of things I do for being more keyboard-driven with Safari:

Use the built-in keyboard shortcuts (in particular, the Cmd+L shortcut)
Use the Vimlike Safari extension: https://apps.apple.com/us/app/vimlike/id1584519802

These tweaks make Safari a much more pleasant experience for me with being keyboard driven.

Development tooling

Apple Shortcuts

Apple Shortcuts supports a surprising amount of functionality. I would highly recommend using Apple Shortcuts to extend the programmability of your iPad.

As an example, check out this post where I use Apple Shortcuts on iOS to turn on/off a Paperspace VM for gaming: https://the-empire.systems/paperspace-cloud-gaming-ios

iSH

https://apps.apple.com/us/app/ish-shell/id1436902243

I use iSH to drastically extend the functionality of my iPad. iSH is an emulated x86 32-bit Alpine Linux environment that runs on iOS and iPadOS. Being able to run command line tools like ssh, git, vim or dig directly on my iPad has made my iPad far more functional.

I have a GitHub repository containing my iSH setup on my iPad: https://github.com/heywoodlh/ish-configs

Tailscale

Tailscale is a user-friendly VPN service based on Wireguard. Although it is not itself specifically a tool for development, it enables me to remain always-connected to the infrastructure I use to support my iPad.

Seemingly, Tailscale does not have the ability to remain always on or use iOS’ on-demand VPN featureset: https://github.com/tailscale/tailscale/issues/1534

The lack of always-on/on-demand VPN hasn’t been an issue with me for Tailscale as I use an automation in Apple Shortcuts to turn on Tailscale when I’m in an app that would need to connect to servers in my Tailnet. For example, with iSH – what I use to primarily SSH into my servers – I have an Apple Shortcuts automation that will connect to Tailscale when I open the iSH app.

Pomerium

Pomerium is also not a development tool – it is an open source, identity-aware proxy. Similar to Tailscale, my Pomerium instance allows me to remotely access my personal infrastructure via HTTPS, and as a result, Pomerium enables me to be more productive in my development workflows on my iPad.

For example, here’s my self-hosted Coder instance made available via my Pomerium instance:

Dedicated Linux server

I have a small VPS running on Vultr that I can SSH into from my iPad that has all the tooling I need that I can’t necessarily run on my iPad directly.

All of the tooling I am using on that server is configured for me via the cross-platform (MacOS, Linux) package manager Nix. My Nix-managed tooling that I use on the server for my iPad is available here: https://github.com/heywoodlh/flakes

Here’s a screenshot of me SSH-ed into my VM for my iPad, running neofetch:

Conclusion

Hopefully these ideas are helpful to any sysadmin, developer or other power user who is also unhappy with the current out-of-the-box state of iPadOS. With the tweaks I’ve listed above, I find myself far more productive on my iPad.

But, maybe someday Apple will make the iPad more than just an iPhone that can work with a keyboard.

Tiny Vim Install

2023-06-18T00:00:00+00:00

Some background

A friend recently pointed me to this repository as a place to grab a self-contained vim binary: https://github.com/dtschan/vim-static

The most recent commit is about 4 years old and the build.sh script uses an old version of Vim. I wanted to see if I could improve on this process and build Vim on an automated, regular cadence and publish the binaries.

So here’s my solution to that: https://github.com/heywoodlh/vim-builds

My build process deviates slightly from the original build.sh script but uses mostly the same process. The most notable deviation in my build is that I’m building whatever version of Vim is available at the main repository at the time of the build, and I’m using GitHub Actions + Docker’s buildx plugin to provide an automated build that produces multiarch (x86_64, and ARM64) Linux binaries.

Installation

Find the latest release and grab the link for the binary of your desired architecture here: https://github.com/heywoodlh/vim-builds/releases

Assuming I was grabbing the latest binary for the 9.0 tag for the x86_64 architecture on my Git repo, these are all the commands necessary for a fully functional Vim installation:

mkdir -p ~/bin
curl -L 'https://github.com/heywoodlh/vim-builds/releases/download/9.0/vim-x86_64' -o ~/bin/vim
chmod +x ~/bin/vim
PATH="~/bin:$PATH" # Add this to your ~/.bashrc or other shell config file to make it persistent

To disable the warning about not being able to read defaults.vim, create an empty ~/.vimrc:

touch ~/.vimrc

Why?

This static Vim binary is tiny and should work out of the box on all Linux distributions. If you aren’t aware, many binaries compiled on a Glibc-centered system (Ubuntu, Arch Linux, etc.) don’t work on a Musl-based system (Alpine Linux). This makes these Vim binaries a potentially ideal candidate for installations where portability is desired (containers, ephemeral environments, etc.).

I’m a huge fan of Vim and love the idea of being able to build/host a totally package-manager-independent Vim binary.

But mostly, I wanted to see if I could do it.

Remote Backups Over SSH With Rsnapshot

2023-04-01T00:00:00+00:00

During a discussion I had with a former coworker at my previous job, he introduced me to Rsnapshot. Rsnapshot is a backup program based on rsync, which allows for simple but effective remote backups over SSH.

I’ve built out my home lab to now support one server, my backup server, SSH-ing into my other servers and backing up the data I care about. This will be an overview on how I have this setup.

Disclaimer: I primarily use NixOS and some Arch Linux for my servers, so some NixOS configuration.nix snippets will be present

Rsnapshot server:

I have one NixOS server with Rsnapshot installed and an SSH key that I’ve generated without a passphrase at /root/.ssh/id_rsa.

At the time of writing, this is my NixOS configuration snippet for rsnapshot:

  services.rsnapshot = {
    enable = true;
    enableManualRsnapshot = true;
    cronIntervals = {
      daily = "0 3 * * *";
      hourly = "0 * * * *";
      weekly = "0 1 * * 1";
      monthly = "0 1 1 * *";
    };
    extraConfig = ''
#Separate everything with tabs, not spaces	
#Convert spaces to tabs in vim with :%s/\s\+/\t/g
snapshot_root	/media/backups/
retain	hourly	24
retain	daily	7
retain	weekly	4
retain	monthly	6

backup	root@ct-1.wireguard:/etc/	ct-1/
    '';
  };

On NixOS, this installs and configures rsnapshot to run on an hourly, daily, weekly and monthly basis. The last line in my extraConfig for rsnapshot shows the actual backup that is going to happen over SSH:

backup	root@ct-1.wireguard:/etc/	ct-1/

This will backup the /etc/ folder on the remote server ct-1.wireguard and place it in a folder named ct-1 – if you combine that with my snapshot_root config, the actual backup path for the /etc/ folder would be /media/backups/ct-1/etc.

If you aren’t on NixOS, the Arch Wiki has a great entry on setting up Rsnapshot:

rsnapshot - ArchWiki

Add all of the remote servers and their paths to your rsnapshot configuration.

Target servers:

On the target servers you only need the following setup:

SSH access, preferably as root – see my recommendations below on reducing security risk
rsync installed

The root user is easiest to use on the remote server over SSH. However, logging in as root is a security risk, so there are a couple of precautions I would take:

Configure sshd_config to only allow root login using SSH keys, prohibiting password authentication, with the following configuration: PermitRootLogin prohibit-password
In setting up /root/.ssh/authorized_keys, restrict the key to only be used from a specific IP address: StackExchange - How to restrict an SSH key to certain IP addresses?
Use a VPN and limit SSH access between your rsnapshot server over the VPN. I like Wireguard for VPN stuff: https://www.wireguard.com/

Profit:

Once this is setup as described, you now have a remote backup solution that works over SSH.

Sync Skyrim Mods Between Devices (with Syncthing)

2023-01-23T00:00:00+00:00

This will be a quick run-down on how I’m syncing mods with Syncthing between two of my devices that have Skyrim Special Edition installed. The two devices are my gaming PC running Windows 10 and my Steam Deck running NixOS. However, this should work with any two devices that can run Syncthing.

Additionally, I suspect that this method should work with other games, I’m just using Skyrim as an example.

The reason for this is pretty simple: I have a Steam Deck and I’d like to sync my Skyrim mods with it – but I already have a modded build on a Windows machine. Additionally, my Steam Deck is running NixOS (Linux) and it’s not currently very convenient to use Vortex (nexusmods.com’s mod manager) on Linux. With this setup I can install mods on my Windows machine and they should sync to my Steam Deck.

Prerequisites:

I’m going to assume that one of your machines already has a modded build of Skyrim, I won’t cover how to actually install mods (it’s trivial on Windows with nexusmods.com and Vortex, Nexus’ Mod Manager).

Both machines should have Skyrim installed – I would suggest starting from a fresh install on the machine that doesn’t yet have the mods/build you want.

Installing Syncthing

I’m not going to provide a detailed walkthrough on how to install Syncthing, but I’ll provide a basic outline:

On Windows (using Chocolatey):

Install Chocolatey: https://chocolatey.org/install

Run the following command as Administrator (search for the Powershell app, right click Powershell, Run as Administrator):

choco install -y syncthing

Note: Chocolatey will install the syncthing.exe executable in C:\ProgramData\chocolatey\bin\syncthing.exe

Set Syncthing to start automatically: https://docs.syncthing.net/users/autostart.html#windows

On Linux:

Use your package manager to install Syncthing.

Start the syncthing service:

sudo systemctl enable --now syncthing@${USER}.service

Alternatively, start it using just your user:

systemctl enable --now --user syncthing.service

Check out the Arch Wiki for more information on Syncthing on Linux: https://wiki.archlinux.org/title/Syncthing

Add Both Devices to Syncthing

With your devices running Syncthing, make sure to add them both so they can share data with each other via Syncthing.

Do the following:

Login to the local Syncthing instance’s web interface: http://localhost:8384
Select Actions > Show ID
On the other device, also login to Syncthing’s web interface and select “Add Remote Device”, filling out the Device ID with the ID from step 2
Back on your original device, a notification should appear in Syncthing’s web interface that your other device would like to be added – accept it

Add Skyrim’s Folders to Syncthing

On your modded machine, access the Syncthing web service with this URL: http://localhost:8384

In Syncthing’s web interface on the device with your already modded Skyrim build, you should add Skyrim Special Edition’s folder as a shared Folder. In my case, my Skyrim Special Edition folder path is at E:\SteamLibrary\steamapps\common\Skyrim Special Edition.

While in Syncthing, you should also add a separate shared folder for the directory on your Windows machine that contains Skyrim’s loadorder.txt and plugins.txt files. By default it is at the following path on Windows:

C:\Users\$USER\AppData\Local\Skyrim Special Edition

Once your Skyrim Special Edition folder is added to Syncthing, select it, select Edit > Sharing and check the box next to the device you want to share it with.

There are three folder paths I would recommend sharing between your devices. I’m going to presume that the sources are all coming from a Windows machine, so I will use the default Windows paths:

C:\Program Files (x86)\Steam\steamapps\common\Skyrim Special Edition: Skyrim game files
C:\Users\$USER\Documents\My Games\Skyrim Special Edition: Save data and local preferences
C:\Users\$USER\AppData\Local\Skyrim Special Edition: contains loadorder.txt

On the other device, I would recommend deleting your freshly installed, unmodded Skyrim Special Edition folder (don’t uninstall Skyrim via Steam) and then go into Syncthing’s web interface and accept the shared folder from the machine with your desired build. I would set the path to the exact location where Skyrim was installed, so that way Steam thinks it is still present on your filesystem.

Really, it doesn’t matter how you do it, mostly just place the Syncthing Skyrim SE folder at a location Steam will be able to view it as present.

For the shared folder in Syncthing containing loadorder.txt, you should place that folder at the default Windows path C:\Users\$USER\AppData\Local\Skyrim Special Edition or if you are on Linux (i.e. Steam Deck) using Valve’s Proton compatibility to run Skyrim, the target path should be ~/.local/share/Steam/steamapps/compatdata/489830/pfx/drive_c/users/steamuser/AppData/Local/Skyrim Special Edition. This path might not exist if you haven’t already run Skyrim the first time, so either run Skyrim or just let Syncthing create that path.

Play!

Now with Syncthing sharing folders between devices, you will be able to have your Skyrim builds synchronize. As long as both of your devices are powered on and syncing your Skyrim builds should stay in sync.

Some Syncthing Tips

File versioning

In Syncthing’s web interface, you should edit the folders and turn on File Versioning. This will prevent files from being deleted if you have a conflict between devices.

Setup a Server

I would highly recommend setting up a device to behave as an always-on Syncthing server. Having a third device that you sync these folders to will ensure that if only one of your two original devices with Skyrim is turned on, the folders will still be synchronized.

Basically, just dedicate a separate device (it can be small, such as a Raspberry Pi – or it could just be any Windows, Mac, Linux machine that is always running) to always run Syncthing and share the folders with that device.

NixOS: Managing GNOME Keyboard Shortcuts and Settings with Home Manager

2023-01-09T00:00:00+00:00

EDIT: June 21, 2023: thanks to u/jtojnar for pointing out the issue on this post with using "@as []" for disabled dconf settings. Shortly after posting this in January, I figured out that "disabled" was one of the ways to disable a setting with GNOME – but I forgot to update this post! I have updated the config shown on this post. Here’s the Reddit post referring to this article: https://reddit.com/r/NixOS/comments/14fenpb/issue_with_declarative_gnome_keyboard_shortcuts/

EDIT: As of June 21, 2023, for reference, here is my GNOME desktop Home-Manager configuration: gnome-desktop.nix

This will be a quick snippet on how I’m using Home Manager in my NixOS configuration to manage GNOME keyboard shortcuts and other various settings.

Use Home Manager as a NixOS config module:

This config basically explains how I’m importing Home Manager (version 22.11) in my /etc/nixos/configuration.nix:

{ config, pkgs, ... }:

let
  home-manager = builtins.fetchTarball "https://github.com/nix-community/home-manager/archive/release-22.11.tar.gz";
in {
  imports = [ <home-manager/nixos> ];
...

Using Home Manager to manage GNOME:

With Home Manager imported, I can now use it’s home-manager.users.<username>.dconf.settings option. I’ll just dump my current home-manager.users.heywoodlh configuration to show what I currently have set as an example (I feel like it’s pretty self-explanatory):

  home-manager.users.heywoodlh = {
    home.stateVersion = "22.11";
    dconf.settings = {
      "org/gnome/shell" = {
        disable-user-extensions = false;
        disabled-extensions = "disabled";
        enabled-extensions = [
          "native-window-placement@gnome-shell-extensions.gcampax.github.com"
          "pop-shell@system76.com"
          "caffeine@patapon.info"
          "hidetopbar@mathieu.bidon.ca"
          "gsconnect@andyholmes.github.io"
        ];
        favorite-apps = ["firefox.desktop" "kitty.desktop"];
        had-bluetooth-devices-setup = true;
        remember-mount-password = false;
        welcome-dialog-last-shown-version = "42.4";
      };
      "org/gnome/shell/extensions/hidetopbar" = {
        enable-active-window = false;
        enable-intellihide = false; 
      };
      "org/gnome/desktop/interface" = {
        clock-show-seconds = true;
        clock-show-weekday = true;
        color-scheme = "prefer-dark";
        enable-hot-corners = false;
        font-antialiasing = "grayscale";
        font-hinting = "slight";
        gtk-theme = "Nordic";
        toolkit-accessibility = true;
      };
      "org/gnome/desktop/wm/keybindings" = {
        activate-window-menu = "disabled";
        toggle-message-tray = "disabled";
        close = ["<Super>q"];
        maximize = "disabled";
        minimize = ["<Super>comma"];
        move-to-monitor-down = "disabled";
        move-to-monitor-left = "disabled";
        move-to-monitor-right = "disabled";
        move-to-monitor-up = "disabled";
        move-to-workspace-down = "disabled";
        move-to-workspace-up = "disabled";
        toggle-maximized = ["<Super>m"]';
        unmaximize = "disabled";
      };
      "org/gnome/desktop/wm/preferences" = {
        button-layout = "close,minimize,maximize:appmenu";
        num-workspaces = 10;
      };
      "org/gnome/shell/extensions/pop-shell" = {
        focus-right = "disabled";
        tile-by-default = true;
        tile-enter = "disabled";
      };
      "org/gnome/desktop/peripherals/touchpad" = {
        tap-to-click = true;
        two-finger-scrolling-enabled = true;
      };
      "org/gnome/settings-daemon/plugins/media-keys" = {
        next = [ "<Shift><Control>n" ];
        previous = [ "<Shift><Control>p" ];
        play = [ "<Shift><Control>space" ];
        custom-keybindings = [
          "/org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom0/"
          "/org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom1/"
          "/org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom2/"
          "/org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom3/"
        ];
      };
      "org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom0" = {
        name = "kitty super";
        command = "kitty -e tmux";
        binding = "<Super>Return";
      };
      "org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom1" = {
        name = "kitty ctrl_alt";
        command = "kitty -e tmux";
        binding = "<Ctrl><Alt>t";
      };
      "org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom2" = {
        name = "rofi-rbw";
        command = "rofi-rbw --action copy";
        binding = "<Ctrl><Super>s";
      };
      "org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom3" = {
        name = "rofi launcher";
        command = "rofi -theme nord -show run -display-run 'run: '";
        binding = "<Super>space";
      };
    };
  };

Using Home Manager this way allows me to have an easy, declarative GNOME setup.

Tips for modifying dconf settings:

If you install dconf-editor you can see what options are available when you run the dconf-editor application.

One of the things that I do when I want to make a setting persist in my config, is I run the following command:

dconf dump / > old-conf.txt

Then I make a change in my GNOME settings and run dconf dump again:

dconf dump / > new-conf.txt

You can now run diff old-conf.txt new-conf.txt to figure out what options in your GNOME settings changed and then codify it in your configuration.nix with the home-manager.users.<username>.dconf.settings attribute.

Installing NixOS on the Steam Deck

2023-01-03T00:00:00+00:00

January 14, 2023 update: added changes to make GNOME touch friendly (using Home Manager to manage GNOME settings) with this commit: https://github.com/heywoodlh/nixos-configs/commit/e20277093d53390e5a5cb0bb516f80022ff0be37

Why not keep SteamOS?

I have a Steam Deck and I really enjoy it. SteamOS on the Steam Deck is based on Arch Linux and I’m very comfortable with Arch as a long-time user. However, there are two issues that I have with the Steam Deck’s base OS:

It wipes out custom changes I’ve made between updates, specifically, I have to reinstall and re-enable Wireguard between SteamOS updates
I’m not a fan of KDE at all – I definitely prefer GNOME

Wireguard is especially important to me on my Steam Deck because I use Wireguard and the open source Moonlight project to remotely connect to my more powerful gaming PC using Nvidia’s proprietary gamestream protocol to do the majority of my gaming.

I’ve been using NixOS on my Linux workstations and Nix on Darwin on my Macs and I enjoy having a declarative/reproducible workstation. So I did some initial research and found that the NixOS community is actively developing stuff for the Steam Deck here: Jovian-NixOS

Installing NixOS on the Steam Deck:

For those uninterested in the rest of the how-to, here’s my current NixOS config on the Steam Deck: https://github.com/heywoodlh/nixos-configs/blob/master/workstation/steam-deck.nix

Requirements/Assumptions:

My desired setup uses GNOME as the desktop environment and installs NixOS as the exclusive operating system on the Steam Deck. If you want anything outside of these constraints, modify this guide to your needs.

The main requirements are the following:

Absolutely required: an external monitor and the necessary dongles/cables to connect the Steam Deck to it (required as of this Github Issue: Jovian-Experiments/Jovian-NixOS/issues/39)
An external keyboard and mouse and necessary dongles/cables to connect to the Steam Deck

Creating an Installer ISO and to Bootable USB:

Assuming you have the Nix package manager installed on a Linux or MacOS machine and git installed on your machine, you should be able to create a GNOME installer ISO with the following command (it takes quite a long time):

git clone https://github.com/Jovian-Experiments/Jovian-NixOS
cd Jovian-NixOS
nix-build -A isoGnome

When the command completes, it should show the resulting ISO in your filesystem.

I’ve uploaded a copy of my ISO to Google Drive, which may save some time if you download it. Feel free to use this link (I’m not malicious but you should definitely be cautious of strangers on the internet):

nixos-steam-deck-22.11-x86_64-linux.iso

Use your preferred method to flash the ISO to a bootable USB. Balena Etcher is a very user friendly option for creating bootable USB drives.

Boot the Bootable USB:

Shut down your Steam Deck. Before powering it on, connect it to an external display – this is absolutely required as of this Github issue: https://github.com/Jovian-Experiments/Jovian-NixOS/issues/39. At least for me, I couldn’t boot from USB until I had an external display connected.

To enter the boot menu, hold down the Volume- and tap the Power button with the bootable USB connected to the Steam Deck.

Install NixOS:

These steps are mostly identical to the NixOS Manual Installation Instructions, just modified for the internal storage as the target drive to install.

Run these commands as the root user in the live environment:

parted /dev/nvme0n1 -- mklabel gpt

parted /dev/nvme0n1 -- mkpart primary 512MB -8GB

parted /dev/nvme0n1 -- mkpart primary linux-swap -8GB 100%

parted /dev/nvme0n1 -- mkpart ESP fat32 1MB 512MB

parted /dev/nvme0n1 -- set 3 esp on

mkfs.ext4 -L nixos /dev/nvme0n1p1

mkswap -L swap /dev/nvme0n1p2

mkfs.fat -F 32 -n boot /dev/nvme0n1p3

mount /dev/disk/by-label/nixos /mnt

mount /dev/disk/by-label/boot /mnt/boot

swapon /dev/nvme0n1p2

nixos-generate-config --root /mnt

Before installing NixOS, you can download my current (as of writing) config for the Steam Deck which sets up GNOME and the relevant modules from the Jovian-NixOS repository:

curl -L 'https://raw.githubusercontent.com/heywoodlh/nixos-configs/ebe30a392abee0426a80158af2f97ad56975d946/workstation/steam-deck.nix' -o /mnt/etc/nixos/steam-deck.nix

You should modify the newly downloaded /mnt/etc/nixos/steam-deck.nix file – particularly lines 4 and 5 to match your username and user description. Additionally, you should modify the rest of the config files in /mnt/etc/nixos/ to work for your

Your /mnt/etc/nixos/configuration.nix can look something like this:

# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
      ./steam-deck.nix
    ];

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "nix-steam-deck"; # Define your hostname.

  # Set your time zone.
  time.timeZone = "America/Denver";

  system.stateVersion = "22.11";

}

Once your config suits your needs, install NixOS:

nixos-install

Set the root password and reboot. Once you login as root, run the following command to change the password for your normal, non-root user:

passwd <username>

Conclusion:

With this setup, by default, NixOS will boot into the handheld Steam UI. For me, it works identically to SteamOS until I use “Switch to Desktop” – which then boots into my preferred desktop environment (GNOME) and preferred OS (NixOS).

Switch back to Steam after running “Switch to Desktop”:

I’m currently working on this, but in order to switch back into Steam’s handheld Steam Deck UI after running “Switch to Desktop” I run the following command while in GNOME:

pkill -9 gnome-shell

Launch Powershell in Tmux with -NoLogo Argument

2022-11-13T00:00:00+00:00

I recently switched to Powershell as my main shell on Linux and MacOS and I couldn’t find any examples online on how to launch Powershell Core, pwsh, in tmux with the -NoLogo argument. So this is my quick solution to that.

I placed this at the beginning of my ~/.tmux.conf:

if-shell "bash -c 'echo ${SHELL} | grep -q pwsh'" "set -g default-command 'pwsh -NoLogo'"

This will run a check using BASH (because if I’m running a system with tmux installed, it will likely have bash installed as well) to see if Powershell Core is my default shell. If it is, then tmux will use pwsh -NoLogo as its default command.

Simple nfdump Setup (in Containers) for Netflow Collection and Analysis

2022-07-20T00:00:00+00:00

This post will outline my simple Netflow collection setup running with some containers with nfdump tooling I setup for this purpose.

Assumptions:

I’m going to make the following assumptions:

You’re in a Unix-like environment
Docker is the targeted container engine
Netflow binaries will be stored in /flows on your system
You already have another system that supports sending Netflow data to a remote host, such as a switch or router (I will not cover how to setup sflow on a Linux machine)

Modify the content to your needs if any of these assumptions don’t line up with your setup.

Why nfdump?

At a previous role I was introduced to nfdump and really liked it. It’s like tcpdump in the fact that it’s a command line tool and provides a user with the ability to inspect the netflow data based on filters such as time or source or destination hosts/ports – but unlike tcpdump, nfdump is used against Netflow binary files that have the captured data on your filesystem.

The repository for nfdump is here: phaag/nfdump

Setup nfcapd

Nfcapd is a daemon included with nfdump that receives Netflow data and writes it to disk for inspection with nfdump.

First, let’s create the /flows directory and make sure that UID 1000 owns it (that UID/GID is used in the container):

sudo mkdir -p /flows
sudo chown -R 1000 /flows

We can spin up an nfcapd receiver with the following docker command:

docker run -d --restart=unless-stopped --name=nfcapd -p 9995:9995/udp -v /flows:/flows heywoodlh/nfcapd:latest

Now that nfcapd is setup, you can send Netflow to UDP port 9995 on your Docker host and it should start writing the received data to binary files in the /flows directory. My container image by default will organize flows into the /flows directory by year/month/day/hour. For example, flows captured between 4:00 p.m. to 5:00 p.m. July 20th, 2022 would be in the following folder /flows/2022/07/20/16.

If you do not like the defaults I have set in my heywoodlh/nfcapd image, you can check out nfcapd’s arguments by running this command:

docker run -it --rm heywoodlh/nfcapd --help

You can then provide your desired nfcapd arguments to the docker run command. If you are going this route I can assume you are technical enough to figure out what you need so I will not cover any thing more of going outside the defaults I have set.

Accessing the Captured Netflow Data:

For actually accessing the data you need to use the nfdump tool. You can either install nfdump on your machine – or just run a Docker container I have setup for this purpose. I will cover how to use the container as that will be a bit more predictable.

Use the following docker command to recursively access ALL of the Netflow data in the /flows/2022 directory:

docker run -it --rm -v /flows:/flows heywoodlh/nfdump -R /flows/2022

As I stated earlier in this post, you can use filters with nfdump to parse out the data you actually want to see. For example, if I want to see all traffic related to destination port 80 from an IP address at 192.168.1.143, I could run this command:

docker run -it --rm -v /flows:/flows heywoodlh/nfdump -R /flows/2022 "dst port 80 and src host 192.168.1.143"

You can access nfdump’s help section with the following command:

docker run -it --rm heywoodlh/nfdump --help

If you install nfdump on your machine directly, all of the above arguments to nfdump should work the same.

The filters provided with nfdump can provide really cool pieces of information about what’s going on with your network. Using nfdump also allows you to have flexibility to build simple but effective monitoring tools around Netflow.

Additional Reading:

Github repo: phaag/nfdump

Man pages: nfcapd, nfdump

Dockerfiles: heywoodlh/nfcapd, heywoodlh/nfdump

Github Actions used to build the images: nfcapd-buildx, nfdump-buildx

A More Secure Linux Desktop

2022-07-06T00:00:00+00:00

This is going to be a guide on some tooling and workflows I use to secure my Linux desktop. As with all security tooling, nothing except for being completely disconnected from the internet can keep you perfectly secure – this guide will serve to add some layers of protection. A lot of this tooling could be made more elegant but rather than focus on making something without flaws, I’m publishing this guide to serve as a starting point.

Why?

Note: this section is all my opinion, feel free to skip this if you just want the details on the workflows/tooling

Linux has a (justified) reputation for being very secure. I think on the server-side this is absolutely true. On the desktop, I feel that there are some big gaps in functionality in regard to tooling compared to some of the security tooling available for Windows or MacOS. My experience is that people consider Linux on the desktop “more secure” but it’s really not very secure by default – it’s just a much less lucrative target for bad actors because of how little market share it has.

My thought process is this: if you were phished and a bad actor got a shell on your machine what could they access? With my security experience as a security engineer and Linux enthusiast, I don’t think I would have any problem eventually getting root on a Linux workstation because there are so few desktop protections enabled by default.

Again, I specifically mean there is a lack of tooling to protect desktop users. For example, on MacOS Little Snitch and Lulu exist to help a user monitor network connections if a non-whitelisted application tries to connect to the internet. Patrick Wardle has developed a gigantic list of tools that help users increase visibility on their machine: [Objective See’s MacOS Tools].

There are a number of tools and workflows that I have found that help close the gaps I have mentioned.

What Gaps?

I will be focusing on what I consider are simple to implement tweaks. I will not be focusing on tools like SELinux that make your system so secure to the point it’s difficult to use.

These are the security gaps that I’ll focus on:

Basic security suggestions
Application autostart monitoring
Application sandboxing
Network application firewalling/monitoring
Container security
Miscellaneous tweaks

Assumptions:

As there is a wide variety of Linux distributions, I will try to avoid distribution-specific instructions (i.e. how to install a package on one distribution) – however, I will admit I have zero interest in Red Hat based distros (RHEL, CentOS, Fedora, Rocky, AlmaLinux, etc.) so most of my suggestions have been tested by me on Arch Linux and Debian-based distributions. Apologies to any Fedora users who find my instructions to be not as compatible with their environment.

I will assume you’re using a systemd-based distribution.

I am going to assume the desktop environment in use will be GNOME.

I will also assume that the reader will implement my opinionated suggestions so I won’t try to cover all alternatives that are available to a tool/workflow that I recommend.

Basic Security:

I’m mostly going to focus on things I do that are specific to desktop systems. I would recommend using this more comprehensive guide for best hardening practices for Linux systems generally:

madaidan’s Linux Hardening Guide – an excellent and detailed document.

Disk Encryption:

This is more of a physical security suggestion so I’ll keep this brief: implementing disk encryption is a great way to secure your data on your hard drive (i.e. if somebody steals your laptop). You should encrypt your drives using full-disk encryption.

If you can’t use full-disk encryption (or have setup an unencrypted system that you don’t want to reinstall) and are on a system with a more aggressive release cycle (Arch Linux or Fedora), check out my entry in the ‘Misc’ section on systemd-homed for a decent alternative.

Use UFW:

UFW is a simple wrapper script around iptables. IPTables is a powerful, but complex Linux firewall and UFW makes it much simpler.

Install UFW on your distribution.

After UFW is installed, enable the service and activate it to block all incoming connections to your machine:

systemctl enable ufw.service
ufw enable

By default this will block all incoming connections and allow outbound connections. You can configure UFW to block outbound connections, but as that can be terribly inconvenient I will not cover doing that.

Note: Docker circumvents IPTables/UFW – I would recommend either setting up rootless Docker, Docker Desktop for Linux, Podman, or follow my instructions for using Lima for running Docker in a VM

ClamAV:

Disclaimer: I can’t stand most proprietary Antivirus products. But I think it’s very useful to know if a known malicious file is on your machine. ClamAV gets you access to a great malware database without all the other buzz-word features and spyware that comes with a lot of antivirus.

Install ClamAV and ClamTK (a graphical interface to ClamAV).

Run the following command to update the local ClamAV database on your machine:

sudo freshclam

Open ClamTK > Settings > Check the following boxes:

Scan files beginning with a dot
Scan directories recursively
Optionally: Scan files larger than 20MB

Back in settings select Scheduler and choose a time to scan your home directory daily.

If you want to be able to scan files from your file manager, I would recommend checking out the plugins:

https://github.com/dave-theunsub/clamtk#plugins

Application Autostart Monitoring:

Prerequisites:

I’ve chosen to use inotify to monitor each these so please install the inotify-tools package. Please also install notify-send so desktop notifications can be sent to alert the user.

Why?

If I were a bad actor on your machine with an unprivileged shell there are a couple of places I would place a malicious application to run on startup (for persistence): the user’s cron jobs, and the ~/.config/autostart directory and the user’s ~/.config/systemd/user directory.

User cron monitoring:

For those unfamiliar with cron jobs: cron is a tool for running jobs on a schedule. You can easily modify a cron to run a job as a particular user on a regular basis. As a bad actor (with an unprivileged shell), I could easily setup a cron to repeatedly attempt to do a malicious task such as establish a reverse shell to a server somewhere.

Create a script in ~/opt/scripts/cron-monitor.sh with the following contents:

#!/usr/bin/env bash

notify-send "Started cron monitoring script"

mkdir -p ~/.tmp/

crontab -l > ~/.tmp/orig-cron.txt

while true
do
    crontab -l > ~/.tmp/new-cron.txt

    if cmp ~/.tmp/orig-cron.txt ~/.tmp/new-cron.txt
    then
	echo 'Crontab unmodified'
    else
	crontab -l > ~/.tmp/orig-cron.txt
	notify-send 'User crontab modified: check crontab with `crontab -l`'
    fi

    sleep 5
done

Make it executable:

chmod +x ~/opt/scripts/cron-monitor.sh

Now create a file at ~/.local/share/applications/cron-monitor.desktop to execute our monitoring script with the following content:

[Desktop Entry]
Name=Crontab Monitoring Script
Exec=bash -c "${HOME}/opt/scripts/cron-monitor.sh"
Icon=org.gnome.Terminal
Type=Application
Categories=Security;

Now, symlink that application to the ~/.config/autostart directory so the application runs when you login:

ln -s ~/.local/share/applications/cron-monitor.desktop ~/.config/autostart

If you open the “Cron Monitoring Script” application it should notify you that the monitoring script has started. You can test that you get notifications with the following command:

crontab -l | { cat; echo "* * * * * echo hello world";  } | crontab -

Make sure to remove the test cron job with crontab -e.

$HOME/.config/autostart monitoring:

On most Linux desktop, desktop applications are defined in a .desktop file with a bunch of parameters defining the application. Any valid .desktop files placed in the ~/.config/autostart directory will be executed when a user logs in to their desired desktop environment such as GNOME. This could be used to create a false, malicious application that can will start when a user logs in.

Create a script in ~/opt/scripts/autostart-monitor.sh with the following contents:

#!/usr/bin/env bash

notify-send "Started autostart monitoring script"

inotifywait --format='%f' -m -e create $HOME/.config/autostart | while read filename
do
    notify-send "Desktop autostart file added: ${filename}"
done

Make it executable:

chmod +x ~/opt/scripts/autostart-monitor.sh

Now create a file at ~/.local/share/applications/autostart-monitor.desktop to execute our monitoring script with the following content:

[Desktop Entry]
Name=Autostart Monitoring Script
Exec=bash -c "${HOME}/opt/scripts/autostart-monitor.sh"
Icon=org.gnome.Terminal
Type=Application
Categories=Security;

Now, symlink that application to the ~/.config/autostart directory so the application runs when you login:

ln -s ~/.local/share/applications/autostart-monitor.desktop ~/.config/autostart

If you open the “Autostart Monitoring Script” application it should notify you that the monitoring script has started. You can test that you get notifications with the following command:

touch ~/.config/autostart/dummy-application.desktop && rm ~/.config/autostart/dummy-application.desktop

$HOME/.config/systemd/user monitoring:

I would say that Systemd is the backbone of most modern Linux systems as it manages all the services. As an unprivileged user you can create user services so that repeated tasks are executed as you which could easily be used for running repeated, malicous tasks for a bad actor to establish persistence.

Create a script in ~/opt/scripts/systemd-monitor.sh with the following contents:

#!/usr/bin/env bash

notify-send "Started systemd monitoring script"

inotifywait --format='%f' -m -e create $HOME/.config/systemd/user | while read filename
do
    notify-send "User systemd file added: ${filename}"
done

Make it executable:

chmod +x ~/opt/scripts/systemd-monitor.sh

Now create a file at ~/.local/share/applications/systemd-monitor.desktop to execute our monitoring script with the following content:

[Desktop Entry]
Name=Systemd User Monitoring Script
Exec=bash -c "${HOME}/opt/scripts/systemd-monitor.sh"
Icon=org.gnome.Terminal
Type=Application
Categories=Security;

Now, symlink that application to the ~/.config/autostart directory so the application runs when you login:

ln -s ~/.local/share/applications/systemd-monitor.desktop ~/.config/autostart

If you open the “Systemd User Monitoring Script” application it should notify you that the monitoring script has started. You can test that you get notifications with the following command:

touch ~/.config/systemd/user/dummy.service && rm ~/.config/systemd/user/dummy.service

Application Sandboxing:

Unlike MacOS, most Linux distributions don’t have application sandboxing at all or very minimal application sandboxing. So if a malicious actor gets a shell by exploiting an application like Firefox they would be able to see anything your user could access. With sandboxing, getting a shell via an application exploit would severely limit what the malicious actor could access.

There are a couple of not-so-invasive methods for implementing restrictions around what an application can access:

AppArmor (an alternative to SELinux that is very non-intrusive in my experience)
Snaps/Flatpaks
Firejail
Bubblewrap
Containers

I’m going to focus on AppArmor and Firejail. I actively dislike Canonical’s approach to Snaps and Flatpak’s sandbox implementation isn’t very good.

AppArmor

AppArmor provides similar functionality to SELinux while not making your system impossible to work with. AppArmor allows you to define clear boundaries for what a process can and cannot access on your system. It can be configured to be more restricted if desired. Additionally, AppArmor can integrate with Firejail.

AppArmor is installed and enabled by default on Ubuntu systems: Ubuntu Docs: AppArmor

Once AppArmor is installed you can check its status with the following command:

sudo aa-status

On Debian-based systems, additional AppArmor profiles can be installed with the following commands:

sudo apt-get update && sudo apt-get install -y apparmor-profiles apparmor-profiles-extra 

On Arch, those profiles can be loaded by enabling and starting the apparmor service:

sudo systemctl enable --now apparmor.service

You can get notifications when the AppArmor policies denies an application:

Arch Wiki: AppArmor – Get desktop notifications on DENIED actions

Additional Reading:

Arch Wiki: AppArmor

Firejail:

After installing Firejail on your machine, use the following command to set up profiles for all of your installed applications that are already defined:

sudo firecfg

I would recommend integrating AppArmor with Firejail:

Arch Wiki: Enable AppArmor support

Ubuntu Manpage: Firejail AppArmor

On Arch you can integrate pacman operations with Firejail:

Arch Wiki: Using Firejail by default

Application Firewall:

On Linux, there aren’t many fully-featured/working application firewalls comparable to Little Snitch on MacOS. For those unfamiliar with what an application firewall does: an application firewall allows a user to permit or block access to network resources per application. For example, if I run curl google.com an application firewall would block curl from accessing google.com until I either permit or block the request. This is beneficial if an application is making undesirable network requests (i.e. a malicious binary attempts to connect to a server, or a program is collecting unwanted analytics, etc.).

There are two application firewalls on Linux that I am aware of:

I’ve never had success with OpenSnitch, but Portmaster has worked very well for me and seems to be very polished with GNOME.

Portmaster Docs: Install on Linux

Once Portmaster is running, open the Portmaster application and go to Settings > Privacy Filter > General > Default Network Action and set it to “Prompt”. This will set Portmaster to prompt you to Allow or Deny every network request to a new host per process. For example, if I run curl google.com and I deny that request in Portmaster any future network request to google.com by curl will be denied.

Container Security:

If you are a Docker user, using the default, privileged Docker daemon is a poor practice on a workstation as it completely bypasses the local firewall. Even worse, if your user has access to that privileged Docker daemon (i.e. added to the docker group) it is fairly trivial to access restricted resources if that user is compromised.

Some alternatives to the privileged Docker Daemon:

For rootless Docker and Podman, the following warning from the Arch Wiki applies: “Rootless Podman relies on the unprivileged user namespace usage CONFIG_USER_NS_UNPRIVILEGED which has some serious security”. However, this CONFIG_USER_NS_UNPRIVILEGED parameter is required for sandboxing in things like Electron applications (for example, the Zoom application on Linux won’t work if this parameter isn’t set). Another note from the Arch Wiki explains more: “Note: The user namespace configuration item CONFIG_USER_NS is currently enabled in linux (4.14.5 or later), linux-lts (4.14.15 or later), linux-zen (4.14.4-2 or later) and linux-hardened. Lack of it may prevent certain sandboxing features from being made available to applications.”

Lima, Rancher Desktop and Docker Desktop all utilize virtual machines to run Docker rather than attaching to the host’s kernel and resources like the privileged Docker daemon does. My preference is Lima as it leverages QEMU and in my experience is the most minimal and simple (for me) of all those options. I prefer using a VM to provide a bit of isolation from my system.

Misc:

Systemd-homed:

If you use Arch Linux or Fedora, you can use systemd-homed to create an ultra-portable home directory for your user.

The security benefits of systemd-homed are the following:

Totally portable home directory
LUKS encrypted home directory
Automatic locking of your home directory when your system is suspended
Better handling of credentials compared to the decades old Shadow system (admittedly, I don’t understand the details as to why or how it’s better but that is a feature touted by the systemd contributors)

If you have already setup your user without systemd-homed, don’t fret as the systemd folks have a nice guide on migrating an existing user to systemd-homed:

systemd.io: Converting Existing Users to systemd-homed managed Users

If you want to use LUKS encryption as the storage mechanism for your home directory create your user with the --storage=luks flag set:

homectl create --storage=luks <username>

Additional reading: Arch Wiki: systemd-homed

The Empire Systems

Moving to heywoodlh.io

Making the iPad Suck Less (for sysadmin, development, etc.)

Why do I own an iPad?

My assumptions

Keyboard-driven workflows

Use Spotlight for switching app focus/workspace switching

Use built-in app keyboard shortcuts

Use Stage Manager

Safari

Development tooling

Apple Shortcuts

iSH

Tailscale

Pomerium

Dedicated Linux server

Conclusion

Tiny Vim Install

Some background

Installation

Why?

Remote Backups Over SSH With Rsnapshot

Rsnapshot server:

Target servers:

Profit:

Sync Skyrim Mods Between Devices (with Syncthing)

Prerequisites:

Installing Syncthing

On Windows (using Chocolatey):

On Linux:

Add Both Devices to Syncthing

Add Skyrim’s Folders to Syncthing

Share Skyrim’s Syncthing Folders With Your Target Device

Play!

Some Syncthing Tips

File versioning

Setup a Server

NixOS: Managing GNOME Keyboard Shortcuts and Settings with Home Manager

Use Home Manager as a NixOS config module:

Using Home Manager to manage GNOME:

Tips for modifying dconf settings:

Installing NixOS on the Steam Deck

Why not keep SteamOS?

Installing NixOS on the Steam Deck:

Requirements/Assumptions:

Creating an Installer ISO and to Bootable USB:

Boot the Bootable USB:

Install NixOS:

Conclusion:

Switch back to Steam after running “Switch to Desktop”:

Launch Powershell in Tmux with -NoLogo Argument

Simple nfdump Setup (in Containers) for Netflow Collection and Analysis

Assumptions:

Why nfdump?

Setup nfcapd

Accessing the Captured Netflow Data:

Additional Reading:

A More Secure Linux Desktop

Why?

What Gaps?

Assumptions:

Basic Security:

Disk Encryption:

Use UFW:

ClamAV:

Application Autostart Monitoring:

Prerequisites:

Why?

User cron monitoring:

$HOME/.config/autostart monitoring:

$HOME/.config/systemd/user monitoring:

Application Sandboxing:

AppArmor

Firejail:

Application Firewall:

Container Security:

Misc:

Systemd-homed: