Laptop as Home Server with Ansible

Here's how I used a trusty old Thinkpad T510 laptop as a home server. This is mostly notes for myself, but perhaps it'll be useful to others too.

The server isn't doing much right now; just running my home weather and power monitoring

• Grafana Dashboards Server
• Prometheus real-time monitoring database
• Alertmanager for Prometheus
• Dynamic DNS (so I can access the above away from home)

Laptops aren't perfect, but they have a few things going for them as servers:

• Low power consumption
• Built-in keyboard and screen for management console
• You can often old laptops hanging around the house

Wipe the laptop

Back up all the data first!

Install an OS

There's plenty of choices. You probably want Linux for a home server; Windows Server is probably overkill. Among Linuxes, I'm very familiar with the Debian/Ubuntu flavor.

I first tried Ubuntu Desktop, with the intention of switching over to Ubuntu Server, but I never really got around to it. It was simpler to set up things like WiFi on the desktop UI than in config files, so I just kept Ubuntu Desktop. It's handy for debugging if something goes wrong too: open the lid and use the desktop environment to fix it.

I burned the image to a USB stick to install, as usual, following ubuntu.com's tutorial.

I enabled full-disk encryption for best security. More on this later.

Disable turning off when lid shuts

I want to shut the lid and have the computer keep running without going to sleep.

I found the answer on AskUbuntu. As usual, there's no GUI option, you have to ensure that /etc/systemd/logind.conf contains:

HandleLidSwitch=ignore

Then restart logind:

systemctl restart systemd-logind.service

I have an Ansible config for ensuring this stays working:

- name: Disable sleep-on-lid down
  become: true
  lineinfile:
    path: /etc/systemd/logind.conf
    regexp: 'HandleLidSwitch'
    line: 'HandleLidSwitch=ignore'

Enable SSH server

Pretty straightforward. Just want to make sure to disable password entry and only use private keys.

Recovering from power outages

From time to time, the machine will turn off, and I need to reboot it. This isn't as automated as I'd like yet.

Wake on LAN

There's a BIOS option in the ThinkPad to enable Wake-On-Lan.

Grab the MAC address from the Wi-Fi dashboard.

On macOS, to trigger the machine to wake-up, I use the wakeonlan program:

$ brew install wakeonlan
$ wakeonlan -i 192.168.1.255 -p 7 ca:fe:ba:aa:aa:be

Port 7 seemed to work for me, but this was just a bit of trial-and-error to find the port.

Remote Unlock Disk Encryption

Setting up full disk encryption with a passphrase makes it difficult to boot up automatically, or remotely.

I followed Hamy's Blog's Instructions.

I learned a lot setting this up. Some background:

• When you boot up your laptop, and get prompted for your full-disk-encryption passphrase, there's actually an operating system running at this stage.
• This OS has a full Linux kernel, and a small busybox userland, and even an init system. You can exit the full-disk-encryption prompt and drop to a shell, and run a few very limited commands.
• This OS lives on /boot, a separate, unencrypted partition on your disk.
• You can install extra stuff into this OS! But it's a real pain in the butt, you're constantly restarting to check if it worked or not.
• In particular, there's a small, statically-linked SSH server called dropbear. Install it with

  $ sudo apt-get install dropbear-initramfs
  

• Configure by dropping public keys into /etc/dropbear-initramfs/authorized_keys and configuring /etc/dropbear-initramfs/config to e.g. change the port by adding DROPBEAR_OPTIONS="-p 4748".
• Then you have to "compile" the initramfs stored in /boot by running:

  $ update-initramfs -u
  

• Finally, restart, and you should be able to connect in through SSH, and unlock by running cryptroot-unlock:

  $ ssh root@192.168.1.13 -p 4748
  # cryptroot-unlock
  <enter your full disk encryption password>
  

I have a few Ansible scripts to configure this behaviour:

# dropbear/tasks/main.yml
- name: install dropbear-initramfs
  become: true
  apt:
    state: latest
    name:
    # https://hamy.io/post/0009/how-to-install-luks-encrypted-ubuntu-18.04.x-server-and-enable-remote-unlocking/#fn:1
    - dropbear-initramfs

- name: Run Dropbear SSH on a different port
  become: true
  # https://hamy.io/post/0009/how-to-install-luks-encrypted-ubuntu-18.04.x-server-and-enable-remote-unlocking/#fn:1
  lineinfile:
    path: /etc/dropbear-initramfs/config
    regexp: 'DROPBEAR_OPTIONS'
    # https://linux.die.net/man/8/dropbear
    line: 'DROPBEAR_OPTIONS="-p 4748"'
  notify: [ 'Update initramfs' ]
  # ssh root@192.168.2.2 -p 4748
  # cryptroot-unlock

- name: add ssh authorized key for dropbear
  become: true
  authorized_key:
    # user: root is important! Don't use 'mark' or dropbear won't use the key
    user: root
    key: "{{ lookup('file', '/Users/mark/.ssh/id_rsa.pub') }}"
    path: /etc/dropbear-initramfs/authorized_keys
  notify: [ 'Update initramfs' ]

Which, whenever something changes, notifies the Update initramfs step to rebuilt /boot:

# dropbear/handlers/main.yml 
---
- name: Update initramfs
  become: true
  command: update-initramfs -u

Managing Updates

I'm using Ansible to automate a bunch of this setup and maintenance. Here are some of my Ansible scripts for this server:

---

- name: install apt packages
  become: true
  apt:
    update_cache: true
    state: latest
    name:
    - arp-scan
    - binwalk
    - ca-certificates
    - python3-pip
    - python3-setuptools
    - fish

- name: Update apt packages to the latest version
  become: true
  apt:
    upgrade: full

- name: remove old apt packages
  become: true
  apt:
    autoremove: true

I'll run these updates about once a month.

Configuring Timezone and Shell

Ansible scripts:

- name: Set timezone to Australia/Sydney
  become: true
  timezone:
    name: Australia/Sydney

- name: use fish shell
  become: true
  user:
    name: "{{ ansible_user }}"
    shell: /usr/bin/fish

Installing Tailscale

I like Tailscale for accessing my server from outside my local network. To install and keep it up to date, I have this Ansible config:

- name: add tailscale apt GPG key
  become: true
  apt_key:
    url: 'https://pkgs.tailscale.com/stable/ubuntu/focal.gpg'

- name: add tailscale apt list
  become: true
  apt_repository:
    repo: 'deb https://pkgs.tailscale.com/stable/ubuntu focal main'

- name: apt update tailscale
  become: true
  apt:
    state: latest
    update_cache: true
    name:
    - tailscale

Initial sign-in to your Google account has to be done manually by SSH'ing into the server and running tailscale up, but after that it just works.

Install Docker

I like using docker-compose to configure my server arguments using a single docker-compose.yml file.

I install Docker with Ansible. There's a bit of rubbish here about cutting the current SSH connection, so that changes to group membership can take effect on the next login. It's not very clean, but it works. Also grab python and the docker-compose binary:

# install_docker/tasks/main.yml

# Docker installation mostly cribbed from:
# https://withblue.ink/2019/07/13/yes-you-can-run-docker-on-raspbian.html
- name: add docker GPG key
  become: true
  apt_key:
    url: 'https://download.docker.com/linux/ubuntu/gpg'

- name: Add docker apt repo
  become: true
  apt_repository:
    repo: 'deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable'

- name: install docker-ce
  become: true
  apt:
    # https://github.com/raspberrypi/linux/issues/3021#issuecomment-508571569
    # The aufs package, part of the "recommended" packages, won't install
    # on Buster just yet, because of missing pre-compiled kernel modules.
    install_recommends: false
    update_cache: true
    state: latest
    name:
    - python3-pip
    - docker-ce

- name: copy /etc/docker/daemon.json
  become: true
  copy:
    src: daemon.json
    dest: /etc/docker/daemon.json
    mode: '0644'
    backup: true
  # have to restart docker to notify this file has changed. SIGHUP not good enough
  notify: ['restart docker']

- name: add user to group docker
  become: true
  user:
    name: "{{ ansible_user }}"
    groups:
    - docker
    append: true

# https://stackoverflow.com/questions/26677064/create-and-use-group-without-restart
# https://docs.ansible.com/ansible/latest/modules/meta_module.html
# reset_connection (added in Ansible 2.3) interrupts a persistent connection (i.e. ssh + control persist)
- name: reset ssh connection to allow user changes to affect 'current login user'
  meta: reset_connection
  # when: 'add_docker_group.changed'
  # Had to remove the above because of:
  # [WARNING]: reset_connection task does not support when conditional

- name: enable & start docker service
  become: true
  systemd:
    name: docker
    state: started
    enabled: true
    daemon_reload: true

- name: pip install python libraries
  pip:
    name:
    # For docker_compose module
    - docker
    - docker-compose
    extra_args: --user

This installs a daemon.json configured to retain logs and export metrics and enable experimental features:

# install_docker/files/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "20m",
    "max-file": "3"
  },
  "metrics-addr" : "0.0.0.0:9323",
  "experimental": true
}

# install_docker/handlers/main.yml
- name: restart docker
  become: true
  service:
    name: docker
    state: restarted

Run the containers

Copy over your docker_compose.yml and any other mounted container config files, and run docker-compose update to grab the latest containers and remove any orphans:

# docker_compose/tasks/main.yml
---
- name: Copy dc to host
  copy:
    src: "{{ inventory_hostname }}/"
    dest: dc

- name: Get latest docker images
  docker_compose:
    project_src: dc
    remove_orphans: true
    pull: true
  register: docker_compose_output

Finally, reload all the containers that need take hot-swappable config files:

- name: reload alertmanager
  command: docker kill --signal=HUP dc_alertmanager_1
  when: '"alertmanager" in ansible_facts'
- name: reload prometheus
  command: docker kill --signal=HUP dc_prometheus_1
  when: '"prometheus" in ansible_facts'
- name: reload blackbox
  command: docker kill --signal=HUP dc_blackbox_1
  when: '"blackbox" in ansible_facts'

All Done

You have an old laptop now redeployed as a server, and a few ansible scripts to keep it up to date and act as documentation for how you set it up. Nice!

There's probably many better ways to do this. These are just the tools that I understand.