Bootstrapping Let's Encrypt on Debian

2019-03-15 - Progress - Tony Finch

I've done some initial work to get the Ansible playbooks for our DNS systems working with the development VM cluster on my workstation. At this point it is just for web-related experimentation, not actual DNS servers.

Of course, even a dev server needs a TLS certificate, especially because these experiments will be about authentication. Until now I have obtained certs from the UIS / Jisc / QuoVadis, but my dev server is using Let's Encrypt instead.

Chicken / egg

In order to get a certificate from Let's Encrypt using the http-01 challenge, I need a working web server. In order to start the web server with its normal config, I need a certificate. This poses a bit of a problem!

Snakeoil

My solution is to install Debian's ssl-cert package, which creates a self-signed certificate. When the web server does not yet have a certificate (if the QuoVadis cert isn't installed, or dehydrated has not been initialized), Ansible temporarily symlinks the self-signed cert for use by Apache, like this:

- name: check TLS certificate exists
  stat:
    path: /etc/apache2/keys/tls-web.crt
  register: tls_cert
- when: not tls_cert.stat.exists
  name: fake TLS certificates
  file:
    state: link
    src: /etc/ssl/{{ item.src }}
    dest: /etc/apache2/keys/{{ item.dest }}
  with_items:
    - src: certs/ssl-cert-snakeoil.pem
      dest: tls-web.crt
    - src: certs/ssl-cert-snakeoil.pem
      dest: tls-chain.crt
    - src: private/ssl-cert-snakeoil.key
      dest: tls.pem

ACME dehydrated boulders

The dehydrated and dehydrated-apache2 packages need a little configuration. I needed to add a cron job to renew the certificate, a hook script to reload apache when the cert is renewed, and tell it which domains should be in the cert. (See below for details of these bits.)

After installing the config, Ansible initializes dehydrated if necessary - the creates check stops Ansible from running dehydrated again after it has created a cert.

- name: initialize dehydrated
  command: dehydrated -c
  args:
    creates: /var/lib/dehydrated/certs/{{inventory_hostname}}/cert.pem

Having obtained a cert, the temporary symlinks get overwritten with links to the Let's Encrypt cert. This is very similar to the snakeoil links, but without the existence check.

- name: certificate links
  file:
    state: link
    src: /var/lib/dehydrated/certs/{{inventory_hostname}}/{{item.src}}
    dest: /etc/apache2/keys/{{item.dest}}
  with_items:
    - src: cert.pem
      dest: tls-web.crt
    - src: chain.pem
      dest: tls-chain.crt
    - src: privkey.pem
      dest: tls.pem
  notify:
    - restart apache

After that, Apache is working with a proper certificate!

Boring config details

The cron script chatters into syslog, but if something goes wrong it should trigger an email (tho not a very informative one).

#!/bin/bash
set -eu -o pipefail
( dehydrated --cron
  dehydrated --cleanup
) | logger --tag dehydrated --priority cron.info

The hook script only needs to handle one of the cases:

#!/bin/bash
set -eu -o pipefail
case "$1" in
(deploy_cert)
    apache2ctl configtest &&
    apache2ctl graceful
    ;;
esac

The configuration needs a couple of options added:

- copy:
    dest: /etc/dehydrated/conf.d/dns.sh
    content: |
      EMAIL="hostmaster@cam.ac.uk"
      HOOK="/etc/dehydrated/hook.sh"

The final part is to tell dehydrated the certificate's domain name:

- copy:
    content: "{{inventory_hostname}}\n"
    dest: /etc/dehydrated/domains.txt

For production, domains.txt needs to be a bit more complicated. I have a template like the one below. I have not yet deployed it; that will probably wait until the cert needs updating.

{{hostname}} {% if i_am_www %} www.dns.cam.ac.uk dns.cam.ac.uk {% endif %}