Stupid SmartOS Tricks

Lets Encrypt on SmartOS

Brian Ewell -

If you think encryption is awesome, then the Let's Encrypt project is pretty much free awesomeness.

Operating as a crowd-funded 501(c)(3) non-profit organization, Let's Encrypt provides a free, automated and open Certificate Authority to the Internet at large.  Which certainly beats paying for some third-party to sign your CSRs.

While I appreciate the goals of let's encrypt, granting root privilege to a script that automatically self-updates through a third-party management pathway and a blatant disregard for the principle of least privilege doesn't really sit well with me.

This guide will outline how to install Let's Encrypt as a local service on the same SmartOS Zone as Nginx, using the webroot method to allow for the verification of our ownership of our web server.  This installation will be slightly more challenging than setting it up on its own VM.  Not only is Let's Encrypt not officially supported for SmartOS, but we will be doing much of its work manually, namely configuring our server for TLS and setting up our own private keys and CSRs.

Because we will be using Let's Encrypt only for the validation and passing of CSRs (and returning of Certificates), we can limit it much more than other installation methods can.  It can operate safely with no privilege escalation on our SmartOS zone, rather than needing a dedicated Linux Branded Zone to use safely with SmartOS.

Installing Lets Encrypt

Since we won't be giving Let's Encrypt any access to root, we will be manually installing the required dependencies:

# pkgin update
# pkgin in -y install gcc49 py27-augeas py27-virtualenv

Create a letsenc system user and su into them:

# useradd -c "Lets Encrypt Agent" -d /var/lib/letsencrypt -s /usr/bin/bash letsenc
# mkdir /var/lib/letsencrypt
# chown letsenc /var/lib/letsencrypt
# su - letsenc

We're going to use certbot-auto to bootstrap our environment.  Download the client and mark it as executable.

$ wget https://dl.eff.org/certbot-auto
$ chmod a+x certbot-auto

Run certbot locally once to download and install all of the python dependencies.  We don't want it attempting to "bootstrap" the platform dependencies (since we've already done that) to be sure to use the --no-bootstrap flag.

$ ./certbot-auto --debug --no-bootstrap

Interrupt (Ctrl-C) lets encrypt when you're asked to switch to the superuser.  Delete the certbot-auto script, create the creds and webroot directories, and then exit back to the root user.

$ rm certbot-auto
$ mkdir creds webroot
$ exit

Generating Keys, CSRs, and Certificates

As root, let's start by generating a Private Key for Nginx.  This key should be owned by and accessible only to Nginx:

# mkdir -p /opt/local/etc/nginx/private/example.net
# cd /opt/local/etc/nginx/private/example.net
# openssl genrsa -out rsa.key 2048

2048-bit RSA is going to be quite secure for a very long time, but if you insist, you can use 3072-bit or 4096-bit RSA keys with Lets Encrypt as well at this time.

Additionally, we'll set up some symlinks in this directory to refer to the CSR and Cert files that will reside in Lets Encrypt's custody.  This is mainly for ease of Nginx configuration.

# ln -s /var/lib/letsencrypt/creds/rsa.csr
# ln -s /var/lib/letsencrypt/creds/rsa.crt

Next, we will need to create the CSR, and issue an initial certificate.  The CSR will be used to request certificates from Lets Encrypt (on behalf of our private key that we're not even going to trust Lets Encrypt with) and the initial certificate will allow our initial domain validation to occur over HTTPS.  While we could do this via HTTP, that would ultimately be more complicated.

Notice: As of 2017, certificates must refer to their domains via the Subject Alternative Names X.509 extension.  therefore I have removed the code snippet that makes reference to setting up the CSR and certificate the old way.

The Subject Alternative Names X.509 extension to allow for multiple domains to be protected by a single certificate (this is much easier than running multiple Lets Encrypt instances).  If you only need to protect a single domain, or want individual certificates per domain, simply specify individual domains here:

# openssl req -new -key rsa.key -out rsa.csr -reqexts tmp \
-subj "/C=US/O=Example Networks/CN=example.net" -config \
<(cat /opt/local/etc/openssl/openssl.cnf <(printf \
"[tmp]\nsubjectAltName=DNS:example.net,DNS:www.example.net"))
# openssl req -new -x509 -days 30 -key rsa.key -out rsa.crt \
-subj "/C=US/O=Example Networks/CN=example.net" -extensions tmp -config \
<(cat /opt/local/etc/openssl/openssl.cnf <(printf \
"[tmp]\nsubjectAltName=DNS:example.net,DNS:www.example.net"))

Configuring Nginx

We will need to configure Nginx to use these keys and certificates and forward a sub-location of the root of every domain we will be encrypting with Lets Encrypt into the Lets Encrypt directory so that domain ownership can be proven.

This is what one of my vhosts look like that I am protecting with Lets Encrypt:

/opt/local/etc/nginx/vhosts/example.enabled

server {
    listen [::];
    server_name example.net;
    return 302 https://example.net$request_uri;
}

server {
    listen [::]:443 ssl http2;
    server_name example.net;
    root /path/to/example.net;

    ssl_certificate private/example.net/rsa.crt;
    ssl_certificate_key private/example.net/rsa.key;

    location /.well-known/acme-challenge {
        root /var/lib/letsencrypt/webroot;
    }

    ...
}

This will forward requests to /.well-known/acme-challenge to /var/lib/letsencrypt/webroot/.well-known/acme-challenge.  Be sure to refresh nginx so that it takes on this configuration before continuing.

# svcadm refresh nginx

Initial Run

Before we do anything, switch back into the let's encrypt user.

# su - letsenc

Let's start by testing the bootstrap (ensuring that Lets Encrypt can actually work before we attempt issuing a real key).

$ ./.local/share/letsencrypt/bin/letsencrypt certonly \
--config-dir ./config \
--logs-dir ./logs \
--work-dir ./work \
--webroot -w /var/lib/letsencrypt/webroot \
--domains example.net \
--email webmaster@example.net \
--text \
--agree-tos \
--keep-until-expiring \
--staple-ocsp \
--csr creds/rsa.csr \
--fullchain-path creds/test.pem \
--test

This will verify that you are able to locally control the webroots of the domains in question, and use the passed CSR to issue a certificate that matches your private key.

Notice: Doing this in testing first is important, as you are only allowed a finite number of successful interactions with the production Certificate Authority per month.

Verify generated certificate (optional)

Now is a great time to see if your Certificate Signing Request matches the Certificate produced by Lets Encrypt.

$ openssl req -in creds/rsa.csr -noout -subject
subject=/C=US/O=Example Networks/CN=example.net

And

$ openssl x509 -in creds/test.pem -noout -subject
subject= /CN=example.net

Lets Encrypt will trim off everything but the CN from the subject.

If you made use of X509 Subject Alternative Names, you can compare them with the following lines:

$ openssl req -in creds/rsa.csr -noout -text | grep "Subject Alternative Name" -A 1
        X509v3 Subject Alternative Name:
            DNS:example.net, DNS:www.example.net

And

$ openssl x509 -in creds/test.pem -noout -text | grep "Subject Alternative Name" -A 1
        X509v3 Subject Alternative Name:
            DNS:example.net, DNS:www.example.net

Lastly, you can easily compare public keys of the CSR and the resulting Certificate like this:

$ diff <(openssl req -in creds/rsa.csr -noout -pubkey) \
       <(openssl x509 -in creds/test.pem -noout -pubkey)

No output means that the public keys are the same.

Generate Production Certificate

Assuming everything looks well, clear the .pem files you've generated, and delete creds/rsa.crt.  This file needs to be manually deleted, as Lets Encrypt refuses to overwrite existing files.

$ rm *.pem creds/test.pem creds/rsa.crt

Next up, run the certificate bootstrap command with production paths (and without the --test flag).

$ ./.local/share/letsencrypt/bin/letsencrypt certonly \
--config-dir ./config \
--logs-dir ./logs \
--work-dir ./work \
--webroot -w /var/lib/letsencrypt/webroot \
--domains example.net \
--email webmaster@example.net \
--text \
--agree-tos \
--keep-until-expiring \
--staple-ocsp \
--csr creds/rsa.csr \
--fullchain-path creds/rsa.crt \

Back as root, renew Nginx and verify that you indeed have a trusted certificate chain.

# svcadm refresh nginx

Automated Renewal

The cron daemon is well suited for running a periodic task such as this:

/root/scripts/letsenc.sh

#!/bin/bash

su - letsenc << EOF
    mv creds/rsa.crt creds/rsa-old.crt
    ./.local/share/letsencrypt/bin/letsencrypt certonly \
        --config-dir ./config \
        --logs-dir ./logs \
        --work-dir ./work \
        --webroot \
        --email webmaster@example.net \
        -w /var/lib/letsencrypt/webroot \
        --domains example.net,www.example.net \
        --text \
        --agree-tos \
        --staple-ocsp \
        --keep-until-expiring \
        --csr creds/rsa.csr \
        --chain-path creds/rsa.crt
EOF
svcadm refresh nginx

What about letsencrypt renew?  Save that for another time.

I recommend scheduling this command to run at most once per month.