Certificates on SmartOS

Encryption. Pretty much the coolest thing since sliced bread. Where would we be without it? Probably not using iPhones, calling our banks and talking to real people to get business done. In short, we'd be in the dark ages.

This guide will go over how to setup X.509 PKI for pretty much anything. We'll be focusing on:

  • How to properly create and handle private keys.
  • Creating self-signed certificates.
  • Creating Certificate Signing Requests for interaction with an external Certificate Authority.
  • Deploying keys and certificates.

We will be saving running your own Certificate Authority for our next post, as it is involved enough to warrant it's own article. If you haven't already, I recommend reading my previous posts on X.509 and Asymmetric Cryptosystems, or at least having a working understanding of X.509 and the differences between RSA and ECC before continuing with reading this post.

Selecting an Implementation

OpenSSL has been the go-to implementation across the UNIX world for pretty much all of recorded history. Unfortunately, it has been taking quite a bit of flak recently for their numerous severe bugs (UPDATED). With seemingly no end to this in sight, many users have begun searching for alternatives.

We will be using OpenSSL for this guide.

In performing research for this article, I found that GnuTLS' certtool (available on SmartOS via pkgin) has an incredibly intuitive command-line interface, much better than OpenSSL in my opinion for simple tasks, however it's still not quite as well supported, ubiquitous or powerful as OpenSSL is. Additionally, OpenSSL's documented flaws have been around it's TLS, not it's key and certificate management.

I also investigated Mozilla's Network Security Services, which ended up in a distant third place during my evaluation, namely for it's incredibly difficult to work with command line utility, certutil.

As well, Solaris' pktool could have been another viable option, except for it's lack of ECDSA support (and apparent lack of any recent development interest), and very specific platform lockin (which really has little impact on a SmartOS specific guide).

LibreSSL could be a viable alternative, however again, for the sake of this document, we don't care about SSL/TLS security, and are instead focused on key and certificate management. This project was forked from OpenSSL less than 2 years ago, so I doubt there has been much change in this area.

Since the Ubuntu project already has an excellent article on using GnuTLS for common key/cert management tasks, re-writing it here seems like a waste of time. If you would like to use GnuTLS, I wholeheartedly recommend using the Ubuntu article as your guide.

Key Generation

Key generation should always occur in place where the key will be used, either in the filesystem or dedicated cryptography hardware such as a Smart Card or Trusted Platform Module (TPM). If that is not an option, creating and moving a card is plan B. Either from one system to another or from filesystem to Smart Card or TPM. The specifics of using TPM for securing private keys are unfortunately beyond the scope of this guide.

Encrypt your private keys early and often. If your keys are going to be used with human intervention, they should always be encrypted, or stored on a PIN locked Smart Card. If they're being transported on USB drive or other removable storage mechanism, they should be encrypted via a strong one-time-password, preferably randomly generated (openssl rand -base64 N where N is a number between 6 and 18`). Decrypt your key once you get it into place (if you're planning on using it unmanned.)

Decisions about key size and cryptographic algorithms should be based on where and for how long these keys will be used. Specific recommendations can be found in my previous blog posts on the subject.

While key length can vary widely, it's recommended to utilize the same cryptograpic algorithm for every certificate in your chain. For instance, if you create a 2048 bit RSA key, it's recommended that the key be signed by another RSA key, not a ECDSA key.


Generating an unencrypted 2048-bit RSA key is quite short and fairly straight forward:

$ openssl genrsa -out rsa.key 2048

Generating an AES256 encrypted 2048-bit RSA key is also fairly straight forward:

$ openssl genrsa -aes256 -out secured.key 2048

If you need to encrypt or decrypt a private key, you can do so using openssl rsa -aes256, or any other supported symmetric cypher flag to encrypt a key with a passphrase.

Various symmetric ciphers are supported by OpenSSL, I recommend the use of Rijndael (AES) or Camellia over DES, 3DES or SEED ciphers.

$ openssl rsa -aes256 -in unsecured.key -out secured.key
writing RSA key
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

To decrypt, just omit the flag:

$ openssl rsa -in secured.key -out unsecured.key
Enter pass phrase for secured.key:
writing RSA key

Note that the original key file will not be deleted by OpenSSL after performing these operations.


Generating a ECDSA key-pair under OpenSSL is a bit more involved. First you must select an ecliptic curve from OpenSSL. For this example, we will be generating a 256-bit ECDSA key.

$ openssl ecparam -list_curves
 secp256k1 : SECG curve over a 256 bit prime field

We will need to name this curve in our generation command.

$ openssl ecparam -name secp256k1 -genkey -out ecc.key

While there is no way to directly generate an encrypted private key directly using OpenSSL, it can be done after the fact using openssl ec -aes256, or any other supported symmetric cypher flag to encrypt a key with a passphrase:

$ openssl ec -aes256 -in unsecured.key -out secured.key
read EC key
writing EC key
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

To decrypt, just omit the flag:

$ openssl ec -in secured.key -out unsecured.key
read EC key
Enter PEM pass phrase:
writing EC key

Update: An ECDSA key-pair can be generated and encrypted by combining the two commands with a pipe:

$ openssl ecparam -name secp256k1 -genkey | openssl ec -aes256 -out secured.key

Self-Signed Certificate

A self-signed leaf certificate is the cryptographic equivalent of a note that says "I am who I say I am." This is generally a horrible idea on a production server, as it side-steps most of X.509's PKI framework. It is however, better than the alternative of not using encryption at all, and is definitely suitable in development and testing.

Since self-signing is done so often, OpenSSL has commands which generate a signing request and signs it with it's own private key, thereby creating the self-signed certificate.

The respective commands are identical for both RSA and ECDSA keys. OpenSSL will (by default) interactively ask you some questions about the certificate you're creating. If the certificate is being created for a server, you need to set the certificate Common Name (OpenSSL) or the certificate dnsName (GnuTLS) to the preferred DNS name of the server you're intending on using this certificate with.

All other subject fields are optional, especially for a self-signed certificate. Note also that self-signed certificates must be CA enabled to allow for their use in issuing certificates, even certificates for their own keys.

To be taken through the interactive OpenSSL interface, use the following line:

$ openssl req -new -x509 -days 365 -key private.key -out public.crt

You can also specify the subject in the command-line, which will allow this command to complete without requiring any additional information. The below example includes just a common name in the subject:

$ openssl req -new -x509 -days 365 -key private.key -out public.crt \
-subj "/CN=subdomain.cospix.net"

Additionally, since the task of generating a key and self-signed certificate is so common, OpenSSL supports performing both actions in a single command. The following command generates a new 2048-bit RSA key and self-signs it with the provided subject:

$ openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout key.pem -out cert.pem \
-subj "/CN=subdomain.cospix.net"

Notice: If you'd rather have your self-signed certificates issued with a cryptographic hash other than the default, you can append that hash as a tag to the above command-lines, basically: -sha256, -sha384 or -sha512

Interacting with Certificate Authorities

Chances are that if you're intending on doing something large scale or commercial with your Certificate, specifically interacting with a large number of peers outside of our immediate control such as using it on a public facing web server or email service, you will want a public Certificate signed by an external Certificate Authority. These external authorities are companies and organizations that have made arrangements for their Certificates to be included in the trusted certificate stores of shipped operating systems.

To have a Certificate Authority issued certificate, you will first need to make a Certificate Signing Request. Fortunately, that's just half of what we did before with our self-signed Certificate. In fact, the commands just omit the -x509 and -days <N> parameters as those commands are specific for issuing certificates. In addition, the output file is a Certificate Signing Request (CSR), instead of a Certificate.

$ openssl req -new -key private.key -out public.csr

As before, OpenSSL will (by default) interactively ask you some questions about the Certificate Signing Request you're creating. If the certificate is being created for a server, you need to set the certificate Common Name (FQDN) to the preferred DNS name of the server you're intending on using this certificate with. In addition, most Certificate Authorities will be expecting more information about the Certificate Subject, such as a name, and area of operation. This information is encoded as the subject field in both CSRs and Certificates. If you prefer, this information can be provided to OpenSSL in the command-line with the following:

$ openssl req -new -key private.key -out public.csr \
-subj "/C=US/ST=Oregon/L=Portland/O=Cospix LLC/CN=subdomain.cospix.net"

When input from the command-line, subject field names are prefixed with a / and separated from their field values by =. A list of common field names is as follows:

Field NameField DescriptionExample
C2-character Country CodeUS
STState NameOregon
LLocality (eg: City) NamePortland
OOrganization NameCospix LLC
OUOrganizational Unit NameNetwork Security
CNCommon Namesubdomain.cospix.net
emailAddressEmail Addressemail@cospix.net

The shrewd reader might notice that these fields are LDIF fields. You are correct.

After creating the Certificate Signing Request, you will need to send it to your preferred Certificate Authority to be issued a certificate. The specifics of that are beyond the scope of this guide. You can use OpenSSL to view the contents of a CSR before sending it to a Certificate Authority. The parameters -text will output a textual representation of the information included in the CSR, while -verify will confirm the signatures in the CSR.

$ openssl req -in public.csr -noout -text -verify

Certificate Authorities will often use a branch key rather than their root key to sign a client's leaf Certificate. This means that you will likely be issued several certificates, more specifically a certificate chain of verification which extends between your public certificate and your trust store. You should just keep this all as one file.

Deploying Keys and Certificates

Traditionally, certificates and keys were kept in /etc/ssl/certs and /etc/ssl/private, however practice has shown that it's better to keep keys and leaf Certificates (or certificate chains) should be kept close to each other, preferably owned by root and readable to the processes via exclusive group which need to make use of them. ie:

$ chmod 640 /opt/local/etc/nginx/vhosts/{site_name.key,site_name.crt}
$ chown root:nginx /opt/local/etc/nginx/vhosts/{site_name.key,site_name.crt}

It's also a good idea to verify the contents of certificates before you actually make use of them, this can be done with both OpenSSL with the following commands:

$ openssl x509 -in public.crt -noout -text

This command produces a full textual representation of the information contained in the certificate. In addition, the following parameters can be used to view the following fields:

  • -subject outputs the subject's subject field
  • -issuer outputs the issuer's subject field
  • -email outputs the subject email address
  • -startdate outputs the notBefore field
  • -enddate outputs the notAfter field
  • -purpose outputs the certificate purposes
  • -dates outputs both the notBefore and notAfter fields
  • -modulus outputs the key modulus (only works on RSA keys)
  • -pubkey outputs the encoded public key
  • -fingerprint outputs the certificate fingerprint

More information can be found by checking the Openssl x509 man page.