SmartOS

Tunnels on SmartOS

Brian Ewell

Mar 11, 2016 • 11 min read

We've all been there: You're managing two private network segments that need to have quick, secure communication between them but are separated by a few hundredths of a second of the wild and open Internet. What to do...

This is a brief guide on setting up a secure IPsec encrypted IP tunnel between two SmartOS Zones (or a SmartOS Zone and another compatible endpoint, such as pfSense.) This guide assumes that each zone is being used as a router via Virtual Ethernet segment or Etherstub which is the private network segment to be bridged, similar to the configuration mentioned in the Joyent wiki. This guide also assumes that no network communication should take place without encryption enabled and that the firewall should be configured to reject packets coming from unsecured interfaces.

If you don't care about encrypting your private network traffic as it makes its way across the public Internet, ignore the sections on IPsec Policy and IKE Configuration. If you don't care about the possibility of an attacker maliciously injecting packets into your network, ignore the section on IP Filter.

Notice: The steps in this guide have been arranged to maximize security. If you're having trouble getting it to work, it may be easier to start with the tunnel/routing section first, then the IP Filter section, and then IPsec Policy/IKE.

Environment

For the sake of the following sections, we will be defining an example environment with two virtualized NAT routers handling routing IPv4 and IPv6 for two private network segments. Their configurations are as follows:

Alice

net0/v4: 1.2.3.4 (public)
net1/v4: 10.0.0.1/24
net1/v6: 2001:470:24:c66::/64

Bob

net0/v4: 5.6.7.8
net1/v4: 10.0.1.1/24
net1/v6: 2001:470:24:c67::/64

Both Alice and Bob's net0/v4 represents their public-facing NIC, which has been labeled with their publicly addressable IPv4 address. net1/v4 represents their attached private IPv4 network (host address and network/masklen) which we're trying to establish secure communication between. net1/v6 represents the IPv6 network prefix used in this private network segment, which should also have it's traffic encrypted.

Since we're going to be establishing a pair of point to point links, we will name each tunnel by the version of IP we're tunneling through, as well as the destination. This may seem like an unnecessary step, but using something generic like tun0 can make it difficult to distinguish which system you're working on, and is almost impossible to deal with in a medium to large scale installation with dozens of interconnecting tunnels.

Link names in SmartOS must end with a digit, so we will append 0 to each. The Alice—Bob tunnel will be referred to as v4_alice0 by Bob, and v4_bob0 by Alice. If both Alice and Bob have native IPv6 connectivity, the tunnel could be established over IPv6 and would be called v6_alice0 and v6_bob0 respectively.

Command Conventions

Any commands that should be run on Alice will be prefixed with alice #. Any commands that should be run on Bob will be prefixed with bob #. Any commands that should be run on both Alice and Bob will be prefixed with #. Many configuration file examples below will be presented for just Alice. To determine Bob's configuration equivalent, replace Alice specific data with Bob specific data.

IPsec Policy

The IPsec policy controls which packets will be protected by IPsec. Policy targets can be specified by port, IP address or tunnel interface, since we're using tunnels, we will be specifying our target via named tunnel link. Below is an example of Alice's IPsec policy:

/etc/inet/ipsecinit.conf:

{tunnel v4_bob0 negotiate tunnel}
    ipsec {encr_algs aes encr_auth_algs sha512 sa shared}

A list of available Encryption and Authorization algorithms is provided by the ipsecalgs command. I recommend the use of aes to encrypt and sha512 to hash, however, you may prefer to use sha1 or sha256 as they will likely be more space efficient. If you would like more input for making decisions about which encryption/hashing algorithms to use in your particular case, please check out the Performance section of this blog post.

After you have configured your IPsec policy, and an appropriate reciprocal policy on Bob, restart the ipsec/policy services on both hosts.

# svcadm restart ipsec/policy

IKE Configuration

Internet Key Exchange is the component of IPsec which passes the critical encryption and authentication information between the secure hosts, effectively bootstrapping IPsec secured communication. While the ESP/AH parameters can be transferred manually between hosts, IKE makes this process so much easier.

I recommend configuring IKE to use a pre-shared key before moving on to using certificates.

IKE Pre-Shared Authentication

While pre-shared authentication is simple and easy to setup, especially if you administer both end-points, it does also come with one major limitation: You cannot use preshared authentication with a dynamically addressed end-point. Below is an example of Alice's ike configuration:

/etc/inet/ike/config:

p1_lifetime_secs 7200
p1_nonce_len 40

p1_xform {
    auth_method preshared
    oakley_group 5
    auth_alg sha512
    encr_alg aes
}

p2_pfs 2

{
    label "v4_bob0"
    local_addr 1.2.3.4
    remote_addr 5.6.7.8
    p1_xform {
        auth_method preshared
        oakley_group 5
        auth_alg sha512
        encr_alg aes
    }
    p2_pfs 5
}

Optionally verify your IKE config file syntax with in.iked:

# /usr/lib/inet/in.iked -c
in.iked: Configuration file /etc/inet/ike/config syntactically checks out.

On either Alice or Bob, generate a random number to use as a key. Since we're using SHA512 in the above example, the minimum recommend preshared key size is 512 bits or 64 bytes. Anything extra is used as additional keying material.

alice # openssl rand -hex 128
0f5451e4e0ad601f64dad77b9a83a0214dd4f460e07935936cfc7faf24c90f1c65609aaebc817ea17894748d999c94324d53a50ba18ce62c7683966674c3d50f93d1333440709f92facd1d10bcfde5b9ccc05e20adfa00a14825025068ba29cac04ba75b3b7e909dfddc01bd84eec2d56fa47a151cd447d0342f4194a910be9c

Next up we're going to create the ike.preshared file. Below is an example of Alice's ike.preshared file:

/etc/inet/secret/ike.preshared:

{
    localidtype IP
    localid 1.2.3.4
    remoteidtype IP
    remoteid 5.6.7.8
    key 0f5451e4e0ad601f64dad77b9a83a0214dd4f460e07935936cfc7faf24c90f1c65609aaebc817ea17894748d999c94324d53a50ba18ce62c7683966674c3d50f93d1333440709f92facd1d10bcfde5b9ccc05e20adfa00a14825025068ba29cac04ba75b3b7e909dfddc01bd84eec2d56fa47a151cd447d0342f4194a910be9c
}

Once you have the appropriate reciprocal files in place on Bob, enable (or restart) the ipsec/ike service in SMF on both Alice and Bob.

# svcadm enable ipsec/ike

IKE Public Key Authentication

Instead of using a pre-shared secret key, SmartOS can use public key cryptography to exchange IPsec keying information. SmartOS supports using both self-signed and CA issued key-pairs, however, there is no support for ECDSA, and RSA keys must be of size 512, 1024, 2048, 3072, or 4096 bits. I recommend using one of the largest three sizes.

While OpenSSL could be used to generate keys, the ikecert command is also capable and will be managing the keys for use in IKE, so we might as well use that.

Generating a Self-Signed Key-Pair

In this scenario, there is no trusted intermediary between Alice and Bob. Each generates their own key-pair, exchanges it and validates it through a side-channel.

alice # ikecert certlocal -ks -m 2048 -t rsa-sha1 \
-D "C=US,O=AliceTech Inc.,OU=Network Security,CN=AliceTech VPN"
Creating private key.
Certificate added to database.
-----BEGIN X509 CERTIFICATE-----
MIIDUzCCAjugAwIBAgIEBfeZ0TANBgkqhkiG9w0BAQUFADBZMQswCQYDVQQGEwJV
...
hVdRqYS1u8aNrae//zkHt9b4Av7Uf1NY6nEAKtBvi2PwTKni5jlD
-----END X509 CERTIFICATE-----

The output can be transferred to Bob for him to import. Pass the certificate into the stdin.

bob # ikecert certdb -a
-----BEGIN X509 CERTIFICATE-----
MIIDUzCCAjugAwIBAgIEBfeZ0TANBgkqhkiG9w0BAQUFADBZMQswCQYDVQQGEwJV
...
hVdRqYS1u8aNrae//zkHt9b4Av7Uf1NY6nEAKtBvi2PwTKni5jlD
-----END X509 CERTIFICATE-----

Bob should also generate a self-signed key-pair and transfer the public component to Alice for her to import into her certificate database.

The commands ikecert certlocal -l and ikecert certdb -l will list the private keys and public certificates in each database.

The commands ikecert certlocal -r <n> and ikecert certdb -r <n> will remove the private key and public certificate at the given slot <n>.

Generating a Key-Pair & Signing Request

In this scenario, there is a trusted intermediary between Alice and Bob, let's call him Charles. Alice and Bob each generate a key-pair and pass the signing request to Charles who will issue a certificate back to each. Assuming that Charles' public key is already securely distributed, there is no need to separately validate the certificates.

alice # ikecert certlocal -kc -m 2048 -t rsa-sha1 \
-D "C=US,O=AliceTech Inc.,OU=Network Security,CN=AliceTech VPN"
Creating private key.
-----BEGIN CERTIFICATE REQUEST-----
MIICvzCCAacCAQAwWTELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDkFsaWNlVGVjaCBJ
...
G7ERhchw/BXbA3ZEYcaAG+jOZ7GibUzCca83F3kdI4y36GA=
-----END CERTIFICATE REQUEST-----

After Charles issues a certificate, Alice and Bob can import it into their certificate databases.

Configuring IKE to use public keys

After your self-signed or CA issued key-pairs are in place, you will need to modify the IKE configuration file to make use of the keys. I've re-included the file here with notes:

/etc/inet/ike/config:

p1_lifetime_secs 7200
p1_nonce_len 40

# The following line is only necessary if you're using a
# Certificate Authority.  Include the DN of the common root.
# This certificate must also have been installed into certdb.
cert_root "C=US, CN=Charles' Certificate Authority DN"
# If you're using the above line, and you would like to ignore
# Certificate Revocation Lists include the following
ignore_crls

# The following lines are necessary if you're using self-signed
# certificates
cert_trust "C=US,O=AliceTech Inc.,OU=Network Security,CN=AliceTech VPN"
cert_trust "C=US,O=BobDesign Ltd.,OU=Network Security,CN=BobDesign VPN"

p1_xform {
    # Changed preshared to rsa_sig, I don't know if this is
    # necessary at this point, but it is done in the docs I'm reading from
    auth_method rsa_sig
    oakley_group 5
    auth_alg sha512
    encr_alg aes
}

p2_pfs 2

{
    label "v4_bob0"
    # Added the following three lines:
    local_id_type dn
    local_id  "C=US,O=AliceTech Inc.,OU=Network Security,CN=AliceTech VPN"
    remote_id "C=US,O=BobDesign Ltd.,OU=Network Security,CN=BobDesign VPN"
    
    local_addr 1.2.3.4
    remote_addr 5.6.7.8
    p1_xform {
        # Changed preshared to rsa_sig
        auth_method rsa_sig
        oakley_group 5
        auth_alg sha512
        encr_alg aes
    }
    p2_pfs 5
}

Optionally verify your IKE config file syntax with in.iked:

# /usr/lib/inet/in.iked -c
in.iked: Configuration file /etc/inet/ike/config syntactically checks out.

Once you have the appropriate reciprocal files in place on Bob, enable the ipsec/ike service in SMF on both Alice and Bob.

# svcadm enable ipsec/ike

If you had previously configured IKE for a pre-shared key, instead restart ipsec/ike and remove any unused entries in /etc/inet/secret/ike.preshared.

# svcadm restart ipsec/ike

IP Filter

The goal of our firewall rules is to ensure that only packets from the authorized remote networks are allowed through the tunnel, and that the tunnel is the only source of packets from that network. We can achieve this with the following firewall rules. Below is an example of Alice's firewall rules:

/etc/ipf/ipf.conf:

# Protecting IPv4 between Alice and Bob
block in on v4_bob0 all
block out on v4_bob0 all
block in from 10.0.1.0/24 to 10.0.0.0/24
block out from 10.0.0.0/24 to 10.0.1.0/22
pass in on v4_bob0 from 10.0.1.0/24 to 10.0.0.0/24
pass out on v4_bob0 from 10.0.0.0/24 to 10.0.1.0/24

The first two lines block all IPv4 traffic on the IP tunnel. The second two lines block all IPv4 traffic between the local and remote network regardless of interface. The last two lines explicitly enable communication between the local and remote network via the IP tunnel.

Unfortunately, SmartOS doesn't properly filter IPv6 packets at this time, so these firewall rules only apply to IPv4. The basic premise should apply the same though with the addition of allowing network traffic through on the link-local address.

After setting the new configuration, restart ipfilter on both hosts:

# svcadm restart network/ipfilter

IP Tunnel

Quite possibly the easiest part of this whole process is setting up the IP Tunnel. SmartOS' commands makes this trivial with dladm, ipadm and route.

Basic Security

From a security perspective, It's often a good idea to discard packets that arrive on interfaces we didn't expect them on. We can do this easily in SmartOS with ipadm set-prop:

# ipadm set-prop -p hostmodel=strong ipv4
# ipadm set-prop -p hostmodel=strong ipv6

There are some additional notes from some older Solaris guides that recommend disabling IP forwarding at a global level and enabling it on a per-interface basis. This is a good idea, but not easily done from within a SmartOS Zone. Instead, we will rely on IP Filter.

Define the Tunnel

Data-Link Admin will allow us to define our IP tunnel with its sub-command create-iptun. It takes a list of parameters, and then a tunnel name. Here is what we're interested in:

-t, --temporary Specifies if the link is temporary, ie: cleared on reboot. This is useful for testing.
-T type, --tunnel-type=type Specifies the type of tunnel to be created, must be one of:
ipv4 A point-to-point, IP-over-IPv4 tunnel. For IPv4 based tunnels.
ipv6 A point-to-point, IP-over-IPv6 tunnel. For IPv6 based tunnels.
-a {local|remote}=<address>,... Address parameters for this tunnel. For our case, we will need to define both a local and remote IP address here.
<iptun-link> The name of the IP tunnel interface.

Establishing the IP tunnel is as easy as running dladm on each end-point, referring to peer visible IP addresses on each side. If you're only testing, use the -t parameter, as the tunnel will be cleared on reboot.

alice # dladm create-iptun -T ipv4 -a local=1.2.3.4,remote=5.6.7.8 v4_bob0
bob # dladm create-iptun -T ipv4 -a local=5.6.7.8,remote=1.2.3.4 v4_alice0

We can verify our IP tunnels on either side by using dladm show-iptun

# dladm show-iptun
LINK            TYPE  FLAGS  LOCAL               REMOTE
v4_bob0         ipv4  --     98.232.133.100      173.164.88.233

If you need to delete an IP tunnel, it can be done using dladm delete-iptun <iptun-link>.

Address the Tunnel

After our IP tunnel has been established, we can use IP Admin to configure the interface for carrying IPv4 and/or IPv6 traffic. This is practical as both protocols can be protected by IPsec which is acting on the tunnel. Here's an overview of the parameters we're interested in:

-t, --temporary Specifies if the address is temporary, ie: cleared on reboot.
-T type Specifies the type of address to be created, we're only concerned with static.
-a {local|remote}=<address>,... Address parameters. For our case, we will need to define both a local and remote IP address here.
<addrobj> The name of the IP address object, in our case <iptun-link>/<protocol>.

This looks quite similar to the dladm line used previously, except we're referring to the internal addresses of the link. For the sake of illustration, we will be showing both IPv4 and IPv6 in our example. In the real world, use proper IPv6 addresses here.

alice # ipadm create-addr -T static -a local=10.0.0.1,remote=10.0.1.1 v4_bob0/v4
bob # ipadm create-addr -T static -a local=10.0.1.1,remote=10.0.0.1 v4_alice0/v4

We'll use addrconf for IPv6, as it's simpler overall

alice # ipadm create-addr -T addrconf v4_bob0/v6
bob # ipadm create-addr -T addrconf v4_alice0/v6

While the repeating of v4 on both ends of the addrobj may seem redundant, it makes it clear at a glance that this is a v4-over-v4 tunnel. At this stage, both end-points should be able to communicate with each other over the tunnel. Verify that the tunnel link is up using ipadm show-if <iptun-link> and ping.

alice # ipadm show-if v4_bob0
IFNAME     STATE    CURRENT      PERSISTENT
v4_bob0    ok       -mp-------46 -46
alice # ping 10.0.1.1
10.0.1.1 is alive
alice # ping 2001::1:1
2001::1:1 is alive

If you need to delete an IP address from the interface, use ipadm delete-addr <addrobj>. If you'd like to clear the entire interface, use ipadm delete-if <iptun-link>.

Route over the Tunnel

While there is connectivity between the routers using their private addresses, there is none between others on the private network segment (10.0.0.2 cant talk to anything in 10.0.1.0/24). We need to add persistent routes to each side so that they will forward traffic to the other side's private network segment. We achieve this with route.

For IPv4, use the following commands on each host.

alice # route -p add 10.0.1.0/24 10.0.1.1
bob # route -p add 10.0.0.0/24 10.0.0.1

For IPv6, we will need to know the configured local and remote IPv6 address used over the link. Find that with ipadm show-addr <addrobj> on either side.

alice # ipaddr show-addr v4_bob0/v6
ADDROBJ           TYPE     STATE        ADDR
v4_cb0/v6         addrconf ok           fe80::246b:bb2a->fe80::5567:e39b

Establish the persistent static route the same way we did with IPv4, with the additional -inet6 flag.

alice # route -p add -inet6 2001:470:24:c67::/64 fe80::5567:e39b
bob # route -p add -inet6 2001:470:24:c66::/64 fe80::246b:bb2a

You can check your established routes by using netstat -rn

Performance

While testing my configuration, I found a surprising performance penalty when using IPv4 over IPv4, as well as when using AES or Blowfish as compared to DES or 3DES. Adjusting the AH algorithm appeared to have little impact, with larger SHA2 hash fields having a barely detectable effect on performance.

I may revisit the topic of IPsec performance in the future with a proper test environment and proper data.