Stupid SmartOS Tricks

2022 Home Router Refresh

Brian Ewell -

With the new hardware up and running, it's time to redeploy new zones. We're going to start with the router zone, and as it's almost 2022, I'm going to call this article my 2022 Home Router Refresh.

This article will overview rebuilding my zone based SmartOS router, and draw from three of my previous ones, specifically Home Router on SmartOS, Hurricane Electric on SmartOS, and Tunnels on SmartOS. If you'd like more of the theory behind why I'm doing what I'm doing, look to those articles, this is going to be much more nuts and bolts.

Lets dive in.

Overview

We will be deploying a router that will be responsible for routing between the isolated private subnets in my new network deployment, as well as providing common home network services such as dynamic host configuration protocol (DHCP), domain name service (DNS), and network address translation (NAT).

In addition, it will be acting as the terminal for a tunnel going out to Hurricane Electric for IPv6 connectivity, as well as IPSec tunnels going to production systems.

Interfaces

The local interfaces that we'll be working with were listed in the previous article, but for simplicity, are the following:

  • Public: net0: nic:admin, vlan_id:5, dhcp, autoconf
  • Infrastructure: net1: nic:admin, vlan_id:1, 172.22.1.0/27, IPv6/64
  • Embedded: net2: nic:admin, vlan_id:2, 172.22.1.32/27, IPv6/64
  • Internal: net3: etherstub:internal, 172.22.1.64/27, IPv6/64
  • External: net4: etherstub:external, 172.22.1.96/27, IPv6/64
  • Secure: net5: nic:admin, vlan_id:3, 172.22.1.128/26, IPv6/64
  • Guest: net6: nic:admin, vlan_id:4, 172.22.1.192/26, IPv6/64

The public interface may end up being directly connected at some point, the 1000BASE-T SFP module I have doesn't play well with the NDC in my compute node, so for now connecting to the public internet through my switch with a dedicated VLAN is how I'm going to proceed forward at this time. Whenever this network switches up to anything faster, there are still two free SFP+ interfaces on the back of the server to handle that interconnect. in the future.

Also, I may end up defining an additional interconnect for the 40GBE interface at the router or experiment with 802.1D bridging via dladm create-bridge, we will see when I get there.

Additionally, I will be creating the following tunnel interfaces:

  • Hurricane Electric: v4_he0
  • Production Tunnels: v4_ze0, v4_ze1

Note that I tend to use the v4_ prefix when referring to IPv4 based tunnels and the v6_ prefix when referring to IPv6 based tunnels.

Zone Manifest

This is the manifest that I used to create router zone:

{
  "image_uuid": "1d05e788-5409-11eb-b12f-037bd7fee4ee",
  "brand": "joyent",
  "alias": "router",
  "hostname": "router",
  "dns_domain": "ewellnet",
  "cpu_cap": 200,
  "max_physical_memory": 128,
  "quota": 10,
  "resolvers": [ "::1" ],
  "nics": [
    {
      "nic_tag": "admin",
      "vlan_id": 5,
      "interface": "net0",
      "ips": ["dhcp","addrconf"],
      "primary": true,
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "interface": "net1",
      "ips": ["172.22.1.1/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "vlan_id": 2,
      "interface": "net2",
      "ips": ["172.22.1.33/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "internal0",
      "interface": "net3",
      "ips": ["172.22.1.65/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "external0",
      "interface": "net4",
      "ips": ["172.22.1.97/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "vlan_id": 3,
      "interface": "net5",
      "ips": ["172.22.1.129/26","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "vlan_id": 4,
      "interface": "net6",
      "ips": ["172.22.1.193/26","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    }
  ]
}

Note that I'm setting a resolver of ::1 in the zone manifest simply because that is ultimately what it will be set to, you will want to temporarily alter /etc/resolv.conf until you have dnsmasq properly configured.

Creation command and login:

# vmadm create -f router.json
Successfully created VM f8510d07-3852-ebf0-83f3-e850f1ceb5fa
# zlogin f8510d07-3852-ebf0-83f3-e850f1ceb5fa

Configuring NAT & Routing

Edit /etc/ipf/ipnat.conf so that it will translate IPv4 packets routed from the private networks out onto the internet:

map net0 172.22.1.0/24 -> 0/32 proxy port ftp ftp/tcp
map net0 172.22.1.0/24 -> 0/32 portmap tcp/udp auto
map net0 172.22.1.0/24 -> 0/32

Enable IPFilter and IPv4 forwarding:

# svcadm enable ipfilter
# routeadm -ue ipv4-forwarding

Then lets verify that it's working properly:

# ipnat -l
List of active MAP/Redirect filters:
map net0 172.22.1.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
map net0 172.22.1.0/24 -> 0.0.0.0/32 portmap tcp/udp auto
map net0 172.22.1.0/24 -> 0.0.0.0/32

List of active sessions:

Installing Dnsmasq

I've come to really appreciate dnsmasq for how small and light it is, and yet how generally capable it is in a home network setting. Sure it isn't the fastest, and it doesn't do everything, but it's fantastic just how much it accomplishes while using minimal resources.

Installation and configuration is a breeze:

# pkgin in dnsmasq

And configuration isn't too difficult either:

/opt/local/etc/dnsmasq.conf:

# DNS specific configuration

# Set DNS cache size
cache-size=8192

# Never forward plain names (without a dot or domain part)
domain-needed

# Never forward addresses in the non-routed address spaces.
bogus-priv

# We don't want dnsmasq to poll resolv files for changes, we will manually notify it with a refresh
no-poll

# Upstream resolvers, use this so that /etc/resolv.conf can refer to localhost
resolv-file=/etc/resolv.upstream

# Domain for local services which should never be forwarded to external resolvers
local=/ewellnet/
domain=ewellnet

# Shared configuration

# Interfaces used for DHCP & DNS
interface=net1
interface=net2
interface=net3
interface=net4
interface=net5
interface=net6

# DHCP specific configuration

# DHCP ranges
dhcp-range=172.22.1.2,172.22.1.30,255.255.255.224,6h
dhcp-range=172.22.1.34,172.22.1.62,255.255.255.224,6h
dhcp-range=172.22.1.130,172.22.1.190,255.255.255.192,6h
dhcp-range=172.22.1.194,172.22.1.254,255.255.255.192,6h

# Read /etc/ethers for static DHCP allocations
read-ethers

# Set the NTP time server addresses to the interface's
dhcp-option=option:ntp-server

# This should be the authoritative DHCP server on the network
dhcp-authoritative

For any static assignments, set records in /etc/hosts and (optionally) /etc/ethers. In my case, the records are suffixed with .ewellnet.

Create the directory at /var/cache and enable the dnsmasq service:

# mkdir /var/cache
# svcadm enable dnsmasq

Local DHCP and DNS should now be up and running.

Secure SSH

Depending on your disposition, you may want to prevent password based logins through SSH or disable the service all together. I usually opt for the latter for home, though I'm sure that'll bite me in the future some time.

# svcadm disable ssh

Tunnel to Hurricane Electric

At the time of this writing, Ziply Fiber still hasn't rolled out IPv6 connectivity to end users. This is okay though, since we have Hurricane Electric.

Login to tunnelbroker.net to set and acquire your necessary credentials. Since these addresses are global, I will be making the following replacements for the below examples:

  • Server IPv4 address: 1.2.3.4
  • Client IPv4 address: 5.6.7.8
  • Server IPv6 address: 2001:470::1
  • Client IPv6 address: 2001:470::2

Run the following commands to bring a Hurricane Electric tunnel up and route default IPv6 traffic over it:

# dladm create-iptun -T ipv4 -a local=5.6.7.8,remote=1.2.3.4 v4_he0
# ipadm create-addr -T static -a local=2001:470::2,remote=2001:470::1 v4_he0/v6
# route -p add -inet6 default 2001:470::1

Test by pinging any well known IPv6 enabled domain:

# ping -A inet6 google.com
google.com is alive

Our router can now access the IPv6 Internet.

IPv6 connectivity to our router isn't terribly helpful on it's own; we'll want to be able to provide routing to our attached devices.

First up, we'll need to start forwarding IPv6 packets:

# routeadm -ue ipv6-forwarding

While I would normally make use of dnsmasq's IPv6 router advertisement features, the ndp daemon is normally already running with addrconf enabled on an interface, so we might as well just use the native tools here, especially since they'll manipulate the routes table on the router for us.

/etc/inet/ndpd.conf:

if net1 AdvSendAdvertisements 1
prefix 2001:470:????::/64 net1
if net2 AdvSendAdvertisements 1
prefix 2001:470:????:1::/64 net2
if net3 AdvSendAdvertisements 1
prefix 2001:470:????:2::/64 net3
if net4 AdvSendAdvertisements 1
prefix 2001:470:????:3::/64 net4
if net5 AdvSendAdvertisements 1
prefix 2001:470:????:4::/64 net5
if net6 AdvSendAdvertisements 1
prefix 2001:470:????:5::/64 net6

And then restart (not refresh) ndp.

# svcadm restart ndp

You should now see functional IPv6 connectivity on every stateless device on your network.

Establish IPSec Policy & Key Exchange

I also run IPSec tunnels to some production zones in a local datacenter. These tunnels will be called ze0 and ze1. First of all, we're going to need to set the encryption policy, specifically what encryption and authentications will be accepted over these tunnels:

/etc/inet/ipsecinit.conf:

{tunnel v4_ze0 negotiate tunnel}
  ipsec {encr_algs aes encr_auth_algs sha512 sa shared}
{tunnel v4_ze1 negotiate tunnel}
  ipsec {encr_algs aes encr_auth_algs sha512 sa shared}

After ensuring that the same settings are present on the remote endpoints, I restart ipsec policy:

# svcadm restart ipsec/policy

Next up is to configure IKE. Normally I would be fancy and use public keys, but pre-shared should be fine in this case:

/etc/inet/ike/config:

p1_lifetime_secs 7200
p1_nonce_len 40

p1_xform {
        auth_method preshared
        oakley_group 5
        auth_alg sha512
        encr_alg aes
}

p2_pfs 2

{
        label "v4_ze0"
        local_addr 1.2.3.4
        remote_addr 5.6.7.8
        p1_xform {
                auth_method preshared
                oakley_group 5
                auth_alg sha512
                encr_alg aes
        }
        p2_pfs 5
}
{
        label "v4_ze1"
        local_addr 1.2.3.4
        remote_addr 5.6.7.9
        p1_xform {
                auth_method preshared
                oakley_group 5
                auth_alg sha512
                encr_alg aes
        }
        p2_pfs 5
}

Generate some large random 64-byte keys for each tunnel. These are example keys only, do not use them production:

# openssl rand -hex 128
0f5451e4e0ad601f64dad77b9a83a0214dd4f460e07935936cfc7faf24c90f1c65609aaebc817ea17894748d999c94324d53a50ba18ce62c7683966674c3d50f93d1333440709f92facd1d10bcfde5b9ccc05e20adfa00a14825025068ba29cac04ba75b3b7e909dfddc01bd84eec2d56fa47a151cd447d0342f4194a910be9c
# openssl rand -hex 128
daa7fafa568601172b01591465e6365f90ad41eb6a778e7730f9ae5ebb9e05b9792fe288a1d7fa9950a2659433ad2b2f31087bc8e3c8ebbc80ef1c1d604847371cf4d1298ebb0b7fe4835b8bd13e969d700312a929ec8c83f63760a8a7b7439441b9e1ec27862c4294214df8ca03bd02eb62c9da6980108e27106c566b3ad7ea

Create some preshared secret files with these keys:

/etc/inet/secret/ike.preshared:

{
        localidtype IP
        localid 1.2.3.4
        remoteidtype IP
        remoteid 5.6.7.8
        key 0f5451e4e0ad601f64dad77b9a83a0214dd4f460e07935936cfc7faf24c90f1c65609aaebc817ea17894748d999c94324d53a50ba18ce62c7683966674c3d50f93d1333440709f92facd1d10bcfde5b9ccc05e20adfa00a14825025068ba29cac04ba75b3b7e909dfddc01bd84eec2d56fa47a151cd447d0342f4194a910be9c
}

{
        localidtype IP
        localid 1.2.3.4
        remoteidtype IP
        remoteid 5.6.7.9
        key daa7fafa568601172b01591465e6365f90ad41eb6a778e7730f9ae5ebb9e05b9792fe288a1d7fa9950a2659433ad2b2f31087bc8e3c8ebbc80ef1c1d604847371cf4d1298ebb0b7fe4835b8bd13e969d700312a929ec8c83f63760a8a7b7439441b9e1ec27862c4294214df8ca03bd02eb62c9da6980108e27106c566b3ad7ea
}

After the reciprocal configuration has been set on the remote side, enable or restart IKE:

# svcadm enable ipsec/ike

Establish IPSec Tunnels & Routing

With all of our IPSec policy and key exchange work out of the way, we can finally test it by establishing some tunnels. We'll do that with dladm:

# dladm create-iptun -T ipv4 -a local=1.2.3.4,remote=5.6.7.8 v4_ze0
# dladm create-iptun -T ipv4 -a local=1.2.3.4,remote=5.6.7.9 v4_ze1

Next up is to bring up a local interface for the tunnel, creating addresses will do this implicitly, so we'll just do that.

# ipadm create-addr -T static -a local=172.22.1.1,remote=10.0.0.1 v4_ze0/v4
# ipadm create-addr -T static -a local=172.22.1.1,remote=10.0.0.2 v4_ze1/v4

After performing the reciprocal steps on the remote host, you should be able to ping the remote side through the tunnel interface:

# ping 10.0.0.1
10.0.0.1 is alive

Next up is to set some static routes. Despite plumbing the interface for IPv6, in my case I'll only be actively using IPv4 over it immediately:

# route -p add 10.0.0.0/28 10.0.0.1
# route -p add 10.0.0.0/28 10.0.0.2

You should now be able to access the remote side of the IPSec tunnel through any subnet attached to this router.

Securing Everything

Next up is to ensure that everything is secure, and only communicating with what it should be. We will achieve this by altering the host model and setting up an inclusive firewall.

Normally I would perform this step earlier and then build into a secure environment. That can make troubleshooting quite difficult, so this time I built everything out first with the intention of locking it all down afterwards.

Strong Host Model

It's often a good idea to discard packets that arrive on unexpected interfaces. We can do this easily in SmartOS by changing the hostmodel to strong by using ipadm set-prop:

# ipadm set-prop -p hostmodel=strong ipv4
# ipadm set-prop -p hostmodel=strong ipv6

This will ensure that any packets that come in through an interface that's doesn't correlate with the interfaces in the routing table are dropped.

Firewall

The next thing to do is to ensure that the router is allowing only the correct traffic flows between subnets using IP Filter. We're going to enact the following policies:

  • IPv6 Traffic from the Internet can reach external services 172.22.1.96/27.
  • Traffic from infrastructure 172.22.1.0/27 can reach the Internet, internal services 172.22.1.64/27, external services 172.22.1.96/27 and the IPSec tunnels v4_ze0 and v4_ze1.
  • Traffic from embedded 172.22.1.32/27 can access internal services 172.22.1.64/27.
  • Traffic from internal services 172.22.1.64/27 can access the Internet, infrastructure 172.22.1.0/27, embedded 172.22.1.32/27, external services 172.22.1.96/27 and secure 172.22.1.128/26.
  • Traffic from external services 172.22.1.96/27 can reach the Internet.
  • Traffic from secure 172.22.1.128/26 can reach the Internet, internal services 172.22.1.64/27 and external services 172.22.1.96/27.
  • Traffic from guest 172.22.1.192/26 can reach the Internet and external services 172.22.1.96/27.
  • Traffic from the work IPSec tunnels v4_ze0 and v4_ze1 can reach infrastructure 172.22.1.0/27.

Note: For the sake of this list, "The Internet" includes IPv4 connectivity via NAT through net0 and IPv6 connectivity via the v4_he0 Hurricane Electric tunnel. Also the IPSec connection between infrastructure and remote infrastructure will likely be constrained further to only allow for administrative console access to the remote network. Next up is to implement these policies in configuration:

/etc/ipf/ipf.conf:

# Traffic from infrastructure (net1) [172.22.1.0/27]
block in on net1 from any to 172.16.0.0/12
pass in quick on net1 from 172.22.1.0/27 to 172.22.1.64/26
pass in quick on net1 from 172.22.1.0/27 to 172.22.1.0/27

# Traffic from embedded (net2) [172.22.1.32/27]
block in on net2 all
pass in quick on net2 from 172.22.1.32/27 to 172.22.1.64/27
pass in quick on net2 from 172.22.1.32/27 to 172.22.1.32/27

# Traffic from internal (net3) [172.22.1.64/27]
block in on net3 from any to 172.16.0.0/12
pass in quick on net3 from 172.22.1.64/27 to 172.22.1.128/26
pass in quick on net3 from 172.22.1.64/27 to 172.22.1.0/25

# Traffic from external (net4) [172.22.1.96/27]
block in on net4 from any to 172.16.0.0/12
pass in quick on net4 from 172.22.1.96/27 to 172.22.1.96/27

block out on net4 from 172.16.0.0/12 to any
pass out quick on net4 proto tcp from 172.16.0.0/12 to 172.22.1.96/27 flags S keep state
pass out quick on net4 proto udp from 172.16.0.0/12 to 172.22.1.96/27 keep state
pass out quick on net4 proto icmp from 172.16.0.0/12 to 172.22.1.96/27 icmp-type echo keep state
pass out quick on net4 from 172.22.1.96/27 to 172.22.1.96/27

# Traffic from secure (net5) [172.22.1.128/26]
block in on net5 from any to 172.16.0.0/12
pass in quick on net5 from 172.22.1.128/26 to 172.22.1.64/26
pass in quick on net5 from 172.22.1.128/26 to 172.22.1.128/26

# Traffic from guest (net6) [172.22.1.192/26]
block in on net6 from any to 172.16.0.0/12
pass in quick on net6 from 172.22.1.192/26 to 172.22.1.96/27
pass in quick on net6 from 172.22.1.192/26 to 172.22.1.192/26

# Traffic from IPsec tunnels (v4_ze0,v4_ze1) [192.168.255.224/27]
block in on v4_ze0 all
pass in quick on v4_ze0 from 10.0.0.0/27 to 172.22.1.0/27

block in on v4_ze1 all
pass in quick on v4_ze1 from 10.0.0.0/27 to 172.22.1.0/27

/etc/ipf/ipf6.conf:

# Traffic from public (he0) [::/0]
block in on he0 from any to 2001:470::/48
pass in quick on he0 from any to 2001:470:0:3::/64

block out on he0 from 2001:470::/48 to any
pass out quick on he0 from 2001:470:0:3::/64 to any
pass out quick on he0 proto tcp from 2001:470::/48 to any flags S keep state
pass out quick on he0 proto udp from 2001:470::/48 to any keep state
#pass out quick on he0 proto icmp from 2001:470::/48 to any icmp-type echo keep state

# Traffic from infrastructure (net1) [2001:470::/64]
block in on net1 from any to 2001:470::/48
pass in quick on net1 from 2001:470::/64 to 2001:470:0:2::/63
pass in quick on net1 from 2001:470::/64 to 2001:470::/64

# Traffic from embedded (net2) [2001:470:0:1::/64]
block in on net2 all
pass in quick on net2 from 2001:470:0:1::/64 to 2001:470:0:2::/64
pass in quick on net2 from 2001:470:0:1::/64 to 2001:470:0:1::/64

# Traffic from internal (net3) [2001:470:0:2::/64]
block in on net3 from any to 2001:470::/48
pass in quick on net3 from 2001:470:0:2::/64 to 2001:470:0:4::/64
pass in quick on net3 from 2001:470:0:2::/64 to 2001:470::/62

# Traffic from external (net4) [2001:470:0:3::/64]
block in on net4 from any to 2001:470::/48
pass in quick on net4 from 2001:470:0:3::/64 to 2001:470:0:3::/64

block out on net4 from 2001:470::/48 to any
pass out quick on net4 proto tcp from 2001:470::/48 to 2001:470:0:3::/64 flags S keep state
pass out quick on net4 proto udp from 2001:470::/48 to 2001:470:0:3::/64 keep state
#pass out quick on net4 proto icmp from 2001:470::/48 to 2001:470:0:3::/64 icmp-type echo keep state
pass out quick on net4 from 2001:470:0:3::/64 to 2001:470:0:3::/64

# Traffic from secure (net5) [2001:470:0:4::/64]
block in on net5 from any to 2001:470:0::/48
pass in quick on net5 from 2001:470:0:4::/64 to 2001:470:0:2::/63
pass in quick on net5 from 2001:470:0:4::/64 to 2001:470:0:4::/64

# Traffic from guest (net6) [2001:470:0:5::/64]
block in on net6 from any to 2001:470:0::/48
pass in quick on net6 from 2001:470:0:5::/64 to 2001:470:0:3::/64
pass in quick on net6 from 2001:470:0:5::/64 to 2001:470:0:5::/64

# Traffic from IPsec tunnels (v4_ze0,v4_ze1)
block in on v4_ze0 all
block in on v4_ze1 all

Reload ipfilter to enable those rules:

# svcadm refresh ipfilter

This enables a stateful IPv4 firewall around external to prevent it from accessing the other network segments without preventing them from accessing it, as well, the IPv6 firewall rules enable two stateful layers, one around the external network segment and another one protecting all networks except for the external segment from access.

Please note: Using a firewall like IP Filter severely limits network performance. While I have taken steps above to mitigate the impact by reducing the firewall rules to their simplest forms, I will be investigating further possible steps to take in the future.

Forwarding Traffic

There are some additional exceptions that need to be set for various services running in this network, namely:

  • Forward NTP traffic udp/123 directed to any of the non-routable router addresses to the SmartOS compute node.
  • Forward Plex traffic tcp/31400 to the Plex Media Server zone.
  • I'm sure many more.

These forwarding rules can be added to /etc/ipf/ipnat.conf along with the map rules we added before:

/etc/ipf/ipnat.conf:

map net0 172.22.1.0/24 -> 0/32 proxy port ftp ftp/tcp
map net0 172.22.1.0/24 -> 0/32 portmap tcp/udp auto
map net0 172.22.1.0/24 -> 0/32

# Plex redirection (public->media)
rdr net0 0.0.0.0/0 port 32400 -> 172.22.1.70 port 32400 tcp

# NTP server redirection (all->hv-1)
rdr net1 172.22.1.1/32   port 123 -> 172.22.1.8 port 123 udp
rdr net2 172.22.1.33/32  port 123 -> 172.22.1.8 port 123 udp
rdr net3 172.22.1.65/32  port 123 -> 172.22.1.8 port 123 udp
rdr net4 172.22.1.97/32  port 123 -> 172.22.1.8 port 123 udp
rdr net5 172.22.1.129/32 port 123 -> 172.22.1.8 port 123 udp
rdr net6 172.22.1.193/32 port 123 -> 172.22.1.8 port 123 udp

IP Filter will need to be refreshed once more:

# svcadm refresh ipfilter

Conclusion

This should be a pretty solid start to a router zone, though I may end up changing some things around, namely:

  • Some of the high throughput communication paths may end up requiring some zones to be moved to different networks to reduce the traffic being filtered through IP Filter and hopefully improve network performance.
  • Adding IPSec policies for mobile devices such as Laptops and Smart Phones to access network resources away from home.
  • Adding QoS for IP telephony traffic.

But for now, this should do.