Stupid SmartOS Tricks

Home Router on SmartOS

Brian Ewell -

A SmartOS zone can serve quite nicely as a NATing router for a home network.  It works incredibly well on the HP N54L Microserver in concert with other VMs.

Note: This guide has been heavily influenced by the SmartOS wiki article on NAT using Etherstubs.

In addition to the SmartOS NAT guide, we will be installing several other services on our router, namely dnsmasq for DNS and DHCP service, as well as optionally haproxy for TCP port forwarding.

Identify your interfaces

Most home routers have two network interfaces, internal and external.  We're going to do the same.  Our internal network interface is the one we're going to connect to our local network and the one we will be NATing packets for.  Our external network interface should be the one directly connected to the wild Internet (or a cable modem), but shouldn't yet have anything configured on.

There are a few different ways this can be done in SmartOS.

The easy way

The simplest configuration is to have two different physical network interfaces in the global zone: one dedicated for admin and one for external.  For simplicity's sake, we will be using the admin network as our internal network.

For this approach, we will need to correctly identify and tag each interface.  Lets check them out on the global zone:

[root@gz ~]# dladm show-phys -m
LINK         SLOT     ADDRESS            INUSE CLIENT
e1000g0      primary  00:11:22:33:44:55  yes  e1000g0
bge0         primary  66:77:88:99:aa:bb   no  bge0

In the above example, e1000g0 is the admin interface while bge0 will become our external interface.  Check the global zone configuration to confirm that admin is 00:11:22:33:44:55 and add a directive to indicate that external is 66:77:88:99:aa:bb.

/usbkey/config

admin_nic=00:11:22:33:44:55
...
external_nic=66:77:88:99:aa:bb
...

If you don't want to reboot, use nictagadm to update the running system with the new tag.

[root@gz ~]# nictagadm add external 66:77:88:99:aa:bb

In the zone configuration below, you should use the following to describe the network interfaces:

...
  "nics": [
    {
      "nic_tag": "external",
      "interface": "net0",
      "ips": ["dhcp"],
      "primary": true,
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "interface": "net1",
      "ips": ["192.168.0.1/24"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    }
  ],
...

The minimalist way

If you have only one physical network interface in the global zone (and a VLAN capable switch), you can still use SmartOS as a home router.  In this case, we will use the untagged admin interface as the internal network and VLAN ID 2 packets as the external interface.

This approach takes no reconfiguration of the global zone, however, you will need to configure your switch to route external frames appropriately, which is beyond the scope of this post.

In the zone configuration below, you should use the following to describe the network interfaces:

...
  "nics": [
    {
      "nic_tag": "admin",
      "vlan_id": 2,
      "interface": "net0",
      "ips": ["dhcp"],
      "primary": true,
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "interface": "net1",
      "ips": ["192.168.0.1/24"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    }
  ],
...

The awesome way

While both of the above configurations are completely valid ways to configure the network, they're both lacking in the performance department.  Think about it: If you've got two or more network interfaces, why haven't you aggregated them yet?

This configuration requires VLAN and LACP capable switching fabric, which isn't common in the home.  The changes to the global zone configuration are as follows:

/usbkey/config:

...
aggr0_aggr=00:11:22:33:44:55,66:77:88:99:aa:bb
aggr0_lacp_mode=active

admin_nic=aggr0
...

While you certainly can manually configure this as well, I recommend rebooting the global zone to apply this.

Since you now effectively have one network interface, follow the minimalist way instructions above.

Configure Zone

Note: In the following VM manifests the default admin network is shared with the site's internal network.  This could be a bad idea if you're using any KVM VMs, as, by default, there is no authentication protecting VNC access to the console of the KVM instance.  This is a non-issue if you're exclusively using Solaris Zones or LX Branded Zones.

Specifying the same MAC address as your last router increases the likelihood of getting the same IP address from your residential ISP's DHCP service.  If you would like the same IP address, I recommend setting the MAC address to match that of your previous router.  If not, I recommend removing the line entirely (#This line).

Notice: If you decided to use one of the alternate network configurations discussed above, change out the nics block below with the one relevant for your configuration.

router.json

{
  "alias": "router",
  "hostname": "router",
  "brand": "joyent",
  "image_uuid": "088b97b0-e1a1-11e5-b895-9baa2086eb33",
  "max_physical_memory": 256,
  "quota": 10,
  "nics": [
    {
      "nic_tag": "external",
      "interface": "net0",
      "mac": "b0:b0:b0:b0:b0:b0",  # This line
      "ips": ["dhcp"],
      "primary": true,
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "interface": "net1",
      "ips": ["192.168.0.1/24"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    }
  ],
  "resolvers": [ "8.8.8.8", "8.8.4.4" ]
}

Create the router VM and login to it.

[root@00-11-22-33-44-55 ~]# vmadm create -f router.json
Successfully created VM 01234567-89ab-cdef-0123-456789abcdef
[root@00-11-22-33-44-55 ~]# zlogin 01234567-89ab-cdef-0123-456789abcdef

Configure Routing

Edit /etc/ipf/ipnat.conf so that it will NAT IPv4 packets routed through the VM.  I recommend uncommenting the rdr line if you're also planning on providing services through haproxy.

map net0 192.168.0.0/24 -> 0/32 proxy port ftp ftp/tcp
map net0 192.168.0.0/24 -> 0/32 portmap tcp/udp auto
map net0 192.168.0.0/24 -> 0/32

# Uncomment the following line if you want the router's external IP address accessible from the internal network
# rdr net1 <public ip>/32 -> 192.168.0.1

Turn on IPv4 forwarding & enable IPFilter

[root@router ~]# routeadm -u -e ipv4-forwarding
[root@router ~]# svcadm enable ipfilter

Verify that NAT is working properly

[root@router ~]# ipnat -l
List of active MAP/Redirect filters:
map net0 192.168.0.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
map net0 192.168.0.0/24 -> 0.0.0.0/32 portmap tcp/udp auto
map net0 192.168.0.0/24 -> 0.0.0.0/32

List of active sessions:

Install dnsmasq and haproxy

[root@router ~]# pkgin in dnsmasq haproxy
reading local summary...
processing local summary...
...
calculating dependencies... done.

nothing to upgrade.
2 packages to be installed (3003K to download, 7790K to install):

dnsmasq-2.75 haproxy-1.6.2

proceed ? [Y/n] y
downloading packages...
...

Configure dnsmasq

Edit /opt/local/etc/dnsmasq.conf.  Below are the highlights from the config file.

# Never forward plain names (without a dot or domain part)
domain-needed

# Never forward addresses in the non-routed address spaces.
bogus-priv

# We only service our internal network
interface=net1

# 200 ip addresses means lots of friends can come over
dhcp-range=192.168.0.50,192.168.0.250,6h

# Use /etc/ethers as a static allocation map
read-ethers

# Set the NTP time server addresses to 192.168.0.9 and 172.16.0.9
dhcp-option=option:ntp-server,192.168.0.9,172.16.0.9

# This should be the authoritative DHCP server on the network
dhcp-authoritative

# Cache of looked up domain names, don't go too crazy here
cache-size=1200

Create a directory at /var/cache and enable dnsmasq.

[root@router ~]# mkdir /var/cache
[root@router ~]# svcadm enable dnsmasq

Configure haproxy

While this package is only necessary if you're providing what should be publicly available services from your internal network, it's still useful to have on hand.  In our example, we're going to use haproxy on our router to forward traffic to another VM on our internal network that's running nginx.

Create /opt/local/etc/haproxy.cfg

global
  log localhost local0
  log localhost local1 notice
  maxconn 4096
  uid 99
  gid 99
  daemon

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option redispatch
  retries 3
  maxconn 1024
  timeout connect 60s
  timeout client 60s
  timeout server 60s

listen http
  bind *:80
  option forwardfor
  balance roundrobin
  server http01 192.168.0.10

Enable haproxy.

[root@router ~]# svcadm enable haproxy

Disable SSH Logins (optional)

Due to the elevated position of this VM in the network, I disable direct ssh access to it, instead preferring to access it through zlogin from the global zone.

[root@router ~]# svcadm disable ssh

Conclusion

In my next blog post, I will be describing how to establish a tunnel to Hurricane Electric's tunnelbroker.net, and provide ipv6 connectivity into the network.