One of the most valuable things a programmer can do with their time is to continue to learn new things.

Not only does this help to maintain the mental flexibility that is critical for tackling problems in ever-changing environments, but is simply makes life more fun.

Half a lifetime ago, I would pick up programming languages at the pace of one every few months. But ever since learning Ruby in the mid 2000s, I just haven't felt the need to anymore. It's almost as if every language I've tried to pick up since has been a disappointment in one way or another: Ruby was the hammer to all of life's nails.

Ruby is certainly not without it's faults. Without turning this into a litany of accusations against the language that I terribly appreciate, I do miss my earlier days of learning languages just for the sake of learning languages.

So that what I'm going to do this year: My New Year's Resolution for 2022 is to master four programming languages that I've at most written a "Hello World" in before this year.

Definition of Mastery

When I say master, I don't mean achieving an equivalent level of fluency to a language in which I have nearly two decades of experience using. That would be ridiculous.

Instead I mean intrinsically understanding and applying the fundamentals of what makes that language stand out, both at an intellectual and reflexive level. And the best way to really showcase that would be to write production software in that language.

So that's what I'm going to do by the end of the year: Four different capstone projects written in four different languages. While I'd like to ensure that they're FOSS and available on my github, that may or may not be possible, depending on what I end up working on.

Here's what I chose to explore:

Elixir

A dynamic, functional language, Elixir leverages the BEAM/OTP run-time environment (also utilized by Erlang) to provide a robust low-latency, vertically and horizontally distributed, fault-tolerant platform upon which to run your code.

I have received multiple recommendations to check out Elixir, along with the Ecto and Phoenix frameworks and after finally looking into it, I have to admit I'm quite excited to start learning this. The biggest weakness I perceive of Rails is it's lack of scalability, which appears to be a complete non-issue with Phoenix.

As far as a specific capstone goes, I will likely utilize Phoenix to write a web-based something or other, basically one of the following:

• A private internet registry (think ICANN) management system to allow for small groups of users to manage and organize address allocations for private Internets (IETF RFC1918).
• The management system for one of the handful of SaaS/PaaS startups I'm involved with.

Dart

Optimized as a client-development oriented language, Dart is a class-based object-oriented garbage-collected language with a C style syntax that compiles to either native or JavaScript, allowing for a unified client development pathway to be applied to multiple platforms (Web, Mobile, Desktop).

As with Elixir, my interest in Dart is primarily focused on a single framework.

Flutter.

I've never really been that excited for single-page web-applications until now, and while I would like to learn about how Phoenix handles templating and HTML, I would also like to completely avoid that all together and just pass structured data to a much more solidly implemented front-end.

As far as projects are concerned, any web-backend I write with Elixir/Phoenix will likely be accompanied by a front-end written in Dart/Flutter.

Erlang

The original language that ran on BEAM/OTP, Erlang compiles to effectively equivalent bytecode that Elixir does, and they can call on each other as well, making learning Erlang an excellent choice after learning Elixir and vice versa.

While I could easily fold my Erlang capstone into the same Phoenix based capstone I proposed for learning Elixir, I'd much rather push myself to do something a bit grander with it. No idea what it'll be specifically, but most likely something that plays well to Erlangs strengths, so likely something that can take advantage of low-latency, vertical and horizontal scalability and distribution, along with robust fault-tolerance:

• Reimplementation of some existing server implementation that doesn't currently scale well.

That's all I'm going to say for now, as I still have plenty of time to figure this out.

Rust

A modern systems-level language that is quite possibly the successor to C, Rust is a high performance multi-paradigm language designed for safety, specifically around concurrent memory access. Rust balance of high and low level features makes it very well suited for a lot of different use cases, and it has already seen adoption in the desktop application, CLI application, system application and operating system spaces.

Beyond functioning quite capably on its own, Rust can also be used to implement Erlang Native Implemented Functions (NIFs) thanks to the Rustler Mix package, making it perfectly suitable for writing procedures that will be used in the Erlang OTP.

As with Erlang, I'm not sure exactly what my specific capstone project will be with Rust, though due to the NIF compatibility, I will likely attempt to combine projects for both of these languages.

Honorable Mentions

There are a few other languages that I would absolutely love to pick up this year, but don't want to risk potentially crowding capstone project time by committing to learning them at this time:

• Go. A statically typed compiled garbage-collected general-purpose language, inspired by C, similar to Rust in some ways and different in others, that's well suited to large concurrent systems.
• Crystal. A statically typed compiled garbage-collected general-purpose object-oriented language syntactically inspired by Ruby and with concurrency inspirations from Go.

If you use package management software commonly across two or more systems, network package caching may be of benefit to you.  In this article, we will setup a package caching solution to service multiple package management systems simultaneously.

Methodology

While caching packages locally in the file system is already common practice with most package management systems, it's difficult to share this local cache with other nearby systems without possible security implications and things getting prohibitively complicated.

Instead, redirecting requests to a local-network intermediary that provides the same interface as the original repository while also managing its own cache is the preferred solution: Most package management systems use HTTP or HTTPS and refer to their repositories via domain name or URL, and many can be reconfigured to access their repositories through alternate means. In cases where this is not trivial, split-horizon DNS records can be used instead.

This technique is used extensively by projects such as Lancache for game management systems but can also easily be applied to other software package management systems such as Pkgsrc, APT, Yum, etc.

Architecture

There will be two independent services involved in implementing this solution: a DNS resolver and a caching HTTP server.

While I will be including notes on the hostnames that need to be configured, the specifics of implementing DNS will be outside the scope of this article. I'm assuming that you have access to your own DNS resolver such as dnsmasq or powerdns-recursor and know how to configure it for split-horizon DNS.

While I used to prefer redirecting the original domain names, I now change the tld to cache.ewellnet instead, for example:

• pkgsrc.joyent.com becomes pkgsrc.joyent.cache.ewellnet.
• archive.ubuntu.com becomes archive.ubuntu.cache.ewellnet.

While this does require the reconfiguration of each host using the cache, overall it's a cleaner approach and allows hosts to easily bypass the cache if need be. Additionally, my search directive under /etc/resolv.conf is set to ewellnet, so I can use the relatively shorter pkgsrc.joyent.cache or archive.ubuntu.cache instead.

For our caching HTTP server, we will be using Nginx within a single zone. It's incredibly high performance, easy to configure, and does its job very efficiently. This should be perfectly performant for most situations, but may be unsuitable in extreme circumstances. This can be remedied by utilizing a cache cluster, which is well outside the scope of today's article.

I will be using a common cache optimized configuration with additional configuration files for each supported package management system. Additionally for this article, I will setup a common package cache for all management systems, but the reader can just as easily partition their caches for sets of management systems.

It is also best that the cache zone be dual-stack. This benefits any IPv6 only hosts on your network that may otherwise not have access to some repositories.

Zone Manifest

We want to ensure that the zone containing the cache will have enough processing power and memory to not be starved, and enough storage space to handle all of the packages you would like to cache.

We will be using the following for our example:

{
  "image_uuid": "1d05e788-5409-11eb-b12f-037bd7fee4ee",
  "brand": "joyent",
  "alias": "cache",
  "hostname": "cache",
  "dns_domain": "ewellnet",
  "cpu_cap": 200,
  "max_physical_memory": 256,
  "quota": 1024,
  "delegate_dataset": true,
  "resolvers": [ "172.22.1.97" ],
  "nics": [{
    "nic_tag": "external0",
    "ips": [ "172.22.1.98/27", "addrconf", "2001:470::98/64" ],
    "gateways": [ "172.22.1.97" ],
    "primary": true
  }]
}

Create it and login.

# vmadm create -f cache.json
Successfully created VM 49ee67da-9e3c-c20a-d925-aa2aa284f95d
# zlogin 49ee67da-9e3c-c20a-d925-aa2aa284f95d

After doing any zone cleanup that you prefer, ensure that a dataset exists to handle your cache. I will often times set a quota to ensure a hard upper limit to size slightly beyond the limit that will be set later in Nginx.

# zfs create -o quota=982G -o mountpoint=/var/www/cache zones/<UUID>/data/cache

Installing & Configuring Nginx

Next, we're going to install Nginx.

# pkgin -y install nginx

Clear out all of the Nginx configuration files; we'll be doing something very specific.

# rm -rv /opt/local/etc/nginx/*

Instead of the default nginx.conf, we'll be using this one, optimized for caching:

/opt/local/etc/nginx/nginx.conf:

user www www;
worker_processes 2;

events { worker_connections 1024; multi_accept on; }

http {
  default_type  application/octet-stream;

  allow 172.22.1.0/24;
  allow 2001:470::/48;
  deny  all;

  server_tokens off;
  tcp_nopush  on;
  sendfile  on;
  gzip    on;

  proxy_buffering          on;
  proxy_buffers            32 8k;
  proxy_cache              default_cache;
  proxy_cache_lock         on;
  proxy_cache_lock_age     5m;
  proxy_cache_lock_timeout 30s;
  proxy_cache_path         /var/www/cache levels=2:2 use_temp_path=on keys_zone=default_cache:128m inactive=4y max_size=980G;
  proxy_cache_revalidate   on;
  proxy_cache_use_stale    error timeout invalid_header updating http_500 http_502 http_503 http_504 http_403 http_404 http_429;
  proxy_cache_valid        1d;
  proxy_cache_valid        any 1m;
  proxy_http_version       1.1;
  proxy_ignore_headers     X-Accel-Redirect X-Accel-Expires X-Accel-Limit-Rate X-Accel-Buffering X-Accel-Charset Expires Cache-Control Vary;
  proxy_temp_path          /var/www/cache/tmp 1;

  proxy_ssl_ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH;
  proxy_ssl_protocols TLSv1.2;
  proxy_ssl_server_name on;
  proxy_ssl_session_reuse on;
  proxy_ssl_verify_depth  4;

  log_format cache '$remote_addr - $remote_user [$time_local] $status $upstream_cache_status $server_name $request_time [$connection:$connection_requests] "$request_method $scheme://$http_host$request_uri $server_protocol" $body_bytes_sent "$http_referer" "$http_user_agent"';
  access_log  /var/log/nginx/cache.log cache;

  server {
    listen [::] default_server ipv6only=off;
    server_name default;
    return 421;
  }

  include cache/*.cache;
}

A quick breakdown of configuration parameters that you will want to tweak:

• worker_processes will limit Nginx to spawning that number of worker processes, this is important on compute nodes with high core counts, and should match your cpu_cap/100. In my case, with a cpu_cap of 200, this value should be 2.
• The allow and deny directives will allow you to whitelist or blacklist IP prefixes. Use allow to whitelist your local network prefixes and then deny all to prevent all others from accessing your cache. In my case, my local network prefixes are 172.22.1.0/24 and 2001:470::/48.

At this point you can enable Nginx to confirm it's happy with its current configuration before adding specific management systems to the cache from the below sections:

# svcadm enable nginx

If you'd like to take a moment here, I recommend reading the Nginx documentation on proxy directives for a more complete understanding of what all has been specified here.

Also, if you want to make full use of all of the below sub-sections of this article, I recommend recompiling Nginx from a build host with sub and cache-purge enabled.

Package Caching

This section will start by demonstrating how to configure pkgsrc package caching and then illustrate the performance differences of caching vs not caching. It will then list additional configured package caching configurations for other operating systems and environments.

Each package management description will include any local changes that need to be made to the system being configured for caching, the DNS records that need to be configured along with the relevant Nginx configuration files. Please replace any references to ewellnet with your site's local domain name, or just omit that domain name entirely, your choice.

I also recommend keeping a terminal open to tail /var/log/nginx/cache.log while configuring this, to monitor for cache misses, hits and revalidations.

Pkgsrc (SmartOS)

The pkgsrc binary package manager is responsible for managing software packages in SmartOS zones. Like most distribution software package managers, it downloads, validates and installs in that order.

Lets get an idea of what sort of performance we can expect out of non-cached pkgsrc. Testing pkgin update in a freshly provisioned zone gives us the following results:

# time pkgin update
reading local summary...
processing local summary...
processing remote summary (https://pkgsrc.joyent.com/packages/SmartOS/2020Q4/x86_64/All)...
pkg_summary.xz                                   100% 2374KB 791.4KB/s   00:03

real    0m6.941s
user    0m3.294s
sys     0m0.318s

That's not bad. Lets see how long it'll take to download a full upgrade:

# pkgin clean
# time pkgin -dy upgrade
calculating dependencies...done.

17 packages to download:
  wget-1.20.3nb10 sudo-1.9.6p1 rsyslog-8.38.0nb10 python38-3.8.6nb1
  postfix-3.5.10 pkgin-20.12.1 pkg_install-20201218 openssl-1.1.1l
  libssh2-1.9.0nb1 libarchive-3.4.3 mozilla-rootcerts-1.0.20201102
  openldap-client-2.4.56 http-parser-2.9.4 npm-6.14.11 nodejs-14.16.1
  nghttp2-1.42.0nb1 curl-7.75.0
74M to download

wget-1.20.3nb10.tgz                     100% 1244KB   1.2MB/s   00:00
sudo-1.9.6p1.tgz                        100% 1866KB   1.8MB/s   00:01
rsyslog-8.38.0nb10.tgz                  100% 1165KB   1.1MB/s   00:01
python38-3.8.6nb1.tgz                   100%   27MB   3.9MB/s   00:07
postfix-3.5.10.tgz                      100% 2175KB   1.1MB/s   00:02
pkgin-20.12.1.tgz                       100%   98KB  98.4KB/s   00:01
pkg_install-20201218.tgz                100% 9201KB   3.0MB/s   00:03
openssl-1.1.1l.tgz                      100% 6488KB   2.1MB/s   00:03
libssh2-1.9.0nb1.tgz                    100%  389KB 389.0KB/s   00:01
libarchive-3.4.3.tgz                    100%  979KB 979.3KB/s   00:01
mozilla-rootcerts-1.0.20201102.tgz      100%  573KB 573.1KB/s   00:00
openldap-client-2.4.56.tgz              100% 1438KB   1.4MB/s   00:00
http-parser-2.9.4.tgz                   100%   48KB  47.8KB/s   00:00
npm-6.14.11.tgz                         100% 5352KB   2.6MB/s   00:02
nodejs-14.16.1.tgz                      100%   14MB   3.6MB/s   00:04
nghttp2-1.42.0nb1.tgz                   100%  296KB 295.9KB/s   00:01
curl-7.75.0.tgz                         100% 1664KB   1.6MB/s   00:01

real    0m40.083s
user    0m0.630s
sys     0m0.306s

40 seconds. Not horrible, but surely we can improve upon this.

DNS Records

Ensure that the following DNS records are configured:

• pkgsrc.joyent.cache.ewellnet points to your cache server.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/joyent.cache:

upstream pkgsrc.joyent.com {
  server pkgsrc.joyent.com:443;
  keepalive 2;
}

server {
  listen [::];
  server_name pkgsrc.joyent.cache pkgsrc.joyent.cache.ewellnet;

  location ^~ pkg_summary.(bz2|gz|xz)$ {
    proxy_pass https://pkgsrc.joyent.com;
    proxy_cache_valid any 1h;
  }

  location / {
    proxy_pass https://pkgsrc.joyent.com;
  }
}

This configuration ensures that requests for normal packages will only be revalidated after the default duration has passed, but requests for the package summary will be revalidated every hour.

Refresh Nginx to enable pkgsrc package caching:

# svcadm refresh nginx

Client Configuration

Pkgsrc clients need to be configured to make use of the local cache. This can be done by altering the repository URL within the configuration of each client:

/opt/local/etc/pkgin/repositories.conf:

...
http://pkgsrc.joyent.cache/packages/SmartOS/2020Q4/x86_64/All

Run pkgin update to confirm that it's able to acquire the repository package summary:

# time pkgin update
cleaning database from https://pkgsrc.joyent.com/packages/SmartOS/2020Q4/x86_64/All entries...
reading local summary...
processing local summary...
processing remote summary (http://pkgsrc.joyent.cache/packages/SmartOS/2020Q4/x86_64/All)...
pkg_summary.xz                          100% 2374KB 593.6KB/s   00:04

real    0m7.952s
user    0m4.332s
sys     0m0.581s

Note that the time is slightly longer, this is due to the clearing out of the previous database and due to the fact that we didn't have the summary cached.  We can also now re-test downloading an upgrade from the cache:

# pkgin clean
# time pkgin -dy upgrade
calculating dependencies...done.

17 packages to download:
  wget-1.20.3nb10 sudo-1.9.6p1 rsyslog-8.38.0nb10 python38-3.8.6nb1
  postfix-3.5.10 pkgin-20.12.1 pkg_install-20201218 openssl-1.1.1l
  libssh2-1.9.0nb1 libarchive-3.4.3 mozilla-rootcerts-1.0.20201102
  openldap-client-2.4.56 http-parser-2.9.4 npm-6.14.11 nodejs-14.16.1
  nghttp2-1.42.0nb1 curl-7.75.0
74M to download

wget-1.20.3nb10.tgz                     100% 1244KB   1.2MB/s   00:01
sudo-1.9.6p1.tgz                        100% 1866KB 933.2KB/s   00:02
rsyslog-8.38.0nb10.tgz                  100% 1165KB   1.1MB/s   00:01
python38-3.8.6nb1.tgz                   100%   27MB   3.9MB/s   00:07
postfix-3.5.10.tgz                      100% 2175KB   2.1MB/s   00:01
pkgin-20.12.1.tgz                       100%   98KB  98.4KB/s   00:00
pkg_install-20201218.tgz                100% 9201KB   3.0MB/s   00:03
openssl-1.1.1l.tgz                      100% 6488KB   2.1MB/s   00:03
libssh2-1.9.0nb1.tgz                    100%  389KB 389.0KB/s   00:00
libarchive-3.4.3.tgz                    100%  979KB 979.3KB/s   00:01
mozilla-rootcerts-1.0.20201102.tgz      100%  573KB 573.1KB/s   00:01
openldap-client-2.4.56.tgz              100% 1438KB   1.4MB/s   00:01
http-parser-2.9.4.tgz                   100%   48KB  47.8KB/s   00:00
npm-6.14.11.tgz                         100% 5352KB   1.7MB/s   00:03
nodejs-14.16.1.tgz                      100%   14MB   4.7MB/s   00:03
nghttp2-1.42.0nb1.tgz                   100%  296KB 295.9KB/s   00:01
curl-7.75.0.tgz                         100% 1664KB   1.6MB/s   00:01

real    0m32.785s
user    0m0.553s
sys     0m0.370s

Not too much quicker, but again, none of this data had been cached yet. Switch the source repository back, re-update, and switch back to the cache again to test it's real performance:

-- Switched back to upstream
# pkgin update
-- Switched back to cache
# time pkgin update
cleaning database from https://pkgsrc.joyent.com/packages/SmartOS/2020Q4/x86_64/All entries...
reading local summary...
processing local summary...
processing remote summary (http://pkgsrc.joyent.cache/packages/SmartOS/2020Q4/x86_64/All)...
pkg_summary.xz                          100% 2374KB 791.4KB/s   00:03

real    0m6.833s
user    0m3.952s
sys     0m0.555s

Not a whole lot to be excited about with pkgsrc update. Lets check on upgrade:

# pkgin clean
# time pkgin -dy upgrade
calculating dependencies...done.

17 packages to download:
  wget-1.20.3nb10 sudo-1.9.6p1 rsyslog-8.38.0nb10 python38-3.8.6nb1
  postfix-3.5.10 pkgin-20.12.1 pkg_install-20201218 openssl-1.1.1l
  libssh2-1.9.0nb1 libarchive-3.4.3 mozilla-rootcerts-1.0.20201102
  openldap-client-2.4.56 http-parser-2.9.4 npm-6.14.11 nodejs-14.16.1
  nghttp2-1.42.0nb1 curl-7.75.0
74M to download

wget-1.20.3nb10.tgz                     100% 1244KB   1.2MB/s   00:00
sudo-1.9.6p1.tgz                        100% 1866KB   1.8MB/s   00:00
rsyslog-8.38.0nb10.tgz                  100% 1165KB   1.1MB/s   00:00
python38-3.8.6nb1.tgz                   100%   27MB  27.3MB/s   00:00
postfix-3.5.10.tgz                      100% 2175KB   2.1MB/s   00:00
pkgin-20.12.1.tgz                       100%   98KB  98.4KB/s   00:00
pkg_install-20201218.tgz                100% 9201KB   9.0MB/s   00:00
openssl-1.1.1l.tgz                      100% 6488KB   6.3MB/s   00:00
libssh2-1.9.0nb1.tgz                    100%  389KB 389.0KB/s   00:00
libarchive-3.4.3.tgz                    100%  979KB 979.3KB/s   00:00
mozilla-rootcerts-1.0.20201102.tgz      100%  573KB 573.1KB/s   00:00
openldap-client-2.4.56.tgz              100% 1438KB   1.4MB/s   00:00
http-parser-2.9.4.tgz                   100%   48KB  47.8KB/s   00:00
npm-6.14.11.tgz                         100% 5352KB   5.2MB/s   00:00
nodejs-14.16.1.tgz                      100%   14MB  14.2MB/s   00:01
nghttp2-1.42.0nb1.tgz                   100%  296KB 295.9KB/s   00:00
curl-7.75.0.tgz                         100% 1664KB   1.6MB/s   00:00

real    0m1.322s
user    0m0.524s
sys     0m0.342s

That's more like it! We can see that package downloads were served from the cache as well with the HIT lines in our logs rather than MISS lines.

Apt (Ubuntu)

The apt package management system is responsible for managing software packages in Ubuntu based Linux distributions. Like most distribution software package managers, it downloads, validates and installs in that order.

DNS Records

Ensure that the following DNS records are configured:

• apt.ubuntu.cache.ewellnet points to your cache server.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/ubuntu.cache:

upstream archive.ubuntu.com {
  server archive.ubuntu.com;
  keepalive 2;
}

server {
  listen [::];
  server_name apt.ubuntu.cache apt.ubuntu.cache.ewellnet;

  location ^~ deb$ {
    proxy_pass http://archive.ubuntu.com;
  }

  location / {
    proxy_pass http://archive.ubuntu.com;
    proxy_cache_valid any 1h;
  }
}

This configuration ensures that requests for normal packages will only be revalidated after the default duration has passed, but requests for the package summary will be revalidated every hour.

Refresh Nginx to enable apt package caching:

# svcadm refresh nginx

Client Configuration

Ubuntu clients need to be configured to make use of the local cache. This can be done by altering the repository URLs within the configuration of each client:

/etc/apt/sources.list:

...
deb http://apt.ubuntu.cache/ubuntu/ focal main restricted
deb http://apt.ubuntu.cache/ubuntu/ focal-updates main restricted
deb http://apt.ubuntu.cache/ubuntu/ focal universe
deb http://apt.ubuntu.cache/ubuntu/ focal-updates universe
deb http://apt.ubuntu.cache/ubuntu/ focal multiverse
deb http://apt.ubuntu.cache/ubuntu/ focal-updates multiverse
deb http://apt.ubuntu.cache/ubuntu/ focal-backports main restricted universe multiverse
deb http://apt.ubuntu.cache/ubuntu/ focal-security main restricted
deb http://apt.ubuntu.cache/ubuntu/ focal-security universe
deb http://apt.ubuntu.cache/ubuntu/ focal-security multiverse

Basically change every reference from archive.ubuntu.com or security.ubuntu.com to apt.ubuntu.cache. Run apt update to confirm that it's able to acquire the repository package summaries and enjoy!

Apt (Debian)

Like Ubuntu, the apt package management system is responsible for managing software packages in Debian based Linux distributions. Like most distribution software package managers, it downloads, validates and installs in that order.

DNS Records

Ensure that the following DNS records are configured:

• apt.debian.cache.ewellnet points to your cache server.
• security.debian.cache.ewellnet points to your cache server. Note that unlike Ubuntu, Debian does handle security updates separately.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/debian.cache:

upstream cdn-fastly.deb.debian.org {
  server cdn-fastly.deb.debian.org;
  keepalive 2;
}

server {
  listen [::];
  server_name apt.debian.cache apt.debian.cache.ewellnet;

  location ^~ deb$ {
    proxy_pass http://cdn-fastly.deb.debian.org;
  }

  location / {
    proxy_pass http://cdn-fastly.deb.debian.org;
    proxy_cache_valid any 1h;
  }
}

upstream security.debian.org {
  server security.debian.org;
  keepalive 2;
}

server {
  listen [::];
  server_name security.debian.cache security.debian.cache.ewellnet;

  location ^~ deb$ {
    proxy_pass http://security.debian.org;
  }

  location / {
    proxy_pass http://security.debian.org;
    proxy_cache_valid any 1h;
  }
}

This configuration ensures that requests for normal packages will only be revalidated after the default duration has passed, but requests for the package summaries will be revalidated every hour.

Refresh Nginx to enable apt package caching:

# svcadm refresh nginx

Client Configuration

Debian clients need to be configured to make use of the local cache. This can be done by altering the repository URLs within the configuration of each client:

/etc/apt/sources.list:

deb http://apt.debian.cache/debian stretch main
deb-src http://apt.debian.cache/debian stretch main

deb http://apt.debian.cache/debian stretch-updates main
deb-src http://apt.debian.cache/debian stretch-updates main

deb http://security.debian.cache/ stretch/updates  main
deb-src http://security.debian.cache/ stretch/updates  main

I could keep going with examples of apt caching on Debian based distributions, but it's all pretty similar, configure Nginx to cache the upstream repository, then configure the client system to access Nginx through a local domain name. This work about the same with Kali, MX, Pop!OS, basically anything.

With that out of the way, lets explore some package managers from different distributions.

DNF (CentOS)

DNF is the updated version of YUM, the default package management system used by CentOS and derivative Linux distributions.

DNS Records

Ensure that the following DNS records are configured:

• yum.centos.cache.ewellnet points to your cache server.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/centos.cache:

upstream mirror.centos.org {
  server mirror.centos.org;
  keepalive 2;
}

server {
  listen [::];
  server_name yum.centos.cache yum.centos.cache.ewellnet;

  location ^~ rpm$ {
    proxy_pass http://mirror.centos.org;
  }

  location / {
    proxy_pass http://mirror.centos.org;
    proxy_cache_valid any 1h;
  }
}

This configuration ensures that requests for normal packages will only be revalidated after the default duration has passed, but requests for the package summaries will be revalidated every hour.

Refresh Nginx to enable yum and dnf package caching:

# svcadm refresh nginx

Client Configuration

CentOS clients need to be configured to make use of the local cache. This can be done by altering the repository URLs within the configuration of each client. There are quite a few files involved, nearly everything in /etc/yum.repos.d:

/etc/yum.repos.d/*:

[appstream]
name=CentOS Linux $releasever - AppStream
baseurl=http://yum.centos.cache/$contentdir/$releasever/AppStream/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

References to mirrorlist should be removed, instead preferring baseurl. Also, don't change these settings for Debuginfo or Sources repos, as we want those to bypass our cache.

Once you're done, run dnf update to confirm everything's working correctly.

XBPS (Void)

Void Linux uses the XBPS package manager. As before we're going to configure DNS, the caching and then each client.

DNS Records

Ensure that the following DNS records are configured:

• xbps.void.cache.ewellnet points to your cache server.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/void.cache:

upstream alpha.de.repo.voidlinux.org {
  server alpha.de.repo.voidlinux.org:443;
  keepalive 2;
}

server {
  listen [::];
  server_name xbps.void.cache xbps.void.cache.ewellnet;

  location ^~ repodata$ {
    proxy_pass https://alpha.de.repo.voidlinux.org;
    proxy_cache_valid any 1h;
  }

  location / {
    proxy_pass https://alpha.de.repo.voidlinux.org;
  }
}

This configuration ensures that requests for normal packages will only be revalidated after the default duration has passed, but requests for the package summaries will be revalidated every hour.

Refresh Nginx to enable xbps package caching:

# svcadm refresh nginx

Client Configuration

Void Linux clients need to be configured to make use of the local cache. This can be done by setting a local repository URL within the configuration of each client:

/usr/share/xbps.d/00-repository-main.conf:

repository=http://xbps.void.cache/current

Once this is set, use xbps-install to ensure that everything is working correctly:

# xbps-install -Su
[*] Updating repository `http://xbps.void.cache/current/x86_64-repodata' ...

Node.js Package Manager (npm)

Other package managers can be configured to make use of a local package cache as well. For instance, the Node.js Package Manager utility (npm).

DNS Records

Ensure that the following DNS record is configured:

• npm.cache.ewellnet points to your cache server.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/npm.cache:

upstream registry.npmjs.org {
  server registry.npmjs.org:443;
  keepalive 2;
}

server {
  listen [::];
  server_name npm.cache npm.cache.ewellnet;

  location / {
    proxy_pass https://registry.npmjs.org;
    proxy_cache_valid any 1h;
  }
}

This approach is revalidation heavy, but unfortunately that's the best we're going to get out of npm, due to how their repository is structured and queried.

Refresh Nginx to enable npm package caching:

# svcadm refresh nginx

Client Configuration

Set the local registry using the npm command-line utility:

# npm set registry http://npm.cache/

And now npm will install packages through your cache.

Python Package Index (pip)

The Python Package Index command-line utility (pip) can also be configured to use a cache. This is a bit more involved than previous as pip normally accesses two different base URIs, https://pipi.org/simple for the repository information which links directly to files normally hosted at https://files.pythonhosted.org/.

However, through clever use of the Nginx ngx_http_sub_module, we should be able to make this work through a single base URI. Note, this module is not currently compiled by default in the build of Nginx distributed via SmartOS pkgsrc, and will need to be custom built to enable this functionality for now.

DNS Records

Ensure that the following DNS record is configured:

• pip.cache.ewellnet points to your cache server.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/pip.cache:

upstream pypi.org {
  server pypi.org:443;
  keepalive 2;
}

upstream files.pythonhosted.org {
  server files.pythonhosted.org:443;
  keepalive 2;
}

server {
  listen [::];
  server_name pip.cache pip.cache.ewellnet;

  location / {
    proxy_pass https://pypi.org;
    proxy_cache_valid 200 301 302 1h;

    sub_filter_once off;
    sub_filter "https://files.pythonhosted.org" "http://pip.cache";
  }

  location /packages {
    proxy_pass https://files.pythonhosted.org;
  }
}

Refresh Nginx to enable pip package caching:

# svcadm refresh nginx

Client Configuration

Configure clients to use the cache registry by using the pip command-line utility:

# pip config set global.index-url http://pip.cache/simple
Writing to /root/.config/pip/pip.conf
# pip config set global.trusted-host pip.cache
Writing to /root/.config/pip/pip.conf

And now pip will install packages through your cache. You can also set this globally if you prefer, as an exercise I leave up to you.

RubyGems (gem)

The RubyGems package management command-line utility (gem) can also be configured to use a cache.

DNS Records

Ensure that the following DNS record is configured:

• rubygems.cache.ewellnet points to your cache server.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/rubygems.cache:

upstream rubygems.org {
  server rubygems.org:443;
  keepalive 2;
}

server {
  listen [::];
  server_name rubygems.cache rubygems.cache.ewellnet;

  location /^~ gem$ {
    proxy_pass https://rubygems.org;
  }

  location / {
    proxy_pass https://rubygems.org;
    proxy_cache_valid 200 301 302 1h;
  }
}

Refresh Nginx to enable gem package caching:

# svcadm refresh nginx

Client Configuration

Set the local sources using the gem command-line utility:

# gem sources --add http://rubygems.cache/
http://rubygems.cache/ added to sources
# gem sources --remove https://rubygems.org/
https://rubygems.org/ removed from sources

And now gem will install packages through your cache.

As should be illustrated by now, many different package management systems can be tweaked to interface to their upstream repositories through a cache. Additional examples of package managers that should work would be Cargo, Hex, and Go, the Package managers for Rust, Erlang/Elixir and Go, and while it would be fun to walk through those and stand them up here as examples, I don't use those languages enough to justify the work. Yet.

Steam

A now classic example of using Nginx to cache software resources like this treads directly into the operational space of Lancache, which is configuring a Steam cache.

Steam has gotten a lot nicer to use with caching in the last few years, but I'd still qualify this section as experimental, as it doesn't yield the performance that I'd like to see out of a steam cache. I will likely be revising this in the near future, but will need better visibility into cache performance that will be available at a later time.

DNS Records

Ensure that the following DNS record is configured:

• lancache.steamcontent.com points to your cache server.

Steam is nice enough to recognize when this record points to a private routing prefix and will redirect all resource requests to this host. Very nice.

Cache Configuration

Ensure that the following configuration file has been added to your Nginx configuration directory in your cache:

/opt/local/etc/nginx/cache/steam.cache:

upstream steam-upstream {
  server cache1-sea1.steamcontent.com;
  server cache2-sea1.steamcontent.com;
  server cache3-sea1.steamcontent.com;
  server cache4-sea1.steamcontent.com;
  server cache1-lax1.steamcontent.com;
  server cache2-lax1.steamcontent.com;
  server cache3-lax1.steamcontent.com;
  server cache4-lax1.steamcontent.com;
  server cache5-lax1.steamcontent.com;
  server cache6-lax1.steamcontent.com;
  keepalive 32;
}

server {
  listen [::];
  server_name lancache.steamcontent.com *.steamcontent.com;
  slice 8m;

  proxy_cache_key lancache.steamcontent.com$uri$is_args$args$slice_range;
  proxy_set_header Range $slice_range;

  location / {
    proxy_pass http://steam-upstream;
  }
}

In my case, I'm using upstream caches located in Seattle and Los Angeles, the two closest locations.

Refresh Nginx to enable Steam caching:

# svcadm refresh nginx

Client Configuration

No client configuration is required. Steam will automatically recognize and make use of this cache.

Cache Partitioning

While all of the above examples use a single shared cache, an approach I prefer, you can also set independent caches per service.

First, disable Nginx.

# svcadm mark maintenance nginx

Create additional ZFS datasets per cache that you'd like to create:

# zfs create -o quota=200G -o mountpoint=/var/www/cache-2 zones/<UUID>/data/cache-2

Register the cache paths in nginx under the http context. Note that the size has been adjusted to reflect the size set for the ZFS dataset:

/opt/local/etc/nginx/nginx.conf:

...
proxy_cache_path /var/www/cache-2 levels=2:2 use_temp_path=on keys_zone=second_cache:128m inactive=4y max_size=190G;
...

Adjust the cache configuration to make use of that cache instead of default_cache, for example with Ubuntu:

/opt/local/etc/nginx/cache/ubuntu.cache:

upstream archive.ubuntu.com {
  server archive.ubuntu.com;
  keepalive 2;
}

server {
  listen [::];
  server_name apt.ubuntu.cache apt.ubuntu.cache.ewellnet;
  proxy_cache second_cache;

  location ^~ deb$ {
    proxy_pass http://archive.ubuntu.com;
  }

  location / {
    proxy_pass http://archive.ubuntu.com;
    proxy_cache_valid 200 301 302 1h;
  }
}

Once you're done, restart Nginx and enjoy having separate caches:

# svcadm clear nginx

Conclusion

While I'm generally happy with the results of this project, there's clearly room for improvement, specifically around Steam and potentially additional game service caching.

Digging further into Lancache and investigating what problems they experienced and how they overcame them is probably the best move from here, as well as investing further into visibility tools to determine how and why Nginx is slowing down requests instead of accelerating them.

But for everything else, this is definitely a good first step.

Network UPS Tools is a project for connecting with many different power devices, and can access UPS data and coordinate with the device to ensure orderly shutdown of a host such as a SmartOS compute node.

NUT flies a bit in the face of the design philosophy of SmartOS. Not only will you need to install it manually (via pkgsrc, which will also need to be installed) in a global zone to use it, but SmartOS would rather rely on ZFS to handle unexpected power events rather than being explicitly shutdown by a UPS monitoring service.

In this article, we will install NUT (and pkgsrc) into the global zone and configure it to interface with a USB attached UPS device and to (optionally) automatically power down the compute node if the UPS batteries run critically low.

Lets get to it.

Installing Pkgsrc into the global zone

The Joyent Pkgsrc website has instructions on installing pkgsrc into the global zone. At the time of this writing, this is the bootstrap that we're going to use:

/root/install-pkgsrc.sh:

#
# Copy and paste the lines below to install the latest 64-bit tools set.
#
BOOTSTRAP_TAR="bootstrap-trunk-tools-20201019.tar.gz"
BOOTSTRAP_SHA="9b7a6daff5528d800e8cea20692f61ccd3b81471"

# Ensure you are in a directory with enough space for the bootstrap download,
# by default the SmartOS /root directory is limited to the size of the ramdisk.
cd /var/tmp

# Download the bootstrap kit to the current directory.  Note that we currently
# pass "-k" to skip SSL certificate checks as the GZ doesn't install them.
curl -kO https://pkgsrc.joyent.com/packages/SmartOS/bootstrap/${BOOTSTRAP_TAR}

# Verify the SHA1 checksum.
[ "${BOOTSTRAP_SHA}" = "$(/bin/digest -a sha1 ${BOOTSTRAP_TAR})" ] || echo "ERROR: checksum failure"

# Verify PGP signature.  This step is optional, and requires gpg.
curl -kO https://pkgsrc.joyent.com/packages/SmartOS/bootstrap/${BOOTSTRAP_TAR}.asc
curl -ksS https://pkgsrc.joyent.com/pgp/DE817B8E.asc | gpg --import
gpg --verify ${BOOTSTRAP_TAR}{.asc,}

# Install bootstrap kit to /opt/tools
tar -zxpf ${BOOTSTRAP_TAR} -C /

# Add to PATH/MANPATH.
PATH=/opt/tools/sbin:/opt/tools/bin:$PATH
MANPATH=/opt/tools/man:$MANPATH

Run this script on the global zone to bootstrap pkgin:

# /root/install-pkgsrc.sh

We should now have pkgsrc available now on the global zone.

If you have a pkgsrc cache available (which we will cover soon), configure pkgin to make use of it:

/opt/tools/etc/pkgin/repositories.conf:

...
http://pkgsrc.joyent.cache/packages/SmartOS/trunk/tools/All

Update and upgrade pkgsrc.

# pkgin update
processing remote summary (http://pkgsrc.joyent.cache/packages/SmartOS/trunk/tools/All)...
database for http://pkgsrc.joyent.cache/packages/SmartOS/trunk/tools/All is up-to-date
# pkgin -y upgrade
...

Install NUT into the Global Zone

This is exceedingly difficult to do using pkgsrc:

# pkgin -y install ups-nut-usb
...

This should install NUT and its USB drivers into /opt/tools as well.

Configuring & testing upsdrvctl

NUT uses a component called upsdrvctl to interface with USB attached devices. This will need to be told about your UPS, specifically which driver and port it should use to access it. The driver you should use is specific to your UPS model, and can be checked using their Hardware Compatibility List. This is an example of a configuration file setup for a SMT2200 attached as the sole UPS to a compute node:

/opt/tools/etc/nut/ups.conf:

[smt2200]
  driver = usbhid-ups
  port = auto
  desc = "American Power Conversion Smart-UPS 2200"

Running the following command will enable the driver and ensure that upsd can connect to it later on. You shouldn't see any error messages here.

# upsdrvctl start
Network UPS Tools - UPS driver controller 2.7.4
Network UPS Tools - Generic HID driver 0.41 (2.7.4)
USB communication driver 0.33
Using subdriver: APC HID 0.96

Configuring & testing upsd

The upsd process is responsible for connecting to and polling information from attached UPSes via upsdrvctl. Its default configuration (of an empty file) is perfectly suitable for a standalone configuration.

If you would like to use upscmd as well, it's best to configure some users who have permission to manipulate the device.

/opt/tools/etc/nut/upsd.users:

[admin]
  password = secret
  actions = SET
  instcmds = ALL

Start upsd when you're ready.

# upsd
Network UPS Tools upsd 2.7.4
fopen /opt/tools/var/db/nut/upsd.pid: No such file or directory
listening on 127.0.0.1 port 3493
listening on ::1 port 3493
Connected to UPS [smt2200]: usbhid-ups-smt2200

You should now be able to query your ups using upsc:

# upsc smt2200 ups.status
Init SSL without certificate database
OL

For reference, OL stands for On-Line Power, OB stands for On-Battery Power, and LB stands for Low-Battery Power. You can also issue this command without any additional options to see all information from the UPS:

# upsc smt2200
Init SSL without certificate database
battery.charge: 68
battery.charge.low: 10
battery.charge.warning: 50
...

The upscmd command can also be used to issue specific commands to your UPS:

# upscmd -l smt2200
Instant commands supported on UPS [smt2200]:

beeper.disable - Disable the UPS beeper
beeper.enable - Enable the UPS beeper
beeper.mute - Temporarily mute the UPS beeper
beeper.off - Obsolete (use beeper.disable or beeper.mute)
beeper.on - Obsolete (use beeper.enable)
load.off - Turn off the load immediately
load.off.delay - Turn off the load with a delay (seconds)
shutdown.reboot - Shut down the load briefly while rebooting the UPS
shutdown.stop - Stop a shutdown in progress

It will also ask you for credentials before allowing any actual commands to be performed.

Configuring & Testing upsmon (optional)

If you would like NUT to actually power down your system once power reaches a critical point, then you will need to configure upsmon, the monitoring component of NUT.

Generally, using this component of NUT is completely optional as SmartOS can generally handle abrupt shutdowns already thanks to ZFS. However if you're doing something crazy like disabling sync writes on certain datasets, or generally like the idea of a graceful shutdown in the global zone, proceed.

A rough layout of of upsmon's flow with our modifications are as follows:

• The UPS goes onto battery power.
• The UPS reaches a low battery state (ups.state goes from OB to LB).
• The upsmon service notices this and generates a NOTIFY_SHUTDOWN event, waits FINALDELAY seconds, creates the POWERDOWNFLAG file and calls SHUTDOWNCMD.
• SMF shuts everything down before finally signaling the UPS to power off and return once the utility power has been restored.
• The system loses power.
• Time passes
• Power returns and the UPS switches back on, restoring supply to the loads.
• All systems reboot and continue on as normal.

Since it interacts with upsd to actually power down the loads, it will need credentials set in upsd.users. Lets do that now. Add the following user:

/opt/tools/etc/nut/upsd.users:

[monuser]
  password = doesitmatter
  upsmon master

Reload upsd to update it with the new information.

# upsd -c reload
Network UPS Tools upsd 2.7.4

Define MONITOR and SHUTDOWNCMD within the upsmon configuration.

/opt/tools/etc/nut/upsmon.conf:

MONITOR smt2200@localhost 1 monuser doesitmatter master
SHUTDOWNCMD /usr/sbin/poweroff

And then start upsmon.

# upsmon
Network UPS Tools upsmon 2.7.4
fopen /opt/tools/var/db/nut/upsmon.pid: No such file or directory
UPS: smt2200@localhost (master) (power value 1)
Using power down flag file /etc/killpower

While the automatic shutdown can be tested, it's probably best to wait until after completing setup of the following section.

Setting up SMF manifests

While the global zone is relatively ephemeral, we do have the option of creating persistent services by placing SMF manifests into /opt/custom/smf, and since we really don't want to have to manually set this all up each time, we're going to do that now.

/opt/custom/smf/nut.xml:

Note: Remove the upsmon instance if you don't want it to start up and attempt to automatically power off your system in the case of a power failure.

Now, you can either restart, or simply import the manifest to enable these services.

# svccfg import /opt/custom/smf/nut.xml

Ensure that they are properly running.

# svcs nut
STATE          STIME    FMRI
online         10:46:59 svc:/pkgsrc/nut:upsdrvctl
online         10:46:59 svc:/pkgsrc/nut:upsd
online         10:46:59 svc:/pkgsrc/nut:upsmon

Conclusion

While I'm still testing this and may continue to update this article, that's basically what it takes to get NUT running in your global zone.

I have to say that I'm disappointed in APC as their SUA1500 model which protects my HP N54L actually has more features exposed through the USB cable than the newer SMT2200-RM2U. Specifically:

• Battery Manufacture Date.
• Battery Temperature.
• Input data, including UPS sensitivity, high and low transfer thresholds, reason for last transfer, and current input voltage.
• Output data, including sinewave frequency and voltage.
• UPS load as a percentage.

While much of this data is available on the physical front panel, that means I can't poll it through upsc for monitoring in metrics, which for me was the entire point of installing NUT in the first place. Additionally, the upscmd command is quite limited, lacking the following features that the SUA1500 has:

• Being able to turn the UPS on.
• Adjust the automatic power-on behavior for the UPS (shutdown.return vs shutdown.stayoff).
• Testing the battery.
• Testing the front panel.

With the new hardware up and running, it's time to redeploy new zones. We're going to start with the router zone, and as it's almost 2022, I'm going to call this article my 2022 Home Router Refresh.

This article will overview rebuilding my zone based SmartOS router, and draw from three of my previous ones, specifically Home Router on SmartOS, Hurricane Electric on SmartOS, and Tunnels on SmartOS. If you'd like more of the theory behind why I'm doing what I'm doing, look to those articles, this is going to be much more nuts and bolts.

Lets dive in.

Overview

We will be deploying a router that will be responsible for routing between the isolated private subnets in my new network deployment, as well as providing common home network services such as dynamic host configuration protocol (DHCP), domain name service (DNS), and network address translation (NAT).

In addition, it will be acting as the terminal for a tunnel going out to Hurricane Electric for IPv6 connectivity, as well as IPSec tunnels going to production systems.

Interfaces

The local interfaces that we'll be working with were listed in the previous article, but for simplicity, are the following:

• Public: net0: nic:admin, vlan_id:5, dhcp, autoconf
• Infrastructure: net1: nic:admin, vlan_id:1, 172.22.1.0/27, IPv6/64
• Embedded: net2: nic:admin, vlan_id:2, 172.22.1.32/27, IPv6/64
• Internal: net3: etherstub:internal, 172.22.1.64/27, IPv6/64
• External: net4: etherstub:external, 172.22.1.96/27, IPv6/64
• Secure: net5: nic:admin, vlan_id:3, 172.22.1.128/26, IPv6/64
• Guest: net6: nic:admin, vlan_id:4, 172.22.1.192/26, IPv6/64

The public interface may end up being directly connected at some point, the 1000BASE-T SFP module I have doesn't play well with the NDC in my compute node, so for now connecting to the public internet through my switch with a dedicated VLAN is how I'm going to proceed forward at this time. Whenever this network switches up to anything faster, there are still two free SFP+ interfaces on the back of the server to handle that interconnect. in the future.

Also, I may end up defining an additional interconnect for the 40GBE interface at the router or experiment with 802.1D bridging via dladm create-bridge, we will see when I get there.

Additionally, I will be creating the following tunnel interfaces:

• Hurricane Electric: v4_he0
• Production Tunnels: v4_ze0, v4_ze1

Note that I tend to use the v4_ prefix when referring to IPv4 based tunnels and the v6_ prefix when referring to IPv6 based tunnels.

Zone Manifest

This is the manifest that I used to create router zone:

{
  "image_uuid": "1d05e788-5409-11eb-b12f-037bd7fee4ee",
  "brand": "joyent",
  "alias": "router",
  "hostname": "router",
  "dns_domain": "ewellnet",
  "cpu_cap": 200,
  "max_physical_memory": 128,
  "quota": 10,
  "resolvers": [ "::1" ],
  "nics": [
    {
      "nic_tag": "admin",
      "vlan_id": 5,
      "interface": "net0",
      "ips": ["dhcp","addrconf"],
      "primary": true,
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "interface": "net1",
      "ips": ["172.22.1.1/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "vlan_id": 2,
      "interface": "net2",
      "ips": ["172.22.1.33/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "internal0",
      "interface": "net3",
      "ips": ["172.22.1.65/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "external0",
      "interface": "net4",
      "ips": ["172.22.1.97/27","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "vlan_id": 3,
      "interface": "net5",
      "ips": ["172.22.1.129/26","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    },
    {
      "nic_tag": "admin",
      "vlan_id": 4,
      "interface": "net6",
      "ips": ["172.22.1.193/26","addrconf"],
      "allow_dhcp_spoofing": "true",
      "allow_ip_spoofing": "true"
    }
  ]
}

Note that I'm setting a resolver of ::1 in the zone manifest simply because that is ultimately what it will be set to, you will want to temporarily alter /etc/resolv.conf until you have dnsmasq properly configured.

Creation command and login:

# vmadm create -f router.json
Successfully created VM f8510d07-3852-ebf0-83f3-e850f1ceb5fa
# zlogin f8510d07-3852-ebf0-83f3-e850f1ceb5fa

Configuring NAT & Routing

Edit /etc/ipf/ipnat.conf so that it will translate IPv4 packets routed from the private networks out onto the internet:

map net0 172.22.1.0/24 -> 0/32 proxy port ftp ftp/tcp
map net0 172.22.1.0/24 -> 0/32 portmap tcp/udp auto
map net0 172.22.1.0/24 -> 0/32

Enable IPFilter and IPv4 forwarding:

# svcadm enable ipfilter
# routeadm -ue ipv4-forwarding

Then lets verify that it's working properly:

# ipnat -l
List of active MAP/Redirect filters:
map net0 172.22.1.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
map net0 172.22.1.0/24 -> 0.0.0.0/32 portmap tcp/udp auto
map net0 172.22.1.0/24 -> 0.0.0.0/32

List of active sessions:

Installing Dnsmasq

I've come to really appreciate dnsmasq for how small and light it is, and yet how generally capable it is in a home network setting. Sure it isn't the fastest, and it doesn't do everything, but it's fantastic just how much it accomplishes while using minimal resources.

Installation and configuration is a breeze:

# pkgin in dnsmasq

And configuration isn't too difficult either:

/opt/local/etc/dnsmasq.conf:

# DNS specific configuration

# Set DNS cache size
cache-size=8192

# Never forward plain names (without a dot or domain part)
domain-needed

# Never forward addresses in the non-routed address spaces.
bogus-priv

# We don't want dnsmasq to poll resolv files for changes, we will manually notify it with a refresh
no-poll

# Upstream resolvers, use this so that /etc/resolv.conf can refer to localhost
resolv-file=/etc/resolv.upstream

# Domain for local services which should never be forwarded to external resolvers
local=/ewellnet/
domain=ewellnet

# Shared configuration

# Interfaces used for DHCP & DNS
interface=net1
interface=net2
interface=net3
interface=net4
interface=net5
interface=net6

# DHCP specific configuration

# DHCP ranges
dhcp-range=172.22.1.2,172.22.1.30,255.255.255.224,6h
dhcp-range=172.22.1.34,172.22.1.62,255.255.255.224,6h
dhcp-range=172.22.1.130,172.22.1.190,255.255.255.192,6h
dhcp-range=172.22.1.194,172.22.1.254,255.255.255.192,6h

# Read /etc/ethers for static DHCP allocations
read-ethers

# Set the NTP time server addresses to the interface's
dhcp-option=option:ntp-server

# This should be the authoritative DHCP server on the network
dhcp-authoritative

For any static assignments, set records in /etc/hosts and (optionally) /etc/ethers. In my case, the records are suffixed with .ewellnet.

Create the directory at /var/cache and enable the dnsmasq service:

# mkdir /var/cache
# svcadm enable dnsmasq

Local DHCP and DNS should now be up and running.

Secure SSH

Depending on your disposition, you may want to prevent password based logins through SSH or disable the service all together. I usually opt for the latter for home, though I'm sure that'll bite me in the future some time.

# svcadm disable ssh

Tunnel to Hurricane Electric

At the time of this writing, Ziply Fiber still hasn't rolled out IPv6 connectivity to end users. This is okay though, since we have Hurricane Electric.

Login to tunnelbroker.net to set and acquire your necessary credentials. Since these addresses are global, I will be making the following replacements for the below examples:

• Server IPv4 address: 1.2.3.4
• Client IPv4 address: 5.6.7.8
• Server IPv6 address: 2001:470::1
• Client IPv6 address: 2001:470::2

Run the following commands to bring a Hurricane Electric tunnel up and route default IPv6 traffic over it:

# dladm create-iptun -T ipv4 -a local=5.6.7.8,remote=1.2.3.4 v4_he0
# ipadm create-addr -T static -a local=2001:470::2,remote=2001:470::1 v4_he0/v6
# route -p add -inet6 default 2001:470::1

Test by pinging any well known IPv6 enabled domain:

# ping -A inet6 google.com
google.com is alive

Our router can now access the IPv6 Internet.

Advertise Routed IPv6 over Hurricane Electric

IPv6 connectivity to our router isn't terribly helpful on it's own; we'll want to be able to provide routing to our attached devices.

First up, we'll need to start forwarding IPv6 packets:

# routeadm -ue ipv6-forwarding

While I would normally make use of dnsmasq's IPv6 router advertisement features, the ndp daemon is normally already running with addrconf enabled on an interface, so we might as well just use the native tools here, especially since they'll manipulate the routes table on the router for us.

/etc/inet/ndpd.conf:

if net1 AdvSendAdvertisements 1
prefix 2001:470:????::/64 net1
if net2 AdvSendAdvertisements 1
prefix 2001:470:????:1::/64 net2
if net3 AdvSendAdvertisements 1
prefix 2001:470:????:2::/64 net3
if net4 AdvSendAdvertisements 1
prefix 2001:470:????:3::/64 net4
if net5 AdvSendAdvertisements 1
prefix 2001:470:????:4::/64 net5
if net6 AdvSendAdvertisements 1
prefix 2001:470:????:5::/64 net6

And then restart (not refresh) ndp.

# svcadm restart ndp

You should now see functional IPv6 connectivity on every stateless device on your network.

Establish IPSec Policy & Key Exchange

I also run IPSec tunnels to some production zones in a local datacenter. These tunnels will be called ze0 and ze1. First of all, we're going to need to set the encryption policy, specifically what encryption and authentications will be accepted over these tunnels:

/etc/inet/ipsecinit.conf:

{tunnel v4_ze0 negotiate tunnel}
  ipsec {encr_algs aes encr_auth_algs sha512 sa shared}
{tunnel v4_ze1 negotiate tunnel}
  ipsec {encr_algs aes encr_auth_algs sha512 sa shared}

After ensuring that the same settings are present on the remote endpoints, I restart ipsec policy:

# svcadm restart ipsec/policy

Next up is to configure IKE. Normally I would be fancy and use public keys, but pre-shared should be fine in this case:

/etc/inet/ike/config:

p1_lifetime_secs 7200
p1_nonce_len 40

p1_xform {
        auth_method preshared
        oakley_group 5
        auth_alg sha512
        encr_alg aes
}

p2_pfs 2

{
        label "v4_ze0"
        local_addr 1.2.3.4
        remote_addr 5.6.7.8
        p1_xform {
                auth_method preshared
                oakley_group 5
                auth_alg sha512
                encr_alg aes
        }
        p2_pfs 5
}
{
        label "v4_ze1"
        local_addr 1.2.3.4
        remote_addr 5.6.7.9
        p1_xform {
                auth_method preshared
                oakley_group 5
                auth_alg sha512
                encr_alg aes
        }
        p2_pfs 5
}

Generate some large random 64-byte keys for each tunnel. These are example keys only, do not use them production:

# openssl rand -hex 128
0f5451e4e0ad601f64dad77b9a83a0214dd4f460e07935936cfc7faf24c90f1c65609aaebc817ea17894748d999c94324d53a50ba18ce62c7683966674c3d50f93d1333440709f92facd1d10bcfde5b9ccc05e20adfa00a14825025068ba29cac04ba75b3b7e909dfddc01bd84eec2d56fa47a151cd447d0342f4194a910be9c
# openssl rand -hex 128
daa7fafa568601172b01591465e6365f90ad41eb6a778e7730f9ae5ebb9e05b9792fe288a1d7fa9950a2659433ad2b2f31087bc8e3c8ebbc80ef1c1d604847371cf4d1298ebb0b7fe4835b8bd13e969d700312a929ec8c83f63760a8a7b7439441b9e1ec27862c4294214df8ca03bd02eb62c9da6980108e27106c566b3ad7ea

Create some preshared secret files with these keys:

/etc/inet/secret/ike.preshared:

{
        localidtype IP
        localid 1.2.3.4
        remoteidtype IP
        remoteid 5.6.7.8
        key 0f5451e4e0ad601f64dad77b9a83a0214dd4f460e07935936cfc7faf24c90f1c65609aaebc817ea17894748d999c94324d53a50ba18ce62c7683966674c3d50f93d1333440709f92facd1d10bcfde5b9ccc05e20adfa00a14825025068ba29cac04ba75b3b7e909dfddc01bd84eec2d56fa47a151cd447d0342f4194a910be9c
}

{
        localidtype IP
        localid 1.2.3.4
        remoteidtype IP
        remoteid 5.6.7.9
        key daa7fafa568601172b01591465e6365f90ad41eb6a778e7730f9ae5ebb9e05b9792fe288a1d7fa9950a2659433ad2b2f31087bc8e3c8ebbc80ef1c1d604847371cf4d1298ebb0b7fe4835b8bd13e969d700312a929ec8c83f63760a8a7b7439441b9e1ec27862c4294214df8ca03bd02eb62c9da6980108e27106c566b3ad7ea
}

After the reciprocal configuration has been set on the remote side, enable or restart IKE:

# svcadm enable ipsec/ike

Establish IPSec Tunnels & Routing

With all of our IPSec policy and key exchange work out of the way, we can finally test it by establishing some tunnels. We'll do that with dladm:

# dladm create-iptun -T ipv4 -a local=1.2.3.4,remote=5.6.7.8 v4_ze0
# dladm create-iptun -T ipv4 -a local=1.2.3.4,remote=5.6.7.9 v4_ze1

Next up is to bring up a local interface for the tunnel, creating addresses will do this implicitly, so we'll just do that.

# ipadm create-addr -T static -a local=172.22.1.1,remote=10.0.0.1 v4_ze0/v4
# ipadm create-addr -T static -a local=172.22.1.1,remote=10.0.0.2 v4_ze1/v4

After performing the reciprocal steps on the remote host, you should be able to ping the remote side through the tunnel interface:

# ping 10.0.0.1
10.0.0.1 is alive

Next up is to set some static routes. Despite plumbing the interface for IPv6, in my case I'll only be actively using IPv4 over it immediately:

# route -p add 10.0.0.0/28 10.0.0.1
# route -p add 10.0.0.0/28 10.0.0.2

You should now be able to access the remote side of the IPSec tunnel through any subnet attached to this router.

Securing Everything

Next up is to ensure that everything is secure, and only communicating with what it should be. We will achieve this by altering the host model and setting up an inclusive firewall.

Normally I would perform this step earlier and then build into a secure environment. That can make troubleshooting quite difficult, so this time I built everything out first with the intention of locking it all down afterwards.

Strong Host Model

It's often a good idea to discard packets that arrive on unexpected interfaces. We can do this easily in SmartOS by changing the hostmodel to strong by using ipadm set-prop:

# ipadm set-prop -p hostmodel=strong ipv4
# ipadm set-prop -p hostmodel=strong ipv6

This will ensure that any packets that come in through an interface that's doesn't correlate with the interfaces in the routing table are dropped.

Firewall

The next thing to do is to ensure that the router is allowing only the correct traffic flows between subnets using IP Filter. We're going to enact the following policies:

• IPv6 Traffic from the Internet can reach external services 172.22.1.96/27.
• Traffic from infrastructure 172.22.1.0/27 can reach the Internet, internal services 172.22.1.64/27, external services 172.22.1.96/27 and the IPSec tunnels v4_ze0 and v4_ze1.
• Traffic from embedded 172.22.1.32/27 can access internal services 172.22.1.64/27.
• Traffic from internal services 172.22.1.64/27 can access the Internet, infrastructure 172.22.1.0/27, embedded 172.22.1.32/27, external services 172.22.1.96/27 and secure 172.22.1.128/26.
• Traffic from external services 172.22.1.96/27 can reach the Internet.
• Traffic from secure 172.22.1.128/26 can reach the Internet, internal services 172.22.1.64/27 and external services 172.22.1.96/27.
• Traffic from guest 172.22.1.192/26 can reach the Internet and external services 172.22.1.96/27.
• Traffic from the work IPSec tunnels v4_ze0 and v4_ze1 can reach infrastructure 172.22.1.0/27.

Note: For the sake of this list, "The Internet" includes IPv4 connectivity via NAT through net0 and IPv6 connectivity via the v4_he0 Hurricane Electric tunnel. Also the IPSec connection between infrastructure and remote infrastructure will likely be constrained further to only allow for administrative console access to the remote network. Next up is to implement these policies in configuration:

/etc/ipf/ipf.conf:

# Traffic from infrastructure (net1) [172.22.1.0/27]
block in on net1 from any to 172.16.0.0/12
pass in quick on net1 from 172.22.1.0/27 to 172.22.1.64/26
pass in quick on net1 from 172.22.1.0/27 to 172.22.1.0/27

# Traffic from embedded (net2) [172.22.1.32/27]
block in on net2 all
pass in quick on net2 from 172.22.1.32/27 to 172.22.1.64/27
pass in quick on net2 from 172.22.1.32/27 to 172.22.1.32/27

# Traffic from internal (net3) [172.22.1.64/27]
block in on net3 from any to 172.16.0.0/12
pass in quick on net3 from 172.22.1.64/27 to 172.22.1.128/26
pass in quick on net3 from 172.22.1.64/27 to 172.22.1.0/25

# Traffic from external (net4) [172.22.1.96/27]
block in on net4 from any to 172.16.0.0/12
pass in quick on net4 from 172.22.1.96/27 to 172.22.1.96/27

block out on net4 from 172.16.0.0/12 to any
pass out quick on net4 proto tcp from 172.16.0.0/12 to 172.22.1.96/27 flags S keep state
pass out quick on net4 proto udp from 172.16.0.0/12 to 172.22.1.96/27 keep state
pass out quick on net4 proto icmp from 172.16.0.0/12 to 172.22.1.96/27 icmp-type echo keep state
pass out quick on net4 from 172.22.1.96/27 to 172.22.1.96/27

# Traffic from secure (net5) [172.22.1.128/26]
block in on net5 from any to 172.16.0.0/12
pass in quick on net5 from 172.22.1.128/26 to 172.22.1.64/26
pass in quick on net5 from 172.22.1.128/26 to 172.22.1.128/26

# Traffic from guest (net6) [172.22.1.192/26]
block in on net6 from any to 172.16.0.0/12
pass in quick on net6 from 172.22.1.192/26 to 172.22.1.96/27
pass in quick on net6 from 172.22.1.192/26 to 172.22.1.192/26

# Traffic from IPsec tunnels (v4_ze0,v4_ze1) [192.168.255.224/27]
block in on v4_ze0 all
pass in quick on v4_ze0 from 10.0.0.0/27 to 172.22.1.0/27

block in on v4_ze1 all
pass in quick on v4_ze1 from 10.0.0.0/27 to 172.22.1.0/27

/etc/ipf/ipf6.conf:

# Traffic from public (he0) [::/0]
block in on he0 from any to 2001:470::/48
pass in quick on he0 from any to 2001:470:0:3::/64

block out on he0 from 2001:470::/48 to any
pass out quick on he0 from 2001:470:0:3::/64 to any
pass out quick on he0 proto tcp from 2001:470::/48 to any flags S keep state
pass out quick on he0 proto udp from 2001:470::/48 to any keep state
#pass out quick on he0 proto icmp from 2001:470::/48 to any icmp-type echo keep state

# Traffic from infrastructure (net1) [2001:470::/64]
block in on net1 from any to 2001:470::/48
pass in quick on net1 from 2001:470::/64 to 2001:470:0:2::/63
pass in quick on net1 from 2001:470::/64 to 2001:470::/64

# Traffic from embedded (net2) [2001:470:0:1::/64]
block in on net2 all
pass in quick on net2 from 2001:470:0:1::/64 to 2001:470:0:2::/64
pass in quick on net2 from 2001:470:0:1::/64 to 2001:470:0:1::/64

# Traffic from internal (net3) [2001:470:0:2::/64]
block in on net3 from any to 2001:470::/48
pass in quick on net3 from 2001:470:0:2::/64 to 2001:470:0:4::/64
pass in quick on net3 from 2001:470:0:2::/64 to 2001:470::/62

# Traffic from external (net4) [2001:470:0:3::/64]
block in on net4 from any to 2001:470::/48
pass in quick on net4 from 2001:470:0:3::/64 to 2001:470:0:3::/64

block out on net4 from 2001:470::/48 to any
pass out quick on net4 proto tcp from 2001:470::/48 to 2001:470:0:3::/64 flags S keep state
pass out quick on net4 proto udp from 2001:470::/48 to 2001:470:0:3::/64 keep state
#pass out quick on net4 proto icmp from 2001:470::/48 to 2001:470:0:3::/64 icmp-type echo keep state
pass out quick on net4 from 2001:470:0:3::/64 to 2001:470:0:3::/64

# Traffic from secure (net5) [2001:470:0:4::/64]
block in on net5 from any to 2001:470:0::/48
pass in quick on net5 from 2001:470:0:4::/64 to 2001:470:0:2::/63
pass in quick on net5 from 2001:470:0:4::/64 to 2001:470:0:4::/64

# Traffic from guest (net6) [2001:470:0:5::/64]
block in on net6 from any to 2001:470:0::/48
pass in quick on net6 from 2001:470:0:5::/64 to 2001:470:0:3::/64
pass in quick on net6 from 2001:470:0:5::/64 to 2001:470:0:5::/64

# Traffic from IPsec tunnels (v4_ze0,v4_ze1)
block in on v4_ze0 all
block in on v4_ze1 all

Reload ipfilter to enable those rules:

# svcadm refresh ipfilter

This enables a stateful IPv4 firewall around external to prevent it from accessing the other network segments without preventing them from accessing it, as well, the IPv6 firewall rules enable two stateful layers, one around the external network segment and another one protecting all networks except for the external segment from access.

Please note: Using a firewall like IP Filter severely limits network performance. While I have taken steps above to mitigate the impact by reducing the firewall rules to their simplest forms, I will be investigating further possible steps to take in the future.

Forwarding Traffic

There are some additional exceptions that need to be set for various services running in this network, namely:

• Forward NTP traffic udp/123 directed to any of the non-routable router addresses to the SmartOS compute node.
• Forward Plex traffic tcp/31400 to the Plex Media Server zone.
• I'm sure many more.

These forwarding rules can be added to /etc/ipf/ipnat.conf along with the map rules we added before:

/etc/ipf/ipnat.conf:

map net0 172.22.1.0/24 -> 0/32 proxy port ftp ftp/tcp
map net0 172.22.1.0/24 -> 0/32 portmap tcp/udp auto
map net0 172.22.1.0/24 -> 0/32

# Plex redirection (public->media)
rdr net0 0.0.0.0/0 port 32400 -> 172.22.1.70 port 32400 tcp

# NTP server redirection (all->hv-1)
rdr net1 172.22.1.1/32   port 123 -> 172.22.1.8 port 123 udp
rdr net2 172.22.1.33/32  port 123 -> 172.22.1.8 port 123 udp
rdr net3 172.22.1.65/32  port 123 -> 172.22.1.8 port 123 udp
rdr net4 172.22.1.97/32  port 123 -> 172.22.1.8 port 123 udp
rdr net5 172.22.1.129/32 port 123 -> 172.22.1.8 port 123 udp
rdr net6 172.22.1.193/32 port 123 -> 172.22.1.8 port 123 udp

IP Filter will need to be refreshed once more:

# svcadm refresh ipfilter

Conclusion

This should be a pretty solid start to a router zone, though I may end up changing some things around, namely:

• Some of the high throughput communication paths may end up requiring some zones to be moved to different networks to reduce the traffic being filtered through IP Filter and hopefully improve network performance.
• Adding IPSec policies for mobile devices such as Laptops and Smart Phones to access network resources away from home.
• Adding QoS for IP telephony traffic.

But for now, this should do.

This is a continuation of the previous article on hardware updates to my homelab, focusing on the configuration of the major hardware components discussed.

No need for much of a preamble here, lets get into it.

Network Configuration

There are a bunch of advanced features to experiment with on this Cisco switch, but for now just getting it up and running should be good enough.

I factory reset it, enabled setup mode and hopped onto the console. After running through the initial setup program and assigning some static IP addresses and passwords, I confirmed I could connect with the new addresses and proceeded with the following configuration steps:

EtherChannel (aka Link Aggregation)

The switch and hypervisor are physically connected via a pair of SFP+ DAC cables. It would be criminal not to configure them for balanced failover. Fortunately, Cisco's EtherChannel and Illumos' Link Aggregation talk to each other, so we'll be setting that up.

I set the following configuration on the switch:

#configure terminal
 interface Te1/0/1
  switchport mode trunk
  channel-group 1 mode active
 interface Te1/0/2
  switchport mode trunk
  channel-group 1 mode active
end

Since I had already configured link-aggregation on the SmartOS side, I also confirmed this was functional on the switch side:

#show etherchannel 1 summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port


Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+--------------------------
1      Po1(SU)         LACP      Te1/0/1(P)  Te1/0/2(P)

Port-channel 1 is currently in use and both 10GBE ports are bundled in it.

Virtual LAN segments

As this switch also supports VLANs, this enables a bunch of my pre-existing hardware, such as IP phones, Wireless Access Points, and SmartOS itself. In roughly sketching some ideas out, I came to the following layout:

• Infrastructure network (vlan id 1, IPv4/27, IPv6/64) dedicated virtual network for network or infrastructure management interfaces: switches, access points, IP phones, iDRACs, hypervisors.
• Embedded network (vlan id 2, IPv4/27, IPv6/64) dedicated virtual network for embedded devices: chromecasts, IoT devices.
• Internal network (internal etherstub, IPv4/27, IPv6/64) dedicated etherstub for internally facing VMs and zones.
• External network (external etherstub, IPv4/27, IPv6/64) dedicated etherstub for externally facing VMs and zones.
• Private network (vlan id 3, IPv4/27, IPv6/64) dedicated virtual network for physically secured devices, workstations, laptops, etc.
• Guest network (vlan id 4, IPv4/27, IPv6/64) dedicated virtual network for guest wireless devices.
• Public network (vlan id 5, IPv4/DHCP) dedicated virtual network for upstream network connectivity.

With this in mind, I set the following configuration on the switch:

#configure terminal
 vlan 2
 name embedded
 vlan 3
 name private
 vlan 4
 name guest
 vlan 5
 name public
end

And then I confirmed with the following command:

#show vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi1/0/1, Gi1/0/2, Gi1/0/3
                                                Gi1/0/4, Gi1/0/5, Gi1/0/6
                                                Gi1/0/7, Gi1/0/8, Gi1/0/9
                                                Gi1/0/10, Gi1/0/11, Gi1/0/12
                                                Gi1/0/13, Gi1/0/14, Gi1/0/15
                                                Gi1/0/16, Gi1/0/17, Gi1/0/18
                                                Gi1/0/19, Gi1/0/20, Gi1/0/21
                                                Gi1/0/22, Gi1/0/23, Gi1/0/24
2    embedded                         active
3    private                          active
4    guest                            active
5    public                           active
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0
2    enet  100002     1500  -      -      -        -    -        0      0
3    enet  100003     1500  -      -      -        -    -        0      0
4    enet  100004     1500  -      -      -        -    -        0      0
5    enet  100005     1500  -      -      -        -    -        0      0
1002 fddi  101002     1500  -      -      -        -    -        0      0
1003 tr    101003     1500  -      -      -        -    -        0      0
1004 fdnet 101004     1500  -      -      -        ieee -        0      0
1005 trnet 101005     1500  -      -      -        ibm  -        0      0

Remote SPAN VLANs
------------------------------------------------------------------------------


Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

Saving the Running Configuration

After ensuring that everything works as intended, the startup switch configuration needs to be overwritten by the current one. This is done with the following:

#copy running-config startup config

Conclusion

While Cisco IOS has quite the learning curve over what I'm used to in switch configuration, it wasn't at all unpleasant to work with once I slowed down and took the time required to understand it.

While the above configuration steps were the bare minimum to get this network up and running, there's a bunch more stuff of interest in that Cisco switch that I would like to dig into in the future.

But as usual, we'll save that for another day.

SmartOS Network Configuration

As I wanted my administrative interface to function over an link aggregation, I set the following in /usbkey/config:

# Aggregation from Intel 2x10GBE Interfaces (e2,e3)
aggr0_aggr=00:00:00:00:00:00,00:00:00:00:00:00
aggr0_lacp_mode=active

# Administrative Interface
admin_nic=aggr0
admin_ip=172.22.1.8
admin_netmask=255.255.255.224
admin_network=172.22.1.0
admin_gateway=172.22.1.1

# Additional Etherstubs
etherstub=external0,internal0

# Common Configuration
hostname=gz-1
dns_domain=ewellnet
dns_resolvers=172.22.1.1
ntp_conf_file=ntp.conf
root_authorized_keys_file=authorized_keys

Some brief highlights:

• aggr0_aggr refers to the interfaces to use in the link aggregation by hardware address.
• aggr0_lacp_mode ensures this side is actively participating in LACP, instead of passively waiting for another active party.
• admin_nic sets the administrative interface, in this case, to the link aggregation. The rest of the admin_ parameters are as set by the SmartOS installation.
• etherstub sets additional etherstubs to be configured upon boot.
• I'm setting my own custom ntp.conf so that I can use my global zone as a network time server across multiple subnets.

/usbkey/config.inc/ntp.conf:

driftfile /var/ntp/ntp.drift
logfile /var/log/ntp.log

# Ignore all network traffic by default
restrict default ignore
restrict -6 default ignore

# Allow localhost to manage ntpd
restrict 127.0.0.1
restrict -6 ::1

# Allow servers to reply to our queries
restrict source nomodify noquery notrap

# Allow local subnets to query this server
restrict 172.22.1.0 mask 255.255.252.0 nomodify

# Time Servers
pool 0.smartos.pool.ntp.org burst iburst minpoll 4
pool 1.smartos.pool.ntp.org burst iburst minpoll 4
pool 2.smartos.pool.ntp.org burst iburst minpoll 4
pool 3.smartos.pool.ntp.org burst iburst minpoll 4

SmartOS Zpool Configuration

After getting a hardware configuration together that worked for Illumos, I spent a few weeks testing various vdev configurations for performance and spatial efficiency. As the tests evolved over that time, I wasn't fully satisfied with the consistency of the methodology and will be rerunning those tests again for my own information as well as to feature in a future article. There were some pretty solid results that shone through through.

• ZFS pools based on three five-drive RAIDZ vdevs significantly outperformed pools based on two eight-drive RAIDZ2 vdevs in terms of sequential read performance (17.9% faster) and storage efficiency (7%).
• ZFS pools with special allocation class vdevs outperformed pools without them in terms of sequential read performance (18.3% faster).

The performance advantages of RAIDZ outweigh the resiliency advantages of RAIDZ2 in my case, as this pool configuration also has a hot-spare, reducing temporal exposure to loss of the pool. As well, critical datasets are regularly replicated off-site.

The zones pool of the new server was manually created during SmartOS installation with the following command:

# zpool create \
  -o autotrim=on -O atime=off \
  -O checksum=edonr -O compression=lz4 \
  -O recordsize=1M -O special_small_blocks=128K \
  zones \
    raidz c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 \
    raidz c1t5d0 c1t6d0 c1t7d0 c1t8d0 c1t9d0 \
    raidz c1t10d0 c1t11d0 c1t12d0 c1t13d0 c1t14d0 \
    spare c1t15t0 \
    special
      mirror c3t1d0 c4t1d0
      mirror c5t1d0 c6t1d0

Parameters of note:

• autotrim=on enables automatic trim for all trim-capable devices in the pool. In this case, all NVMe SSD based special vdev leaf devices.
• atime=off disables file access time updates, reducing metadata writes and improving throughput. This is a standard parameter used by SmartOS zones pools.
• checksum=edonr uses the edonr checksum instead of fletcher for filesystem checksum calculations. I had found during previous testing that out of all of the cryptographicly strong checksum algorithms available to ZFS, edonr performed the best. This should be re-verified.
• compression=lz4 explicitly uses lz4 for block compression, which is almost universally a good idea. This could also be set to compression=on and will use the ZFS default compression algorithm, which will attempt to balance compression speed with compression ratio.
• recordsize=1M raises the maximum record size from 128K to 1M. This improves performance for large sequential file access by keeping RAIDZ stripes on individual leaf devices relatively large (36K-256K) and ensures that a range of file sizes fit on the normal vdevs instead of the special vdevs, thanks to the next parameter.
• special_small_blocks=128K allows for data blocks up to 128K to also be stored on the NVMe SSDs instead of the hard drives. This should drastically improve random IO and overall throughput.

The combination of the last two parameters effectively creates a hybrid storage pool. All metadata and blocks of up to 128K go to one class of storage while blocks between 128K and 1M. By adjusting recordsize and/or special_small_blocks different storage properties can be achieved for different datasets.

Cache only metadata for swap zvol

I've never liked the idea of swap pages being cached in the ARC, and that happens by default in SmartOS. Fortunately it's easy to switch that behavior on and off at anytime:

# zfs set primarycache=metadata zones/swap

The above command will ensure that only metadata for zones/swap makes its way into the ARC, preserving it for normal file access. I can't really foresee a case where I would want to reverse this, perhaps with L2ARC devices installed in this pool? Either way it's rather trivial to revert back to the normal behavior with:

# zfs inherit primarycache zones/swap

Calming the Dragon (fans)

It was a bit of a surprise when I started this server up the first time after adding a non-certified-by-dell PCIe card. If that sounds like a bit of a shakedown, it is. It also sounded like the building was about to take off. I, as many before me had, discovered that Dell is very conservative when it comes to cooling devices that their firmware can't monitor the temperatures of. And by conservative, I mean liberal with the cooling.

Some people solve this problem by completely disabling the automatic thermal profiles and manually stepping up and down the fan speed via ipmitool and some cron scripts that run every minute.

And struck me as a horrible idea.

It would be so much better to continue to let the dedicated firmware that monitors system temperature to track component temperatures and adjust airflow to correct for it. Just if there was only a way to tell it not to worry about those PCIe devices behind the curtain.

Fortunately, at least someone at Dell agrees with me.

It turns out you can disable the third-party PCIe cooling response, preventing it from loudly complaining about additional PCIe devices in the system. The only utility required to change this behavior is ipmitool which is already part of the SmartOS global zone.

To check the current cooling response status, run the following command:

# ipmitool raw 0x30 0xCE 0x01 0x16 0x05 0x00 0x00 0x00

The following response means the third-party cooling response is disabled. Quiet.

 16 05 00 00 00 05 00 01 00 00

The following response means the third-party cooling response is enabled. Loud.

 16 05 00 00 00 05 00 00 00 00

To disable the third-party cooling response, run the following command:

# ipmitool raw 0x30 0xCE 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x01 0x00 0x00

To enable the third-party cooling response, run the following command:

# ipmitool raw 0x30 0xCE 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x00 0x00 0x00

This setting appears to be maintained across power cycles, but if you do find yourself in a situation where you need to run cards that run hot (like NV1604s), it would be wise to re-enable the default behavior to avoid that potential fire hazard.

So much for regular posts to this blog in 2021.

A lot of what I intended to write about this year hinged on access to better hardware. Not that I couldn't have done it with the N54L, it was more a case of only wanting to do it once instead of twice. So instead I focused on updating my homelab with the intent of writing about it once everything was in place.

Yeah. That took a little longer than anticipated. Months longer.

But hey, better late than never, right? Lets get into it.

Please note: this article is going to be focused on hardware. Software and networking configurations will be covered in a subsequent post.

Updated Demarcation Point

Well before signing up for Ziply fiber in 2019, my father and I made a weekend project of running some conduit vertically though the house from the mechanical room in the basement to the attic for horizontal cable runs throughout the house.

This turned out to be quite fortunate, since Optical Network Terminals need to be powered, and there was no readily available circuits on the external wall where my installer initially wanted to place the ONT. I volunteered installing the ONT in the mechanical room instead, and after running one additional conduit through the attic for the fiber and placing up some plywood, that's exactly what I got.

That ONT looked a little lonely up there all by itself, so I just had to add a few more things. These are some photos of the build, and what it looks like today.

Major components from right to left: Ziply enclosure (housing a FOG420 ONT), SiliconDust HDHomeRun HDHR5-4K network tuner with 3D printed mounting bracket from an excellent manufacturer on eBay, vertically mounted Cisco Catalyst 2960-S-24PD-L PoE switch on a 1U vertical mounting bracket. Power is routed along the bottom and data is routed along the top supported with Southwire J-Hooks and Allen Tel Distribution Rings on the left side of the board. Cables are secured to the distribution rings using Velcro straps.

The major regions of this board are ethernet on the left, coax (from a large antenna in the attic) in the middle, and the Gigabit Passive Optical Network (GPON) ONT on the right. There's still plenty of room in the middle of the board for additional network tuners or even a cable modem if I'm desperate.

Additional things to do on the demarcation point:

• Dress and properly anchor the antenna coax, and ethernet cables that run vertically down between the HDHomeRun and the ONT.
• Obtain the correct power cable for the server to bypass the UPS with one of it's power supplies. Might just do this with a flush cable extension, more on that later.
• Properly dress the power cables going along the bottom of the board.

Updated An Actual Core Switch

I've been running without a proper switch for years now. While this generally hadn't been an issue, I recently upgraded to EnGenius Wi-Fi 6 2x2 EWS357AP access points, and my new home hypervisor hardware has 10GBE ports, meaning I was going to outgrow the desktop switch I had sitting next to my current server. Any core switch was going to have the following requirements:

• 802.3af PoE support for Access Points and IP Phones. 802.3at PoE support a bonus, but no specific requirement for it.
• SFP+ ports to connect to the nearby hypervisor via DAC cables.
• 802.3ad Link Aggregation Control Protocol (LACP) support, because why connect a server over a single 10GBE link when you can connect over two?
• 802.1Q VLAN support so that I can properly sequester devices and run multiple isolated SSIDs.

While there are plenty of new switches that would fulfil these requirements, they can end up being pretty expensive. Enter the Cisco Catalyst 2960-S-24PD-L, a switch that has every feature I was looking for at a fraction of the cost on the used market.

Sure it's loud, and takes forever to boot up, but my new server is louder, and how often will I be rebooting this switch? My only real complaint with it so far is how generally finicky Cisco IOS is. This is probably just learning curve though, and again, is probably inconsequential once I get the switch configured as I need it.

A New Ceiling-Mounted Rack

I wanted to ensure that all of my homelab gear took up as little space as possible, as close to the demarcation point as possible. The natural solution was to bolt it all right to the ceiling.

This custom 7U (with room for an additional 1U above) rack is lag bolted to the ceiling joists. It perfectly occupies the space here; providing just enough clearance around the front and back of the rack for servicing the demarcation point, the back of the rack, and deploying gear into and removing gear from the front of the rack. While 7U may seem like an odd decision, it's just enough space for two 2U UPSes, a 2U (or two 1U) server, and an automatic transfer switch for any critical components that don't have redundant power supplies, which is the likely maximum deployment of hardware to this space.

Updated Server Hardware

While I'm still very fond of my HP N54L, I realistically outgrew it for a core hypervisor years ago in all categories:

• CPU (AMD Turion II Neo, 2-cores, 2.2Ghz): I can peg one core and nearly peg the other when running a speedtest over uncapped GPON using a SmartOS zone as a NATing router, which only hits around 75% advertised bandwidth. Real-time Plex transcoding is out of the question, as is line-speed rsync (via rsyncd, so no ssh overhead) over gigabit, all due to CPU limitations.
• Memory (16GB DDR3 ECC): Upon getting this system I immediately upgraded the memory to the maximum supported 16GB. This was great at the time, and I'd often see ZFS ARC hit rates of 99%, but as time went on and I ran more and more workloads on it, I found my c values and hit rates dropping, sometimes as low as 90% for stretches. I know, 90%, boo-hoo! This was more an issue that there's no additional room for memory upgrades if the need arises.
• Storage capacity (4x HGST 8TB SATA Hard Drives): I've been running at over 90% capacity on my ZFS pool for a year, and 99% for the last few months on a 4-drive RAIDZ based pool.
• Storage bandwidth: Besides some weird missing space issues in this pool, probably due to the disadvantageous 4-drive RAIDZ layout, the system doesn't have the best sequential read performance either, topping out at just over 300MBps. While this is perfectly fine for a gigabit limited network file server, it's less suitable for local containers, and a complete non-starter for any significant network upgrades.
• IO Expandability: The N54L comes with a single on-board gigabit ethernet port and PCIe 3.0 x16 and x1 ports. While the x1 port is pretty suitable for a second gigabit ethernet NIC, the fact that there's only a single x16 port means you're going to either be limited to improved networking (a quad 10GBE NIC or dual 40GBE NIC) or improved storage (NVMe card), but not both, which defeats the purpose of either.

Fortunately, a good friend was selling a server he had just grown out of for an excellent price. Enter the Dell R730XD that came with the following configuration:

• 2x Intel Xeon E5-2678v3 CPUs. Base clock 2.5GHz Turbo clock: 3.1GHz Cores: 12 Threads: 24 TDP: 120W
• 128GB DDR4 ECC Memory
• 16x Hitachi 8TB SAS Hard Drives
• Dell PERC H730 Mini Mono RAID Controller
• Dell F6PCP Emulex based Quad 10GBE SFP+ Network Daughter Card
• Dell iDRAC8 Enterprise
• Dell Rear Flex Bay 2.5" Drive Backplane Kit
• Two out of the three riser kits (missing Riser1)
• Dell PowerEdge 2U Ready Rails

I added the following additional hardware components:

• Dell 68M95 Intel X710 based Quad 10GBE SFP+ Network Daughter Card, as the Emulex based one wasn't recognized by drivers under Illumos and was flaky under FreeBSD. While this could probably be fixed, the Intel based card wasn't too expensive used and the cost could be recouped by selling the Emulex card.
• Dell PERC HBA330 Mini Mono Host Bus Adapter, as the H730 was flaky under Illumos while being load tested. The H730 could probably be resold for more than the cost of the HBA330, but I'm more apt to keep this around to try to figure out what's going on with the driver.
• Cisco QLogic QL45412HLCU dual port 40GB QSFP+ NIC intended for use in experiments with high speed connectivity to a single remote system (workstation) to see what that's like.
• ASUS Hyper M.2 x16 PCIe v3.0 x4 v2. A 4x4 M.2 NVMe to PCIe v3.0 x16 expansion card that allows for four M.2 NVMe drives to be connected to the host to experiment with special vdevs. This is the main reason I opted for the R730XD over the R720XD, as PCIe bifurcation is required to use this card effectively, and is only available on the later model.
• Dell Riser 1 (adding 3 PCIe v3.0 x8 slots) did this for the NV1604s mentioned below, but since I ended up not using them, this part wasn't required. I may end up loading them up with Supermicro AOC-SLG3-2M2 cards if I like special vdevs on ZFS and need more space, or perhaps Intel Optane drives to experiment with L2ARC/SLOG.
• Two Flashtec NV1604 NVRAM PCIe v3.0 x8 NVMe drives for use as mirrored SLOG devices. After having re-read the documentation, it turns out that these cards need to be directed to store their contents to flash upon every power-loss event (I'm really wishing storage review mentioned that). While this does improve flash longevity, it's also quite unfortunate, as it makes them ill-suited for use as SLOG devices: either a third party process will have to run in the global zone to manage these cards, or ZFS will have to be extended to do so. Neither of these approaches are overly palatable, so for now I'm running without any dedicated logging devices.
• Dell PowerEdge 2U Cable Management Arm. Completely unnecessary, but it makes the cable management that much better.

The ASUS Hyper card ended up conflicting with the power distribution cables of the drive midplane within the R730XD, so I had it milled down there, and around where the card support arm of the case would swing out to support the weight of the card. This is what it looks like after those modifications.

This server hardware has expansion opportunities in all directions, and should suit my needs for quite a few years to come.

A New Uninterruptable Power Supply

Since the new server was going to be rack mounted to the ceiling next to the demarcation point in the mechanical room, it really made sense to buy a rack mount UPS, as my other UPSes (all desktop form factor) would be out of place there. Used APCs work great, so I picked up an APC Smart-UPS (SMT2200RM2U) for cheap, added new batteries and tested to ensure that it works.

It works.

Getting the shelf to mount properly was something else though. The 10-32 screws on the front were milled down to fit just flush on the front rails and pairs of nylon washers were used on the back that also just fit in the rail holes to ensure that the rear screws were perfectly aligned when mounting. It was annoying, and took some work, but ultimately, the UPS mounted perfectly in the rack.

Since the server has dual redundant power supplies, one is plugged into the UPS while the other will be plugged into the upstream surge suppressor once I find the correct length and low-profile NEMA receptacle cable. This may change down the road if I get a second UPS, but this is fine for now, as my switch gear and ONT are also plugged into this UPS: if there's a power delivery issue inside the device, I'm going to have problems across the board.

Conclusion

With this all in place, I can start to focus on software updates and actually starting to make use of this. Look forward to more content next week!

Or maybe next year.

SmartOS manifests, also sometimes referred to as zone manifests, are JSON files that describe the resources and permissions to be allocated and granted to a given guest zone on SmartOS.  They are used by SmartOS global zones to instance discrete guest zones, either through Triton or directly on the global zone command-line through vmadm.

Since having the wrong manifest for a SmartOS zone can significantly impact its operation, it seems worth it to dedicate a full article to the topic.  We'll be looking at examples of the different types of SmartOS manifests, as well as exploring the specific properties of each.

Please note that the properties covered in this article are limited to ones I've found useful.  A more authoritative source of all of this information is probably the SmartOS wiki and the vmadm manual page.

I tend to keep my manifests on file in the global zone when running SmartOS as a stand-alone containerizor, usually under /usbkey/vmcfg/ or /usbkey/manifests/ for easy re-creation of zones.

Native Zones

Native SmartOS zones are an Illumos based isolated environment that passes everything through to the global zone kernel with no translation at all, except for the isolation provided by virtue of being a zone.

A slightly modified example manifest from the SmartOS wiki:

{
 "brand": "joyent",
 "image_uuid": "1d05e788-5409-11eb-b12f-037bd7fee4ee",
 "alias": "test-smartos",
 "hostname": "test-smartos",
 "cpu_cap": 200,
 "max_physical_memory": 1024,
 "quota": 20,
 "delegate_dataset": true,
 "resolvers": ["8.8.8.8", "208.67.220.220"],
 "nics": [
  {
   "nic_tag": "admin",
   "ips": ["dhcp"]
  }
 ]
}

For ease of reading, these properties tend to be grouped into the following general arrangements.  Properties marked with an asterisk (*) are required for each brand.

Administrative

• brand*: String that must be set to joyent or joyent-minimal for this zone type.
• image_uuid: String representing the image UUID that this zone should be instanced from.  Images are managed by imgadm.
• alias: String used for display/lookup purposes from outside the guest zone.
• hostname: String used to configure the guest zone's hostname on creation.

CPU/Process

• cpu_cap: Integer representing the percentage of each CPU core available to this zone.  A value of 300 represents up to 3 full CPU cores.
• cpu_shares: Integer representing the number of fair share scheduler (FSS) shares for this zone.  Only meaningful relative to other zones on the system and only applies when there is CPU contention between zones.  A value of 25 will mean this zone will only have access to \(\frac{1}{4}\) as much CPU time as another zone with the default value of 20.
• max_lwps: Integer representing the maximum number of threads a zone is allowed to run.  The default value of 2000 should be pretty reasonable.

Memory

• max_locked_memory: Integer representing the number of MiB of memory this zone is allowed to lock.  Locked memory are pages that are explicitly marked as non-swappable, and cannot exceed its default value of max_physical_memory.
• max_physical_memory: Integer representing the number of MiB of memory this zone is allowed to use.  The default value is 256.
• max_swap: Integer representing the number of MiB of virtual memory this zone is allowed to use.  This value must be greater than its default value of max_physical_memory or 256 whichever is greater.

Storage

• quota: Integer representing the number of GiB that this zone ZFS dataset should have its quota set to.
• delegate_dataset: Boolean that determines if a ZFS dataset will be delegated to this zone on creation.  If set to true, this zone will get a dataset at <zoneroot dataset>/data (default of: zones/<uuid>/data.)  This dataset can be configured many different ways to optimize for databases, snapshots, etc.
• indestructible_delegated: Boolean that determines if the delegated ZFS dataset should have a zfs hold set on it to enable two-step deletion.  Use this if you're really unsure about accidentally deleting your data.
• indestructible_zoneroot: Same as above but for the entire guest zone.
• filesystems: Array of JSON objects representing additional filesystems that would be outside of normal operation to be mounted within zones.  Below are the required parameters:
• filesystem.*.type: String representing the type of filesystem to be mounted, lofs for a bind mount, pcfs for a pc filesystem, tmpfs etc.
• filesystem.*.source: String representing the source directory from the scope of the global zone, primarily useful for lofs mounts.
• filesystem.*.target: String representing the mountpoint from the scope of the guest zone.
• filesystem.*.raw: String representing a raw device to be associated with the source filesystem, most often, this should be a device file for a drive.
• filesystems.*.options: Array of strings representing the mount options for this filesystem when it is mounted into the zone.  Eg: ["ro", "nodevices"]
• fs_allowed: String representing filesystem types this zone is allowed to mount.  If you're building SmartOS, you will want this as: "ufs,pcfs,tmpfs"
• tmpfs: Integer representing the number of MiB this zone is allowed to use for its tmpfs mounted at /tmp.  Cannot exceed its default value of max_physical_memory.
• zfs_filesystem_limit, zfs_snapshot_limit: Integers representing the limits on the number of ZFS filesystems and snapshots a zone can have.  Useful when combined with delegate_dataset to prevent runaway resource consumption.
• zfs_io_priority: Integer representing the zone's IO priority when operating on a system with IO contention.  Zones with values less than (or greater than) the default value of 100 will have their IO throttled (or prioritized) when both try to use all available storage IO.

Network

• resolvers: Array of strings representing DNS resolvers that will be assigned to /etc/resolv.conf upon zone creation.
• maintain_resolvers: Boolean that determines if vmadm should update guest zone resolvers when the above property is updated.  Default: false
• nics: Array of JSON objects representing a guest zone's network interfaces.  Below are the required parameters:
• nics.*.primary: Boolean representing which vnic should be used for this zone's default gateway and nameserver values.  Only useful with multiple nics.
• nics.*.nic_tag: String representing which physical nic or etherstub that this vnic should be associated with.
• nics.*.vlan_id: Integer representing what vlan tag should be used for this vnic.
• nics.*.interface: String representing the interface name this zone will use for this interface.  Always in the format of netX where  X is an integer \(\geq 0\).  This parameter is primarily useful for configuring zones with multiple nics.
• nics.*.mac: String representing the MAC address of a vnic.  This is useful when interfacing with external systems expecting a specific MAC address.
• nics.*.ips: Array of strings representing IPv4 CIDR or IPv6 CIDR addresses for a given vnic.  The special strings "dhcp" and "addrconf" can be used as well to represent the use of DHCPv4 and SLAAC or DHCPv6, respectively.
• nics.*.gateways: Array of strings representing IPv4 addresses that this zone should use as network gateways.  If multiple gateways are specified, OS-specific behavior will apply (eg round robin on SmartOS).  Not required if using DHCP.
• nics.*.routes: JSON object that maps network destinations to gateways.  Destinations (keys) can be either IP addresses or IP Subnetworks in CIDR notation.  Gateways can be either IP addresses or in the form of nics[0] or macs[aa:bb:cc:12:34:56].
• nics.*.allow_dhcp_spoofing, nics.*.allow_ip_spoofing: Booleans that determine if this zone vnic should be granted certain permissions.  DHCP spoofing is required for DHCP servers.  IP spoofing is required for routers.
• nics.*.allowed_ips: Array of strings representing additional IP addresses from which this vnic is allowed to send traffic.  This is useful for IP address failover schemes between multiple zones.
• nics.*.blocked_outgoing_ports: Array of integers representing port numbers to which this vnic is prevented from sending traffic.  Eg: [80, 443, 8080]

Additional Properties

• limit_priv: String representing the list of privileges that will be available to this zone.  The default is normally fine, but some applications may require special permissions to run properly, for instance FreeSwitch apparently needs "default,proc_clock_highres,proc_priocntl" to enable the use of high resolution timers with very small time values and for better control over its scheduling class, both probably important for low latency voice.  See man 5 privileges.
• customer_metadata: JSON object representing metadata to be associated with this VM.  This data can be accessed from within the guest zone by using the mdata-get command, even through this object, eg:

"customer_metadata": {
 "root_authorized_keys": "ssh-ed25519 <key data>",
 "user-script": "/usr/sbin/mdata-get root_authorized_keys > /root/.ssh/authorized_keys"
}

Linux Branded Guest Manifests

Linux Branded SmartOS zones are a Linux user-space with an additional translation layer that converts Linux ABI calls from the user-space into Illumos ABI calls before passing them on to the Illumos kernel, effectively allowing Linux user applications to operate under an Illumos kernel.

A slightly modified example manifest from the SmartOS wiki:

{
 "brand": "lx",
 "kernel_version": "4.2.0",
 "image_uuid": "63d6e664-3f1f-11e8-aef6-a3120cf8dd9d",
 "alias": "test-debian9",
 "hostname": "test-debian9",
 "cpu_cap": 400,
 "max_physical_memory": 4096,
 "quota": 1000,
 "resolvers": ["192.168.180.1", "8.8.8.8"],
 "nics": [
  {
   "nic_tag": "external",
   "vlan_id": 180,
   "ips": ["192.168.180.182/24"],
   "gateways": ["192.168.180.1"]
  }
 ]
}

The properties of Linux branded zones are almost identical to SmartOS zones, with the following differences:

Administrative

• brand*: String that must be set to lx for this zone type.
• kernel_version: String representing the version of Linux to report/emulate.

As of January 2021, not all ABI functionality of the latest Linux kernels is supported by the Linux translation layer, meaning that many modern distributions fail to function correctly.  This is being worked on.

HVM Guest Manifests

Hardware Virtual Machine (HVM) Guest zones contain a hardware virtualization suite utilizing either KVM or Bhyve to emulate hardware for any operating system that can run as a guest.

A slightly modified example manifest from the SmartOS wiki:

{
 "brand": "bhyve",
 "alias": "test-debian10",
 "hostname": "test-debian10",
 "vcpus": 4,
 "ram": 4096,
 "disks": [
  {
   "image_uuid": "9bcfe5cc-007d-4f23-bc8a-7e7b4d0c537e",
   "model": "virtio",
   "boot": true
  }
 ],
 "resolvers": ["208.67.222.222", "8.8.4.4"],
 "nics": [
  {
   "nic_tag": "admin",
   "ips": ["10.33.33.33/24"],
   "gateways": ["10.33.33.1"],
   "model": "virtio",
   "primary": true
  }
 ]
}

While there's quite a bit of divergence between the properties of OS (joyent and lx branded zones) and HVM (kvm and bhyve branded zones), most of the OS properties actually still apply, only to the zone performing the virtualization, not to the guest.

Please also note that some of these properties are specific to bhyve while others are specific to kvm.  I will try to illustrate which is which below:

Administrative

• brand: String representing which hardware virtualization suite to use for this VM.  Must be either kvm or bhyve.
• bhyve_extra_opts, qemu_extra_opts: Strings representing additional bhyve and kvm command-line parameters to be appended to the end of the commands.  While this was intended for debugging, it's also generally useful.
• boot: String representing the boot order for kvm VMs. Expected format is order=X* where X is either c for the hard drive, d for the first CD-ROM drive, and n for network boot.  eg: order=cdn would boot from the hard drive, CD-ROM drive, and network, in that order.
• bootrom: String representing the bootrom to use under bhyve.  Values are either bios, uefi or a path to a custom bootrom binary relative to the guest zone root.

CPU/Process

• vcpus: Integer representing the number of virtual CPUs the guest will see.  This property can be used with cpu_cap and cpu_shares to more closely control CPU utilization.

Memory

• ram: Integer representing the number of MiB of memory that will be made available to the guest kernel.  This should be used in place of max_physical_memory as it will need to allocate additional memory to handle the requirements of bhyve or qemu.

Storage

• disks: Array of JSON objects representing disks that should be associated with this VM.
• disks.*.block_size: Integer representing the block size of the disk.  This property can only be set during disk creation, and cannot be set when cloning a disk.
• disks.*.boot: Boolean representing if this disk should be bootable.
• disks.*.guest_block_size: String representing the device block size reported to the guest.  By default, the block size of the underlying device is reported to the guest.  This setting will override the default value.  It also supports reporting of both physical and logical block sizes using a string in the form of "logical size/physical size", eg: "512/4096" to look like a 512e drive.  Values must always be powers of 2.
• disks.*.image_uuid: String representing the dataset from which to clone this disk.  These images are managed by imgadm.
• disks.*.refreservation: Integer representing the size of this refreservation in MiB.
• disks.*.size: Integer representing the size of this disk in MiB.  This property is mutually exclusive from image_uuid, and is useful for creating empty disks.
• disks.*.media: String representing whether this disk is a "disk" or a "cdrom".
• disks.*.model: String representing the driver that should be used by the guest to access this disk.  Should be one of "virtio", "ide" or "scsi".
• disk_driver: String representing the default values for disks.*.model above.
• flexible_disk_size: Integer representing the number of MiB of storage space that a bhyve instance may use for its disks and snapshots of those disks.  This value should be larger than \(\sum_{d}\).

Network

• nics.*.allow_unfiltered_promisc: Boolean representing if this guest should be able to utilize multiple MAC addresses, eg: running SmartOS with vnics.  Really only suitable for testing containerizors from within a VM.
• nics.*.model: String representing the driver that should be used by the guest to access this vnic.  Should be one of "virtio", "e1000" or "rtl8139".
• nic_driver: String representing the default values for nics.*.model above.

Additional Properties

• vnc_port: Integer representing the TCP port that the VNC server attached to this VM will listen on.  0 (default) will choose a port at random, -1 will disable VNC server.
• vnc_password: String representing the password which will be required when authenticating to the VNC server.  This password will be visible from the global zone, and is limited to a maximum of 8 characters.

Plex is a client-server media player suite combined with a hybrid video streaming service, and is currently the most popular do-it-yourself method of media streaming.

While the Plex Media Player is open source, the Plex Media Server is closed source, and only distributed by Plex to a discrete set of platforms, excluding Illumos or SmartOS.  Fortunately, Plex works fine within an LX branded zone, which is exactly how we will be setting it up today.

SmartOS Zone Configuration

While Void Linux's incredibly light default memory footprint would be ideal for this, Plex tends to update their media server software regularly, and doesn't distribute packages for Void's package manager (XBPS); I would prefer a well integrated update mechanism, so this project is probably best suited for a Debian or Ubuntu image.

This zone will also be using a read-only lofs filesystem mount between this zone and the file server zone featured in a previous article.  This ensures the principle of least privilege, in that the Plex Media Server can only read media (and not modify or delete it) while also isolating our bulk storage in a different zone.  This specific approach unfortunately blocks this zone from automatically booting (as the source path is not available until the other zone has fully booted), and it will need to be manually started after your global zone boots.

Below is an example manifest, please note that {uuid} refers to the uuid of the bulk storage zone:

{
  "image_uuid": "63d6e664-3f1f-11e8-aef6-a3120cf8dd9d",
  "brand" : "lx",
  "kernel_version": "4.20",
  "alias": "plex",
  "hostname": "plex",
  "cpu_cap": 200,
  "max_physical_memory": 2048,
  "quota": 20,
  "delegate_dataset": true,
  "filesystems": [
    {
      "type": "lofs",
      "source": "/zones/{uuid}/root/home/brian/media",
      "target": "/media",
      "options": [ "ro" ]
    },
  ],
  "resolvers": [ "1.1.1.1" ],
  "nics": [
    {
      "nic_tag": "admin",
      "ips": [ "dhcp" ],
      "primary": true
    }
  ],
  "customer_metadata": {
    "root_authorized_keys": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDrStZlHS0yfE8n71meairBGvFnc5mlDFNKAJy7tQMi2",
    "user-script": "/usr/sbin/mdata-get root_authorized_keys > /root/.ssh/authorized_keys"
  }
}

For more details about the properties configured for this manifest, please read this article.

We're expecting regular and predictable CPU and memory usage per transcoded stream from this zone, if you're experiencing transcoding performance issues, you may need to increase the values for cpu_cap and max_physical_memory.  In testing I've found that 1GiB of memory is suitable for a single viewer who is direct streaming video.  These requirements are likely much higher for multiple transcoding users.

The quota value of 20GB is fine for many collections, as that only applies to this zone (which only holds the plex databases and meta-files, not the actual media files, which have been mounted from another zone).  Depending on how large your media collection grows, this value may need to be increased.

Ansible Play

This Ansible play is relatively simple and straight forward.  It's broken up into two parts, a common-debian role that mirrors the common role for smartos:

• The PATH variable includes executables under /native.
• The hostname has been set.
• The default delegated ZFS filesystem has been unmounted.
• All packages have been upgraded.
• SSH server configured to only accept public key authentication.

As well as a plex role that performs plex specific configuration on the zone:

• The Plex package signing key has been added to apt.
• The Plex repository has been added to apt.
• A ZFS filesystem is mounted at /var/lib/plexmediaserver.
• A recordsize tuned ZFS filesystem has been mounted for the Plex sqlite3 databases.
• Ownership of /var/lib/plexmediaserver is set to 999:999 (necessary for plex to install to it.)
• Installing Plex Media Server
• Enabling Plex Media Server
• Ensuring that system will update all packages regularly, including Plex Media Server.

Below is an example ansible play:

---
- name: 'Testing Plex'
  hosts: plex
  roles:
  - plex
  vars:
    hostname: plex

Plex Configuration

This Plex Media Server will now need to be associated with your Plex account.  Point a web browser to https://<ip address>:32400/web to continue.

Port Forwarding

If your network employs NAT, you may want to port forward your plex server at your router to ensure that clients outside of your local network can have access.

If you are using a SmartOS Zone to route IP traffic for your network, ensure that the following line exists within /etc/ipf/ipnat.conf, where <external ip> is your router's public IP address and <internal ip> is your plex server's IP address:

# Plex Redirection
rdr net0 <external ip>/32 port 32400 -> <internal ip> port 32400 tcp

Conclusion

The read-only lofs filesystem mount might be better employed mounting a ZFS source filesystem that exists outside of any zone, with a second read-write lofs filesystem mount connecting that same source filesystem to another zone to allow for editing.  This configuration might end up being more trouble than it's worth as well, since ZFS and non-ZFS filesystem mounting is not interlaced when Zones boot up.

For now, I just remind myself to manually start the Plex zone whenever I reboot the global zone.  Perhaps there's an opportunity to extend SmartOS to respect Zone boot order, specifically allowing Zones to specify dependency zones that need to be booted up before they can boot.

Also, a special thanks to the Lights and Shapes blog for writing about this significantly earlier than I have.

While TrueNAS is a powerful tool for setting up simple and easy network addressable storage, there's a lot of unnecessary feature overlap between TrueNAS and SmartOS.  Both operating systems prefer direct hardware access, and passing through ZFS volumes to HVM for TrueNAS to format as ZFS would just be sacrilege to both parties.

Fortunately, it's trivial to configure a lightweight SmartOS zone to act as a network addressable storage (NAS) provider, albeit without the graphical configuration and monitoring of TrueNAS, but instead with the raw expressive power of configuration files!

Seriously though, the configuration is not that hard, and results in a solution that's both minimalistic and unassuming.  Unlike TrueNAS, the approach documented here will bifurcate Samba configuration from snapshots/replication and system resource monitoring, two topics that we will address separately with more robust, hypervisor encompassing solutions in later articles.

Why Samba?

The official SmartOS wiki describes a method for serving SMB from the kernel.  While this works, it doesn't quite offer the flexibility that Samba offers you.

Unmap Drives

Depending on how you've configured Samba, if you're upgrading from a previous SmartOS virtual machine image, it can be helpful to disconnect any drive mappings from your previous NAS before proceeding.

SmartOS Zone Configuration

We'll be providing NAS service from an isolated SmartOS zone.  Below is an example manifest:

{
  "image_uuid": "1d05e788-5409-11eb-b12f-037bd7fee4ee",
  "brand": "joyent",
  "alias": "samba",
  "hostname": "samba",
  "cpu_cap": 100,
  "cpu_shares": 25,
  "max_physical_memory": 256,
  "quota": 20480,
  "zfs_io_priority": 25,
  "delegate_dataset": true,
  "resolvers": [ "10.0.0.1" ],
  "nics": [
    {
      "nic_tag": "admin",
      "ips": [ "10.0.0.3/24" ],
      "gateways": [ "10.0.0.1" ],
      "primary": true
    }
  ],
  "customer_metadata": {
    "root_authorized_keys": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDrStZlHS0yfE8n71meairBGvFnc5mlDFNKAJy7tQMi2",
    "user-script": "/usr/sbin/mdata-get root_authorized_keys > /root/.ssh/authorized_keys"
  }
}

For more details about the properties configured for this manifest, please read this article.

Be sure to set the resolvers, nics and root_authorized_keys parameters as appropriate for your environment.  Create the zone, and assign its IP address to a relevant Ansible host inventory.

Samba Configuration

Use the samba ansible role from ansible-smartos-tricks to configure Samba on your zone.  This will automatically deploy a ZFS filesystem at /home if a dataset has been delegated, and configure Samba as is normally comes configured when installed directly by pkgin.

Additionally, this role can modify the Samba configuration through the samba variable.  Included below is my default Samba configuration modifications that specify a workgroup name, limit remote host access, and include shadow_copy2 on the homes shares, configured to treat any ZFS snapshots as VSS snapshots made available over Samba:

---
- name: 'Configuring Samba Servers'
  hosts: samba
  roles:
  - samba
  vars:
    vim:
      colorscheme: elflord
    samba:
      global:
        'workgroup': 'LOCALNET'
        'hosts allow': '192.168.0. 127. fe80::'
      homes:
        'vfs objects': 'shadow_copy2'
        'shadow:snapdir': '.zfs/snapshot'
        'shadow:sort': 'desc'
        'shadow:format': 'r%Y%m%d%H%M%S'
        'shadow:snapdirseverywhere': 'yes'
        'shadow:crossmountpoints': 'yes'

For a stand-alone server, both system and corresponding Samba users will need to be created in the zone to access the samba share, this can be done as follows:

[root@samba ~]# useradd -m brian
144 blocks
[root@samba ~]# smbpasswd -a brian
New SMB password:
Retype new SMB password:
Added user brian.

At this point, you should be able to connect to the server and login, and access your shares.

Managing ZFS Snapshots

The vfs_shell_snap module can be added to your Samba shares, allowing remote users to manage the manual creation and deletion of snapshots on your shares.  I prefer not doing this, instead leaving it to automatic server side processes, as control over this functionality may be hijacked by ransomware and maliciously used against the user.

ClamAV Configuration

Samba also ships with the vfs_virusfilter module that allows for the scanning and filtering of virus files on Samba file servers with an external anti-virus scanner, such as ClamAV.  While this may seem cool, ClamAV requires over 1GB of additional memory to operate.

The following ansible play enables ClamAV along side Samba on this zone and configures Samba to scan files with ClamAV before serving them to clients.

---
- name: 'Configuring Samba Servers'
  hosts: samba
  roles:
  - clamav
  - samba
  vars:
    vim:
      colorscheme: elflord
    samba:
      global:
        'workgroup': 'LOCALNET'
        'hosts allow': '192.168.0. 127. fe80::'
      homes:
        'vfs objects': 'virusfilter shadow_copy2'
        'virusfilter:scanner': 'clamav'
        'virusfilter:socket path': '/var/clamav/clamd.sock'
        'shadow:snapdir': '.zfs/snapshot'
        'shadow:sort': 'desc'
        'shadow:format': 'r%Y%m%d%H%M%S'
        'shadow:snapdirseverywhere': 'yes'
        'shadow:crossmountpoints': 'yes'

Dataset Transfer

Datasets from a previous file server zone can be transferred to this new file server zone using some previous documentation on the topic.

Active Directory

Starting from version 4.0, Samba is able to function as an Active Directory Domain Controller.  Since this configuration should usually be utilized with multiple DCs for failover reasons, we'll save it for another article.

Remap Drives

Once you are satisfied with your file server zone, be sure to remap any drives you had previously unmapped from your workstations.

Multicast DNS

If you'd like your shares to be auto-discoverable in the MacOS X finder, enable the multicast DNS service with the following command:

[root@samba ~]# svcadm enable svc:/network/dns/multicast:default

This should probably be managed via Ansible, but it seems kind of silly to configure a role for it now.  Perhaps later.

Conclusion

While this isn't the only way to run a SMB file server from within a SmartOS Zone, it is a nice way to do it.  If there's interest, I might end up benchmarking Samba vs the official recommended approach from the SmartOS Wiki.

It's easy to have a love-hate relationship with Infrastructure Automation.

On one hand, the principle of Infrastructure Automation is fantastic: Using portable and well defined modules of code to ensure consistent deployments of software and configurations to potentially hundreds or thousands of physical or virtual machines is how we should have always done system administration, and in retrospect, it's a shame that it took cloud computing for all of us to see that.  The improvements to fault detection, consistency and delivery time are massive boons to rapidly expanding deployments maintained by relatively small teams.

On the other hand, the implementation of Infrastructure Automation is... less than fantastic.  Chef and Puppet both initially present as resource hogs that are not trivial to setup or manage and have questionable economies of scale.  SaltStack, while better, wasn't nearly as easy to get up and running as it should have been.  Terraform and CloudFormation both look interesting, but appear to be focused as provisioning tools instead of configuration management tools, and in the case of the later, is AWS only.  It seems like Terraform also requires a third party provisioning provider to use with SmartOS, which while I'm not opposed to using, I'd rather not have to learn Terraform with that added complexity.

That leaves us with Ansible, which while easy to get started with, both in configuration and use, quickly becomes less palatable as your deployments increase in complexity.  This specifically manifests in how Ansible uses YAML to describe tasks.  Simple tasks are easy, but as soon as flow control structures such as branching or looping are introduced to your plays, the YAML structures become anything but minimal.  There appears to be a complete ignorance of the DRY principle as well with an incredible preference to write everything multiple times, but it's what we've got for now, so we're going to go with it.

Stupid SmartOS Tricks

That leads us to changes in how articles will be published to this blog moving forward.

In the principle of not repeating oneself (DRY), articles will focus on the overarching intentions and concerns one might encounter when deploying something to SmartOS.  Instead of fully describing all required commands and configuration steps to achieve a given outcome, articles will instead link to a companion Ansible role in the accompanying ansible-smartos-tricks playbook that will perform that deployment.

This should also be very helpful to ensure consistent deployment of software for the previously-mentioned plans of benchmarking SmartOS against FreeNAS and Proxmox.

The roles in this playbook are organized in the following fashion:

• A common role that will perform all boiler plate configurations to any base SmartOS zone including cleaning up and managing ZFS datasets, and disabling inetd and sac as recommended in this article.
• Service roles that depend on common and zero or more other service roles and install and configure commonly required programs and services including mysql, neo4j, nginx, postgresql, redis and samba.

Installation

Ansible plays should be run from their own isolated SmartOS zone, as this system will need root access to any other system configured by it.  Below is an example manifest:

{
  "image_uuid": "1d05e788-5409-11eb-b12f-037bd7fee4ee",
  "brand": "joyent",
  "alias": "ansible",
  "hostname": "ansible",
  "cpu_cap": 100,
  "max_physical_memory": 1024,
  "quota": 10,
  "resolvers": [ "10.0.0.1" ],
  "nics": [
    {
      "nic_tag": "admin",
      "ips": [ "10.0.0.2/24" ],
      "gateways": [ "10.0.0.1" ],
      "primary": true
    }
  ]
}

If you have any questions about the properties chosen for this manifest, please read this article.

Create the zone, zlogin to it, install git, clone ansible-smartos-tricks locally and then run ./ansible-bootstrap.sh from within the ansible-smartos-tricks directory.

Output has been omitted for brevity:

[root@home-gz ~]# vmadm create -f ansible.json
[root@home-gz ~]# zlogin <uuid or "ansible<tab>">
[root@ansible ~]# pkgin -y install git
[root@ansible ~]# git clone https://github.com/brianewell/ansible-smartos-tricks
[root@ansible ~]# cd ansible-smartos-tricks
[root@ansible ~/ansible-smartos-tricks]# ./bootstrap.sh

The bootstrap script will handle all of the rest of the configuration for you, including installing Redis locally and configuring ansible to utilize it to cache remote host facts, significantly improving ansible performance, as well as automatically ensuring the existence of an SSH key-pair to authenticate to remote systems when configuring them.

Using SmartOS Tricks

Ansible uses SSH to connect to and configure remote systems, and this project specifically uses ed25519 keypairs to handle authentication.  While you could manually copy the public key to each system you'd like to configure, it's much easier and more consistent to include the key at the end of each manifest under the customer_metadata key.  An example:

{
...
  "customer_metadata": {
    "root_authorized_keys": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDrStZlHS0yfE8n71meairBGvFnc5mlDFNKAJy7tQMi2",
    "user-script": "/usr/sbin/mdata-get root_authorized_keys > /root/.ssh/authorized_keys"
  }
}

So far I've just been using static hosts in Ansible host inventories.  You can either remember the IP address you set a given host to, or use vmadm to discover it from the global zone:

[root@gz ~]# vmadm list -o alias,nics.0.ips
ALIAS       NICS.0.IPS
doudna      10.0.0.3/24,addrconf
plex        10.0.0.4/24,addrconf
router      10.0.0.1/24,addrconf
metrics     10.0.0.5/24,addrconf
ansible     10.0.0.2/24,addrconf
test        dhcp,addrconf

Please note that zones using DHCP will not report their IP address through vmadm.  Instead, it's probably best to login to the zone and check using ipadm show-addr from within the zone:

[root@test ~]# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net0/?            dhcp     ok           10.0.0.201/24
lo0/v6            static   ok           ::1/128

Place any hosts that you would like Ansible to configure into the ansible static hosts inventory file.  In this example we will include 10.0.0.201 as a member of the test inventory:

[root@ansible ~]# cat /etc/ansible/hosts
[test]
10.0.0.201

You can now create plays directly within ansible-smartos-tricks that refer to the provided roles, an example that applies the common role to test hosts:

[root@ansible ~]# cat ~/ansible-smartos-tricks/common.yml
---
- name: 'Common Role'
  hosts: test
  roles:
  - common
  vars:
    vim:
      colorscheme: elflord

Future Plans

It may seem by the title of this article that this is some kind of endorsement for Ansible, but I am honestly confused how software as misery inducing as this can be so popular.  Most 'sophisticated' Ansible plays appear to be written by closet masochists who enjoy typing a lot , and while that probably also says something about me, it also looks to be an incredible opportunity to do better in the space of Infrastructure Automation.

I've already started designing an infrastructure automation replacement for Ansible.

For now though, I will be using Ansible with SmartOS until my research indicates that a different tool would be preferable, either of my own or someone else's design.

What started years ago as a wild goose chase for a scalable first-class hyper-converged infrastructure ultimately led to the fantastic discovery of Joyent SmartOS.

While I've barely looked back since, the world has changed significantly since then, especially with the exodus of multiple high profile Joyent engineers from that company.  I occasionally wonder how my conclusions would have changed had I been searching in today's technological climate.

In this article, we'll explore the strengths and weaknesses of SmartOS in the context of today's open-source hyper-convergence space.

SmartOS (Illumos)

Joyent SmartOS is an Illumos based open-source hypervisor/containerizor that integrates Crossbow, DTrace, KVM, Bhyve, ZFS and Zones into a light weight in-memory solution which can boot from either a local USB drive, a boot dataset embedded into the primary storage pool via piadm, or over PXE.

Pros:

• Lightweight ephemeral in-memory global zone that is relatively immutable, improving security and enabling easy upgrades by simply re-deploying a boot image and rebooting.
• Supports both containers (zones) for maximum performance and HVM (KVM or Bhyve) based guests for maximum flexibility.
• Strong default isolation between guests without any additional configuration.  HVMs are run from within a zone, providing an additional layer of security between the guest and the host.
• Zones can be Linux (lx) branded, which allows for Linux user-spaces to exist natively on Illumos.  Debian, Ubuntu, and CentOS zone images are included by default.
• A ZFS dataset can be delegated to a zone, allowing the zone to define and configure their own children datasets.
• DTrace is accessible from both inside and outsize of zones, allowing for incredibly detailed instrumentation of production deployments.
• Crossbow network virtualization allows for complex virtual networks to be configured between guests of a single hypervisor, or bridged between multiple hypervisors.
• Guests can be tightly constrained by to follow very specific CPU, memory, file system and network restrictions.
• Scalable, up to thousands, or down to a single host.
• Support for Docker, Kubernetes and Object Store (Manta) through Triton Data Center.
• Rapid Updates.  Joyent usually has new releases of SmartOS available every two weeks.
• Strict local node storage architecture ensures low file system latency and node independence.
• Uses NetBSD's pkgsrc for package management.
• Optional management layers such as Triton DataCenter and Project Fifo allow for large clusters of hosts to be easily managed, as well as providing additional features.

While none of these pros are completely unique to SmartOS, I have yet to find a project that incorporates all of them so well, even today.  I'm probably biased.

Cons:

• DTrace doesn't work across the HVM boundary.
• Strict local node storage architecture means migration between compute nodes requires a ZFS send/recv to push guest data from the source node to the destination node, making instant migrations between hypervisors impossible.
• SmartOS limits crossbow configurations to specific conventions.
• Does not support as wide an array of hardware or release new drivers as quickly as Linux does.
• LX branded zones do not support the latest Linux kernel interfaces, making them ill suited for the latest versions of leading Linux distributions.
• Illumos has many fewer active developers than Linux does.
• Illumos ZFS rather than OpenZFS.

While I don't give the first three points on this list of cons much attention, the later ones have become a much bigger deal-breaker in the past few years.  Linux has long been the focal point of performance improvements and technological innovation in this space, and any technological capital that may have been built up under Sun has almost certainly been eclipsed by Linux at this time.

Maybe.

There are certainly important performance metrics in which Linux surpasses Illumos.  There are also technologies in the Linux ecosystem that completely fail to solve their intended problems.  The best example is probably epoll: that took over a decade for Linux to "get right", despite prime working examples from multiple other predating operating system implementations.

And that's fine.

While the number of developers contributing to a project doesn't necessarily have a causal effect on the quality of that project, it's still quite concerning to observe the continuing ablation of Illumos engineering and outreach talent from Illumos is very concerning for the future of the operating system.

Probably my most significant concern is the combination of the above point and what appears to be the divergence of Illumos from OpenZFS.  Illumos had enjoyed being the implementation of reference for years, but that changed early last year with what was likely a self-inflicted wound on Illumos' part and the ZFS on Linux repo being renamed to openzfs/ZFS, that is no longer the case.  Illumos ZFS will either need to maintain feature parity and ideally binary compatibility with OpenZFS, or port OpenZFS into Illumos.

The bad news: both of these prospects require Illumos kernel developer time, which is at a premium right now.  The good news: this has clearly registered as a priority for the Illumos developers, and work appears to have been made towards porting OpenZFS into Illumos.

Lastly: I'm not sure what exactly Samsung's intentions with Joyent are.  Since purchasing Joyent in 2016, there has been a marked change in the way that Joyent does business, beginning with sweeping changes to the way it communicates to the public about its product offerings, and probably culminating in the 2019 closure of the Joyent Public Cloud.  Besides the lack of any recent innovations and exodus of top-tier talent, it generally feels like Joyent is just treading water, and that's not a good position to be in for the long run.

Alternatives

As of January 2021, there are numerous projects that overlap quite heavily with SmartOS.  Lets briefly review some of them.

TrueNAS Core (FreeBSD)

iXsystems' TrueNAS Core is a FreeBSD based open-source Network Addressable Storage (NAS) Operating System that provides data accessibility through SMB, AFP, NFS, iSCSI, SSH, rsync, and FTP/TFTP, all managed through a nice shiny web interface.  While it's main use case is as a NAS, it can also locally run FreeBSD Jails and BHyve, giving it hypervisor and containerizor functionality.

TrueNAS includes DTrace and OpenZFS, both of which are well supported on FreeBSD based operating systems.  The installation process, while quite straight forward, does rely on installation to separate boot media which, unlike an ephemeral in-memory image, is continually being written to during normal operation.  This will wear out SD cards and USB flash drives given enough time, meaning that your best bet will be to install directly to a hard drive or solid state drive.  In most of my configurations, such drive space comes at a premium since I'm usually using that space for my primary storage pool instead.  This also makes it a bit less convenient to upgrade.

While FreeBSD based operating systems should have access to Open vSwitch, it is unclear how accessible this feature is through TrueNAS.  Meaning that custom network configurations may not be easily established without additional work.

In the past, iXsystems had been rather inconsistent in their release schedule and some of their updates have been full of regressions (lookin' at you FreeNAS 10).  This appears to have been ironed out with completely after the TrueNAS re-branding.

TrueNAS Core is definitely worth looking back into, as they have made major leaps and bounds on their platform since I last directly used FreeNAS.

Docker Engine (Linux)

Docker is the world's most adopted containerizor solution, and can be directly installed into a pre-existing Linux installation.  Docker is based on Linux Containers (LXC) and utilizes Linux cgroups to isolate processes from each other and create virtual environments, similarly to FreeBSD's Jails and Illumos' Zones.

Having worked with the precursor to Linux Containers, I probably would have been at home with the feature set of LXC and the convenience and consistency of Docker.  However, there were no solidly integrated light-weight docker-based solutions that integrated ZFS at the time I was searching for a solution to my problem at the time, which is how I ended up switching to SmartOS.

There are a few well-packaged and delivered solutions now though.

Proxmox VE (Linux)

If Proxmox Virtual Environment existed then as it does now, I would probably never have looked any further.  It's a hypervisor/containerizor with a pretty web-based interface that has buttons and graphs, making it visually appealing and generally easy to use.  It ships with OpenZFS, actually supports PCIe hardware passthrough to virtual machines, has what appears to be solid Open vSwitch integration and can scale up with it's clustering support.  It looks to be generally adaptable to various contortions that I'd be sure to put it into.  It's basically perfect.

Almost perfect.

Like TrueNAS, Proxmox VE needs to be installed onto a physical drive, both for the same reasons and with the same caveats.

While the UX is very nice, it is easier than it would seem to misconfigure things at times.  I know that's ridiculous coming from someone who primarily works on CLIs.  Dockers can be run from the Proxmox VE host, but due to the differences between "application containers" (docker) and "system containers" (pct), that practice is discouraged in the official documentation.  Yes, they're both LXC based, but they're different above that layer, and apparently incompatible to manage.

As with TrueNAS, Proxmox VE is definitely something that's worth looking into.

OpenStack, OpenQRM, OpenNebula, oVirt (Linux)

I don't know why but I'm just not interested in any of these projects.  They present as generally less suitable than the other options already listed above, usually due to appearing too large and inflexible.  They're just not exciting.

I may end up exploring some of these projects in the future, but there are currently no foreseeable plans to do so.

Oxide

Technology is always moving, and while they have yet to release any products, the cloud technology company that Bryan Cantrill and quite a few other Joyent talent is at the center of deserves an honorable mention here.

If Cantrill's public speaking event around the time of his leaving of Joyent is any indication, Oxide will be an attempt at building a full cloud-scale operating system using the Rust programming language and all of the experience and expertise that team embodies.  It will definitely be worth keeping an eye out for any significant announcements.

Conclusion

While SmartOS has been a good fit over the last decade, there are looming uncertainties on the horizon which lead me to question if that will still be the case come 2030.

Fortunately, there are also a lot of options moving forward.

If all goes as planned, these options will be benchmarked against SmartOS and each other on bare metal running both simple and complex workloads.  Keep an eye out for that hopefully sooner rather than later.

For as long as there have been multi-user operating systems, there has been the need to switch between those users.  Clearly, this can be done by directly starting a session as a given user, or even logging in again through localhost, but this approach tends to break down when manipulating system users (which are never meant to be directly logged into) or performing complex cross-user automation.

Today we will be exploring the command-line methods available on SmartOS for executing commands as other users, namely su and sudo.

su

The switch user (su) command executes a new shell owned by the specified user (or root if no user is specified).  This effectively allows the ownership of a session to be changed without logging off to assume the role of the new user.

Non-superusers attempting to switch users will be prompted for the login credentials of the user being switched to, just as they might be if they were logging in directly from a terminal.  Superusers are never prompted for login credentials when using su.

A few examples:

# su - brian
$ su -
Password:
#

The - parameter before the username further configures the login environment with the following additional changes:

• The LC*, LANG and TZ environment variables from the specified user's environment are also propagated to the new shell.
• Sets the MAIL environment variable to /var/mail/new_user.

Any parameters after the user will be passed to the executing shell, effectively emulating sudo's general functionality:

# su - brian -c whoami
brian

Additionally, the behavior of su can be modified by altering configuration parameters in /etc/default/su, specifically the following:

• SULOG all attempts to use su are logged to the specified file.
• CONSOLE if defined, all attempts to su to the superuser are logged to the console.
• PATH sets the default path of a shell spawned by su.
• SUPATH sets the default path of a superuser shell spawned by su.
• SYSLOG uses syslog to log all su attempts.

This command is the original and the simplest of the three, but you still may want to read the man page for su for additional information.

sudo

The sudo command permits users to execute commands as other users as allowed by a sudo specific security policy.  This effectively allows the ownership of a single command to be changed without disrupting the rest of the session to assume the role of the new user.  The major differences between su and sudo are as follows:

• sudo allows any command to be run as a trailing parameter, not just the user's shell.  sudo can also be passed the -i parameter to open an interactive shell, effectively emulating the functionality of su.
• sudo checks escalations against a security policy, allowing for fine-grained control over privilege escalation.
• sudo prompts users for the originating user's credentials while su prompts users for the credentials of the user being switched to.

By default, the security policy is configured in /opt/local/etc/sudoers.

Notice: the sudoers file should always be edited with visudo instead of directly.

Beyond global parameters, the sudoers file specifies host, user and command aliases:

User_Alias ADMINS = brian, notbrian, alsonotbrian
Cmnd_Alias PROCESSES = /usr/bin/nice, /bin/kill, /usr/bin/renice, /usr/bin/pkill
Cmnd_Alias REBOOT = /sbin/halt, /sbin/reboot, /sbin/poweroff

As well as user privilege specifications:

root ALL=(ALL) ALL

This specification allows root to run any command as any user.

%sudoers ALL=(root) /bin/kill, (operator) /bin/ls

This specification allows a member of the sudoers group to run /bin/kill as root and /bin/ls as the operator user.

If the included flexibility wasn't enough, sudo is also a plugin-based architecture, which can be extended in many different ways.  I would recommend thoroughly reading the sudo and sudoers manpages, as sudo is as complicated as su is simple, and the entire scope of its functionality is way beyond the scope of this brief post.

Conclusion

If you need to escalate yourself to a superuser role or need to quickly and simply switch into another role, su should be your go-to command.  It's simple, direct, and requires very little additional configuration or tweaking.

If you're working in a more complex multiuser environment and finer grained access control is a requirement, sudo is going to be your weapon of choice.  Additionally, I find sudo more convienent if I need to perform a single command as a different user rather than entirely switching my context to them.

Ultimately, depending on the context, I use both.

Additionally, SmartOS supports an additional privilege escalation framework in profiles and Role Based Access Control (RBAC), however that is significantly more complicated than even sudo, and will be the topic of a future article.

SmartOS Zones make for excellent blank slates to do development or production work from.

Except as it turns out, they're not blank slates.  The two most minimal Zone images, base and minimal start out with over a dozen running processes on them.

What are those processes and what functionality do they provide?  Which of them can we disable if we need or want to?

Environment

Since SmartOS Zones version 16.2.0 released yesterday, let's spin up a base Zone image and check out our running processes.

Here's the manifest I used for this demonstration:

{
        "brand": "joyent",
        "image_uuid": "13f711f4-499f-11e6-8ea6-2b9fb858a619",
        "alias": "base_test",
        "hostname": "base_test",
        "max_physical_memory": 256,
        "quota": 20,
        "resolvers": [ "8.8.8.8", "8.8.4.4" ],
        "nics": [ {
                        "nic_tag": "admin",
                        "ip": "dhcp"
                } ]
}

Processes

Notice: I performed this test using version 16.2.0 of both base-64 and minimal-64.  Besides a few slight deviations (minimal calls rsyslogd -c5 -n and base does not) all running processes were the same.

Immediately after logging in, I polled the process list:

# ps ax
   PID TT       S  TIME COMMAND
  7239 ?        S  0:00 zsched
  7299 ?        S  0:00 /sbin/init
  7323 ?        S  0:00 /lib/svc/bin/svc.startd
  7328 ?        S  0:02 /lib/svc/bin/svc.configd
  7384 ?        S  0:00 /lib/inet/ipmgmtd
  7607 ?        S  0:00 /usr/sbin/nscd
  7625 ?        S  0:00 /sbin/dhcpagent
  7626 ?        S  0:00 /usr/lib/pfexecd
  7774 ?        S  0:00 /opt/local/sbin/rsyslogd
  7779 ?        S  0:00 /usr/sbin/cron
  7782 ?        S  0:00 /usr/lib/inet/inetd start
  7784 ?        S  0:00 /usr/lib/saf/sac -t 300
  7787 ?        S  0:00 /usr/lib/saf/ttymon
  7788 ?        S  0:00 /usr/lib/utmpd
  7831 ?        S  0:00 /usr/lib/ssh/sshd
  7994 pts/2    S  0:00 /usr/bin/login -z global -f root
  7995 pts/2    S  0:00 -bash
  8062 pts/2    O  0:00 ps -ax
  7796 console  S  0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p base_test console login:

We can immediately discount the three processes tied to the pts/2 terminal since those processes are associated with our active login.

zsched

Each active zone has an associated kernel process, named zsched.  This process owns all kernel threads doing work on behalf of the zone, and enables the zones subsystem to keep track of per-zone kernel threads.

This process is critical to the proper functioning of a zone, and as such, it is not possible to disable this process from within the zone.

init

In traditional UNIX, init is the "father of all processes" that was responsible for spawning and restarting service processes that made up the running operating system.

Since Solaris 10, most of this responsibility has now been offloaded to the Service Management Facility (SMF).  Init is now primarily responsible for initializing core components of SMF (namely svc.startd and svc.configd) and restarting them if they fail.

This process is automatically restarted by the Illumos kernel if it is killed, and as such, it is not possible to disable this process from within the zone.

svc.startd

This process is the master process management daemon for the Service Management Facility subsystem.  It's responsible for starting, stopping, restarting, and signaling services based on administrative requests as well as system or application failures.

While this process can be disabled (/etc/inittab), doing so would disable SMF entirely, and is not recommended.

svc.configd

This process is the configuration repository daemon for the Service Management Facility subsystem.  It is responsible for maintaining the configurations for all services on the system, as well as passing administrative requests for services to be started, stopped, restarted, or signaled to the master process management daemon (described above).

This process is automatically started by svc.startd and cannot be independently disabled.

ipmgmtd

This process handles administrative events for network IP interfaces and IP/TCP/UDP/SCTP/ICMP tunables.  It is managed by SMF and provides the back-end that ipadm uses.

While this process can be disabled (with the service identifier svc:/network/ip-interface-management:default), doing so would prevent network configuration, and is not recommended.

In testing with a non-networked SmartOS Zone, I was unable to get svc:/network/physical:default to properly online at all, with or without ipmgmtd.

nscd

This process provides a cache for most name service requests, improving local and network lookup performance.  It specifically provides cache services for the following databases:

• passwd
• group
• hosts
• ipnodes
• exec_attr
• prof_attr
• user_attr
• ethers
• rpc
• protocols
• networks
• bootparams
• auth_attr
• services
• netmasks
• printers
• projects

While this process can be disabled (with the service identifier svc:/system/name-service-cache:default) it really should be kept on due to the performance advantage it provides.

dhcpagent

This process implements the client half of the dynamic host configuration protocol (DHCP) on Solaris/Illumos.  It will only be running when the zone has network interfaces configured to use DHCP, and as such, should never be manually enabled or disabled.

pfexecd

This process manages the Solaris/Illumos Role Based Access Control (RBAC) system.

It is managed by SMF (with the service identifier svc:/system/pfexec:default) and probably shouldn't be disabled at risk of disrupting normal system operation.

rsyslogd

This process provides a reliable message logging service for processes which do not handle their own logging.

It is managed by SMF (with the service identifier svc:/pkgsrc/rsyslog:default) and probably shouldn't be disabled at risk of disrupting normal system operation.

cron

This process is able to start other processes as other users at specified dates and times, making it very convenient for running regularly scheduled commands.  SmartOS already makes use of cron to perform periodic operations (such as rotating logs and checking for vulnerabilities in installed packages).

It is managed by SMF (with the service identifier svc:/system/cron:default) and while it could be disabled, I can't really think of a situation where I'd recommend it.

inetd

This process is a delegated restarter for inet services.  It is currently part of SMF, and is quite similar to svc.startd with the added functionality of optionally listening for network requests for services.  Out of the box, it is responsible for maintaining the following services:

• svc:/network/nfs/rquota:default The remote quota service (for remote NFS clients accessing local shares)
• svc:/network/rpc/gss:default The daemon that generates and validates security tokens between the kernel rpc and the GSS-API layers.
• svc:/network/security/ktkt_warn:default Notifies users when their Kerberos tickets are about to expire or automatically renews them before they expire.
• svc:/network/rpc/rex:default RPC remote execution.
• svc:/network/login:eklogin Remote login (rlogin) service (encrypted+kerberos).
• svc:/network/login:klogin Remote login (rlogin) service (kerberos).
• svc:/network/login:rlogin Remote login (rlogin) service.
• svc:/network/rexec:default Remote execution service.
• svc:/network/shell:default Remote shell server.
• svc:/network/shell:kshell Remote shell server (kerberos).

Inetd is managed by SMF (with the service identifier svc:/network/inetd:default) and unless you're using NFS or rlogin (which has all but been replaced by ssh) I recommend that you disable this service.

You can also check with inetadm before you disable it to see if it would disrupt any services.

sac

The Service Access Controller (SAC) appears to be part of the Service Access Facility or the subsystem that manages terminal connectivity into the system.  Port monitors (ttymon, see below) as described by SAF would be the rough Linux equivalent of a getty, and SAC manages those terminal monitors.

SAC is managed by SMF (with the service identifier svc:/system/sac:default) and unless you're making extensive use of TTYs, I would recommend disabling this service as it poses no apparent disruption to the system.

utmpd

This process is responsible for maintaining the user accounting databases (utmp/utmpx) in cases where individual processes are unable to correctly update the database, usually failing to properly terminate a session when they close.

It is managed by SMF (with the service identifier svc:/system/utmp:default) and probably shouldn't be disabled at risk of creating possibly erroneous user accounting databases.

sshd

This is the OpenSSH daemon and is responsible for providing the server end-point for secure encrypted communications via SSH.  Chances are that you will want to keep this one on unless you never intend on logging directly into this zone from the network (instead, going through the global zone and zlogin).

OpenSSH is managed by SMF (with the service identifier svc:/network/ssh:default) and should only be disabled if you do not want SSH logins to be possible at all.

ttymon

Besides using SAC/SAF, SMF also can call Port Monitors directly, and this service is an example of that.

As far as I can tell, this ttymon instance connects directly to the virtual console you would connect to with the vmadm console <uuid> command from the global zone.

It is directly managed by SMF (with the service identifier svc:/system/console-login:default) and while I would normally recommend disabling it, it does appear to still be required.

Conclusion

Without too much effort, we've developed a rough idea of what the default processes of a SmartOS zone are, as well as which ones can be disabled without too much of an impact on zone functionality.

In most normal circumstances, I would recommend disabling both the inetd and sac services unless they are required in your specific case:

# svcadm disable svc:/network/inetd:default svc:/system/sac:default

In situations where you do not need or want to support SSH login access, you can also safely disable the sshd process entirely.

# svcadm disable svc:/network/ssh:default

Consider that you really should establish some other mechanism to perform maintenance if you do this.

Minecraft.

It's one of those games that can appeal to many different people in many different ways.  I got into playing under the assumption that modded Minecraft was the norm, and that calculating the optimum designs for automated power production and gargantuan resource-gathering apparatuses was the entire point of the game.

Today, we'll explore how to robustly host Minecraft on SmartOS, either as a simple vanilla server, a Minecraft Forge server with mods, a Spigot server or as a BungeeCord proxy server.

Zone setup

We'll be operating within a current standard base-64 zone.  A base-32 zone should work as well, assuming your server never exceeds 4GB of memory.  You will likely need to tune max_physical_memory, cpu_cap and quota for your specific Minecraft server, and are likely to change over time.  If you plan on hosting multiple Minecraft servers from this zone, your resource usage is going to skyrocket.  Plan accordingly.

First, let's install Java.  Some older versions of Minecraft may require Java 7, but you really should prefer using Java 8, as it's significantly more performant than it's predecessor.

Also, I'd recommend installing tmux.  It's the easiest way to periodically access a server console.

Lastly, you will want to install git if you're planning on building a Spigot server (which must be built from source).  This additional package is unnecessary with both vanilla Minecraft and Minecraft Forge.

# pkgin in openjdk8 tmux git

I prefer cleaning up the delegated dataset cruft.  We can create a dataset mounted under /var/db/minecraft as well.

# UUID=$(sysinfo | json UUID)
# zfs set mountpoint=none zones/$UUID/data
# rmdir -p /zones/$UUID
# zfs create -o mountpoint=/var/db/minecraft -o quota=8G zones/$UUID/data/minecraft

Next, we can create our Minecraft group and user, set up limits and set the proper permissions for their home directory.

# groupadd -g 900 minecraft
# useradd -u 900 -g minecraft -d /var/db/minecraft -s /bin/bash \
  -c "Minecraft user" minecraft
# projadd -U minecraft -G minecraft -c "Minecraft server" \
  -K "process.max-file-descriptor=(basic,65536,deny)" minecraft
# chown minecraft:minecraft /var/db/minecraft
# chmod 700 /var/db/minecraft

You will need to unlock the minecraft account if you want it to be able to run its own crontab (helpful later on).

# passwd -N minecraft

And then su into them to install Minecraft.

# su - minecraft

Installing Minecraft

There are effectively three different Minecraft servers that are popular today:

• (Vanilla) Minecraft from Mojang is the easiest to install and maintain, and generally uses the least resources of the three.
• Spigot, a high-performance Minecraft server that supports Bukkit plugins, and is compatible with Vanilla clients.
• Minecraft Forge, which supports client and server side mods and is arguably the most powerful of the three (also generally considered the most resource intensive).  Minecraft Forge is also required if you're using a Minecraft Forge client and is generally incompatible with the vanilla client.

We'll cover how to install all three below.

Notice: Each of these sub-sections represents a different type of server install, and should be done separately.

Minecraft server

The Vanilla Minecraft server can be downloaded directly from Majong.

As the Minecraft user, use wget to download the Java Archive file.

$ wget https://s3.amazonaws.com/Minecraft.Download/versions/1.10.2/minecraft_server.1.10.2.jar -O server.1.10.2.jar

Agree to the EULA.

$ echo eula=true > eula.txt

And then start the server.

$ java -server -jar server.1.10.2.jar nogui

If all went well, you should have a functional vanilla Minecraft server.

Spigot server

Due to a legal battle between the Mojang and the developers of Spigot, the later project must be downloaded as source code and compiled locally (this is a legal requirement, not a technical one).  Fortunately, the developers of Spigot have written a tool that does just that.

First, we will need to download this jar file.

$ wget https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar

And then just run it to build against the latest version of Minecraft.

$ java -jar BuildTools.jar

If you want to build against a specific version of Minecraft, that can be specified through the --rev flag.

$ java -jar BuildTools.jar --rev 1.9.4

Wait for the build process to complete (it can take quite a while) and if all goes well, you'll have a shiny new spigot-1.10.jar and craftbukkit-1.10.jar file to work with.

Agree to the EULA.

$ echo eula=true > eula.txt

And then start the Spigot server.

$ java -server -jar spigot.1.10.jar nogui

If all went well, you should have a functional Spigot Minecraft server.  Shut down the server and delete all of the additional files and directories produced by BuildTools.

> stop
...
$ rm -r .m2 BuildData BuildTools.log.txt Bukkit CraftBukkit Spigot apache-maven-3.5.0 work

You can also delete BuildTools.jar if you don't intend on using it again.

$ rm BuildTools.jar

Minecraft Forge server

Minecraft Forge is most popularly distributed at the core of Modpacks: Community designed combinations of community developed Minecraft mods that have been balanced for reasonable gameplay.

Popular mod-pack projects such as Feed the Beast and ATLauncher ship client launchers that allow a user to download and install hundreds of different versions of dozens of different mod-packs.  These launchers can also produce server packs which are suitable to be uploaded to and extracted on the SmartOS zone we're working on.

Feed the Beast distributes their server mod-packs as zip files directly from their CDN, making them the ideal example to use for this section.

Use wget to download and unzip to decompress a server mod-pack.

$ wget http://ftb.cursecdn.com/FTB2/modpacks/FTBInfinity/2_0_0/FTBInfinityServer.zip
...
$ unzip FTBInfinityServer.zip

Download the vanilla Minecraft server and launch wrapper by running the FTBInstall script.

$ sh FTBInstall.sh

Agree to the EULA.

$ echo eula=true > eula.txt

And then start the Minecraft server.

$ java -server -Xms1G -Xmx4G -jar FTBServer-1.7.10-1448.jar nogui

Notice: Minecraft Forge takes the longest to start of the three different server types, and won't work without extra memory.  In the above example: 4G.

Once this test is complete, you can remove much of the install cruft that comes with FTB mod-pack servers.  We will be replacing their start scripts with something more robust in the optimization section below.

$ rm FTBInfinityServer.zip FTBInstall.* ServerStart.*

BungeeCord Server

BungeeCord is a Minecraft proxy server that sits between your clients and one or more Minecraft servers (as it's part of the Spigot project, it's safe to assume Spigot servers are supported), and enables large-scale server deployments.  Details about the configuration of BungeeCord are available on their website.

Download the BungeeCord server.

wget https://ci.md-5.net/job/BungeeCord/lastSuccessfulBuild/artifact/bootstrap/target/BungeeCord.jar

Configure it, and then start the server.

$ java -server -jar BungeeCord.jar

Server optimizations

Getting a Minecraft server to start and run is just the first step.  This guide will also focus on getting your server to run well, and to make the most of the SmartOS platform.  What follows are a series of sections that focus on optimizing different aspects of running a Minecraft server on SmartOS.

Optimized Java parameters

Java Virtual Machines can be started with parameters which adjust their default behavior, and while the start commands in the above examples allowed us to start and test our Minecraft servers just fine, they're the bare minimum and hardly optimal for the daily operation of a high volume server.

The optional parameters we'll be examining below should be applied immediately after the java command:

$ java <parameters> -jar server.1.10.2.jar nogui

We'll break them up into the following three classifications.

Essential parameters

These parameters should always be set for a Minecraft server.

• -XmsM: Sets the initial size of the heap to M.  This is the memory consumed by the JVM when first starting.  If you want to start with 1GB, the parameter would be: -Xms1G.
• -XmxM: Sets the maximum size of the heap to M.  This is the maximum amount of memory that can be consumed by the JVM and should always be greater than or equal to -XmsM.  If you want to limit the server to 4GB, the parameter would be: -Xmx4G.
• -XX:+UseConcMarkSweepGC or -XX:+UseG1GC: Enables the use of the CMS garbage collector for the old generation.  It's recommended that you use this GC when application latency requirements cannot be met by the throughput garbage collector.  The G1 garbage collector (+UseG1GC) is another alternative, but it might not be as memory efficient.
• -XX:+UseLargePages: SmartOS supports large pages and enabling this should translate to a performance improvement.

Optional parameters

These parameters may be useful for a Minecraft server in certain situations.

• -d64: Forces JVM to be 64-bit.  This is only practical on a base-multiarch image, as java on base-32 and base-64 will always be one or the other.
• -server: Selects the Java HotSpot Server VM.  This is only necessary with the 32-bit JVM, as the 64-bit JVM only supports the Server VM.
• -showversion: Displays JVM version information before continuing execution of the application.  This is practical for logging purposes.
• -XX:+AggressiveOpts: Enables the use of aggressive performance optimization features which are expected to become the default in upcoming Java releases.  This option may improve performance, but that performance may come at the cost of stability.
• -XX:MinHeapFreeRatio=N, -XX:MaxHeapFreeRatio=N: Sets the minimum and maximum allowed percentage of free heap space (0 to 100) after a garbage collection event.  If the free space after a garbage collection event is less than the minimum by the given percentage than the JVM will attempt to allocate additional memory to the heap until the difference is greater than the minimum given percentage.  If the free space after a garbage collection event is greater than the maximum by the given percentage than the JVM will attempt to free memory from the heap until the difference is less than the maximum given percentage.  The current defaults of 40% and 75% should be suitable, but I've seen configurations as extreme as 5% and 10%.
• -XX:MaxGCPauseMillis=10: Sets a target for the maximum garbage collection pause time (in milliseconds).  This is a soft goal, and the JVM will make it's best effort to achieve it.  In the case of Minecraft, 10ms is unreasonably short, but this should help to tune the JVM to run shorter more frequent garbage collection cycles.

Depreciated parameters

Despite often being recommended by other guides, these parameters should not be used on SmartOS.

• -Xincgc: Enables incremental garbage collection.  This option was deprecated in Java 8 with no replacement.  Use CMS, G1, or let the JVM decide instead.
• -XX:PermSize=M: Sets the space allocated to the permanent generation that triggers garbage collection if it is exceeded.  This parameter has been deprecated in Java 8 and superseded by the -XX:MetaspaceSize parameter, which shouldn't really be tuned either.
• -XX:+UseParNewGC, -XX:+UseParallelGC: Explicitly tells the JVM which garbage collection strategy should be used.  Neither of these garbage collectors should be selected (UPN will be automatically enabled with CMS).
• -XX:+CMSIncrementalPacing: Enables automatic adjustment of the incremental mode duty cycle based on statistics collected while the JVM is running.  This option has been deprecated in Java 8 with no replacement.
• -XX:+CMSClassUnloadingEnabled: Enables class unloading when using the concurrent mark-sweep (CMS) garbage collector.  This option is enabled by default, meaning there's no reason to enable it explicitly.
• -XX:ParallelGCThreads=2: Sets the number of threads used for parallel garbage collection.  This should be automatically determined by the JVM.
• -XX:-UseVMInterruptibleIO: Thread interrupts before or with EINTR for I/O operations results in OS_INTRPT.  I could find no additional documentation describing what this parameter does, so I can't really recommend using it.

Example

Based on the above parameters, I came up with the following JVM options for a Minecraft server running on SmartOS.  It's not perfect, but it leaves most of the finer configuration up to the JVM and out of our hair.

$ java -Xms256M -Xmx4G -XX:+UseConcMarkSweepGC -XX:+UseLargePages \
-jar server.1.10.2.jar nogui

If you'd prefer something more complicated, there are other recommendations out there, but I would recommend you cross-reference them with the Java8 command line documentation to ensure you're not specifying a default or depreciated parameter.

Now with that sorted, you can either take this string of options and write them into a startup.sh script to manually call from within your Minecraft directory, or you can start up Minecraft the fun way.

Minecraft and SMF

SmartOS ships with the Solaris Service Management Facility, which is the ideal tool to ensure our Minecraft server stays running.  And since we're on SmartOS, It'd be a shame not to use it!

Note: Minecraft can either be managed directly by SMF or wrapped with tmux.  Direct is simpler, but you will be unable to issue commands directly to the Minecraft server console, which will cause issues later on.  For the sake of completeness, we will explore both options below.

Import SMF Manifest

As root, download the following SMF manifest:

This manifest has been configured for a default Minecraft server instance wrapped with tmux, if you just want to use Java, uncomment the java only sections and comment out the tmux wrapped sections (both the exec_method and propval tags).

Edit the value_node tags to reflect the java parameters you want to use with your server.  This step can be skipped and done later.

Edit the server propval tag to reflect your specific minecraft server.  This can be done via symlink or skipped and done later as well.

# su - minecraft
$ ln -sf server.1.10.2.jar server.jar
$ logout

Import the manifest and enable the minecraft service.

# svccfg import minecraft-single-smf.xml
# svcadm enable minecraft

You should now have an SMF managed Minecraft server.

Edit Java parameters

Java parameters can be changed and updated directly within SMF without the need to delete and re-import the manifest.

For instance, if you want to expand the initial heap (-XmsN) from 256M to 1G, you'll need to remove the old value and replace it with a new value:

# svccfg -s minecraft
svc:/gameserver/minecraft> delpropvalue options/parameters -Xms256M
svc:/gameserver/minecraft> addpropvalue options/parameters -Xms1G
svc:/gameserver/minecraft> exit

You can also completely clear a property list and replace it with new values (note, the astring: is only required for the first value and the quotations are optional):

# svccfg -s minecraft
svc:/gameserver/minecraft> delprop options/parameters
svc:/gameserver/minecraft> addpropvalue options/parameters astring: "-Xms1G"
svc:/gameserver/minecraft> addpropvalue options/parameters "-Xmx4G"
svc:/gameserver/minecraft> addpropvalue options/parameters "-XX:+UseConcMarkSweepGC"
svc:/gameserver/minecraft> addpropvalue options/parameters "-XX:+UseLargePages"
svc:/gameserver/minecraft> exit

You can also check the pending property list:

# svccfg -s minecraft listprop options/parameters

If you're happy with what you have, refresh (commit) the configuration and restart the Minecraft service:

# svccfg -s minecraft:default refresh
# svcadm restart minecraft

Edit server JAR

Along with the parameters, the Minecraft server string can be changed and updated directly within SMF without the need to delete and re-import the manifest.

For example, if you want to update the server jar to server.1.10.2.jar, set the new value using svccfg (quotations are optional):

# svccfg -s minecraft setprop options/server="server.1.10.2.jar"

Pending configuration data can be read with svccfg:

# svccfg -s minecraft listprop options/server
options/server  astring  server.1.10.2.jar

If the pending configuration is correct, refresh (commit) the configuration and restart the Minecraft service:

# svccfg -s minecraft:default refresh
# svcadm restart minecraft

Delegated authorization

By default, only the superuser can manage the Minecraft service state and properties.  Permission to alter the state or properties of the Minecraft service can be independently granted to additional users through Role-Based Access Control (RBAC).

First, define authorization descriptions under /etc/security/auth_attr:

solaris.smf.manage.minecraft:::Manage Minecraft Service States::
solaris.smf.value.minecraft:::Change Values of Minecraft Service Properties::

Then authorize one or more users with these descriptions.  For example, the following line will grant state and property management permission to the Minecraft user:

# usermod -A solaris.smf.manage.minecraft,solaris.smf.value.minecraft minecraft

This user is now able to reboot the server and adjust the configuration options discussed above.

# su - minecraft
$ svcadm restart minecraft

This is especially practical if you want to grant one or more users access to reboot your server without having any access to its files:

# useradd -m -A solaris.smf.manage.minecraft brian
# passwd brian
...
# su - brian
$ svcadm restart minecraft
$ ls /var/db/minecraft/
ls: cannot open directory '/var/db/minecraft/': Permission denied

Now Brian can log in and reboot the server without any additional access to server data.

Snapshots > backups

What is possibly the best reason to host Minecraft on SmartOS is the first-class availability of ZFS.  Setting up a dataset to contain the world directory along with cron-driven periodic snapshots completely supersedes any functionality provided by CraftBukkit plugins or MinecraftForge mods.

Setup the World Dataset

First, switch your Minecraft server into maintenance state if it's been enabled.

# svcadm mark maintenance minecraft

Move the current world directory to a temporary location.

# mv /var/db/minecraft/world /var/db/minecraft/world_tmp

Create a new dataset for the world data.  Optionally set a quota (20G in our example).

# zfs create -o quota=20G zones/$(sysinfo | json UUID)/data/minecraft/world

Chown the root of the new dataset and copy the contents of the old world directory to the new (this may take some time on established worlds).  Delete the temporary directory when you're done.

# chown minecraft:minecraft /var/db/minecraft/world
# mv /var/db/minecraft/world_tmp/* /var/db/minecraft/world/
# rmdir /var/db/minecraft/world_tmp

If you're done in maintenance mode, you can now clear your Minecraft service.

# svcadm clear minecraft

Delegated ZFS management

Normally ZFS administration is handled by the superuser, but authorizations over actions can be delegated to other users on a per dataset basis.  In our case, we want our Minecraft user to be able to create snapshots of the world dataset so that the periodic snapshot script can be managed under that user.

# zfs allow -lu minecraft snapshot zones/$(sysinfo|json UUID)/data/minecraft/world

Destroying and renaming snapshots can be achieved by granting destroy and rename permissions to any descendant datasets (of which snapshots are).  Note that this will allow any user to destroy or rename any descendant dataset.

# zfs allow -du brian destroy,rename zones/$(sysinfo|json UUID)/data/minecraft/world

You can review what permissions have been granted by calling zfs allow <dataset>:

# zfs allow zones/$(sysinfo|json UUID)/data/minecraft/world

And you can revoke permissions with zfs unallow:

# zfs unallow -du brian destroy,rename zones/$(sysinfo|json UUID)/data/minecraft/world

Setup periodic snapshots

The easiest way to create snapshots of the world directory is by using a script that disables auto-saving, issues an explicit save creates the snapshot, and then re-enables auto-saving.

I put together a simple implementation in bash that does these things, based on the script described in this blog post.

Notice: This script needs to be able to communicate with the Minecraft server console to function.  It's designed to interface with tmux, as configured in the previous section, and will not work in a java-only configuration.

As the minecraft user, copy the following script to the Minecraft directory:

Be sure to adjust snappath to reflect your dataset name.  Add the script to your crontab scheduled to run as often as you want it to.

For example, here's a crontab that will snapshot your world dataset every 15 minutes:

0,15,30,45 * * * * ./snapshot.sh

Notice: If you get errors about Minecraft not being allowed to execute cronjobs, you will need to unlock the account as root.

# passwd -N minecraft

I am aware that this script does not manage existing snapshots (expiring old ones or rotating snapshots) or send snapshots to other systems for remote replication.  I leave extending this script to encompass additional functionality as an exercise for the reader.

Conclusion

This is pretty much everything you need to get a single instance of Minecraft up and running within a single SmartOS zone.  I had a bunch more notes that delved into advanced Minecraft multitenancy but I figured that was a bit much for a single blog post, so we'll save that for another day.

One of the more annoying differences I noticed in the switch from Linux to SmartOS was while connecting to a terminal via PuTTY: The former supports using the Home and End keys to navigate to the beginning and end of a command-line buffer while the latter does not, instead spitting tilde characters into the command-line wherever you happen to have the cursor.

My Google-fu was weak and unfocused (see: distracted) at the time, so I'd usually just spend a few minutes fruitlessly searching for an answer until deciding to give up and focus on the original problem I was working on, trying my best to ignore the fact that I couldn't zip back and forth across a multi-row command like I was used to being able to on Linux.

It finally got bad enough that I committed myself to finding a solution before I moved back to my original task.  This blog post is the culmination of my findings.  Thanks to geedoubleya and rofrol from Stack Exchange for their work on this answer which set me on the right track for researching this article.

Bonus: Most of this guide should apply well to any modern readline installation!

Bash and Readline

What I was used to were features of GNU Bash and more specifically GNU Readline, the library that provides the line-editing and history functionality for Bash.  Fortunately, with SmartOS, my problem was one of misconfiguration, not of missing software, as both Bash and Readline are already installed in most (if not all?) SmartOS images.

You can actually confirm this by testing against the default Readline key bindings.  The relevant ones are as follows:

• beginning-of-line (ctrl-a) moves the cursor to the beginning of the line.
• end-of-line (ctrl-e) moves the cursor to the end of the line.

And while we could just stick with those, where's the fun in limiting ourselves to that?  Readline supports reconfiguration via a simple configuration file (~/.inputrc or /etc/inputrc).

Readline configuration

Readline configuration file syntax supports two common definition structures: Variables can be set with the form set <variable> <value> and key bindings can be defined with the form keyname: function-name or "key sequence": function-name forms.

Setting variables will alter the run-time behavior of Readline.  Variables that I found interesting include:

• bell-style set to none if you don't want to hear a terminal ring again.
• colored-stats set to on if you want to see completions in different colors to indicate their file type.
• editing mode set to either emacs (default) or vi depending on your religious affiliation.
• enable-keypad set to on if you want to make your life easier with arrow keys.
• expand-tilde set to on if you want to expand the tilde to a full path.
• history-size control the number of lines stored in the history buffer (and saved to .bash_history).
• horizontal-scroll-mode set to on will cause a command-line to scroll across the bottom of the screen instead of wrap to a new line.
• keymap sets the overall keymap for key binding commands, options include emacs (default), emacs-standard, emacs-meta, emacs-ctlx, vi, vi-move, vi-command and vi-insert.
• keyseq-timeout sets the timeout duration in milliseconds for Readline to wait when reading an ambiguous key sequence.
• mark-symlinked-directories set to on if you want symlinks to directories to have a / appended to their names.

Key bindings will allow you to map specific functions or macros to keys.  Functions that I found interesting and relevant include:

• beginning-of-line moves the cursor to the beginning of the current line.
• end-of-line moves the cursor to the end of the current line.
• forward-char moves the cursor forward one character.
• backward-char moves the cursor backward one character.
• forward-word moves the cursor forward to the end of the next word.
• backward-word moves the cursor backward to the start of the current or previous word.
• clear-screen clears the screen and redraws the current line at the top of the screen.
• redraw-current-line redraws the current line in its current location.
• previous-history moves back through the history list, displaying the previous command.
• next-history moves forward through the history list, displaying the next command.
• beginning-of-history displays the first entry of the history list.
• end-of-history displays the last entry of the history list, the current line being entered.
• history-search-backward moves backward through the history list, displaying lines that match your partially typed command.
• history-search-forward moves forwards through the history list, displaying lines that match your partially typed command.

There's a bunch of other stuff in the documentation (like conditional blocks and a BUNCH of other bindable functions) that while I found interesting, didn't have a really practical place to apply it while writing this article.  You may want to give it a proper read if you'd like to know more.

When testing your configuration, you can prompt a reload by using the key sequence Ctrl-X Ctrl-R, instead of logging out and back in again.

Guest Zone configuration

As mentioned before, the system-wide configuration file is located at /etc/inputrc which can be overridden by individual users with a file at ~/.inputrc.  If you tend to switch roles into different users often, I recommend using the system-wide configuration file, as those configuration directives will be applied to all users on the system.

Global Zone Configuration

Since SmartOS Global Zones are refreshed on reboot, we'll need to set up a transient SMF manifest to load our settings into the global zone each time it boots up.

Download the above manifest to /opt/custom/smf/readline.xml and copy your inputrc file to /usbkey/config.inc/inputrc.  This will ensure that your SmartOS global zone will copy /usbkey/config.inc/inputrc from persistent memory into the ramdisk (/etc/inputrc) upon each boot.

My Inputrc

While a vim user, I prefer not having to deal with that editor's intricacies while I'm in bash.  Also, I tend to ssh from Windows, PuTTY handles my copy buffer, so all I really want is for readline to handle is command-line navigation.  I originally wanted to do a pretty straightforward interface where basic arrow keys provided navigation and modifier keys (Alt, and Ctrl) increased magnitude, but it turns out that arrow keys in a terminal are just "Metafied" ASCII, and pressing the Alt key while pressing one of those keys is just the equivalent of a normal ASCII sequence.  If I were to attempt these bindings, I would end up with normal ASCII in my terminal: Not exactly what I was looking for.

So, for now, I'm stuck using Home, End, PageUp and PageDown just like the rest of us mortals.

/etc/inputrc:

"\e[1~": beginning-of-line
"\e[4~": end-of-line
"\e[5~": history-search-backward
"\e[6~": history-search-forward
"\e[3~": delete-char