On to SmartOS

As I continued to research the potential of FreeNAS Plugins (FreeBSD Jails), something was becoming increasingly obvious: I really should be using something else.

BSD Jails were great if I wanted to run in a FreeBSD environment, but what about Linux?  Or even Windows?  While FreeNAS was great in that it kept the storage layer close (and responsive) to the compute layer, it lacked the flexibility to consider it a serious type-1 hypervisor.

I had also been evaluating VMWare ESXi 5 at a non-profit organization I volunteer at and while I generally liked it, I kept having the same nagging question pop up:  VMFS (the filesystem for direct attached storage under ESXi) doesn't address bitrot, or really, any data storage concerns, as it expects to be attached to a storage layer that deals with data validity and reliability for it.  It had the exact opposite problem as FreeNAS had: Much more flexibility, but at the cost of robust datastore reliability.

Additionally, ESXi can only do full VMs, not light-weight containerization like FreeBSD can do with BSD Jails.

The solution many had come up with was running ESXi and FreeNAS on two separate pieces of hardware and connecting them via NFS or iSCSI.  This makes sense logically, as each will cover for the weaknesses of the other: FreeNAS handles storage and jailed processes while ESXi handles full VMs, but that would mean I would need to purchase a second N54L (or some other piece of hardware) if I wanted to run this at home, and it came with its own set of limitations:

• Gigabit Ethernet supports at best, 100MB/s data transfer over the wire where each of the drives I had support around 150MB/s of sustained throughput.  This would hard limit my ESXi machine at 100MB/s of reading and writing at any given moment.
• Network latency and iSCSI/NFS latency would become unacceptably high when the connection was saturated, which meant either dealing with it or developing a much more complicated network.

Adding an additional computer and possible additional network hardware (through a four-port gigabit NIC wouldn't have been that bad) wasn't really high on my list of things to do when I had originally set out to simplify and consolidate my home network.

I could have also attempted to run FreeNAS under ESXi and hardware passthrough the HBA to FreeNAS which would then in turn export storage via NAS or iSCSI to ESXi to run it's additional VMs on.  This is apparently also done quite often, but it strikes me as incredibly delicate, and susceptible to circular dependencies.  It also presents two different interfaces for setting up containerized and full VMs (one through FreeNAS, and the other through ESXi) which would make fair resource sharing difficult.

There had to be a better solution, and as it turned out, there was.

The Solution

What I was actually looking for was a converged storage/hybrid-virtualization platform that sat directly atop ZFS and was able to provision both hybrid and full VMs.  All disk IO is local and fairly distributed between containers and VMs alike.  Any NAS functionality could be implemented directly within a container or from a VM.  At the time though, typing "virtualization" and "ZFS" presented a vastly different landscape then it does now, fortunately, Joyent was still on the first page.

At first glance, Joyent's Smart Data Center (SDC) was entirely outside the scope of what I was looking for: A massive data-center spanning "operating system" that could provision VMs either through direct customer interaction or REST API, account for their operation, spin them up and down, and manage access and billing.  But at its core, SDC operates on SmartOS: A small, in-memory based read-only Illumos distribution that provisions Zones or VMs (KVM) over a local ZFS pool.

In other words, it was almost exactly what I wanted.

Having never seriously worked with Illumos or Solaris before, I didn't know what to expect from a Zone, and as it turned out, Joyent never ported the AMD processor support from Linux KVM to Illumos, so I actually wouldn't have access to full VMs at home either as long as I continued to use AMD hardware (N54L).

That last part almost stopped me.

I would have to port all of my development and storage platforms over to an operating system I knew little if anything about.  What an opportunity to learn something new.

Besides ZFS and KVM, two other major features provided by SmartOS (Illumos) were also quite appealing: Crossbow and DTrace.

Crossbow is Illumos' virtual network (vNIC/vSwitch) implementation which allows for resource provisioning all the way to the port level.  Besides being able to distribute networking contention issues more efficiently than the traditional Solaris networking stack, it also provides QoS functionality which apparently actually makes sense to configure.

Dynamic Tracing (DTrace) is a tracing framework which could instrument and inspect nearly anything running in production with minimal performance impact.  It's basically the end-all cudgel of software debugging that can be used to beat your problems into submission.  Or so I'm told.

The Plan

While I already had a long history of using Linux in strange and interesting ways, I had basically consolidated most of my daily activities down to what would fit into a few simple zones.

• A router that could receive an IPv4 address via DHCP (WAN) and was capable of providing network services such as IPSec, NAT, DHCP, DNS and port forwarding to a Local Area Network.
• A File Server that could run Samba.
• A web-development server that could run Nginx, Ruby on Rails, PHP, and various database management systems.

As it turned out, all of this was within SmartOS' capabilities.