While SmartOS was designed by Joyent to be installed on thousands of machines simultaneously at datacenter scales, it can also be used in much smaller deployments of a handful, or even just a single machine.

This is excellent in the home visualization space, as SmartOS integrates the hypervisor and storage subsystem into a single component, removing the need for any storage infrastructure, and making it even more attractive for use in the home or small office.

It also allows for the deployment of Illumos Zones OR full Virtual Machines (contained within zones) allowing you to get both maximum performance and flexibility out of your hardware.

What follows is a brief guide on setting up and configuring SmartOS for use in a stand-alone environment like this.  It specifically outlines the tweaks I personally made to my installation at home.  And really that's the only point of it.

SmartOS Architecture

SmartOS boots from a live image which provides the Illumos kernel and a root ramdisk.  The zones ZFS pool is imported, SMF manifests are imported, and the system (optionally) starts previously running zones and VMs.  If the zones pool is not found, SmartOS will go through its installation process.

Notice: Despite looking like a (relatively) normal UNIX environment, the global zone is quite specialized, and should not be used as a general purpose operating environment.  It should only be reserved for tasks that are ill-suited for running within a Zone or VM.

Hardware

I've installed SmartOS on my HP N54L Microserver, which supports everything but KVM.  Since I also use one of the Zones as my home router, I've added an additional Network Interface, as described in my previous blog post.

The latest SmartOS live images are available at wiki.smartos.org.  The ISO is nice and simple but can get costly to download and re-burn each time you decide to upgrade to the latest build image.  I recommend using the USB image, especially in my case since the N54L has an onboard USB port for booting from.

Ubuntu has an excellent disk imager utility which I recommend using to burn USB images to USB drives on Windows.

Installation

Installing SmartOS is as simple as burning the image to the medium and booting from it.  Depending on the read speed of your CD or USB drive, booting can take a while, and since SmartOS was designed for unattended use, it's not overly verbose on the console while it's booting.

Configuration

If there is no configured zones pool on your hardware, you will be taken through a series of prompts to provide required configuration information.

 SmartOS Setup
Joyent                                       https://wiki.smartos.org/install
-------------------------------------------------------------------------------

You will have a chance to review and correct your answers, as well as a
chance to edit the final configuration, before it is applied.

At the prompts, if you type ^C you will be placed into a shell.  When you
exit the shell the configuration process will resume from where it was
interrupted.

Press [enter] to continue


Provide basic networking information, select the disks to be used in the storage pool, and specify a root password.

2016 Update: The configuration options for a new installation of SmartOS has changed over the last year.  Now, in addition to basic network information, static DNS servers and NTP servers can be set, as well as a default search domain.  Additionally, you can specify your Zpool configuration (raidz2, mirrored, or manual)

You will have a chance to edit the configuration file before it's written to the storage pool, and then the configurator will create the storage pool, several critical datasets, populate them with files and write the configuration to the pool, then reboot.

Tweaks

While it's easy to get started right away using SmartOS, this is exactly the time to ensure that your experience is going to be the best that it can be.

Checksum

By default, SmartOS uses Fletcher4 as it's checksum.

[[email protected] ~]# zfs get checksum
NAME           PROPERTY  VALUE     SOURCE
zones          checksum  on        default
zones/archive  checksum  on        default
zones/config   checksum  on        default
zones/cores    checksum  on        default
zones/dump     checksum  noparity  local
zones/opt      checksum  on        default
zones/swap     checksum  on        default
zones/usbkey   checksum  on        default
zones/var      checksum  on        default


While this is normally just fine, I prefer the security of a secure hashing algorithm, and fortunately, ZFS delivers.

2016 Update: SmartOS now ships with several new hashing algorithms, namely sha512 (fast), skein (faster), and edonr (fastest), all of which are superior to fletcher in collision resistance and sha256 in performance.  I recommend using one of these instead of sha256, especially if you value disk performance.

[[email protected] ~]# zfs set checksum=edonr zones


We will leave the noparity local option alone for zones/dump.

Compression

I'm a fan of always using compression on disk.  The lz4 compression algorithm is a high-performance replacement for lzjb, which features significantly faster compression and decompression, as well as a moderately higher compression ratio than it's predecessor.  Let's see how SmartOS makes use of this compression algorithm by default.

[[email protected] ~]# zfs get compression
NAME           PROPERTY     VALUE     SOURCE
zones          compression  off       default
zones/archive  compression  lzjb      local
zones/config   compression  off       default
zones/cores    compression  gzip      local
zones/dump     compression  off       local
zones/opt      compression  off       default
zones/swap     compression  off       default
zones/usbkey   compression  off       default
zones/var      compression  off       default


Unfortunately, not at all.  Let's fix that.

[[email protected] ~]# zfs set compression=lz4 zones
[[email protected] ~]# zfs inherit compression zones/archive


Besides enabling lz4 for everything, we ensure that zones/archive inherits it from zones (as lz4 > lzjb).  However, we do leave the zones/cores with gzip and zones/dump without compression.

2016 Update: If the lz4_compress feature is enabled on your storage pool, lz4 will be the default compression algorithm used (by compression=on), and will no longer need to be explicitly set by name.

ARC and Swap

If there's not enough free memory, the kernel will start paging memory out to persistent storage.  If that happens, we don't need to cache it in free memory as well (seeing as we won't have much of that left).  Because of this, I always limit ARC caching of swap to metadata only:

[[email protected] ~]# zfs set primarycache=metadata zones/swap