Installing SmartOS
SmartOS was designed by Joyent to be installed on thousands of machines simultaneously at datacenter scales. It can also be used in much smaller deployments of a handful, or even just a single machine.
This is excellent in the home virtualization space, as SmartOS integrates the hypervisor and storage subsystem into a single hardware component, removing the need for independent storage infrastructure, ultimately making it even more attractive for use in the home or small office.
It also allows for the deployment of Illumos Zones OR full Hardware Virtualization (contained within zones) allowing you to get both maximum performance and flexibility out of your hardware.
What follows is a brief guide on setting up and configuring SmartOS for use in a stand-alone environment like this. It specifically outlines the tweaks I personally made to my installation at home.
SmartOS Architecture
SmartOS boots from a live image which provides the Illumos kernel and a root ramdisk. The zones ZFS pool is imported, SMF manifests are imported, and the system (optionally) starts previously running zones and VMs. If the zones pool is not found, SmartOS will go through its installation process.
Notice: Despite looking like a (relatively) normal UNIX environment, the global zone is quite specialized, and should not be used as a general purpose operating environment. It should only be reserved for tasks that are ill-suited for running within a Zone or VM.
Hardware
I've installed SmartOS on my HP N54L Microserver, which supports everything but KVM. Since I also use one of the Zones as my home router, I've added an additional Network Interface, as described in my previous blog post.
Downloading
The latest SmartOS live images are available at wiki.smartos.org. The ISO is nice and simple but can get costly to download and re-burn each time you decide to upgrade to the latest build image. I recommend using the USB image, especially in my case since the N54L has an onboard USB port for booting from.
Ubuntu has an excellent disk imager utility which I recommend using to burn USB images to USB drives on Windows.
Installation
Installing SmartOS is as simple as burning the image to the medium and booting from it. Depending on the read speed of your CD or USB drive, booting can take a while, and since SmartOS was designed for unattended use, it's not overly verbose on the console while it's booting.
Configuration
If there is no configured zones pool on your hardware, you will be taken through a series of prompts to provide required configuration information.
SmartOS Setup
Joyent https://wiki.smartos.org/install
-------------------------------------------------------------------------------
You must answer the following questions to configure your SmartOS node.
You will have a chance to review and correct your answers, as well as a
chance to edit the final configuration, before it is applied.
At the prompts, if you type ^C you will be placed into a shell. When you
exit the shell the configuration process will resume from where it was
interrupted.
Press [enter] to continue
Provide basic networking information, select the disks to be used in the storage pool, and specify a root password.
2016 Update: The configuration options for a new installation of SmartOS has changed over the last year. Now, in addition to basic network information, static DNS servers and NTP servers can be set, as well as a default search domain. Additionally, you can specify your Zpool configuration (raidz2, mirrored, or manual)
You will have a chance to edit the configuration file before it's written to the storage pool, and then the configurator will create the storage pool, several critical datasets, populate them with files and write the configuration to the pool, then reboot.
ZFS Property Tweaks
2021 Update: While it's easy to get started right away using SmartOS, this is the perfect time to ensure that your performance is going to be the best that it can be by tweaking ZFS properties on the zones dataset. Recommended commands are posted immediately below with explanations following:
[root@gz ~]# zfs set checksum=edonr compress=lz4 recordsize=1M zones
[root@gz ~]# zfs set primarycache=metadata zones/swap
[root@gz ~]# zfs inherit compression zones/archiveChecksum
By default, SmartOS uses Fletcher4 as it's checksum.
[root@gz ~]# zfs get checksum
NAME PROPERTY VALUE SOURCE
zones checksum on default
zones/archive checksum on default
zones/config checksum on default
zones/cores checksum on default
zones/dump checksum noparity local
zones/opt checksum on default
zones/swap checksum on default
zones/usbkey checksum on default
zones/var checksum on default
While this is fine, I prefer using a cryptographically secure hashing algorithm.
2016 Update: SmartOS now ships with several new hashing algorithms, namely sha512 (fast), skein (faster), and edonr (fastest), all of which are superior to fletcher in collision resistance and sha256 in performance. I recommend using one of these instead of sha256.
We will leave the noparity local option alone for zones/dump.
Compression
2021 Update: Enabling compression by default on ZFS is a must for SmartOS, especially when enabling a 1M recordsize as described below. While ZFS does support dynamic block allocation, upto the limit specified by recordsize, it doesn't directly support block suballocation, meaning that all blocks of a given object in ZFS will have the same logical size. If you're storing data without compression, \(logical size = physical size\). In the case of large files with a 1M recordsize, the last part of the file will occupy a full 1MiB block, unless compression is enabled.
With compression enabled, all logical blocks are (opportunistically in the case of lz4) compressed and stored in a smaller physical allocation, significantly improving storage performance and capacity. The empty space of the tailing 1M block will be compressed and stored in much less physical space.
All datasets should be compressed with lz4 (or hopefully zstd in the near future) except for zones/cores which should stay as gzip, and zones/dump which should stay uncompressed.
If the lz4_compress feature is enabled on your storage pool, lz4 will be the default compression algorithm used (by compression=on), and will no longer need to be explicitly set by name. It is nice however to know which specific compression algorithm was used, which is why I still explicitly refer to the compression algorithm by name.
Recordsize
2021 Update: ZFS now supports records up to one megabyte in size instead of 128 kilobytes. This is great for bulk storage of large files, and is generally a good idea to enable.
This also means that ZFS datasets that are used for database storage will be subjected to even more significant write amplifications than they would have under a 128KB recordsize. Be sure to set recordsize appropriately for any database workloads.
ARC and Swap
If there's not enough free memory, the kernel will start paging memory out to persistent storage. If that happens, we don't need to cache it in free memory as well (seeing as we won't have much of that left). Because of this, I always limit ARC caching of swap to metadata only.
Platform Image Management
2021 Update: SmartOS now supports platform images installed directly to zones/boot, eliminating the need for a dedicated boot device. This can be accomplished using the following command on the global zone:
[root@gz ~]# piadm bootable -e zonesThis will take a while as it downloads and installs the latest available platform image.
Platform images can be updated using the following commands:
[root@gz ~]# piadm install <platform image identifier> zones
[root@gz ~]# piadm activate <platform image identifier> zonesConclusion
And that's what I do to a SmartOS installation before downloading images. Please let me know what you do to your SmartOS installations in the comments section below.