Ansible on SmartOS

It's easy to have a love-hate relationship with Infrastructure Automation.

On one hand, the principle of Infrastructure Automation is fantastic: Using portable and well defined modules of code to ensure consistent deployments of software and configurations to potentially hundreds or thousands of physical or virtual machines is how we should have always done system administration, and in retrospect, it's a shame that it took cloud computing for all of us to see that.  The improvements to fault detection, consistency and delivery time are massive boons to rapidly expanding deployments maintained by relatively small teams.

On the other hand, the implementation of Infrastructure Automation is... less than fantastic.  Chef and Puppet both initially present as resource hogs that are not trivial to setup or manage and have questionable economies of scale.  SaltStack, while better, wasn't nearly as easy to get up and running as it should have been.  Terraform and CloudFormation both look interesting, but appear to be focused as provisioning tools instead of configuration management tools, and in the case of the later, is AWS only.  It seems like Terraform also requires a third party provisioning provider to use with SmartOS, which while I'm not opposed to using, I'd rather not have to learn Terraform with that added complexity.

That leaves us with Ansible, which while easy to get started with, both in configuration and use, quickly becomes less palatable as your deployments increase in complexity.  This specifically manifests in how Ansible uses YAML to describe tasks.  Simple tasks are easy, but as soon as flow control structures such as branching or looping are introduced to your plays, the YAML structures become anything but minimal.  There appears to be a complete ignorance of the DRY principle as well with an incredible preference to write everything multiple times, but it's what we've got for now, so we're going to go with it.

Stupid SmartOS Tricks

That leads us to changes in how articles will be published to this blog moving forward.

In the principle of not repeating oneself (DRY), articles will focus on the overarching intentions and concerns one might encounter when deploying something to SmartOS.  Instead of fully describing all required commands and configuration steps to achieve a given outcome, articles will instead link to a companion Ansible role in the accompanying ansible-smartos-tricks playbook that will perform that deployment.

This should also be very helpful to ensure consistent deployment of software for the previously-mentioned plans of benchmarking SmartOS against FreeNAS and Proxmox.

The roles in this playbook are organized in the following fashion:

• A common role that will perform all boiler plate configurations to any base SmartOS zone including cleaning up and managing ZFS datasets, and disabling inetd and sac as recommended in this article.
• Service roles that depend on common and zero or more other service roles and install and configure commonly required programs and services including mysql, neo4j, nginx, postgresql, redis and samba.

Installation

Ansible plays should be run from their own isolated SmartOS zone, as this system will need root access to any other system configured by it.  Below is an example manifest:

{
  "image_uuid": "1d05e788-5409-11eb-b12f-037bd7fee4ee",
  "brand": "joyent",
  "alias": "ansible",
  "hostname": "ansible",
  "cpu_cap": 100,
  "max_physical_memory": 1024,
  "quota": 10,
  "resolvers": [ "10.0.0.1" ],
  "nics": [
    {
      "nic_tag": "admin",
      "ips": [ "10.0.0.2/24" ],
      "gateways": [ "10.0.0.1" ],
      "primary": true
    }
  ]
}

If you have any questions about the properties chosen for this manifest, please read this article.

Create the zone, zlogin to it, install git, clone ansible-smartos-tricks locally and then run ./ansible-bootstrap.sh from within the ansible-smartos-tricks directory.

Output has been omitted for brevity:

[root@home-gz ~]# vmadm create -f ansible.json
[root@home-gz ~]# zlogin <uuid or "ansible<tab>">
[root@ansible ~]# pkgin -y install git
[root@ansible ~]# git clone https://github.com/brianewell/ansible-smartos-tricks
[root@ansible ~]# cd ansible-smartos-tricks
[root@ansible ~/ansible-smartos-tricks]# ./bootstrap.sh

The bootstrap script will handle all of the rest of the configuration for you, including installing Redis locally and configuring ansible to utilize it to cache remote host facts, significantly improving ansible performance, as well as automatically ensuring the existence of an SSH key-pair to authenticate to remote systems when configuring them.

Using SmartOS Tricks

Ansible uses SSH to connect to and configure remote systems, and this project specifically uses ed25519 keypairs to handle authentication.  While you could manually copy the public key to each system you'd like to configure, it's much easier and more consistent to include the key at the end of each manifest under the customer_metadata key.  An example:

{
...
  "customer_metadata": {
    "root_authorized_keys": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDrStZlHS0yfE8n71meairBGvFnc5mlDFNKAJy7tQMi2",
    "user-script": "/usr/sbin/mdata-get root_authorized_keys > /root/.ssh/authorized_keys"
  }
}

So far I've just been using static hosts in Ansible host inventories.  You can either remember the IP address you set a given host to, or use vmadm to discover it from the global zone:

[root@gz ~]# vmadm list -o alias,nics.0.ips
ALIAS       NICS.0.IPS
doudna      10.0.0.3/24,addrconf
plex        10.0.0.4/24,addrconf
router      10.0.0.1/24,addrconf
metrics     10.0.0.5/24,addrconf
ansible     10.0.0.2/24,addrconf
test        dhcp,addrconf

Please note that zones using DHCP will not report their IP address through vmadm.  Instead, it's probably best to login to the zone and check using ipadm show-addr from within the zone:

[root@test ~]# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net0/?            dhcp     ok           10.0.0.201/24
lo0/v6            static   ok           ::1/128

Place any hosts that you would like Ansible to configure into the ansible static hosts inventory file.  In this example we will include 10.0.0.201 as a member of the test inventory:

[root@ansible ~]# cat /etc/ansible/hosts
[test]
10.0.0.201

You can now create plays directly within ansible-smartos-tricks that refer to the provided roles, an example that applies the common role to test hosts:

[root@ansible ~]# cat ~/ansible-smartos-tricks/common.yml
---
- name: 'Common Role'
  hosts: test
  roles:
  - common
  vars:
    vim:
      colorscheme: elflord

Future Plans

It may seem by the title of this article that this is some kind of endorsement for Ansible, but I am honestly confused how software as misery inducing as this can be so popular.  Most 'sophisticated' Ansible plays appear to be written by closet masochists who enjoy typing a lot , and while that probably also says something about me, it also looks to be an incredible opportunity to do better in the space of Infrastructure Automation.

I've already started designing an infrastructure automation replacement for Ansible.

For now though, I will be using Ansible with SmartOS until my research indicates that a different tool would be preferable, either of my own or someone else's design.