Stupid SmartOS Tricks

NUT in the Global Zone

Brian Ewell -

Network UPS Tools is a project for connecting with many different power devices, and can access UPS data and coordinate with the device to ensure orderly shutdown of a host such as a SmartOS compute node.

NUT flies a bit in the face of the design philosophy of SmartOS. Not only will you need to install it manually (via pkgsrc, which will also need to be installed) in a global zone to use it, but SmartOS would rather rely on ZFS to handle unexpected power events rather than being explicitly shutdown by a UPS monitoring service.

In this article, we will install NUT (and pkgsrc) into the global zone and configure it to interface with a USB attached UPS device and to (optionally) automatically power down the compute node if the UPS batteries run critically low.

Lets get to it.

Installing Pkgsrc into the global zone

The Joyent Pkgsrc website has instructions on installing pkgsrc into the global zone. At the time of this writing, this is the bootstrap that we're going to use:

/root/install-pkgsrc.sh:

#
# Copy and paste the lines below to install the latest 64-bit tools set.
#
BOOTSTRAP_TAR="bootstrap-trunk-tools-20201019.tar.gz"
BOOTSTRAP_SHA="9b7a6daff5528d800e8cea20692f61ccd3b81471"

# Ensure you are in a directory with enough space for the bootstrap download,
# by default the SmartOS /root directory is limited to the size of the ramdisk.
cd /var/tmp

# Download the bootstrap kit to the current directory.  Note that we currently
# pass "-k" to skip SSL certificate checks as the GZ doesn't install them.
curl -kO https://pkgsrc.joyent.com/packages/SmartOS/bootstrap/${BOOTSTRAP_TAR}

# Verify the SHA1 checksum.
[ "${BOOTSTRAP_SHA}" = "$(/bin/digest -a sha1 ${BOOTSTRAP_TAR})" ] || echo "ERROR: checksum failure"

# Verify PGP signature.  This step is optional, and requires gpg.
curl -kO https://pkgsrc.joyent.com/packages/SmartOS/bootstrap/${BOOTSTRAP_TAR}.asc
curl -ksS https://pkgsrc.joyent.com/pgp/DE817B8E.asc | gpg --import
gpg --verify ${BOOTSTRAP_TAR}{.asc,}

# Install bootstrap kit to /opt/tools
tar -zxpf ${BOOTSTRAP_TAR} -C /

# Add to PATH/MANPATH.
PATH=/opt/tools/sbin:/opt/tools/bin:$PATH
MANPATH=/opt/tools/man:$MANPATH

Run this script on the global zone to bootstrap pkgin:

# /root/install-pkgsrc.sh

We should now have pkgsrc available now on the global zone.

If you have a pkgsrc cache available (which we will cover soon), configure pkgin to make use of it:

/opt/tools/etc/pkgin/repositories.conf:

...
http://pkgsrc.joyent.cache/packages/SmartOS/trunk/tools/All

Update and upgrade pkgsrc.

# pkgin update
processing remote summary (http://pkgsrc.joyent.cache/packages/SmartOS/trunk/tools/All)...
database for http://pkgsrc.joyent.cache/packages/SmartOS/trunk/tools/All is up-to-date
# pkgin -y upgrade
...

Install NUT into the Global Zone

This is exceedingly difficult to do using pkgsrc:

# pkgin -y install ups-nut-usb
...

This should install NUT and its USB drivers into /opt/tools as well.

Configuring & testing upsdrvctl

NUT uses a component called upsdrvctl to interface with USB attached devices. This will need to be told about your UPS, specifically which driver and port it should use to access it. The driver you should use is specific to your UPS model, and can be checked using their Hardware Compatibility List. This is an example of a configuration file setup for a SMT2200 attached as the sole UPS to a compute node:

/opt/tools/etc/nut/ups.conf:

[smt2200]
  driver = usbhid-ups
  port = auto
  desc = "American Power Conversion Smart-UPS 2200"

Running the following command will enable the driver and ensure that upsd can connect to it later on. You shouldn't see any error messages here.

# upsdrvctl start
Network UPS Tools - UPS driver controller 2.7.4
Network UPS Tools - Generic HID driver 0.41 (2.7.4)
USB communication driver 0.33
Using subdriver: APC HID 0.96

Configuring & testing upsd

The upsd process is responsible for connecting to and polling information from attached UPSes via upsdrvctl. Its default configuration (of an empty file) is perfectly suitable for a standalone configuration.

If you would like to use upscmd as well, it's best to configure some users who have permission to manipulate the device.

/opt/tools/etc/nut/upsd.users:

[admin]
  password = secret
  actions = SET
  instcmds = ALL

Start upsd when you're ready.

# upsd
Network UPS Tools upsd 2.7.4
fopen /opt/tools/var/db/nut/upsd.pid: No such file or directory
listening on 127.0.0.1 port 3493
listening on ::1 port 3493
Connected to UPS [smt2200]: usbhid-ups-smt2200

You should now be able to query your ups using upsc:

# upsc smt2200 ups.status
Init SSL without certificate database
OL

For reference, OL stands for On-Line Power, OB stands for On-Battery Power, and LB stands for Low-Battery Power. You can also issue this command without any additional options to see all information from the UPS:

# upsc smt2200
Init SSL without certificate database
battery.charge: 68
battery.charge.low: 10
battery.charge.warning: 50
...

The upscmd command can also be used to issue specific commands to your UPS:

# upscmd -l smt2200
Instant commands supported on UPS [smt2200]:

beeper.disable - Disable the UPS beeper
beeper.enable - Enable the UPS beeper
beeper.mute - Temporarily mute the UPS beeper
beeper.off - Obsolete (use beeper.disable or beeper.mute)
beeper.on - Obsolete (use beeper.enable)
load.off - Turn off the load immediately
load.off.delay - Turn off the load with a delay (seconds)
shutdown.reboot - Shut down the load briefly while rebooting the UPS
shutdown.stop - Stop a shutdown in progress

It will also ask you for credentials before allowing any actual commands to be performed.

Configuring & Testing upsmon (optional)

If you would like NUT to actually power down your system once power reaches a critical point, then you will need to configure upsmon, the monitoring component of NUT.

Generally, using this component of NUT is completely optional as SmartOS can generally handle abrupt shutdowns already thanks to ZFS. However if you're doing something crazy like disabling sync writes on certain datasets, or generally like the idea of a graceful shutdown in the global zone, proceed.

A rough layout of of upsmon's flow with our modifications are as follows:

  • The UPS goes onto battery power.
  • The UPS reaches a low battery state (ups.state goes from OB to LB).
  • The upsmon service notices this and generates a NOTIFY_SHUTDOWN event, waits FINALDELAY seconds, creates the POWERDOWNFLAG file and calls SHUTDOWNCMD.
  • SMF shuts everything down before finally signaling the UPS to power off and return once the utility power has been restored.
  • The system loses power.
  • Time passes
  • Power returns and the UPS switches back on, restoring supply to the loads.
  • All systems reboot and continue on as normal.

Since it interacts with upsd to actually power down the loads, it will need credentials set in upsd.users. Lets do that now. Add the following user:

/opt/tools/etc/nut/upsd.users:

[monuser]
  password = doesitmatter
  upsmon master

Reload upsd to update it with the new information.

# upsd -c reload
Network UPS Tools upsd 2.7.4

Define MONITOR and SHUTDOWNCMD within the upsmon configuration.

/opt/tools/etc/nut/upsmon.conf:

MONITOR smt2200@localhost 1 monuser doesitmatter master
SHUTDOWNCMD /usr/sbin/poweroff

And then start upsmon.

# upsmon
Network UPS Tools upsmon 2.7.4
fopen /opt/tools/var/db/nut/upsmon.pid: No such file or directory
UPS: smt2200@localhost (master) (power value 1)
Using power down flag file /etc/killpower

While the automatic shutdown can be tested, it's probably best to wait until after completing setup of the following section.

Setting up SMF manifests

While the global zone is relatively ephemeral, we do have the option of creating persistent services by placing SMF manifests into /opt/custom/smf, and since we really don't want to have to manually set this all up each time, we're going to do that now.

/opt/custom/smf/nut.xml:

Note: Remove the upsmon instance if you don't want it to start up and attempt to automatically power off your system in the case of a power failure.

Now, you can either restart, or simply import the manifest to enable these services.

# svccfg import /opt/custom/smf/nut.xml

Ensure that they are properly running.

# svcs nut
STATE          STIME    FMRI
online         10:46:59 svc:/pkgsrc/nut:upsdrvctl
online         10:46:59 svc:/pkgsrc/nut:upsd
online         10:46:59 svc:/pkgsrc/nut:upsmon

Conclusion

While I'm still testing this and may continue to update this article, that's basically what it takes to get NUT running in your global zone.

I have to say that I'm disappointed in APC as their SUA1500 model which protects my HP N54L actually has more features exposed through the USB cable than the newer SMT2200-RM2U. Specifically:

  • Battery Manufacture Date.
  • Battery Temperature.
  • Input data, including UPS sensitivity, high and low transfer thresholds, reason for last transfer, and current input voltage.
  • Output data, including sinewave frequency and voltage.
  • UPS load as a percentage.

While much of this data is available on the physical front panel, that means I can't poll it through upsc for monitoring in metrics, which for me was the entire point of installing NUT in the first place. Additionally, the upscmd command is quite limited, lacking the following features that the SUA1500 has:

  • Being able to turn the UPS on.
  • Adjust the automatic power-on behavior for the UPS (shutdown.return vs shutdown.stayoff).
  • Testing the battery.
  • Testing the front panel.