Stupid SmartOS Tricks

Minecraft on SmartOS

Brian Ewell -

Minecraft.

It's one of those games that can appeal to many different people in many different ways.  I got into playing under the assumption that modded Minecraft was the norm, and that calculating the optimum designs for automated power production and gargantuan resource-gathering apparatuses was the entire point of the game.

Today, we'll explore how to robustly host Minecraft on SmartOS, either as a simple vanilla server, a Minecraft Forge server with mods, a Spigot server or as a BungeeCord proxy server.

Zone setup

We'll be operating within a current standard base-64 zone.  A base-32 zone should work as well, assuming your server never exceeds 4GB of memory.  You will likely need to tune max_physical_memory, cpu_cap and quota for your specific Minecraft server, and are likely to change over time.  If you plan on hosting multiple Minecraft servers from this zone, your resource usage is going to skyrocket.  Plan accordingly.

First, let's install Java.  Some older versions of Minecraft may require Java 7, but you really should prefer using Java 8, as it's significantly more performant than it's predecessor.

Also, I'd recommend installing tmux.  It's the easiest way to periodically access a server console.

Lastly, you will want to install git if you're planning on building a Spigot server (which must be built from source).  This additional package is unnecessary with both vanilla Minecraft and Minecraft Forge.

# pkgin in openjdk8 tmux git

I prefer cleaning up the delegated dataset cruft.  We can create a dataset mounted under /var/db/minecraft as well.

# UUID=$(sysinfo | json UUID)
# zfs set mountpoint=none zones/$UUID/data
# rmdir -p /zones/$UUID
# zfs create -o mountpoint=/var/db/minecraft -o quota=8G zones/$UUID/data/minecraft

Next, we can create our Minecraft group and user, set up limits and set the proper permissions for their home directory.

# groupadd -g 900 minecraft
# useradd -u 900 -g minecraft -d /var/db/minecraft -s /bin/bash \
  -c "Minecraft user" minecraft
# projadd -U minecraft -G minecraft -c "Minecraft server" \
  -K "process.max-file-descriptor=(basic,65536,deny)" minecraft
# chown minecraft:minecraft /var/db/minecraft
# chmod 700 /var/db/minecraft

You will need to unlock the minecraft account if you want it to be able to run its own crontab (helpful later on).

# passwd -N minecraft

And then su into them to install Minecraft.

# su - minecraft

Installing Minecraft

There are effectively three different Minecraft servers that are popular today:

  • (Vanilla) Minecraft from Mojang is the easiest to install and maintain, and generally uses the least resources of the three.
  • Spigot, a high-performance Minecraft server that supports Bukkit plugins, and is compatible with Vanilla clients.
  • Minecraft Forge, which supports client and server side mods and is arguably the most powerful of the three (also generally considered the most resource intensive).  Minecraft Forge is also required if you're using a Minecraft Forge client and is generally incompatible with the vanilla client.

We'll cover how to install all three below.

Notice: Each of these sub-sections represents a different type of server install, and should be done separately.

Minecraft server

The Vanilla Minecraft server can be downloaded directly from Majong.

As the Minecraft user, use wget to download the Java Archive file.

$ wget https://s3.amazonaws.com/Minecraft.Download/versions/1.10.2/minecraft_server.1.10.2.jar -O server.1.10.2.jar

Agree to the EULA.

$ echo eula=true > eula.txt

And then start the server.

$ java -server -jar server.1.10.2.jar nogui

If all went well, you should have a functional vanilla Minecraft server.

Spigot server

Due to a legal battle between the Mojang and the developers of Spigot, the later project must be downloaded as source code and compiled locally (this is a legal requirement, not a technical one).  Fortunately, the developers of Spigot have written a tool that does just that.

First, we will need to download this jar file.

$ wget https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar

And then just run it to build against the latest version of Minecraft.

$ java -jar BuildTools.jar

If you want to build against a specific version of Minecraft, that can be specified through the --rev flag.

$ java -jar BuildTools.jar --rev 1.9.4

Wait for the build process to complete (it can take quite a while) and if all goes well, you'll have a shiny new spigot-1.10.jar and craftbukkit-1.10.jar file to work with.

Agree to the EULA.

$ echo eula=true > eula.txt

And then start the Spigot server.

$ java -server -jar spigot.1.10.jar nogui

If all went well, you should have a functional Spigot Minecraft server.  Shut down the server and delete all of the additional files and directories produced by BuildTools.

> stop
...
$ rm -r .m2 BuildData BuildTools.log.txt Bukkit CraftBukkit Spigot apache-maven-3.5.0 work

You can also delete BuildTools.jar if you don't intend on using it again.

$ rm BuildTools.jar

Minecraft Forge server

Minecraft Forge is most popularly distributed at the core of Modpacks: Community designed combinations of community developed Minecraft mods that have been balanced for reasonable gameplay.

Popular mod-pack projects such as Feed the Beast and ATLauncher ship client launchers that allow a user to download and install hundreds of different versions of dozens of different mod-packs.  These launchers can also produce server packs which are suitable to be uploaded to and extracted on the SmartOS zone we're working on.

Feed the Beast distributes their server mod-packs as zip files directly from their CDN, making them the ideal example to use for this section.

Use wget to download and unzip to decompress a server mod-pack.

$ wget http://ftb.cursecdn.com/FTB2/modpacks/FTBInfinity/2_0_0/FTBInfinityServer.zip
...
$ unzip FTBInfinityServer.zip

Download the vanilla Minecraft server and launch wrapper by running the FTBInstall script.

$ sh FTBInstall.sh

Agree to the EULA.

$ echo eula=true > eula.txt

And then start the Minecraft server.

$ java -server -Xms1G -Xmx4G -jar FTBServer-1.7.10-1448.jar nogui

Notice: Minecraft Forge takes the longest to start of the three different server types, and won't work without extra memory.  In the above example: 4G.

Once this test is complete, you can remove much of the install cruft that comes with FTB mod-pack servers.  We will be replacing their start scripts with something more robust in the optimization section below.

$ rm FTBInfinityServer.zip FTBInstall.* ServerStart.*

BungeeCord Server

BungeeCord is a Minecraft proxy server that sits between your clients and one or more Minecraft servers (as it's part of the Spigot project, it's safe to assume Spigot servers are supported), and enables large-scale server deployments.  Details about the configuration of BungeeCord are available on their website.

Download the BungeeCord server.

wget https://ci.md-5.net/job/BungeeCord/lastSuccessfulBuild/artifact/bootstrap/target/BungeeCord.jar

Configure it, and then start the server.

$ java -server -jar BungeeCord.jar

Server optimizations

Getting a Minecraft server to start and run is just the first step.  This guide will also focus on getting your server to run well, and to make the most of the SmartOS platform.  What follows are a series of sections that focus on optimizing different aspects of running a Minecraft server on SmartOS.

Optimized Java parameters

Java Virtual Machines can be started with parameters which adjust their default behavior, and while the start commands in the above examples allowed us to start and test our Minecraft servers just fine, they're the bare minimum and hardly optimal for the daily operation of a high volume server.

The optional parameters we'll be examining below should be applied immediately after the java command:

$ java <parameters> -jar server.1.10.2.jar nogui

We'll break them up into the following three classifications.

Essential parameters

These parameters should always be set for a Minecraft server.

  • -XmsM: Sets the initial size of the heap to M.  This is the memory consumed by the JVM when first starting.  If you want to start with 1GB, the parameter would be: -Xms1G.
  • -XmxM: Sets the maximum size of the heap to M.  This is the maximum amount of memory that can be consumed by the JVM and should always be greater than or equal to -XmsM.  If you want to limit the server to 4GB, the parameter would be: -Xmx4G.
  • -XX:+UseConcMarkSweepGC or -XX:+UseG1GC: Enables the use of the CMS garbage collector for the old generation.  It's recommended that you use this GC when application latency requirements cannot be met by the throughput garbage collector.  The G1 garbage collector (+UseG1GC) is another alternative, but it might not be as memory efficient.
  • -XX:+UseLargePages: SmartOS supports large pages and enabling this should translate to a performance improvement.

Optional parameters

These parameters may be useful for a Minecraft server in certain situations.

  • -d64: Forces JVM to be 64-bit.  This is only practical on a base-multiarch image, as java on base-32 and base-64 will always be one or the other.
  • -server: Selects the Java HotSpot Server VM.  This is only necessary with the 32-bit JVM, as the 64-bit JVM only supports the Server VM.
  • -showversion: Displays JVM version information before continuing execution of the application.  This is practical for logging purposes.
  • -XX:+AggressiveOpts: Enables the use of aggressive performance optimization features which are expected to become the default in upcoming Java releases.  This option may improve performance, but that performance may come at the cost of stability.
  • -XX:MinHeapFreeRatio=N, -XX:MaxHeapFreeRatio=N: Sets the minimum and maximum allowed percentage of free heap space (0 to 100) after a garbage collection event.  If the free space after a garbage collection event is less than the minimum by the given percentage than the JVM will attempt to allocate additional memory to the heap until the difference is greater than the minimum given percentage.  If the free space after a garbage collection event is greater than the maximum by the given percentage than the JVM will attempt to free memory from the heap until the difference is less than the maximum given percentage.  The current defaults of 40% and 75% should be suitable, but I've seen configurations as extreme as 5% and 10%.
  • -XX:MaxGCPauseMillis=10: Sets a target for the maximum garbage collection pause time (in milliseconds).  This is a soft goal, and the JVM will make it's best effort to achieve it.  In the case of Minecraft, 10ms is unreasonably short, but this should help to tune the JVM to run shorter more frequent garbage collection cycles.

Depreciated parameters

Despite often being recommended by other guides, these parameters should not be used on SmartOS.

  • -Xincgc: Enables incremental garbage collection.  This option was deprecated in Java 8 with no replacement.  Use CMS, G1, or let the JVM decide instead.
  • -XX:PermSize=M: Sets the space allocated to the permanent generation that triggers garbage collection if it is exceeded.  This parameter has been deprecated in Java 8 and superseded by the -XX:MetaspaceSize parameter, which shouldn't really be tuned either.
  • -XX:+UseParNewGC, -XX:+UseParallelGC: Explicitly tells the JVM which garbage collection strategy should be used.  Neither of these garbage collectors should be selected (UPN will be automatically enabled with CMS).
  • -XX:+CMSIncrementalPacing: Enables automatic adjustment of the incremental mode duty cycle based on statistics collected while the JVM is running.  This option has been deprecated in Java 8 with no replacement.
  • -XX:+CMSClassUnloadingEnabled: Enables class unloading when using the concurrent mark-sweep (CMS) garbage collector.  This option is enabled by default, meaning there's no reason to enable it explicitly.
  • -XX:ParallelGCThreads=2: Sets the number of threads used for parallel garbage collection.  This should be automatically determined by the JVM.
  • -XX:-UseVMInterruptibleIO: Thread interrupts before or with EINTR for I/O operations results in OS_INTRPT.  I could find no additional documentation describing what this parameter does, so I can't really recommend using it.

Example

Based on the above parameters, I came up with the following JVM options for a Minecraft server running on SmartOS.  It's not perfect, but it leaves most of the finer configuration up to the JVM and out of our hair.

$ java -Xms256M -Xmx4G -XX:+UseConcMarkSweepGC -XX:+UseLargePages \
-jar server.1.10.2.jar nogui

If you'd prefer something more complicated, there are other recommendations out there, but I would recommend you cross-reference them with the Java8 command line documentation to ensure you're not specifying a default or depreciated parameter.

Now with that sorted, you can either take this string of options and write them into a startup.sh script to manually call from within your Minecraft directory, or you can start up Minecraft the fun way.

Minecraft and SMF

SmartOS ships with the Solaris Service Management Facility, which is the ideal tool to ensure our Minecraft server stays running.  And since we're on SmartOS, It'd be a shame not to use it!

Note: Minecraft can either be managed directly by SMF or wrapped with tmux.  Direct is simpler, but you will be unable to issue commands directly to the Minecraft server console, which will cause issues later on.  For the sake of completeness, we will explore both options below.

Import SMF Manifest

As root, download the following SMF manifest:

This manifest has been configured for a default Minecraft server instance wrapped with tmux, if you just want to use Java, uncomment the java only sections and comment out the tmux wrapped sections (both the exec_method and propval tags).

Edit the value_node tags to reflect the java parameters you want to use with your server.  This step can be skipped and done later.

Edit the server propval tag to reflect your specific minecraft server.  This can be done via symlink or skipped and done later as well.

# su - minecraft
$ ln -sf server.1.10.2.jar server.jar
$ logout

Import the manifest and enable the minecraft service.

# svccfg import minecraft-single-smf.xml
# svcadm enable minecraft

You should now have an SMF managed Minecraft server.

Edit Java parameters

Java parameters can be changed and updated directly within SMF without the need to delete and re-import the manifest.

For instance, if you want to expand the initial heap (-XmsN) from 256M to 1G, you'll need to remove the old value and replace it with a new value:

# svccfg -s minecraft
svc:/gameserver/minecraft> delpropvalue options/parameters -Xms256M
svc:/gameserver/minecraft> addpropvalue options/parameters -Xms1G
svc:/gameserver/minecraft> exit

You can also completely clear a property list and replace it with new values (note, the astring: is only required for the first value and the quotations are optional):

# svccfg -s minecraft
svc:/gameserver/minecraft> delprop options/parameters
svc:/gameserver/minecraft> addpropvalue options/parameters astring: "-Xms1G"
svc:/gameserver/minecraft> addpropvalue options/parameters "-Xmx4G"
svc:/gameserver/minecraft> addpropvalue options/parameters "-XX:+UseConcMarkSweepGC"
svc:/gameserver/minecraft> addpropvalue options/parameters "-XX:+UseLargePages"
svc:/gameserver/minecraft> exit

You can also check the pending property list:

# svccfg -s minecraft listprop options/parameters

If you're happy with what you have, refresh (commit) the configuration and restart the Minecraft service:

# svccfg -s minecraft:default refresh
# svcadm restart minecraft

Edit server JAR

Along with the parameters, the Minecraft server string can be changed and updated directly within SMF without the need to delete and re-import the manifest.

For example, if you want to update the server jar to server.1.10.2.jar, set the new value using svccfg (quotations are optional):

# svccfg -s minecraft setprop options/server="server.1.10.2.jar"

Pending configuration data can be read with svccfg:

# svccfg -s minecraft listprop options/server
options/server  astring  server.1.10.2.jar

If the pending configuration is correct, refresh (commit) the configuration and restart the Minecraft service:

# svccfg -s minecraft:default refresh
# svcadm restart minecraft

Delegated authorization

By default, only the superuser can manage the Minecraft service state and properties.  Permission to alter the state or properties of the Minecraft service can be independently granted to additional users through Role-Based Access Control (RBAC).

First, define authorization descriptions under /etc/security/auth_attr:

solaris.smf.manage.minecraft:::Manage Minecraft Service States::
solaris.smf.value.minecraft:::Change Values of Minecraft Service Properties::

Then authorize one or more users with these descriptions.  For example, the following line will grant state and property management permission to the Minecraft user:

# usermod -A solaris.smf.manage.minecraft,solaris.smf.value.minecraft minecraft

This user is now able to reboot the server and adjust the configuration options discussed above.

# su - minecraft
$ svcadm restart minecraft

This is especially practical if you want to grant one or more users access to reboot your server without having any access to its files:

# useradd -m -A solaris.smf.manage.minecraft brian
# passwd brian
...
# su - brian
$ svcadm restart minecraft
$ ls /var/db/minecraft/
ls: cannot open directory '/var/db/minecraft/': Permission denied

Now Brian can log in and reboot the server without any additional access to server data.

Snapshots > backups

What is possibly the best reason to host Minecraft on SmartOS is the first-class availability of ZFS.  Setting up a dataset to contain the world directory along with cron-driven periodic snapshots completely supersedes any functionality provided by CraftBukkit plugins or MinecraftForge mods.

Setup the World Dataset

First, switch your Minecraft server into maintenance state if it's been enabled.

# svcadm mark maintenance minecraft

Move the current world directory to a temporary location.

# mv /var/db/minecraft/world /var/db/minecraft/world_tmp

Create a new dataset for the world data.  Optionally set a quota (20G in our example).

# zfs create -o quota=20G zones/$(sysinfo | json UUID)/data/minecraft/world

Chown the root of the new dataset and copy the contents of the old world directory to the new (this may take some time on established worlds).  Delete the temporary directory when you're done.

# chown minecraft:minecraft /var/db/minecraft/world
# mv /var/db/minecraft/world_tmp/* /var/db/minecraft/world/
# rmdir /var/db/minecraft/world_tmp

If you're done in maintenance mode, you can now clear your Minecraft service.

# svcadm clear minecraft

Delegated ZFS management

Normally ZFS administration is handled by the superuser, but authorizations over actions can be delegated to other users on a per dataset basis.  In our case, we want our Minecraft user to be able to create snapshots of the world dataset so that the periodic snapshot script can be managed under that user.

# zfs allow -lu minecraft snapshot zones/$(sysinfo|json UUID)/data/minecraft/world

Destroying and renaming snapshots can be achieved by granting destroy and rename permissions to any descendant datasets (of which snapshots are).  Note that this will allow any user to destroy or rename any descendant dataset.

# zfs allow -du brian destroy,rename zones/$(sysinfo|json UUID)/data/minecraft/world

You can review what permissions have been granted by calling zfs allow <dataset>:

# zfs allow zones/$(sysinfo|json UUID)/data/minecraft/world

And you can revoke permissions with zfs unallow:

# zfs unallow -du brian destroy,rename zones/$(sysinfo|json UUID)/data/minecraft/world

Setup periodic snapshots

The easiest way to create snapshots of the world directory is by using a script that disables auto-saving, issues an explicit save creates the snapshot, and then re-enables auto-saving.

I put together a simple implementation in bash that does these things, based on the script described in this blog post.

Notice: This script needs to be able to communicate with the Minecraft server console to function.  It's designed to interface with tmux, as configured in the previous section, and will not work in a java-only configuration.

As the minecraft user, copy the following script to the Minecraft directory:

Be sure to adjust snappath to reflect your dataset name.  Add the script to your crontab scheduled to run as often as you want it to.

For example, here's a crontab that will snapshot your world dataset every 15 minutes:

0,15,30,45 * * * * ./snapshot.sh

Notice: If you get errors about Minecraft not being allowed to execute cronjobs, you will need to unlock the account as root.

# passwd -N minecraft

I am aware that this script does not manage existing snapshots (expiring old ones or rotating snapshots) or send snapshots to other systems for remote replication.  I leave extending this script to encompass additional functionality as an exercise for the reader.

Conclusion

This is pretty much everything you need to get a single instance of Minecraft up and running within a single SmartOS zone.  I had a bunch more notes that delved into advanced Minecraft multitenancy but I figured that was a bit much for a single blog post, so we'll save that for another day.