Graceful UPS shutdowns for your ESXi server with Centos

Background

When I initially setup my home ESXi server the UPS was an afterthought.  I vaguely knew that ESXi could be setup to trigger a shutdown from a UPS and i figured if I had Vmware Guest utilities installed on all my vm’s that would take care of safe shutdown.  Searching around most of the methods were either network shutdown (my UPS only has a USB connection) or homebrew.  If it’s homebrew already I might as well learn what this is all about and build my own solution.

 

Base setup

The heart of the UPS monitor/shutdown service will be a small vm running Linux. I chose Centos since it is reasonably light-weight feel free adapt this to your favorite distro-of-choice.

  1. Download Centos 6.3
  2. Create a new VM in ESX with the following:
    • 512 MB ram
    • 8 gig hdd – Thin provisioned
  3. Install Centos as normal
  4. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 – set onboot to yes reboot
  5. yum upgrade
  6. After installation add the USB controller and pass-through device from ESX.  You should see something labeled “APC”
  7. yum install usbutils
  8. lsusb – see if apc is there

 

Install APC Stuff

The default package repo in Centos did not have the apcups packages so we need to add “epelrepo” (http://elrepo.org/tiki/tiki-index.php)

  1. Add epel repo
    • rpm -Uvh http://ftp.osuosl.org/pub/fedora-epel/6/i386/epel-release-6-7.noarch.rpm
  2. yum install apcupsd
  3. chkconfig apcupsd on
  4. edit /etc/apcupsd/apcupsd.conf and set the following options
    • UPSNAME – any name will do – UPSNAME myups01
    • UPSCABLE – usb – UPSCABLE usb
    • UPSTYPE – usb – UPSTYPE usb
    • DEVICE – leave this blank – DEVICE
  5. service apcupsd start
  6. apcaccess (to test). You should see something like the output below

Email alerts

Next we’ll setup UPS alerts so the UPS can warn you of power outages, battery status, etc. The detail and options available will be different for each UPS to make sure to review the apcupsd manual for details. Remember if you intend to got notifications of power outages your modems, routers, switches, etc. need to be powered at that time :-)

  • yum install sendmail
  • chkconfig sendmail on
  • service sendmail start
  • Edit /etc/aliases and setup and alias for root, perferably an in internal and external user ex. root:  user,user@gmail.com
  • Run /etc/apcupsd/changme this will simulate a “battery needs changing” alert and should trigger an email to be sent

Install Vmware Tools

At this point we have the apcupsd tools installed, we can monitor the battery and have the default alerting rules setup.  Our plan will be to have a manually specified list of vms and the order to shut them down in.  The purpose for these is we may be running vms that don’t support Vmware Tools, thus can’t do a “safe shutdown” when ESXi shuts down.

When apcupsd triggers a shudown (senses loss of power) we’ll but using our Centos install to control the shutdowns of all of our vms and the ESX server itself.  However we don’t have a good way to tell our vm to shut itself down (it is possible we could add our instance to the end of the shutdown script or do a simple shutdown -h now … this seemed like a cleaner solution at the time)

  • ESX Console -> Vm -> Guest -> Install Tools
  • mount /dev/cdrom /mnt
  • cp /mnt/VMwareTools* /tmp
  • cd /tmp
  • tar xzvf VMWareTools...
  • cd vmware-tools-distrib
  • ./vmware-install.pl (choose all defaults)
  • reboot and confirm that ESX summary for the vm shows vmware tools running
  • Test “guest” reboot through VM -> Power -> Restart Guest

Test Default UPS shutdown

  • Safely shutdown all ESX vms “just in case” :-)
  • Leave the Centos vm running
  • Unplug the power
  • tail /var/log/apcupsd.events and watch the UPS events. By default when 3 mins left will initiate a shutdown

Shut Down Everything

At this point we have tested all the pieces of our glorious UPS shutdown system.  Time to put the final pieces in place to do the actual work.

Setup SSH access

To accomplish the shutdowns we will need to run some commands on the ESX server, which means our Centos VM will need ssh access to the server.

Following this guide create a ssh key and add it to the ESX servers authorized keys.  Make sure to follow this guide since the authorized_keys file isn’t in the usual location

 Shutting down the VMs

In ESXi there is a built in command language to allow us to do some of our regular maintenance tasks from the command line.  One of these is to send a “shutdown signal” to the vm.  Since we want to control the order our VMs are shutdown we need to get a list of the VMs and their unique IDs first

Log on to your ESX server and run the vim-cmd vmsvc/getallvms command and note the VM names, IDs and the order you want to shut them down in.

Next create a script similar to the following and place it somewhere on your ESX server.

https://gist.github.com/zpeters/3909385

Wrapping It All Up

After you’ve tested remote access to the ESX server (run something like ssh root@1.1.1.1 "ls" and making sure you get output) and you’ve done a test run of your shutdown script. It’s time to tell apcupsd to run this instead of the traditional ‘shut myself down’ command.

  • edit /etc/apcupsd/apccontrol and find the line for doshutdown
  • Replace the traditional “shutdown -h ….” line with ssh root@1.1.1.1 "/scripts/shutdown-all-vms.sh"

And that is pretty much it.  You have a small Centos instance monitoring your UPS battery state, capable of initiating shutdown (or any other commands you wish) on a power outage.  Timing for power outages, how long you wait to shutdown the host, etc. are up to you, however the defaults are reasonable.

Have fun and enjoy.

 

Bonus – CGI interface

As a bonus I’m throwing in instructions on how to setup the CGI interface that will allow you to see the UPS/Battery status in real time.

  • yum install apcupsd-cgi
  • chkconfig httpd on
  • service httpd start
  • edit /etc/sysconfig/iptables (add a line to allow 80 in – copy the line for ssh/22)
  • service iptables restart
  • edit /etc/httpd/conf.d/apcupsd.conf (comment out Allow from lines and add Allow from all)
  • service httpd reload
  • Browse to http://1.1.1.1/apcupsd

 

References

Share

5 Comments

  1. Bernhard Kather says:

    Hello Zach,

    many thanks for this solutions, I looking long time for this using APC UPC, UPCUPSD and VMware.

    I have no experience with Linux/Centos and I have on problem, may be you could help me?

    I have more than one hosts connect to the APC UPS and I want to conect the other host trought the NET function of APCUPSD. But always coms an error: commlost

    The apcupcsd.conf:
    UPSCABLE ether
    UPSTYPE net
    DEVICE 192.168.10.17:3551
    POLLTIME 10

    Could you help me or any idea?

    Best regards
    Bernhard

  2. Anonymous says:

    Did you ever find “a good way to tell the vm (running apcupsd)” to shut itself down?

  3. Zach Peters says:

    I ended up just putting a “shutdown -h now” at the end of the script with a “sleep” before it :-)

  4. Ken Shane says:

    Specifically, where at the end did you put the shutdown statement?

    How long is the sleep?

    Did it work for you? :)

    # leave the ups station running to continue exeuting this script

    # sleep for X to ensure everything shutdown
    echo “Sleeping for 300 seconds…”
    sleep 300s

    # shutdown the esx server
    echo “Shutting down self…”
    poweroff

  5. Zach Peters says:

    If all of your VMs support VMware tools the easy way is to just configure them to shutdown with the host in the VMware client and then reduce this script to the “power off” command. In my case I ended up modifying the command I’m running from the ups VM to the following “ssh -l root 192.168.1.2 ‘/scripts/shutdown-all-vms.sh’ && shutdown -h now”. That way once the main script does its job, the ups VM will shut itself down.

    Hit me up if you have any questions or find a better approach :-)

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre user="" computer="" escaped="">