HHVM upstart script

pjv · November 3, 2014, 12:12pm

If you follow along the rtcamp tutorial for installing HHVM on Ubuntu and use the apt-get install, HHVM is installed with an init.d script that will start and stop the daemon, but does not re-start it if it crashes. The install also did not (for me on 14.04) run the rc script to make the daemon start up on boot.

In my testing, I found that the daemon was crashing sometimes and then things slowed down considerably on the site because first nginx would try to send the request to HHVM and then it would wait 10 seconds for it to time out and then it would send the request to php-fpm as a backup. That happens with every request - nginx doesn’t somehow figure out that HHVM is down for the count. I think there are some nginx directives that you can use in the upstream block to make this better, but I thought that the best thing would be to just respawn the HHVM daemon if it crashed.

So I decided to replace the init.d script with a simple upstart script (goes into /etc/init) that would keep an eye on hhvm and respawn if it quit.

First I stopped hhvm: sudo service hhvm stop

Then created /etc/init/hhvm.conf with the following contents:

UPDATE: see latest script in this gist

Then moved the init.d script out of the way: sudo mv /etc/init.d/hhvm ~

Then restarted hhvm sudo service hhvm start

Now upstart is running and supervising the HHVM daemon for you. It will automatically start up on reboot and if it crashes it will respawn for you.

NOTE: read through the rest of this thread for some caveats and updates.

rahul286 · November 3, 2014, 12:17pm

Good work.

Thanks for sharing your script. We faced same problem in past. We will give your script a try.

pjv · November 3, 2014, 2:12pm

I should mention that the script ignores /etc/default/hhvm. To be complete it should probably be sourcing that file for any additional arguments, but when I looked at the file, everything in it was commented out, so for simplicity, I just excluded it from the init script.

In the future, if some HHVM update starts putting anything in there, I’d modify the init script to pay attention.

pjv · November 24, 2014, 3:19pm

My VPS had a hardware problem this morning and was rebooted. Unfortunately the upstart script did not successfully restart the HHVM process. Something about creating the required /var/run/hhvm directory on startup was not happening. I’m still investigating and will update the upstart script when i figure out how to fix it.

rahul286 · November 24, 2014, 4:15pm

Thanks for update.

I hope your issue will be solved soon.

pjv · November 24, 2014, 4:20pm

After a bit of research I found what the problem was. I have re-written the upstart script to handle creation of the /var/run/hhvm directory on first run. The syntax of this new version checks out fine, but I am unable to test it at this time on my production server. If anyone wants to try it on a spare server they can reboot, that would be awesome. I put the code into a gist here.

And copy it below as well:

# hhvm - HipHop VM
#
# The HipHopVM server provides a high performance PHP stack and web server.
# modified by pjv from original found here: http://stackoverflow.com/questions/19013516/upstart-script-for-hhvm-hiphop

description     "HHVM server"
author "pjv https://gist.github.com/pjv/2e9ab32d8d9884bf79a4"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 10 5
umask 022

# Location of config and executable
env SERVER=/usr/bin/hhvm
env SERVER_CONFIG_FILE="/etc/hhvm/server.ini"
env PHP_CONFIG_FILE="/etc/hhvm/php.ini"
env RUN_AS=www-data

pre-start script
  if [ ! -d /var/run/hhvm ]; then
    mkdir -p -m0755 /var/run/hhvm
    chown $RUN_AS:$RUN_AS /var/run/hhvm
  fi
end script

exec $SERVER --mode daemon --config $PHP_CONFIG_FILE --config $SERVER_CONFIG_FILE --user $RUN_AS

pjv · December 17, 2014, 1:54pm

Just an update on this upstart script. I have now tested the new version in two ways: 1. it successfully started up the HHVM server/daemon on a system reboot, and 2. this morning something made HHVM crash and upstart started it right back up immediately after it died.

So it seems like this is a good working upstart script for HHVM.

mattylb · December 18, 2014, 7:59pm

That’s pretty awesome @pjv - I hope this makes its way into a future EE release as an option. Great contribution!

rahul286 · December 19, 2014, 3:10pm

@mattylb we already added this script to our hhvm checklist - https://github.com/rtCamp/easyengine/issues/199

alexblajan · December 29, 2014, 4:24am

We can also use ps-watcher for monitoring HHVM (or other processes) and start it automatically. Just saying…

$ apt-get install ps-watcher

Edit /etc/ps-watcher.conf (you might have to create it) and add the following lines:

[hhvm]
occurs = none
action = service hhvm restart

Enable ps-watcher to start:

$ sed -i -e 's/# startup=1/startup=1/g' /etc/default/ps-watcher

rahul286 · December 29, 2014, 11:00am

@alexblajan ps-watcher is nice. Thanks for sharing.

pjv · January 18, 2015, 2:05pm

When updates come in for HHVM, the update will re-create the start script in /etc/init.d. Maybe there is a better way than this, but I run the following after each HHVM update:

sudo service hhvm stop; sudo mv /etc/init.d/hhvm ~/unused/hhvm-init-d ; sudo service hhvm start

obviously you need to either create ~/unused or pick a place where you want to save off the packaged init script.

JDdiego · January 28, 2015, 6:17am

Can you break this part down for those new to server side command line:

How and where do you create “~/unused”

pjv · January 28, 2015, 12:47pm

No offense intended, but if creating ~/unused (or some preferred alternative) is opaque to you, using EE, trying to run a server with HHVM, etc. is probably beyond your current skillset and you might want to back up a little bit and get some basic linux shell training.

That said,

HOW: you make a directory on the command line with the command mkdir

WHERE: is actually already in what I wrote because the ‘~’ is a shortcut in the linux shell for your home directory, so if you type mkdir ~/unused you will create a directory called “unused” inside your home directory.

(But you might not want to keep it there. You can put it (the start script) anywhere you want or even just throw it away [ rm /etc/init.d/hhvm]. The idea is just to get it out of /etc/init.d so it won’t be executed by the system automatically. Again, all of this is basic linux shell / sysadmin stuff that you really need to already know if you are trying to set up and maintain a server with EE).

JDdiego · January 29, 2015, 12:37pm

Thanks for the breakdown, yes I am in the process of learning and this has helped me, and I’m sure it would help others who land on it as well.

alexblajan · February 4, 2015, 9:33pm

I found another good one: monit. it seems its running in the background, daemon, and watches over HHVM. if it uses to much ram/cpu or goes down, it restarts/runs HHVM. See what’s better for EE

JDdiego · February 8, 2015, 6:27am

I’m trying monit and followed these directions to make a file to watch HHVM: https://codeable.io/community/speed-up-wp-admin-redis-hhvm/

check process hhvm with pidfile /var/run/hhvm/pid
group hhvm
start program = "/usr/sbin/service hhvm start" with timeout 60 seconds
stop program  = "/usr/sbin/service hhvm stop"
if failed unixsocket /var/run/hhvm/hhvm.sock then restart
if mem > 400.0 MB for 1 cycles then restart
if 5 restarts with 5 cycles then timeout

But the odd thing is when my HHVM crashes it’s not only a HHVM restart that’s required but I found running ee stack restart and ee clean gets it working again. I’m not familiar with monit, but is there a way it or something else can automatically flush FastCGI and restart the ee stack in addition to restarting hhvm when hhvm crashes?

alexblajan · February 10, 2015, 9:15am

I really don’t know. I’m using the script from pjv

umamakesh · February 22, 2015, 1:04am

monit is awesome

alexblajan · February 22, 2015, 1:38pm

Yes it is