Erlang & Ejabberd on OpenShift

I’m mixing it up a little here – this post is about the backend for our GrafPad site and not related to Python or RapydScript. We will probably be using an XMPP for some new GrafPad features. Sure, there are Python XMPP servers, but we won’t touch the XMPP server code, we’ll only talk to it, so language doesn’t matter. We’re going to use the most proven XMPP server, ejabberd, which happens to written in Erlang. Unfortunately, setting up ejabberd, or any Erlang application for that matter, is a nontrivial task on OpenShift. In most places Erlang likes to automatically bind to 0.0.0.0 and/or 127.0.0.1, which is something not accessible on OpenShift. Even when you provide an IP list to serve on, Erlang interally adds 127.0.0.1. On top of that, Erlang likes to use ports that are not allowed on OpenShift.

ejabberd on OpenShift After messing around for a day or 2, I discovered that these issues cannot be overcome by configs – the changes had to be made in the original Erlang source. Also, fair warning – my goal was to get something working. The solution I have works, but it’s not production ready. I have a repo with the modified Erlang/OTP source, so feel free to make this better!

There are 2 core problems that need to be solved. It turns out the issues all come from 2 things:

  • You cannot bind to 0.0.0.0 or 127.0.0.1
  • The only usuable ports are 8080, and 15000-30000.

Solving these problems is tricky. I originally set out to find everywhere where something binded to 0.0.0.0 or 127.0.0.1 and change where they bind to, but there were still issues starting it up even after changing the values everywhere obvious in the sources (grepping for {0,0,0,0} and 0.0.0.0). I ended up chaging the binding code to change the IP to the $OPENSHIFT_DIY_IP if the address is 0.0.0.0 or 127.0.0.1. I liked this solution a lot better since it accomplishes exactly what I want, and I won’t have to make changes to libraries that try to bind to 0.0.0.0.

So how can you get this setup? It should be pretty easy using my Erlang source. I thought about putting this in a repo that would build when the repo was pushed, but the build takes more than 1 hr so it always gets stopped midway though. I saw messages like Shell command '.../.openshift/action_hooks/build' exceeded timeout of 3516. Instead you have to SSH onto the machine and run the following script manually:

cd $OPENSHIFT_TMP_DIR
wget https://github.com/charleslaw/otp/archive/openshift.zip -O openshift.zip
unzip openshift.zip
cd $OPENSHIFT_TMP_DIR/otp-openshift
sed -i 's/{0,0,0,0}/'"{${OPENSHIFT_DIY_IP//[.]/,}}"'/g'  ./lib/erl_interface/src/connect/eirecv.c
ERL_ROOT=$OPENSHIFT_DATA_DIR/erl_home
./otp_build autoconf
./configure --prefix=$ERL_ROOT --without-termcap
make
make install

After it is done, you can test it by starting epmd and seeing that it works:

$OPENSHIFT_DATA_DIR/erl_home/bin/epmd -address $OPENSHIFT_DIY_IP -debug

Next, you have to install ejabberd. You can do this running this script:

ERL_ROOT=$OPENSHIFT_DATA_DIR/erl_home

cd $OPENSHIFT_TMP_DIR
wget http://downloads.sourceforge.net/expat/expat-2.1.0.tar.gz
tar xzvf expat-2.1.0.tar.gz
cd expat-2.1.0
./configure --prefix=$ERL_ROOT
make
make install

cd $OPENSHIFT_TMP_DIR
wget http://github.com/processone/ejabberd/archive/v2.1.13.tar.gz -O v2.1.13.tar.gz
tar xvf v2.1.13.tar.gz
cd ejabberd-2.1.13/src/
export PATH=$ERL_ROOT/bin:$PATH
./configure --prefix=$ERL_ROOT
make
make install

sed -i 's/localhost/$OPENSHIFT_DIY_IP/g' $ERL_ROOT/sbin/ejabberdctl
sed -i 's/localhost/'"$HOSTNAME"'/g' $ERL_ROOT/etc/ejabberd/inetrc
sed -i 's/127,0,0,1/'"${OPENSHIFT_DIY_IP//[.]/,}"'/g' $ERL_ROOT/etc/ejabberd/inetrc
sed -i 's/127,0,0,1/'"${OPENSHIFT_DIY_IP//[.]/,}"'/g' $ERL_ROOT/etc/ejabberd/ejabberdctl.cfg
sed -i 's/5280/8080/g' $ERL_ROOT/etc/ejabberd/ejabberd.cfg

EDIT: (added based on feedback) Make sure port 8080 is free. If you run ps -ef and see a ruby app running, you’ll need to kill it (If you really want to make sure it’s running on port 8080 run netstat -tulpn | grep $OPENSHIFT_DIY_IP).

Next you can start ejabberd running the following 2 commands, which you’ll want to put in your .openshift/action_hooks/start script:

$OPENSHIFT_DATA_DIR/erl_home/bin/epmd -address $OPENSHIFT_DIY_IP &
$OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl start

If there are no errors, you should be all set! At this point you’ll want to configure ejabberd to your own liking. You will need a user to login to the admin interface. If you don’t want to keep localhost as an XMPP host, change it in $OPENSHIFT_DATA_DIR/erl_home/etc/ejabberd.cfg. You’ll need an admin user on one of your hosts, so run the following command (using your host instead of localhost):

$OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl register admin localhost password1234

Connect to OpenShift My app was at http://erl-jabberserver.rhcloud.com so I went to http://erl-jabberserver.rhcloud.com/admin and logged in using username [email protected] / password1234. Success! I can also connect to it using any XMPP software that supports BOSH. The default setup does not have encryption, so make sure not to require it for testing.

So this works for us since we’re only experimenting with this at this point. But it could be better. Specifically, right now when port 0 is specified, I pick a random port between 20001 & 30000. This should probably instead pick a port that is not used in that range. There may be other issues that I just haven’t run into, but this is a good start.

Is Canonical becoming Microsoft-like?


Ubuntu (Canonical’s main product) sports “Linux for human beings” as it’s motto. With its main goal to make Linux easier for everyday use, it’s no surprise it has become the most widely-used distribution. I myself still run it as the main OS on both of my computers. However, next time I’ll need to reinstall or update, chances are that I’ll switch to Debian or Fedora instead.

I’ve been noticing Canonical treating Ubuntu more and more like Microsoft treated Vista or Windows ME. One of my biggest annoyances is the effect of their set-in-stone 6-month release cycle on the quality of included software. I like getting a new shiny OS every 6 months, but I don’t like being stuck with unstable, buggy beta versions of software just because the developers wanted some new flashy feature and couldn’t wait an extra couple weeks for it to be tested properly. Remember Firefox 3 beta becoming the default in Hardy Heron long-term release? Putting a beta browser into a release you plan to support for 3 years doesn’t sound like a good idea to me. I also remember pulseaudio, which replaced alsa in 8.04 before it was thoroughly tested, and caused me grief in both, Flash and Boxee until I decided to remove it and rebuild alsa again until pulse support got better in later versions.

Karmic came with its own can of worms as well. After installing it, I was unable to boot into my Hackintosh anymore. The culprit? Grub 2 beta replaced the original Grub bootloader, and support for OSX loading was not completely there yet. Again, why are you bundling BETA software into your releases, Canonical? I’m starting to feel like I’m using Windows again, except instead of SP1 to save me, Canonical releases a new version of Ubuntu that usually ends up fixing one thing, but breaking another. Fast forward to Lucid Lynx, another long-term release. All of a sudden the Window control get moved to the opposite side. I was able to move them back again after a few minutes of research on Google, but why mess with something that works, Canonical? I doubt you’ll appeal to the Mac users just by moving the buttons to the wrong side.

The second problem with 10.04 is the new XulRunner, which completely removes all hulahop support that several other applications, including Pyjamas-Desktop have depended on in the past. Obviously, Pyjamas-Desktop package, which was added to Ubuntu package manager in 9.10, wouldn’t work anymore. Canonical’s solution? Remove the entire Pyjamas package, as if it was never there. Any bugs opened against this on launchpad automatically get ignored, since the affected package (Pyjamas-Desktop) is no longer in current version of Ubuntu. How convenient? Screwing the little guy to make work easier for themselves has long been Microsoft’s specialty (ignoring web standards, pushing their own propriatary formats while ignoring existing ones, etc.), but it seems Canonical might soon take the cake. After being confronted about this behavior by Pyjamas developers, Canonical added the Pyjamas package back (since it does not depend on hulahop), Pyjamas-Desktop is gone for good, however, unless it gets ported from XulRunner to WebKit.

Another minor annoyance I have with Canonical is their push towards GPL-software and drivers. I have no problem with that, unless (as is often the case with drivers) the proprietary version is free and actually does a better job than the GPL one. A good example is the NVIDIA and ATI drivers, as well as Wifi drivers. Just like Microsoft seems paranoid of all open-source, Canonical seems paranoid of all closed-source.

Since I’ve stayed with 9.10 (Karmic) to use Pyjamas-Desktop, I’m not sure what new surprises Maverick brings, but chances are I won’t have to find out.