Erlang & Ejabberd on OpenShift

I’m mixing it up a little here – this post is about the backend for our GrafPad site and not related to Python or RapydScript. We will probably be using an XMPP for some new GrafPad features. Sure, there are Python XMPP servers, but we won’t touch the XMPP server code, we’ll only talk to it, so language doesn’t matter. We’re going to use the most proven XMPP server, ejabberd, which happens to written in Erlang. Unfortunately, setting up ejabberd, or any Erlang application for that matter, is a nontrivial task on OpenShift. In most places Erlang likes to automatically bind to 0.0.0.0 and/or 127.0.0.1, which is something not accessible on OpenShift. Even when you provide an IP list to serve on, Erlang interally adds 127.0.0.1. On top of that, Erlang likes to use ports that are not allowed on OpenShift.

ejabberd on OpenShift After messing around for a day or 2, I discovered that these issues cannot be overcome by configs – the changes had to be made in the original Erlang source. Also, fair warning – my goal was to get something working. The solution I have works, but it’s not production ready. I have a repo with the modified Erlang/OTP source, so feel free to make this better!

There are 2 core problems that need to be solved. It turns out the issues all come from 2 things:

  • You cannot bind to 0.0.0.0 or 127.0.0.1
  • The only usuable ports are 8080, and 15000-30000.

Solving these problems is tricky. I originally set out to find everywhere where something binded to 0.0.0.0 or 127.0.0.1 and change where they bind to, but there were still issues starting it up even after changing the values everywhere obvious in the sources (grepping for {0,0,0,0} and 0.0.0.0). I ended up chaging the binding code to change the IP to the $OPENSHIFT_DIY_IP if the address is 0.0.0.0 or 127.0.0.1. I liked this solution a lot better since it accomplishes exactly what I want, and I won’t have to make changes to libraries that try to bind to 0.0.0.0.

So how can you get this setup? It should be pretty easy using my Erlang source. I thought about putting this in a repo that would build when the repo was pushed, but the build takes more than 1 hr so it always gets stopped midway though. I saw messages like Shell command '.../.openshift/action_hooks/build' exceeded timeout of 3516. Instead you have to SSH onto the machine and run the following script manually:

cd $OPENSHIFT_TMP_DIR
wget https://github.com/charleslaw/otp/archive/openshift.zip -O openshift.zip
unzip openshift.zip
cd $OPENSHIFT_TMP_DIR/otp-openshift
sed -i 's/{0,0,0,0}/'"{${OPENSHIFT_DIY_IP//[.]/,}}"'/g'  ./lib/erl_interface/src/connect/eirecv.c
ERL_ROOT=$OPENSHIFT_DATA_DIR/erl_home
./otp_build autoconf
./configure --prefix=$ERL_ROOT --without-termcap
make
make install

After it is done, you can test it by starting epmd and seeing that it works:

$OPENSHIFT_DATA_DIR/erl_home/bin/epmd -address $OPENSHIFT_DIY_IP -debug

Next, you have to install ejabberd. You can do this running this script:

ERL_ROOT=$OPENSHIFT_DATA_DIR/erl_home

cd $OPENSHIFT_TMP_DIR
wget http://downloads.sourceforge.net/expat/expat-2.1.0.tar.gz
tar xzvf expat-2.1.0.tar.gz
cd expat-2.1.0
./configure --prefix=$ERL_ROOT
make
make install

cd $OPENSHIFT_TMP_DIR
wget http://github.com/processone/ejabberd/archive/v2.1.13.tar.gz -O v2.1.13.tar.gz
tar xvf v2.1.13.tar.gz
cd ejabberd-2.1.13/src/
export PATH=$ERL_ROOT/bin:$PATH
./configure --prefix=$ERL_ROOT
make
make install

sed -i 's/localhost/$OPENSHIFT_DIY_IP/g' $ERL_ROOT/sbin/ejabberdctl
sed -i 's/localhost/'"$HOSTNAME"'/g' $ERL_ROOT/etc/ejabberd/inetrc
sed -i 's/127,0,0,1/'"${OPENSHIFT_DIY_IP//[.]/,}"'/g' $ERL_ROOT/etc/ejabberd/inetrc
sed -i 's/127,0,0,1/'"${OPENSHIFT_DIY_IP//[.]/,}"'/g' $ERL_ROOT/etc/ejabberd/ejabberdctl.cfg
sed -i 's/5280/8080/g' $ERL_ROOT/etc/ejabberd/ejabberd.cfg

EDIT: (added based on feedback) Make sure port 8080 is free. If you run ps -ef and see a ruby app running, you’ll need to kill it (If you really want to make sure it’s running on port 8080 run netstat -tulpn | grep $OPENSHIFT_DIY_IP).

Next you can start ejabberd running the following 2 commands, which you’ll want to put in your .openshift/action_hooks/start script:

$OPENSHIFT_DATA_DIR/erl_home/bin/epmd -address $OPENSHIFT_DIY_IP &
$OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl start

If there are no errors, you should be all set! At this point you’ll want to configure ejabberd to your own liking. You will need a user to login to the admin interface. If you don’t want to keep localhost as an XMPP host, change it in $OPENSHIFT_DATA_DIR/erl_home/etc/ejabberd.cfg. You’ll need an admin user on one of your hosts, so run the following command (using your host instead of localhost):

$OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl register admin localhost password1234

Connect to OpenShift My app was at http://erl-jabberserver.rhcloud.com so I went to http://erl-jabberserver.rhcloud.com/admin and logged in using username [email protected] / password1234. Success! I can also connect to it using any XMPP software that supports BOSH. The default setup does not have encryption, so make sure not to require it for testing.

So this works for us since we’re only experimenting with this at this point. But it could be better. Specifically, right now when port 0 is specified, I pick a random port between 20001 & 30000. This should probably instead pick a port that is not used in that range. There may be other issues that I just haven’t run into, but this is a good start.

This entry was posted in Languages, Products And Services and tagged , , by Charles Law. Bookmark the permalink.

About Charles Law

Charles started out in Systems Engineering, designing algorithms, but became a programmer once discovering Python's clean syntax. He has experience using many popular products and services in production environments including web2py, uwsgi, AWS, and OpenShift, as well as experience setting up front-ends in Javascript/Python. He tends to cover his experiences, and notes, from setting up servers, and talks about his experiments with different front-end tools.

12 thoughts on “Erlang & Ejabberd on OpenShift

  1. Hi, I have been trying to create embedded Cartridge for Couchbase and the problem we are facing is Erlang likes to automatically bind to 0.0.0.0 and/or 127.0.0.1,

    So Instead of $OPENSHIFT_DIY_IP i tried to use the private IP of Cartridge. I was m able to launch epmd but I am getting error while trying to execute $OPENSHIFT_DATA_DIR/erl_home/bin/erl. Do you have any idea how to resolve this?

  2. Hi, thanks for the great guide!

    I followed everything successsfully up to here: Next you can start ejabberd running the following 2 commands, which you’ll want to put in your .openshift/action_hooks/start script

    there is no error and ejabberd seems to be started, but the next command: $OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl register admin localhost password1234

    give error:

    Failed RPC connection to the node ‘[email protected]′: {‘EXIT’, {badarg, [{ets,lookup, [local_config, ejabberdctl_access_commands], []}, {ejabberd_config, get_local_option, 1, [{file, "ejabberd_config.erl"}, {line,590}]}, {ejabberd_ctl, get_accesscommands, 0, [{file, "ejabberd_ctl.erl"}, {line,236}]}, {ejabberd_ctl, process,1, [{file, "ejabberd_ctl.erl"}, {line,199}]}, {rpc, ‘-handle_call_call/6-fun-0-’, 5, [{file, "rpc.erl"}, {line,205}]}]}}

    Commands to start an ejabberd node: start Start an ejabberd node in server mode debug Attach an interactive Erlang shell to a running ejabberd node live Start an ejabberd node in live (interactive) mode

    Optional parameters when starting an ejabberd node: –config-dir dir Config ejabberd: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/etc/ejabberd –config file Config ejabberd: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/etc/ejabberd/ejabberd.cfg –ctl-config file Config ejabberdctl: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/etc/ejabberd/ejabberdctl.cfg –logs dir Directory for logs: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/var/log/ejabberd –spool dir Database spool dir: /var/lib/openshift/52c9674d5973ca7734000180/app-root/data//erl_home/var/lib/ejabberd –node nodename ejabberd node name: [email protected].1

    I’m not sure what causes the error… Is there something I missed?

    Thanks for the advice…

    • So I read this and had no idea what was going on. I had never seen this before, so I went to go test it on my setup, and I saw the error, what luck! For me it turned out that there was something running on port 8000 (/var/lib/openshift/526698c55973ca91ce000224/app-root/runtime/repo//diy/testrubyserver.rb) so I killed it and then restarted ejabberd. Let me know if this works, and I can make sure to put it in the instructions.

      • Hi, I am also encountering the same problem. How do you check for the processes running on port 8000? I do not have testrubyserver.rb or any other script as it is vanilla diy cartridge that I just recreated…

        I follow the whole process and now I get exactly the same error:

        [ejabberd-jibostore.rhcloud.com src]\> $OPENSHIFT_DATA_DIR/erl_home/sbin/ejabberdctl register admin localhost password1234 Failed RPC connection to the node ‘[email protected]′: {‘EXIT’, {badarg, [{ets,lookup, [local_config, ejabberdctl_access_commands], []}, {ejabberd_config, get_local_option, 1, [{file, "ejabberd_config.erl"}, {line,590}]}, {ejabberd_ctl, get_accesscommands, 0, [{file, "ejabberd_ctl.erl"}, {line,236}]}, {ejabberd_ctl, process,1, [{file, "ejabberd_ctl.erl"}, {line,199}]}, {rpc, ‘-handle_call_call/6-fun-0-’, 5, [{file, "rpc.erl"}, {line,205}]}]}}

        Commands to start an ejabberd node: start Start an ejabberd node in server mode debug Attach an interactive Erlang shell to a running ejabberd node live Start an ejabberd node in live (interactive) mode

        Optional parameters when starting an ejabberd node: –config-dir dir Config ejabberd: /var/lib/openshift/52e5c76f5973ca731a000002/app-root/data//erl_home/etc/ejabberd –config file Config ejabberd: /var/lib/openshift/52e5c76f5973ca731a000002/app-root/data//erl_home/etc/ejabberd/ejabberd.cfg –ctl-config file Config ejabberdctl: /var/lib/openshift/52e5c76f5973ca731a000002/app-root/data//erl_home/etc/ejabberd/ejabberdctl.cfg –logs dir Directory for logs: /var/lib/openshift/52e5c76f5973ca731a000002/app-root/data//erl_home/var/log/ejabberd –spool dir Database spool dir: /var/lib/openshift/52e5c76f5973ca731a000002/app-root/data//erl_home/var/lib/ejabberd –node nodename ejabberd node name: [email protected].1

  3. I found the ps aux | less command and managed to kill the testrubyserver.rb process. I didn’t know how it turns out to be there as well… but it is still giving the same error… I also tried killing of all the other processes but none works… Is it correct that it tries to connect to localhost?

  4. Hi,

    I’m looking through your OTP code changes and checking if some of them could be avoided by simply tweaking some configuration parameters.

    • The port for EPMD could be changed by simply passing the -port option or by setting the ERL_EPMD_PORT environment variable
    • Even if the loopback interface is automatically added internally, you should be able to specify the interface to be used by simply setting the inet_dist_use_interface application environment variable in the kernel Erlang application (this needs verification).
    • You can set the variables inet_dist_listen_min and inet_dist_listen_max in the Kernel application to set the range of ports which will be used by the Erlang distribution protocol, so that it fits your requirements

    Let’s try to make this working without touching OTP. If not possible, we can provide a patch.

    • Awesomse!

      I think I may have tried the the port and inet_dist_use_interface ideas already, but I cannot find my notes on doing this. I did (do?) not know Erlang, and never worked with it before so I would not be at all surprised if my solution missed something obvious. I am excited to see what you find!

  5. Hi. Thank you for the instructions.

    I followed this guide and got ejabberd installed on openshift, but I cannot connect clients. I believe everything has been successfully installed. The only thing I did not know was where to place the action hooks. My server is up and running and I can log into the admin interface. However I cannot connect clients to the server. I am at a loss for why. My server is located at: http://urlgone.com/b6d001/ (this link will expire in 30 days). Is anyone able to be of assistance?

  6. Hi, I followed your guid succesfully, but the port 8080 seems to be not opened. I can connect on ejabberd admin page but I cannot connet to the server using clients on that port, could you pls help me? thanx

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>