Build Your Own Chat System with Jabber/XMPP

by Silke Meyer

Yet another instant messaging system? Fair enough, most people already have a variety of messenger apps on their phones. Isn’t Jabber this nerdy communication network that never made it to the masses? Why is it interesting to even consider it as an addition or a replacement for the state-of-the-art messengers we have nowadays?

The Jabber/XMPP network is decentralized, meaning that there is not a sole provider running servers for it. Anyone can install a Jabber server using free software. All of those servers communicate with each other over the open XMPP standard. Think of email: Anyone can send emails to anyone, no matter who is with which email provider.

A free (as in freedom) and decentralized communications infrastructure has several benefits: It allows us to run D.I.Y. services. You can build your own network and decide whether it is meant to be open to the public or for a closed group. As a server admin, you own the data of your network instead of handing them over to a third party. The more servers there are, the more choice users have to select the provider they trust most. Also, the more servers there are, the harder it is to harm or to control the network. Running your own infrastructure allows you to gain a certain independence from companies’ communication services. Companies may always decide to shut down even popular services, and central corporate chat services often use your phone number as a unique identifier for your account. This comes in handy to find your contacts, but it implies that you don’t have the option to use separate accounts for separate contexts because you only have one phone number. It then becomes impossible to choose which contacts can find you on the network. Anyone you ever gave your phone number to can see that you have an account.

That said, decentralization also has downsides: Quite an important one is that server providers have to keep their servers up to date or else they may cut their users off the network. In the XMPP network, servers that still rely on deprecated server-to-server encryption like TLS 1.0 are no longer connected to servers with a state of the art configuration that only allows TLS 1.2. With many volunteers running Jabber servers, it’s likely that not everyone takes care of maintenance all the time. Obviously, a decentralized network doesn’t provide a central address book, instead, users have to actively spread their addresses. With email, this has been working for many years. But in the context of instant messaging, a few big companies made us expect that we can just join the network and be connected to everyone we know. In short, decentralization brings some inconvenience.

So what is Jabber?

Jabber was the original name of the communication protocol which is now called XMPP, Extensible Messaging and Presence Protocol. It was started by Jeremie Miller at the end of the 1990s and first, it was all about real-time communication, presence information (“Are my contacts online right now?”), and server-side contact list maintenance. The Internet Engineering Task Force (IETF) has had an XMPP working group since 2002. There are several official XMPP specifications, the most important one being RFC 6120 “Extensible Messaging and Presence Protocol (XMPP): Core”. The term “extensible” implies that there are a plethora of official extensions to XMPP which are called XMPP Extension Protocols (XEPs).

Some of these extensions are the reason why Jabber has become interesting again. They add new features to transform XMPP into a more state-of-the-art messaging system. Most people expect a modern messenger to accomplish more or less the following tasks: It should run on multiple devices simultaneously, keeping all conversations in sync between them. It should deal with the fact that mobile internet connections are unstable. Battery drain is a no-go. Finally, it should offer end-to-end encryption of messages, preferably with an intuitive way of verifying your contacts. These features have been implemented in XMPP with the following XEPs:

• Message Carbons (XEP-0280) makes the server send the messages to all devices an account has connected from.
• Message Archive Management (XEP-0313) introduces an archive that stores all messages in a database so that offline devices can come and collect them later.
• Stream Management (XEP-0198) allows session resumption for devices with a flaky internet connection.
• Client State Indication (XEP-0352) lets XMPP clients inform the server whether users are actively using the app or whether it is running in the background. To save battery life, the server won’t send unnecessary information to idling devices, e.g. “The contact is typing a message right now”.
• Push Notifications (XEP-0357) enable the XMPP server to wake up mobile devices to make them pull new messages. (This is very simplified! Any server cannot just push notifications to your phone.)
• OMEMO (XEP-0384) implements end-to-end encryption between more than two devices. It is a recursive acronym that stands for OMEMO Multi-End Message and Object Encryption. The usual Off-The-Record encryption (OTR) can only encrypt messages for one receiving device at a time.¹ Features like Message Carbons don’t help if one device can receive an OTR encrypted message while the other connected devices only see the unreadable crypto stuff. This is the problem OMEMO fixes. The status of OMEMO is still experimental and most XMPP clients haven’t implemented it yet.
• HTTP File Upload (XEP-0363) opens an HTTP server over which images or other files can be shared even in multi-user chatrooms. Conversations for Android seems to be the only client to support OMEMO encryption for this right now.

Note that for most of the above-mentioned XEPs, it is not sufficient to activate the modules on the server because the XMPP clients also have to support them, e.g. Message Carbons or Message Archive Management. On the contrary, end-to-end-encryption is obviously not handled by the servers but by XMPP clients.

Set up your own XMPP server

Are you ready to try it out? If you have set up a web server at some point, you can install up your own XMPP server with just a bit of reading. Please be careful with copy and paste tutorials—it is really recommended to read documentation and ask people about the parts you don’t understand! Here are the most important steps:

• Choose the server software. The most common two XMPP servers are ejabberd, written in Erlang, and Prosody which is written in Lua. I chose Prosody because back when I started, it was easier to configure than ejabberd. (This has probably changed by now.) The Prosody project offers a package repository for APT-based Linux distributions where you have the choice between the stable release (package “prosody”) and nightly builds (package “prosody-0.10”). I recommend the 0.10 nightly builds because the stable 0.9 release is not compatible with all newer modules.
• In Prosody, most of the interesting features have been written as modules so you want to get the Prosody community modules and include their path into Prosody’s configuration file. Think of a way to update them regularly, otherwise they might stop working at some point.
• If you want to offer Message Archive Management, create a database and configure it.
• Of course, you need to get a domain and an SSL certificate, e.g. from the certification authority Let’s Encrypt. Configure your server to require both client-to-server and server-to-server TLS encrypted connections. Read the Prosody documentation about security and the Prosody documentation about “Advanced SSL/TLS configuration”² and decide if you need any of it. Keep the defaults if you don’t understand certain settings! You might want to limit TLS to the protocol version 1.2 by adding this line to the SSL configuration:

      ssl = {
       protocol = "tlsv1_2";
      }
  

• Once the basic server is up and running do some security tests against it, such as Cryptcheck or xmpp.net which offers separate checks for client-to-server (c2s) and server-to-server (s2s) connections.
• An optional cool thing to do is to run the XMPP server as a Tor Hidden Service for XMPP clients with extra privacy features like TorMessenger (experimental) or Conversations for Android. You’ll need to install Tor on the server and edit /etc/tor/torrc as shown below. Restart the Tor service and look up the .onion-URL that was generated for your server in /var/lib/tor/server1.tld/hostname. Publish this address for your users.

  DataDirectory /var/lib/tor
  HiddenServiceDir /var/lib/tor/server1.tld/
  HiddenServicePort 5222 server1.tld:5222 # for client-to-server connections
  HiddenServicePort 5555 server1.tld:5555 # optional: port for file transfer proxy if you use mod_proxy65
  

• Configure the server to send a welcome message to new users where you explicitly recommend turning on end-to-end encryption.

Now you are good to go into the details. Your choices will depend on your target group: Who are you setting the server up for? Is it a manageable amount of people you maybe even know in person? Or is it meant to be a public server where anyone can register an account?

• Make some decisions about the features you want to offer. In Prosody you can find them among the community modules, each of which has a short documentation page. If the server is meant to be used by a closed group, discuss what you need.
• Make some decisions about data retention and ensure it’s accessible to your users. Ask yourself if you want to limit the number of accounts, e.g. due to server resources. Can inactive accounts stay forever or do you want to do some housekeeping from time to time? Note that to remove orphaned accounts you have to keep a log of your users’ last logins. How do you want to react to bots registering accounts on your server? (Yes, spam over XMPP exists, while spam protection does not.) What data will your server log and for how long will you keep the logs? What do the logs tell about your users and do you really need to know all that for debugging purposes?
• Think about where to spread the word about your new service. If it is meant to be a public server, you can register it in the Public XMPP Server Directory and/or in the XMPP Compliance Tester’s Github repository to appear in their respective lists of servers.

Considerations on privacy of user data

Now that you are responsible for other people’s sensitive data, keep in mind that the same features you want for convenience mean that you’ll increase the amount of user data you retain. Without a message archive or offline messages, you obviously don’t save any messages on the server but at the cost of convenience. Actually, privacy concerns are the main reason some providers choose to not turn features on. An interesting read is riseup.net’s discussion of their XMPP server.

Let’s have a look at the database! First of all, you can see people’s contact lists, the so-called rosters. This is what the entries look like. User alice@server1.tld has bob@server2.tld in their contact list, more precisely in the contact group “Buddies”.

mysql> select * from prosody where store = "roster" and user = "alice" limit 1\G;
*************************** 1. row ***************************
host: server1.tld
user: alice
store: roster
key: bob@server2.tld
type: json
value: {"groups":{"Buddies":true},"subscription":"both"}

Similarly, you can look up who people have added to their blocklists! This is about as sensitive of data as the contact list is. User alice@server1.tld has blocked shut-up@server5.tld.

mysql> select * from prosody where store = "blocklist" limit 1\G;
*************************** 1. row ***************************
host: server1.tld
user: alice
store: blocklist
key: shut-up@server5.tld
type: boolean
value: true

Below, you can see what an OMEMO encrypted message looks like: the payload itself is encrypted but you retain all sorts of clear text metadata. alice@server1.tld sent an encrypted message to bob@server2.tld. According to the UNIX timestamp, this happened on Feb 9th 2017 at 11:02:19 GMT. Alice’s client is Conversations, the encryption is OMEMO (“axolotl”).

mysql> select * from prosodyarchive\G;
*************************** 7. row ***************************
sort_id: 322419
host: server1.tld
user: alice
store: archive2
key: fd060ccd-5ce7-4162-a588-7c3b0df7c820
when: 1486638139
with: bob@server2.tld
type: xml
value: <header>MwohBd+ZQ7pJCaREonBQug9kAp29mpiri8hh3RjRTgDQ==</header>9pPikdwmCgpBfA/elK9yL6lYH1Ho0za2

We would obviously see a plain text message here if Alice had not used end-to-end encryption. There is an unofficial Prosody module to enforce OTR encryption from the server’s side. This is a very good first step, but next, it should support both OTR and OMEMO encryption. I really hope that some Lua and crypto programmers will figure out how to do this and add it to Prosody.

In addition to the database, you also collect your users’ data in the file system: offline messages from clients that don’t support Message Archive Management are stored in the file system, as well as the files they share via HTTP Upload. Check out the directory /var/lib/prosody.

This brings us to the housekeeping tasks: In my experience, people are really keen to have the new features. So, depending on the use case for the server, you should create a data retention policy. Keeping the archived messages and the file uploads for a week or a month is probably enough. The next thing you will find out is that Prosody does not offer many settings for cleaning up from time to time. So now, you can either make a code contribution to Prosody (which would be greatly appreciated!) or help yourself with sysadmin workarounds, e.g. cronjobs:

# crontab -e
# Delete uploaded files that are older than 7 days
@daily /usr/bin/find /var/lib/prosody/http_upload -type f -mtime +7 -delete; find /var/lib/prosody/http_upload -mindepth 1 -type d -empty -delete
# Delete offline messages that are older than 7 days
@daily /usr/bin/find /var/lib/prosody/server1\%2etld/offline -type f -mtime +7 -delete
# Delete archived entries that are older than 7 days\
@daily week=$( date --date="7 day ago" +"%s" ); /usr/bin/mysql -e "delete from prosody.prosodyarchive where `when`<=$week"

As you can see, there is still room for improvement: I am looking forward to seeing OMEMO leave experimental status. It could then be a good option to enforce end-to-end encryption with XMPP. Also, it would be cool to be able to hide the contact list and blocklist from server admins. This would be a big change to how the servers and clients currently work, but hashing this information and comparing hashes could be a way to achieve it. Like the majority of free software projects, Prosody is developed by few people, but anyone can contribute.

Note to Jabber users

My experience from running a public XMPP server is that nobody actually asks who you are to find out if you are trustworthy. (How could they know, after all?) People just register accounts. Obviously, many of those who use Jabber know about the features and they want them. They only contacted me for one reason: features that stopped working. Out of curiosity, I looked at some samples of archived messages and uploaded files. I was glad to see that they were all encrypted! I would still like to end with a few hints for Jabber users:

• Please be aware of what your server admin can find out about you! Choose a trustworthy provider.
• Do a feature comparison of clients before choosing one. You will only be able to benefit from the new features if your Jabber client is actively developed to support them. The recommended client for Android is called Conversations. For iOS, it is ChatSecure and for desktop computers, it's most likely Gajim although there are features missing in Gajim (e.g. OMEMO encrypted file uploads).
• Convince your contacts to switch to modern XMPP clients, otherwise all the cool features don't make any sense for you.
• Verify your encrypted connections if possible. Only then can you be warned if a connection was intercepted.

Interesting links

• The (Sad) State of Mobile XMPP in 2014 by Georg Lukas
• The State of Mobile XMPP in 2016 by Daniel Gultsch
• Encrypted Instant Messaging Recommendations January 2017 by Marcel Ackermann
• Blind Trust Before Verification by Daniel Gultsch
• Riseup.net's Discussion of XEPs and User Privacy for their XMPP server
• XMPP Compliance Tester by Daniel Gultsch
• IM Observatory – Testing the Security of the Jabber/XMPP Network
• CryptCheck

Silke is a free software advocate based in Berlin, Germany. She currently works in the operations team of a hosting provider and she loves to build communications infrastructure with free software.