Agregador de noticias

Ignite Realtime Blog: Openfire 4.1.0 Final Release

Planet Jabber - 21 December, 2016 - 15:20

Christmas has come early.


Our github repository - GitHub - igniterealtime/Openfire: A XMPP server licensed under the Open Source Apache License. - reached a thousand and one stars today, so we've celebrated by releasing Openfire 4.1.0, containing over 150 fixes and new features. I feel I should personally thank Daryl Herzmann and Guus der Kinderen in particular for the gargantuan effort they've both put into this release, but we're also very lucky to have a large number of developers within the community providing us with high-quality patches via Pull Requests. This is, once again, a huge collective effort.


Changlog: Openfire Changelog





openfire_4_1_0.dmgWindowsd590ebf8e3ca429b1d738833e14419c3dd514110openfire_4_1_0.exeBinary (tar.gz)


openfire_4_1_0.tar.gzBinary (zip)


openfire_4_1_0.zipSource (tar.gz)cf458223b8acf23c84219f343d5d14bebfe9a963openfire_src_4_1_0.tar.gzSource (zip)a4a637354be17031be64f7cff9443584dde4022dopenfire_src_4_1_0.zipRPM Installer (32-bit JRE)


openfire-4.1.0-1.i686.rpmRPM Installer (64-bit JRE)


openfire-4.1.0-1.x86-64.rpmRPM Installer (No JRE)


openfire-4.1.0-1.noarch.rpmDEB Installer9ea6b83a7aa9229be0358385e02985a9df04418eopenfire_4.1.0_all.deb


You can download it from here:


Ignite Realtime: Downloads


As a reminder, Openfire's source code and development can be found on Github and an active developer discussion occurs within our groupchat.  Please consider helping out with Openfire's development!  We can use help in about all areas of development, but especially:

  • experienced packaging developers to improve the install/upgrade experience on various platforms
  • developers interested in transitioning Openfire to Maven
  • testing of Bamboo generated builds


Thank you for your interest in Openfire!

ProcessOne: ejabberd 16.12 beta1

Planet Jabber - 15 December, 2016 - 14:57

In 2016, we made several major code refactors and improvements on ejabberd. From Elixir support, to test suite and code clean up and modernization.

After 16.09 which brings a lot of improvements, the last version for 2016 includes a big refactor we have been preparing for long time, and a couple of new features that pursue the central effort on API, commands and web integration started early this year.

ejabberd 16.12 includes: a new API permissions framework for commands, a new more reliable BOSH module, major code refactor, more integration and unit tests, improved support for Erlang/OTP R19, compatibility with rebar3 build system and many fixes and optimizations.

New BOSH module

This new BOSH implementation improves BOSH users experience, as it’s more robust and efficient and overall over lower latency.

New API permissions framework

This new framework is one of the big changes in this release. It makes our API very flexible, allowing administrators to fine grain access to some users to a group of API, depending on the path used to call a given command.

While it should make it possible to implement any kind of access control on top of ejabberd management API, it also makes it much easier to configure ejabberd API for simple needs.

ejabberd API operations are organised around the concept of commands. ejabberd modules provide many commands, but the mechanism is generic and any module can provide its own set of commands.

All commands can be exposed through interfaces. Available interfaces are: ejabberdctl command-line tool, ejabberd ReST API and ejabberd XML-RPC API. The XML-RPC API still works but is deprecated in favor of the ReST API. You should migrate to ReST if you are using it.

Finally, ReST API can be accessed through two authentication mechanisms: HTTP Basic Authentication and OAuth 2.

Each command interface can have different restrictions based on how exposed and sensitive the commands are.

Note that the following configuration snippets assume ejabberd API listeners are properly configured, as defined in API listener configuration

By default, when no api_permission option is provided, ejabberd would use the following default permissions:

api_permissions: "console commands": - from: - ejabberd_ctl - who: all - what: "*" "admin access": - who: - admin - oauth: - scope: "ejabberd:admin" - admin - what: - "*" - "!stop" - "!start"

This will grant access to all commands when ejabberdctl is used (this is what “console commands” group does), and additionally each command that is authenticated by user that match acl auth from any source will be able to call all commands except start and stop.

It’s possible to extend this by adding new section like this:

api_permissions: ... "allow statbot to get server stats" - who: - user: "" - what: - connected_users_number - num_active_users

This will allow user statbot to execute commands connected_users_number and num_active_users.

You can even relax the need to authenticate on ReST API, for example, if you only want to restrict the API to localhost usage, for admin tasks:

api_permissions: "local admin can use all commands" - who: - ip: "" - what: - "*" - "!stop" - "!start"

As you can see, for simple cases, the API permission management is now very simple. Yet, the system is powerful and can accommodate your most advanced needs.

For more details, see full api permission documentation.

You can now rely on ejabberd hundreds of commands to integrate in a precise and secure way with your back ends.

Major code refactor

Finally, the most important change (in terms of line of code and impact on ejabberd code base) is refactor of XMPP packet handling in the entire code base. This improvement makes code simpler, safer, and smaller. ejabberd now uses a dedicated XMPP library, which helps developers packing/unpacking XMPP packets.

Example is always better than words:

Don’t build entire XML element, but rely on common “templates”


#xmlel{name = <<"identity">>, attrs = [{<<"category">>, <<"pubsub">>}, {<<"type">>, <<"pep">>}]} IQ#iq{type = error, sub_el = [Error, SubEl]}


#identity{category = <<"pubsub">>, type = <<"pep">>} xmpp:make_iq_result(IQ, Error); Match packets efficiently


normal_state({route, From, <<"">>, #xmlel{name = <<"message">>, attrs = Attrs, children = Els} = Packet}, StateData) -> case is_user_online(From, StateData) of true -> case fxml:get_attr_s(<<"type">>, Attrs) of <<"groupchat">> -> ...


normal_state({route, From, <<"">>, #message{type = Type, lang = Lang} = Packet}, StateData) -> case is_user_online(From, StateData) of true when Type == groupchat -> ... Don’t use macros to build error response


jlib:make_error_reply(Packet, ?ERRT_BAD_REQUEST(Lang, Txt));


xmpp:make_error(Packet, xmpp:err_bad_request(Txt, Lang));

The code is safer, as all XMPP processing in the core of ejabberd can be typed-checked through static analyzer like Dialyzer.

Overall, this change paves the way to improvements of ejabberd API and will make it much more pleasant for contributors to write new modules and extensions. Improving the contribution documentation is next on our list and you are welcome to join the effort.

This big refactor impacts the entire code base. Now that all tests are successful, we provide an early beta before the final 16.12 release. 16.12-beta1 is not yet intended for production use. We are waiting for your feedback to check that special cases did not sleep through our careful refactor. Please, test it and send us feedback in your use case, so that we can catch the tricky bugs before the final version is released.


This is just a summary of the most relevant ones:

API / integration
  • New API permissions framework
  • Add configurable weight for ejabberd commands
  • add_rosteritem: Support several groups separated by ;
  • create_rooms_file: Fix reading room jids from file
  • delete_old_messages: Fix command for SQL backends
  • send_message: Don’t duplicate the message
  • New BOSH module
  • Use fxml_gen XML generator
  • Use our new stand-alone XMPP library instead of jlib.erl
  • Don’t let MAM messages go into offline storage
  • Add xdata generator for XMPP data form
  • Get rid of excessive (io)list_to_binary/1 calls
  • Add authentication support to mod_http_fileserver
  • ejabberd_http: Handle missing POST data gracefully
  • Use inets instead of lhttpc in http_p1
  • Add http_p1.erl, rest.erl, and oauth2 ReST backend for OAuth2 tokens
  • Create room on configuration request as per XEP-0045, 10.1.3
  • Ensure that presence_broadcast room option is stored
  • Fix conference disco#items when running multiple virtual hosts
  • Fix Result Set Management (RSM) for conference disco#items
  • Introduce muc_invite hook
  • Make the constant MAX_ROOMS_DISCOITEMS configurable
  • mod_carboncopy: Don’t copy MUC private messages
  • Store the flag “Allow Subscription” room option in database
  • When getting list of subscribed rooms, also check temporary ones
Relational databases support
  • Append ; to privacy_list_data exporting lines
  • Improve relational database import
  • Make build system compatible with rebar3
  • Produce ejabberd.service and fix for systemd usage
  • Bugfix: Don’t let MAM messages go into offline storage
  • Delete obsolete module mod_configure2
  • Bugfix: Ignore offline sessions in ejabberd command statistics
  • Rename #error{} record to #stanza_error{}

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

Ignite Realtime Blog: Openfire 4.1.0 Beta

Planet Jabber - 14 December, 2016 - 18:52

A Year and a Day - TV Tropes


On the 13th of December, 2015, we released Openfire 4.0.0 Beta.


Now, a year later, we have a new Beta to try. This one has numerous major security fixes and enhancements, and a huge long Openfire Changelog to read through.




openfire_4_1_0_beta.dmgWindows1834ff1278e0249f4fe8662536683aabd6b839c5openfire_4_1_0_beta.exeBinary (tar.gz)


openfire_4_1_0_beta.tar.gzBinary (zip)


openfire_4_1_0_beta.zipSource (tar.gz)625e92e1dd76c99ebde70b3ae7de6a651f08d4d5openfire_src_4_1_0_beta.tar.gzSource (zip)9d9fe67f5e62f3d31d2ab6d6a4ebf282a1e01254openfire_src_4_1_0_beta.zipRPM Installer (32-bit JRE)


openfire-4.1.0-0.2.beta.i686.rpmRPM Installer (64-bit JRE)


openfire-4.1.0-0.2.beta.x86-64.rpmRPM Installer (No JRE)


openfire-4.1.0-0.2.beta.noarch.rpmDEB Installer9cce98aadd53cb3995b355fd54543459ab80e056openfire_4.1.0.beta_all.deb


Please do try this, but not in production of course. Unless you're a mad and crazy bunch, like we are. You can download it from here:


Ignite Realtime: Beta Downloads


As a reminder, Openfire's source code and development can be found on Github and an active developer discussion occurs within our groupchat.  Please consider helping out with Openfire's development!  We can use help in about all areas of development, but especially:

  • experienced packaging developers to improve the install/upgrade experience on various platforms
  • developers interested in transitioning Openfire to Maven
  • testing of Bamboo generated builds


Thank you for your interest in Openfire!

ProcessOne: Build awesome Internet of Things with ProcessOne

Planet Jabber - 13 December, 2016 - 12:33

You probably don’t know this, but ProcessOne has been delivering enterprise-grade solutions for device communication since before it was called the Internet of Things. IoT market is growing at a rapid pace, and by 2018 there will be more internet-connected devices than PCs, smartphones and tablets combined!

But back in 2008, we already implemented production-ready infrastructure for IoT. Based on XMPP, of course, it powered the community of Nabaztags – ambient electronic devices in the shape of a rabbit that could communicate with text and voice messages, play music and internet radio, as well as speak various news: weather, stocks, rss, email etc. Its community continued to be very live and open, and keeps supporting the rabbits through private servers.

After the Nabaztag project we were approached by Kodak to create an IoT infrastructure for digital, connected photo frames. Through 2010 we developed XMPP-based connection backend for several series of Kodak digital frames.

Another notable example of our long-term commitment to the Internet of Things is the now decade-long cooperation with IMA Téléassistance, a major European player in security and monitoring of individual and commercial property. With the growing popularity of Smart Home solutions, IMA Téléassistance recently entered this market with additional connected products like electricity plugs, light and motion sensors or smoke detectors. With our proven IoT backend, they had the confidence to expand their device pool.

At ProcessOne, we believe connected devices will power the big change in enterprise and infrastructure operations. It is our opinion that your business will need IoT infrastructure to cut costs, empower employees and optimize processes. And we are here to help you do that – See more and contact us today! »

Peter Saint-Andre: Musical Finds

Planet Jabber - 13 December, 2016 - 00:00
Although I don't live or eat in a consistently paleo fashion, I do like some paleo foods. One of my favorites is grain-free granola. It also happens to be very simple to make. Here's how....

Peter Saint-Andre: Grain-Free Granola Recipe

Planet Jabber - 9 December, 2016 - 00:00
In Walden, Thoreau wrote:...

ProcessOne: ejabberd Development with Docker

Planet Jabber - 5 December, 2016 - 11:31

We have released a Docker image to help you get started with ejabberd development easily.

Docker image for ejabberd developers is available from Docker Hub: ejabberd/mix

Thanks to this image, you can build ejabberd with dependencies provided in our Docker image, without the need to install any specific tools (beside Docker) directly on your own machine.

Please note that this image can likely be reused as is to build other Erlang or Elixir software.

Pulling ejabberd development Docker image

You can pull ejabberd image for developer with this Docker command:

docker pull ejabberd/mix

Once done, you are ready to build ejabberd from source and add your custom plugin modules.

Building ejabberd from source

You can build ejabberd from source with all dependencies, with the following commands:

git clone docker run --rm -v $(pwd):$(pwd) -w $(pwd) ejabberd/mix do deps.get, deps.compile, compile

Alternatively if you do not have Git installed, you can do retrieve compressed source code archive and then build ejabberd:

wget unzip ejabberd-master cd ejabberd-master docker run --rm -v $(pwd):$(pwd) -w $(pwd) ejabberd/mix do deps.get, deps.compile, compile Run ejabberd with mix command-line tool attached

During development, you will need to run ejabberd with debug console attached.

First, you can customize the configuration used by ejabberd in development mode by editing the file config/ejabberd.yml. As a default, it will run ejabberd with console attached on domain “localhost”:

docker run --rm -it -p 5222:5222 -p 5280:5280 -v $(pwd):$(pwd) -w $(pwd) --entrypoint="/usr/bin/iex" ejabberd/mix -S mix

You can then directly create a user from Elixir shell:

Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false] Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> :ejabberd_auth.try_register("test", "localhost", "passw0rd") {:atomic, :ok}

Finally, you can connect with user “test@localhost” (password: passw0rd) on server on localhost port 5222 and use those parameters to connect with an XMPP client.

Get into the container

If you want to run Erlang command line, you can do so by opening a shell inside the container and then running the command as desired:

docker run -it -v $(pwd):$(pwd) -w $(pwd) --entrypoint="/bin/sh" ejabberd/mix Troubleshooting Clock resync

If you have warning about file timestamp being out of sync (Like ‘Clock skew detected’), you may want to force resync your clock before running the build. Docker on MacOS does not force clock resync of Docker after the laptop went to sleep.

You can force clock resync as follow:

docker run -it --rm --privileged --entrypoint="/sbin/hwclock" ejabberd/mix -s

You can check if the clock of your laptop is in sync with the one inside Docker with the following command:

docker run --rm --entrypoint="/bin/sh" ejabberd/mix -c date -u && date -u Send us feedback

This image is under development and improvement, so please do not hesitate to send us your feedback, so that we can improve ejabberd development experience.

The goal is to provide a simple easy way to develop ejabberd plugins from any OS, whether you use Linux, MacOS or Windows.

Enjoy !

Peter Saint-Andre: Philosophy as a Way of Life

Planet Jabber - 2 December, 2016 - 00:00
Most people have no idea how the web works and don't understand the role they play in the web ecosystem. Partly this is because our mental model of the web dates back to the early days of "Web 1.0", when folks used the web to read physics papers and personal weblogs and such, somewhat as they would visit their local public library and read books and magazines and newspapers sitting on the shelves. My, how times have changed!...

Ignite Realtime Blog: Openfire 4.0.4 is Released

Planet Jabber - 1 December, 2016 - 14:53

The Ignite Realtime Community is pleased to announce the 4.0.4 release of Openfire.  Openfire is a cross-platform real-time collaboration server based on the XMPP (Jabber) protocol. This release signifies our ongoing effort to produce a stable 4.0 series release while continuing to work toward a 4.1 release. A changelog of fixes is available and indicates 16 issues resolved.


As always, please report issues in the Community Forums and you can download this release from our website.  Here are the sha1sum's for the downloads available.





openfire_4_0_4.dmgWindows3d342c6f45abb9fc6aff990cf25d879ee20470edopenfire_4_0_4.exeBinary (tar.gz)


openfire_4_0_4.tar.gzBinary (zip)


openfire_4_0_4.zipSource (tar.gz)e947ffdf4746d3d0a7efb7cf25d198c05668ab54openfire_src_4_0_4.tar.gzSource (zip)fbd8e4c143b7c4841fcb231db2c5974d90137b4fopenfire_src_4_0_4RPM Installer


openfire-4.0.4-1.i386.rpmDEB Installerb16d334a90c86e7b15e9ec44162ec01ca700434dopenfire_4.0.4_all.deb


As a reminder, Openfire's source code and development can be found on Github and an active developer discussion occurs within our groupchat.  Please consider helping out with Openfire's development!  We can use help in about all areas of development, but especially:

  • experienced packaging developers to improve the install/upgrade experience on various platforms
  • developers interested in transitioning Openfire to Maven
  • testing of bamboo generated builds


Thank you for your interest in Openfire!

Ignite Realtime Blog: Spark 2.8.2 Released

Planet Jabber - 25 November, 2016 - 19:00

The Ignite Realtime community has just released Spark 2.8.2 and it can be downloaded from Ignite Realtime: Downloads .


This is a regular update. But there is a change, which can affect new users. "Accept all certificates" option on the Adanced menu on the Login screen won't be enabled by default for new users from now on. This option makes Spark automatically accept expired, self-signed and other incorrect TLS certificates. Which makes it less secure. If you are using self-signed certificates, your new users will have to enable this option manually (once). Or you can add this option to a preconfigured file, if you distribute it to new users. Old users won't be affected by this (those upgrading straight from 2.7 version will still be affected). We encourage server admins to use certificates from trusted authorities (e.g. Let's Encrypt). This version also includes 4.1.9 update for Smack, which has a fix for a critical security vulnerability.


UPDATE (2016/11/29): it looks like Spark is not saving settings on Advanced menu, if a user has never opened it. For such users "Accept all certificates" setting will be disabled after updating to 2.8.2 from any version. They will have to go to Advanced menu and enable it, if they encounter "Unable to verify a certificate" error.


Those who are using Client Control plugin will have to update it to version 2.1.0 as Spark 2.8.2 is already modified to work with that version and it will miss many menus and checkboxes if being used with an older version of that plugin. 2.1.0 version requires Openfire 4.0.0 at least. If you are using older version of Openfire and want to keep using Client Control, you might want to hold of updating to Spark 2.8.2.


We also would like to mention SparkMeet plugin provided by our long time contributor Dele Olajide. Hopefully we will find a way to bundle this plugin with Spark or provide other options to use it. You can also vote here and tell what you think about it. Initial commit is done and its in Spark's source code sparkmeet plugin - initial release by deleolajide · Pull Request #253 · igniterealtime/Spark · GitHub.


For a complete list of changes please check Spark Changelog


As usually we encourage new developers to join Spark project and provide patches. Those familiar with Smack can join the development easier, as we are now using the latest stable version (4.1.9). Patches can be attached in the forums or submitted as PRs on GitHub.


Here are the contributors to this release (besides myself):

freeseawind · GitHub  contributed new skin for Spark - LittleLuck

Alexander198961 (Aleksander Kovtunenko) · GitHub  fixed the issue with Spark freezing on exit because of a faulty plugin

Michael  added more settings to and Client Control plugin, fixed showing of an empty Unfiled group and a few other issues

Martini26  updated Polish translation

speedy  fixed an issue with systray icon switching to regular one when it should be DND or Away and a few issues with invisible mode

nicoben (Nico Ben) · GitHub  updated Italian translation


Here are sha1 checksums for the downloads:

7a45319a723368b6c830a1c1a7287fb207140b4f spark_2_8_2.dmg

dd9c533868d42fc0ef651d544f463ff1fba5d21c  spark_2_8_2.exe

fb324011e7f1a61b5e793b2c6b1778306bf19102  spark_2_8_2_online.exe

20dbc5940a450ab3176f1dcf4d4dfd01f90b4b06  spark_2_8_2.tar.gz

813e110f6715f7f72af28e2ee90ed2bd9fa11005  spark-2.8.2.rpm

1409189a0877a203ef3fe16410a520056fe214d7  spark-2.8.2.src.rpm

565ff2947b4cca69f73f6818f18d49571db6bf78  spark_2.8.2.deb

Peter Saint-Andre: I'm Just a Thing on the Internet

Planet Jabber - 25 November, 2016 - 00:00
The other day I finished reading Nicholas Carr's book The Glass Cage: Automation and Us. Carr's argument about automation, and technology more generally, is that it is not an unmitigated good. If technology is not made to serve human ends, then humans will end up serving technology. We can see this clearly with social networking services. As the saying goes, if you're not paying then you're the product. Yet it goes much deeper than that. Tools such as the ubiquitous "like" button give you the pleasant feeling that you are interacting with your 500 friends on Facebook or Twitter or Instagram, but that feeling is illusory. Aristotle observed over 2300 years ago that true friendship requires spending time together; yet when clicking buttons on a website you are merely engaged in the simulacra of friendship, not the real thing. (I gather that Jacob Silverman makes this point quite forcefully in his book Terms of Service, which I have not yet read.)...

Ignite Realtime Blog: Smack Security Advisory 2016-11-22

Planet Jabber - 22 November, 2016 - 07:23

A critical security vulnerability has been found in Smack. Please upgrade immediately to Smack 4.1.9. Like all Smack releases with the same major and minor version numbers, 4.1.9 is a drop in replacement for all Smack 4.1 releases. Smack 4.1.9 is available on Maven Central.


The Ignite Realtime community would like to thank Sylvain Sarméjeanne for discovering and reporting the vulnerability to

Fanout Blog: Moving from polling to long-polling

Planet Jabber - 21 November, 2016 - 18:42

If you’ve built a REST API that clients poll for updates, you’ve probably considered adding a realtime push mechanism. Maybe you’ve been putting it off due to the added complexity, or the impact it might have on your API contract. These are valid concerns, but push doesn’t have to be that complicated.

In this article we’ll discuss how to update an API to use long-polling. It assumes:

  1. You have an existing REST API.
  2. You have clients repeatedly polling this API.

Long-polling is not the same as “plain” polling. With long-polling, the server delays the response to the client if there is no new data yet. This enables the server to respond instantly whenever the data does change. Aside from providing actual realtime updates, what’s great about long-polling is that technically it’s still RESTful, requiring hardly any changes to your API contract or client code.

Of course, long-polling may not be as efficient as streaming mechanisms like Server-Sent Events or WebSockets, but it’s inarguably more efficient than plain-polling. Let’s compare:

Mechanism Latency Load Plain-polling As high as the polling interval (e.g. 5 second interval means updates will be up to 5 seconds late) High Long-polling Zero Order of magnitude reduction

Long-polling is a great way to dip your feet in the realtime waters without having to dramatically change your API contract and client code.


Jack Moffitt: Servo Interview on The Changelog

Planet Jabber - 21 November, 2016 - 00:00

The Changelog has just published an episode about Servo. It covers the motivations and goals of the project, some aspects of Servo performance and use of the Rust language, and even has a bit about our wonderful community. If your curious about why Servo exists, how we plan to ship it to real users, or what it was like to use Rust before it was stable, I recommend giving it a listen.

Valerian Saliou: Announcing Providence: A Spam Filter For XMPP

Planet Jabber - 20 November, 2016 - 15:15

Fighting spam on decentralized & federated messaging networks (ie. XMPP) is "essentially war with a multi-headed hydra, when 3 new heads are instantly grown up when you cut off just one." — A. on the XSF mailing lists.

After going through an extensive search on the available ways to fight XMPP spam and reading a lot of emails on the XMPP Standards Foundation mailing lists, I came to a sad conclusion: for now, neither users nor server operators have satisfying ways to alleviate the problem of spam on XMPP. This is huge, considering some users receive tens of spammy messages a day.

As I was once the operator of (a public XMPP server that has grown very popular), I became aware of the problem of spam very early. First, with automated accounts which got created from our once-open In-Band Registration (IBR) and were used to attack our legitimate users, and then from external accounts from the federation (from both legitimate and illegitimate XMPP servers). Spam attacks add a tremendous load to your infrastructure, by overloading user offline message database, and bother your users to a point they'd want to stop using your service and move to another server, or even consider stopping using XMPP at all. Thus, I decided to take action and contribute further to the XMPP community, following my work on Jappix and Giggle.

This post announces the inception of an open-source project that aims to solve the problem of spam on the XMPP network. The following post will go through explaining what is considered as spam in the XMPP world, what are the odds if we do nothing to fix it and how other protocols cope (successfully) with spam. I will then suggest a set of solutions that may allow to prevent spam at the server-level, that can be plugged to existing XMPP servers on the market. Then, I will set a timeline for the project. This post is subject to change, as I collect more data and ideas from the community.

If you want to stay in touch with the Providence community and discuss with other Providence users, join the Providence groupchat on XMPP.

If you have any comment, or you would like to discuss the post in private, or contribute to the Providence project, feel free to chat with me on XMPP, or send me an email.

XMPP SPIM A Brief Description Of XMPP SPIM

In this post, Spam over XMPP will be referred to as SPIM (Spam + IM).

XMPP, as a federated, Extensible Messaging and Presence Protocol, may allow the following kinds of SPIM (vectors of attack):

  • Messaging SPIM: SPIM received from XMPP messages (with types chat, groupchat, or even normal or headline)
  • Presence SPIM: SPIM received via roster + presence subscribtion requests from spammy JIDs (mostly with type subscribe)
  • XMPP Protocol Extension SPIM: SPIM that exploits weaknesses / features of largely implemented XEPs, such as VJUD (vCard directory).

This post and subsequent project will only cover Messaging SPIM and Presence SPIM. Considering that no SPIM on any XMPP Protocol Extension has been reported so far on the XSF mailing lists, and being aware of the complexity of hardening the wide range of available XMPP Protocol Extensions, this will be deferred to later (if it becomes needed in the future).

All XMPP servers participating in the federated network are likely to be subject to SPIM. Not only outgoing SPIM, which can be easily prevented by restricting and controlling registrations (accounts hosted on a server, used to send SPIM to accounts to local/remote users). Indeed, incoming SPIM is much harder to control and alleviate as it may come from anywhere (legitimate and illegitimate servers - illegitimate servers being setup and used for the sole purpose of sending SPIMs), under any form (Message stanzas, Presence stanzas, IQ stanzas), and varying among XMPP users (the langage used, the content it promotes). The problem is very similar to email SPAM.

Example Of SPIMs

To give some emphasis to this post, I collected some SPIM messages, that I anonymized. I maintain a non-exhaustive corpus of them contained in the Providence project repository, for test and training purposes.

You can find one of them there (Russian SPIM):

Приглашаем Вас на RAMP (marketplace), это Огромный русский (Silk Road), посвященный продаже ПАВ c 2012 года. Вас ждет самый огромный выбор исключительно НАТУРАЛЬНЫХ веществ от проверенных продавцов со всей России. Полный каталог магазинов (работает только через TOR браузер): А здесь вы можете почитать отзывы и пообщаться с людьми, найти себе работу, открыть свой магазин и узнать массу полезной информации по веществам. (работает только через TOR браузер): [LINKS OBFUSCATED]

The SPIM above basically tries to promote a Tor-based marketplace, as an alternative to the long-defunct Silk Road. I obfuscated the Onion links that followed the message to avoid promoting it there.

It has been reported on the XSF Mailing Lists that some (if not all) of SPIMs are originating from the Matanga botnet. See discussion thread here.

Some well-known XMPP servers are known to host spammy JIDs, among which:

  • (I got an email from Tigase founder: they are aware of this and working on mitigating outgoing spam)

In order to know which JID to target for SPIM, I suspect that spammers made up a list of JIDs to spam by cross-linking it to lists of known emails. Indeed, I received spam the day after setting up a new XMPP server on a previously unused XMPP address, which has however been used as my main email address for a long time. Another technique they may use is the following: they list all users registered on an XMPP server though the VJUD (Virtual Jabber User Directory) service that some XMPP servers expose (mostly ejabberd).

State Of The Market (2016)

As of 2016, some measures have been taken to provide operators ability to filter spam:

Server modules have been / are being developed:
  • Prosody modules: mod_spam_reporting
  • ejabberd modules: (upcoming Advanced Blocking module)
Some interesting talks have been given: Ideas are being discussed: Things have been improved in the defaults of XMPP servers:

Good practices have been implemented in the way XMPP server are shipped. Default configurations come with disabled registration and disabled public user search (ejabberd & Prosody)

But so far, operators have no "no-brainer" way to start blocking SPIM efficiently. Email server operators have a a decent number of open-source filters to help them block spam, and it's working decently well.

The Future If We Do Nothing

If no elegant solution is proposed / implemented to help block / help reduce the amount of SPIM XMPP users receive, we may soon see the following happening:

  • Users stopping launching their XMPP client: to avoid the burden of filtering spam.
  • Users stopping using XMPP at all: there's too much spam out there.
  • Operators applying unproper emergency solutions, among which: blocking IPs (or even blocks), blocking domains, blocking alphabets (I start seeing operators doing those)
  • Server offline message databases collapsing: under the amount of SPIM they retain (which should be delivered to the users once they come back online)

When I say that an elegant solution needs to be designed, I mean it in a way it will stop operators implementing SPIM blocking practices that break the fundamentals principles of a truly open XMPP network. Solving a problem via dirty configuration tricks is not sustainable on the long run.

SPIM/SPAM Prevention Techniques Community Ideas

Users on the XSF mailing lists have proposed the following solutions. Here are my comments on those ideas:

  • Use client-based anti-spams: not portable, waste of time for client developers as everyone will implement its own code. Some popular clients will remain unprotected and may not want to implement such a system.
  • Automated CAPTCHA responses to unknown message senders: would generate a lot of outgoing traffic to perform checks under large SPIM attacks, definitely not user-friendly.
  • Block IPs: block remote IPs, as a single IP can host multiple XMPP domains (more efficient than blocking domains). Though, it is very easy to switch a server to a new IP today using popular cloud hosting providers (eg: DigitalOcean allows this very easily).
  • Refuse servers with open In-Band Registration (IBR): would break the federation in its current state, and would not be very efficient as you'd need servers to proceed checks against each other by trying to connect to other's IBR services. Plus, a closed IBR doesn't state that the server don't contain spammy accounts. Take for instance a server that was setup for the sole purpose of sending SPIM can register accounts using internal server commands, and yet not expose an open IBR: it would be detected as non-spammy as its IBR appears closed, although it hosts spammy JIDs.
  • Enforcing servers to implement an anti-spam: how do you ensure remote servers implement an anti-spam solution? The only way I see it is to perform a remote check, for which a "yes, I implement an anti-spam system" response can be spoofed by spammy servers.
  • Deny all messages from JIDs not in roster: too strict, as people may want to speak without being subscribed to each other. Plus, it only prevents Message SPIMs, but don't prevent at all Presence Subscription SPIMs.
  • Block alphabets: as most SPIM messages are written in Russian (Cyrillic), some have suggested to block any message containing characters from the Cyrillic alphabet. Note that this is an horrifying solution, as we should not discriminate users by their language. Even if the current SPIM is mostly Russian in Cyrillic, I would definitely not like to see federated servers start blocking Cyrillic as they may discriminate what some of their users may say. As some said on the XSF ML, this is very Nazi, and I couldn't agree more with that.
Proposed Solutions

Given my positive experience of using SPAM filters for other protocols (email), I would suggest:

  1. Host the filter on the server, implement SPIM reporting XEPs on the clients: keep the complexity on the server, show simple SPIM reporting buttons to the user. Possibly, give the user the ability to view messages that have been retained on the server because categorized as SPIM, and provide a way to report them as non-SPIM. Clients which don't implement SPIM reporting features still benefit from server-side blocking. Reduces c2s network overhead on clients (critical for XMPP on mobile devices).
  2. Give a score to each incoming message from users who are not in roster: bypass the SPIM filters if the message is coming from a JID which is in the user's roster. JIDs don't need to be authenticated as spoofing is not possible on XMPP.
  3. Calculate message score as an aggregate of multiple filters: Bayesian Filter, Karma Filter, Authentication Filter (more on that below).
  4. Run a public DNSBL service, used by federated servers: will short-circuit the filter and block IPs that are being very spammy.

I came up with some more advanced ideas to alleviate SPIM:

  1. Proof-of-work mechanism on XMPP servers: given that sending a message has a large cost (eg. computer resources) relative to receiving a message, the economics would be no more in the favor of spammers, which need to send a massive amount of messages cheaply. See Hashcash as a starting point, which is used to fight email SPAM. As XMPP is extensible, we could slowly roll over such a protocol for Message and Presence stanzas, to end up enforcing it on the network (as the federation did with encryption in 2014).

The latter idea, which is, getting economics at work against spammers, would make Providence obsolete on the long-run. This should be considered as a long-term solution, that still needs approval from the community, as well as a standard (XEP). Then, both clients and servers will need to implement it, until we reach a point where it is wildly deployed. Only then the XSF can flip the switch and enforce the proof-of-work mechanism in the protocol. Finally defeating those pesky spammers.

However, there are also interesting counter-arguments against Hashcash, pointing out that it can be easily defeated by large botnets.

More on what I propose is being described below.

A (Short) Study On How Email Fight SPAM

There are always benefits to reviewing how others do similar things. As XMPP don't have any past with fighting SPIM, and given the similarities between how the XMPP and Email/SMTP network is operated, it's worth giving a shot to how SPAM is being fougth with email:

1) As email addresses can be spoofed and email content can be altered, senders address/message authenticity needs to be verified using SPF and/or DKIM/DMARC.

Though XMPP doesn't have the spoofing problem, DMARC is still interesting for XMPP for its automated abuse reporting capabilities. Indeed, DMARC provides a way to define a target abuse email in DNS, used by federated servers to report incoming emails that failed DMARC validation for the sender server. If we put this in parallel to XMPP, a feature similar to DMARC could be used to automatically report messages that were marked as SPIM, in a distributed way. Using this mechanism securely implies enforcing DNSSEC. This would provide a way for server operators to be aware of outgoing SPIM activity on their server, and take action in to removing spammy accounts. We can already register some server administrator/abuse addresses using XEP-0157.

2) Email Bayesian filters are very efficient at categorizing HAM from SPAM alone, indistinctively of any sender reputation. I receive a lot of email SPAM and I use a Bayesian filter on my laptop (SpamSieve), which is able to filter SPAM with an accuracy of 98%, and generates very few false positives; less than 2-3 legitimate emails go to the Junk folder per month. SpamAssassin implements a Bayesian filter on the server, and is very popular on email servers. However, some Bayesian filters are not so efficient; their accuracy relies on fine-tuning of Bayesian hyperparameters.

The SPAM emails I receive look, in some ways, very similar to the XMPP SPIM I collected in Providence test corpus. Thus, I predict that using a Bayesian filter on XMPP messages from unknown senders (ie. not in roster) would be as efficient. The Bayesian database has to be user-specific, as it stores information on the words the user is used to send/receive in legitimate messages, and the words that have been reported as spam. It uses a probabilistic model that is very simple and very efficient. As everyone don't use the same words or speak about the same topics, a Bayesian filter is very efficient at detecting anomalies in messages that drift from the word dictionary the user is used to employ. If the Bayesian filter fails at predicting the spamicity of a message, we can still rely on the adjustment filters described right below.

3) Email blocking lists are terribly efficient at preventing spammy SMTP servers from delivering emails (ie. servers that only/mostly send SPAM messages). Public DNSBL are maintained by centralized authorities and used by the federated network of SMTP servers.

DNS-based blocking lists can also work very well for XMPP. The technique relies on the DNS infrastructure and is very efficient, but requires central authorities to exist, and be trusted by federated servers as a reliable source of trust. DNSBL should be used as a last-resort, circuit-breaker solution to prevent very spammy hosts that are involved in large SPIM attacks to trigger more elaborate filters, such as a Bayesian filter. Indeed, DNSBL requires minimal resources, while other filters may use more (if not a lot of) CPU / RAM / disk resources to return a score.

4) Email greylisting is good at minimizing the impact of a sudden SPAM attack on an email server, but it may add some latency for some legitimate emails in being accepted by the SMTP server. Though, it gives the idea of a reputation system with temporary throttling of unknown sending servers.

A similar, fine-grained reputation system can be used for XMPP. A reputation filter will help throttle a sender at a level depending on its past reputation. If the server is known to be less likely to be spammy (given an aggregate result of multiple reputation scoring methods), then it is less likely to be throttled when sending many messages. The other way, if it is unknown to the reputation database, its throttle level should be lower. And finally, if it is known to the reputation database as more likely to be spammy, its throttle level should be very low (ie. it cannot send a lot of messages per unit of time).

If you would like to learn more on the topic of email SPAM, I would recommend giving a look at those essays from Paul Graham, notably A Plan for Spam. The latter explains the fundamentals of detecting SPAM with the Bayesian approach.

A Proper Solution: Providence

To alleviate the problem of SPIM on the federated XMPP network - in a proper way - I propose Providence. Providence is a SPIM filter, dedicated to XMPP, that runs on the server. It is a complex aggregate of simple filters, that are known to work well with generalizing.

What Providence stands for: Providence aims at being a lightweight spam/ham classifier for XMPP servers, and will be built in Rust - a robust systems programming language backed by Mozilla. Providence would run as a daemon on the XMPP server and would be reachable either via an UNIX socket or via a TCP socket (thus, it can also be hosted on a dedicated server, and connected to the XMPP server through a LAN/VLAN; which is useful for large server deployments). A TCP-based Providence protocol will be built (named Providence Network Protocol; aKa PNP), inspired by the SpamAssassin Network Protocol.

Providence will be implementation-independant: meaning that it won't rely on the specifics of any XMPP server / software on the market. Full-fledged XMPP servers, as well as dedicated components, or even advanced XMPP clients, will be able to consume Providence services in a simple way.

What Providence means: the "providence" word is defined by the Oxford British Dictionary as "timely preparation for future eventualities" and "the protective care of God or of nature as a spiritual power". Though I am not a believer, I actually like the latter definition, as one can picture what the system does: it protects you from bad people.

Where Providence is hosted: the Providence project is available at: Its development should start by the end of the first semester of 2017.

Providence makes things easy and elegant: the Grand Goal is to provide a well-designed, long-term solution to the problem of SPIM on XMPP. I expect to make it easy for XMPP server operators to install it. It should be pre-configured by default and won't require too much maintenance. Therefore, I plan to maintain both DEB, RPM and Pacman packages for popular Linux distributions.

As well, from the perspective of an operator, it should be easily pluggable to any popular XMPP server. Providence will ship with mod_providence modules for ejabberd, Prosody and (probably) Openfire. Those modules will implement the PNP and provide some additional filtering logic on the top of it. We can even imagine Providence binary coming as part of XMPP servers and auto-configured, so that when an operator installs, eg. Prosody, he doesn't even have to consider installing the Providence standalone package. Although, running Providence as a standalone package is definitely better for portability, scalability and upgrade purposes.

Filter Features

Below, you can find an exhaustive list of all filters Providence will implement, which make up what I call the Spam/Ham Decisional Pipeline.

Each filter unit returns a score, that is aggregated in a weighed sum. Providence gives more importance to the Bayesian Filter, over Karma Filter and Authentication Filter. Though, the Karma Filter can negatively affect the total score given the sender server has a reputation for spamming users. The weight for each filter has not yet been specified, as I still need to give some thought to it to find the Providence Magic Formula.

  • Bayesian Filter: most accurate filter for spam detection, is tied down to user activity and learns / adapts (the messages the user is used to receive / not used to)
    • Server-wide Bayes Database: local training database, linked to domain/virtualhost (is this really needed / safe?)
    • User-wide Bayes Database: local training database, linked to users JIDs
  • Karma Filter: reputation filter, weaker than Bayesian filter but still useful to improve accuracy
    • JID Karma Database: holds the reputation of a remote (offending/or not) JID - ponderated w/ spam/ham reports + number of messages this user sends to people not in roster (in local server) + if the user has a vcard w/ basic info (especially: avatar)
    • Sender Client Crawler: crawl the sender client and discover its capabilities (given the message origin resource, this can help detect clients that don't reply to CAPS requests and thus may be spam bots)
    • Sender User Rate Limit: apply a per sending user rate-limit threshold for sending parties that are not in recipient user roster
    • Sender User Blacklist: holds status on the number of spam reports for the sending user (using spam reporting XEP on recipient clients)
    • IP Karma Database: holds the reputation of a remote IP (prevents multiple spammy hosts attacks on same IP)
    • Sender IP Rate Limit: apply a per sending IP rate-limit threshold for sending parties that are not in recipient user roster (an IP may hold multiple spamming XMPP hosts)
    • Sender AS Greylist: some Autonomous Systems are less responsive to abuse requests, and those are know to hold more spammers (eg: AS from Russia and USA)
    • Sender IP Blacklist: IPs known to be only related to spam can be safely blocked by a public mechanism similar to DNSBL
  • Authentication Filter: gives a technical quality score to a server (the high complexity of a setup suggests the server is less likely to be spammy)
    • TLS Certificate Hostname Match Checker: does the remote server certificate matches server hostname?
    • TLS Certificate Valid Authority Checker: is the remote server certificate validated by a trusted authority?
    • TLS Certificate Expire Checker: is the remote server certificate still valid? (not expired)
    • SRV Records Validity Checker: advanced setups may have valid SRV records active; a spammer won't bother with this
    • DNSSEC Authentication Checker: advanced setups may have valid DNSSEC records active; a spammer won't bother with this
    • DANE/TLSA Checker: given the server has DNSSEC active, attempt to perform a DANE verification if there is a TLSA record
    • Reverse DNS Checker: proper server setups have a proper reverse DNS that resolves back to the server IP
    • Number Of XMPP Virtual Hosts On IP: approximate how many XMPP hosts/domains are active on the server IP (a spammy server may have more than normal)
    • SMTP Checker: checks if the XMPP server domain holds a valid SMTP MX record (checks if the domain also handles email, and thus more likely to be a long-term setup)
    • Website Checker: checks if the XMPP server has a website for users to land on (hammy XMPP servers are most likely to have a website)
    • In-Band Registration Checker: registrations must not be open to the wild (Web redirect or CAPTCHA-protected)
    • XMPP CAPS Checker: checks XMPP server capabilities (the more services the server provides, the more likely it is to be legit)
  • Granular Filter Cache: retain a cache of each filter score result to speed up further checks (with pre-defined granular TTL values)

Filters are grouped in nested pipelines, as shown on the following (simplified) schema:

This list of filters can also be found in the project's file. It may be subject to change as I refine the theories behind the filters.

Training Features

Providence keeps efficient over time thanks to client training of the filters. Clients that implement some SPIM reporting XEPs will be able to make the Providence filters more efficient at SPIM filtering. I call this the Training Feedback Loop:

  • Bayesian Training: training of Bayesian filter
    • Server-wide Bayes Training: adjust probabilistic rules for spammy/hammy words server-wide (beware of database flood attacks) (is this really needed / safe?)
    • User-wide Bayes Training: adjust probabilistic rules for spammy/hammy words user-wide
  • Karma Filter: training of karma filter
    • JID Karma Training: adjust the reputation of a remote (offending/or not) JID
    • Host Karma Training: adjust the reputation of a remote host (+ sub-hosts)
    • IP Karma Training: adjust the reputation of a remote IP

This list of training features can also be found in the project's file. It may be subject to changes.

User Privacy & Bayesian Databases

As Providence will maintain a Bayesian database for each account on the server (this is needed as each user use a different word dictionnary), user privacy needs to be taken into account very seriously.

Some XMPP users are self-hosted and thus, have total trust in the server they rely on. Some others are using public servers which they also trust, but for which they are less likely to accept seeing words extracted from the corpus of their messages and being collected in a database.

Considering the worst-case scenario of a server hack, a leaked Bayesian database could potentially reveal, among words, private information that has been exchanged at any point in time (eg. passwords). Thus, Providence will need to remove sensitive information from corpuses before storing words in the database. Such a "message cleaning" system cannot be 100% reliable, though.

I still need to think about a proper, elegant way to do it. I can picture a system where the XMPP server holds an user-specific Providence database key that's encrypted with the user XMPP account's password, and transmitted to Providence for each operation related to the user, in such a way the server operator cannot see it, thus decrypt any of Providence user databases. Though, I can also picture weaknesses to such a system.

Avoiding False Positives

Well-trained Bayesian filters can still "leak" a few legitimate messages and categorize them as SPIM.

Thus, we need a way for the user to be able to fetch all undelivered SPIM messages at any point in time, similarly to how email works with Junk / Spam folders. We still need a XEP to formalize it. Servers need to implement a Junk message box. Clients also need to implement such a XEP to provide the user a way to "list, delete or train Messages as non-SPIM", for those that have been categorized as SPIM.

Similarly, Presence Subscription requests that have not been delivered to the user may still be found in the Junk folder, and the user should be able to train the filters and mark a Presence Subscription stanza as legitimate. This would trigger a feedback on Providence Bayesian and/or Reputation databases.

Interoperability With Existing XEPs

Some already-existing XEPs may be amended or implemented as part of a working Providence implementation:

Though, a proper implementation of Providence on the XMPP server may involve new XEPs for features such as the retrieval and management of Messages & Presences classified as SPIM (the Junk box).

A Schedule For Initial Tests

Providence should not be used in "full mode" when the project is still in Alpha phase, as it may still prevent too many legitimate messages from being delivered to users if the filters are not well-tuned, and the Providence Magic Formula isn't yet perfect.

Thus, I propose a plan for Alpha tests, allowing the community to report any weakness in the filters, and benchmark how Providence does in real-world situations:

  • Build test fixtures of SPIM corpus: Providence will build a database of plaintext SPIM corpus, used to train the test filters and improve our development process. You are more than welcome to submit all the SPIM messages you receive (more on that below). This will help train the Providence Magic Formula and perform automated unit and acceptance tests on the filters.
  • Call for community tests: Providence will provide a simple framework for initial testing in "passive mode" on real / production XMPP servers, where the server operator get logs of all Providence decisions, containing the corpus of each message. A Providence toolkit will provide a way to report wrong decisions to the Providence project. This can be useful on small servers where it is okay to log all non-OTR user messages. Regarding larger public servers, I need to think of a way end users can enroll in the Providence test program and get reports of blocked messages directly to their JID via chat message. Then, the user can reply whether the decision is valid or invalid, without the server operator seeing the corpus of any message.
What Server Operators Should Do Now [Or Not Do] (Before Providence Is Available)

If you are an XMPP server operator, there are some actions you should take now to ensure your server is at least safe from Outgoing SPIM:

  1. Disable In-Band Registration (IBR): open servers are being targeted by spammers to register user accounts and use them as a relay for their spam. If you host a public server, you'd better enable a CAPTCHA or allowing users to register from a Web form only (the latter is not very XMPP standard though).
  2. Do not block servers: avoid blocking servers suspected to be spammy, as some may also host legitimate accounts. On the long run, operators will forget about those configuration rules being still active and this will hurt the federated network.
  3. Do not blacklist words: avoid blocking words that you know are spammy, as they may also be used by some of your users (and anyway, this is not a proper way to deal with the issue). Also, avoid blocking alphabets (eg. Cyrillic alphabet).

Also, ensure of the following to avoid Incoming SPIM (ie. SPIM targeting your users):

  1. Disable VJUD services and public directory services: they may help a spammer collect a list of your user's JIDs.
  2. Rate limit (suspect) incoming messages: if the Message / Presence sender JID is not in user's roster and is sending stanzas to different users in a short period of time, you may rate limit the sending server.

You can also help build the Providence test database by submitting the anonymized XML stanzas of SPIM messages your users receive. Ensure to remove the to attribute on stanzas, but keep the from (ie. the sender JID). You can either email-me an archive of the XML files, or fork the Providence repository on GitHub, add them to the fixtures folder and submit a Pull Request (PR). I will be happy to accept it!

I will keep the community posted on any substantial progress on Providence. I will come up with a technical paper on the Providence Magic Formula as well as an early Alpha version by the end of the 1st semester of 2017.

Peter Saint-Andre: Living Deliberately

Planet Jabber - 18 November, 2016 - 00:00
The recent statement by Elon Musk that skeptics of autonomous vehicles are effectively killing people has set me to thinking, especially in conjunction with my current reading of The Glass Cage by Nicholas Carr. Given that 40,000 or more Americans die each year in car accidents, Musk certainly has a point. Yet when I extrapolate this line of argument forward to a wide range of human activities, I start to wonder where our freedom and humanity will go. Eventually we might hear similar arguments against human doctors and nurses, human teachers, even human parents. We humans are messy - we get tired, we make mistakes, we go astray, we are imperfect. Yet we can be ethical, we can be caring, we can learn from our mistakes, we can inspire each other to live up to our ideals. To what extent will we lose what makes us human if we cede all activities to machines? Will we become mere pets, doted over (we hope!) but never taking initiative, never doing anything dangerous, never doing anything interesting? Last year I heard Steve Wozniak argue that becoming pets of the machines would be an ideal life; I strongly disagree....

Erlang Solutions: MongooseIM platform 2.0.0: higher value and lower friction

Planet Jabber - 16 November, 2016 - 11:10

If you haven’t heard of MongooseIM before, it is a mobile messaging platform that provides massive scalability and allows mobile and web app builders to add chat features to their new or existing apps.

In the past months we worked hard on improving MongooseIM for all, and we managed to significantly lower the pain of integration by making it easier and faster; and lowering the barrier of entry for developers. The new features allow the addition of innovative real-time, social features to existing apps. Globally, the MongooseIM platform enables higher acquisition, retention, and engagement, by leveraging high-density, real-time network effects.

In this post we will walk you through all the changes that differentiate the MongooseIM platform on the market. You can also join our webinar on 24 November at 16:00, where myself and my colleague Ludwik Bukovski will talk about the most important features and answer your questions. Enjoy the read and let others read it too, so share and comment below!

Executive summary


The purpose of this article is to update MongooseIM enthusiasts who are already familiar with the previous versions of MongooseIM.

MongooseIM has made a huge jump forward with version 2.0.0 and its platform. Here are the major changes:

  • Pivot from standalone server to a platform: you now benefit from the consistency of a platform, as we provide validated and consistent components, both on the client-side and the server-side so that you can integrate faster
  • A REST API, both on the client-side and the server-side, for a more natural and logical integration with the modern developer culture
  • A modern and simple group chat called “MUC light” (Multi-User Chat light), highly useful from fast prototyping to massively scalable production systems
  • PubSub, or Publish-and-Subscribe, for social innovation, in order to increase the value of your network, thus accelerating your growth
  • Android and iOS libraries allow you to move at light-speed and deliver incredibly fast iterations with full confidence
  • WombatOAM plugins, for high transparency of running systems, both for optimisation purposes and metrics for business visibility
  • Useful documentation improvements that will help you plan and anticipate better
  • Larger range of tests for better trust in every single code change
  From server to platform, for consistency


This change is important for both those in tech roles (developer, devops, sysadmin) and those in executive positions (CTO, COO, founder, owner).

Integrating a discrete, standalone open source server is often a difficulty: although client/server interoperability is guaranteed by the strict use of open standards, the available third party software is not always consistent with the server in terms of features: some may lack in one side or the other.

The consequences are that you still have to code for yourself, and potentially contribute the missing pieces. This slows you down and is quite costly.

We now offer feature parity with the server. Features that have been designed, tested, and validated in real life together with the client and the server. Either we used our own backend software components, or we contributed to third party open source software components.. MongooseIM platform is made of:

  • MongooseIM server
  • WombatOAM system
  • XMPPFramework and Jayme for iOS
  • Smack and Retrofit for Android
  • Escalus, an XMPP client written in Erlang, for functional testing

For all components in the platform, we consistently support features: the same as those available on the server. It allows you to integrate much faster, with significantly lower costs, and bring your products to market much faster.

REST API: lowering the barrier of entry


Maybe the most important of the new features on the platform is the REST API on two sides of the server: server-server for backend integration, and client/server for client development.

Frontend, client-side: light speed development

This improvement will make a lot of client developers happy, regardless if they are mobile or web developers.

The problem lies in the fact that XMPP and XML are losing some love these days, as most developers use REST APIs with JSON everywhere, and it is difficult to learn XMPP and async programming.

Consequently, most developers are reluctant to use XMPP/XML, thus it becomes difficult to find good developers and manage the hiring process and the turnover.

Our solution is a very simple REST API, for all the most basic use cases. We will slowly extend it overtime, according to your needs.

With this REST API, we are lowering down the barrier of entry for developers, because it is easier and more natural to code and test. Your direct benefit is that hiring becomes much easier.

Note: We are definitely keeping XMPP at the core of MongooseIM, and are fully committed to the XMPP community. We are simply offering a new, parallel REST API, that starts simply and will improve overtime.

Backend, server-side: natural and logical technology for integration

A REST API also exists for backend developers and sysadmin, CTOs and architects.

Until today, you could only use the hook system and the command line to integrate MongooseIM.

The consequence was minimalistic integration, and thus MongooseIM remained almost standalone in an infrastructure.

Here comes the server-side REST API.

The benefits are obvious: easy and automatable integration and testing, as it is a natural/logical choice in the backend side of an infrastructure.

MUC light: mobile to the core, massive simplicity, monster scalability


Client-side developers, and UX designers will finally be thrilled by what this feature brings for their users.

The XMPP standard for group chat is MUC, which was designed in the 90s is fully based on presence, and has lots and lots of features.

The outcome is that, as a user, when you lose your network connection (switch to/from wifi, tunnel, uncovered zone), you then lose any link to the MUC group chats you are present in. You lose all messages that get exchanged meanwhile and you have to join again when your client recovers the network. The great number of features is also an overall complaint, being too complex for users and developers.

Our solution is MUC light, a presence-less, and much simpler group chat. You still are in your group chats when you lose your network connection. And you get all your messages, thanks to archiving. It has a very simple and basic set of use cases.

Your developers, UX designers, and ultimately your users are now able to use a modern group chat, independent of the connectivity. MUC light is far simpler for users and developers, and has greater robustness and fault-tolerance, and lower bandwidth consumption on the server-side.

PubSub: real-time non-chat applications


Another huge feature will make UX designers and client-developers love MongooseIM.

The issue with a chat system is that you can design real-time features, but all are based on chat philosophy. All real-time apps that are not chat, like social features, lack some concrete specificities.

The consequence is that everything looks like chat user experience.

The solution is the publish-and-subscribe design pattern, as Wikipedia puts it nicely. “[...] senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead characterize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.“ It is a very flexible and quite generic design approach.

This allows significant space for innovation, in areas such as real-time non-chat apps, and social features, helping you to deepen your user network and thus achieve massive value increase, for higher acquisition, retention, and engagement.

Android and iOS: consistency, trust, lower costs


Here, we have targeted the mobile developers, mainly iOS and Android.

Some third-party open source libraries had features that were not present in MongooseIM, and the other way around.

The direct effects were that there was a high need for more coding and testing, and consequently a general slow down in your project.

So we contributed to the libraries the same features that were present in the server. Of course, testing passed, as we designed the platform with consistency by default.

It is now much faster and easier to integrate the MongooseIM platform in your apps, providing a lower time-to-market, and far lower costs of development and maintenance.


On iOS, we use and contribute to:


On iOS, we use and contribute to:

  WombatOAM plugins


WombatOAM is an awesome tool for sysadmins and erlang developers.

Sometimes it is difficult to see and understand what’s happening in a live system written in Erlang/Elixir (as MongooseIM is).

Developers and sysadmins fall short of capacity to analyse and fix.

WombatOAM helps you with full visibility on metrics and over your live system.

You then have a very high capacity to maintain and optimise it.



Documentation is an improvement for all.

We had a few issues: lack of clarity or visibility, sometimes people didn’t even know some features and concepts existed.

Thus some people failed to see the value, and the benefits from our awesome features and capabilities.

So we added some precisions to the existing docs, and added some new pages.

Now you can have a far clearer vision, and higher know-how on the MongooseIM platform.



Tests are much needed by all developers, and also sysadmins and architects.

The problem is always the same, with functional tests coverage, and their repetitions.

A lack of tests generates a lot of uncertainty and doubt among product teams and their users, which in turn ends up consuming a lot of time.

Our solutions are automated tests and continuous deployments.

We inject a fair amount of determinism, as we catch our issues very early on. We consequently generate trust with reliable code, and some guarantee on non-regressions of all kinds.

Functional tests, continuous integration and deployment

Our platform works out of the box, and the functional test results are available publicly.

Continuous load testing

As we have to fulfill our commitment to advanced scalability, we are working on running continuous load testing.



MongooseIM 2.0.0 brings a massive change, and thus a small revolution in the world of messaging platforms!

We have greatly increased MongooseIM’s value, as we brought consistency thanks to the platform approach and design, and our contributions to the ecosystem. We also brought straightforward integration in infrastructures, which makes development on the platform way faster and easier. You benefit from higher trust in our codebase, with our continuous efforts on our testing strategy. Also now you have a clearer vision with our documentation and our roadmap.

We have also highly reduced the main frictions by lowering costs, lowering time to market, lower the barrier of entry, and lowering maintenance.

MongooseIM platform 2.0.0 is enabling you to differentiate with our technical advances and optimisations, and is opening a large range of potential innovations.

Call to action!


We encourage you to show some simple support and spread the word. But only if you agree and if you do so voluntarily. Please feel free to:

  • Star our repository on GitHub
  • Follow our account on Twitter
  • Star our Docker image on DockerHub

Visit the MongooseIM platform product page, and subscribe to mailing list!

This massive acceleration of MongooseIM platform is just a beginning. To be continued...

Fanout Blog: Five years of Fanout

Planet Jabber - 14 November, 2016 - 19:10

A note from our founder, Justin.

Holy cow, it’s been five years. Five years ago today, Fanout was registered as a company. Back then, all we had were some scribbles in a text file. These days we have a production-ready product and many paying customers. I want to give a big thanks to our investors, advisors, mentors, early adopters, friends, and everyone else who helped get us here.

I also want to take this moment to share some of our experiences over the years.


Peter Saint-Andre: Freedom is Messy

Planet Jabber - 11 November, 2016 - 00:00
This week, while America was losing its mind over the elections, I decided to unplug even further from the emotionally and mentally frenzied approach to life that is becoming more and more the norm. Among other things, I stopped using Twitter. More important, I finally dove into Nicholas Carr's book The Shallows, which describes the damaging effects of how we typically interact with computers and the Internet. As a result, I've started reforming my work habits: engaging in more "deep work" (I highly recommend Cal Newport's book by that name), shutting down all apps but the one or two I'm using to complete my current task, refraining from constant checks of email or chatrooms (Twitter isn't even on that list anymore!), and in general focusing on one thing at a time instead of fooling myself about my nonexistent ability to multitask.

ProcessOne: Elixir Paris Meetup #8 Summary

Planet Jabber - 10 November, 2016 - 14:38

We add a great Paris.ex meetup in last july and video of the talks (in French) are now available.

We had three different talks:

Rails to Phoenix: Feedback coming from the rewrite of a SaaS application: Thibaut Barrère, Rails developer, gave us his feedback and tips discovered while rewriting his Rail application in Elixir in an incremental way.

Here is the video of the talk:

Slides De Rails à Phoenix – retour d’expérience sur une réécriture d’application SaaS

Background Processing (Exq, Sidekiq): Bryan Frimin explains how and why to use background processing tools and libraries in Elixir.

Here is the recording of the talk:

Phoenix Presence: the real time component of Phoenix: Mickaël Rémond demoed a basic chat server developed with Phoenix Presence, along with a step-by-step tutorial to build it. In the process, he had to migrate his application from Phoenix 1.0 to Phoenix 1.2 and describes the needed changes on the application code.

Here is the video of the talk:

Slides: Phoenix Presence: Le service temps réel de Phoenix

If you live in Paris and enjoy Elixir, don’t hesitate to watch the previous videos: Paris.ex videos

See you soon in a meetup !

Distribuir contenido