Planet Jabber

Distribuir contenido
Planet Jabber - http://planet.jabber.org/
Actualizado: hace 20 mins 12 segs

Erlang Solutions: Build a complete iOS messaging app using XMPPFramework - Tutorial Part 1

2 December, 2016 - 10:00
YAXT??! Yet another XMPP tutorial?

 

Well, this is going to be another tutorial, but I'm going to try to make it a little bit different. This is an XMPP tutorial from an iOS developer's perspective. I'll try to answer all the questions I had when I started working in this area. This journey is going to go from no XMPP knowldege at all to having a fully functional instant messaging iOS app using this cool protocol. We are going to be using the super awesome (yet overwhelming at the beginning...) XMPPFramework library, and the idea is also to also mix in some iOS concepts that you are going to need for your app.

What's XMPP?

 

From Wikipedia: Extensible Messaging and Presence Protocol (XMPP) is a communications protocol for message-oriented middleware based on XML.

This basically means XMPP is a protocol for exchanging stuff. What kind of stuff? Messages and presences. We all know what messages are, but what about presences? A presence is just a way of sharing a "status", that's it. You can be 'online', 'offline', 'having lunch', or whatever you want. Also there's another important word: Extensible meaning it can grow. It started as an instant messaging protocol and it has grown into multiple fields for example IoT (Internet of Things). And last, but not least: every piece of information we are going to exchange under this protocol is going to be XML. I can heard you complaining but... Come on, it's not that bad!

Why do we need XMPP? Why not just REST?

 

Well what other options do we have? On the one hand, a custom solution means building everything from scratch, that takes time. On the other hand, we have XMPP, a super tested technology broadly used by millions of people every day, so we can say that's an advantage over a custom approach.

Everytime I talk about XMPP, someone asks me 'Why not just REST?'. Well, there is a misconception here. REST is not a protocol, it's just a way of architecting a networked application; it's just a standarized way of doing something (that I love btw). So let's change the question to something that makes more sense: "Why not just build a custom REST chat application?". The first thing that comes to my mind is what I already explained in the previous paragraph, but there is something else. How do I know when someone has sent me a message? For XMPP this is trivial: we have an open connection all the time so, as soon as a message arrives to the server, it will send us the message. We have a full-duplex. On the other hand, the only solution with REST is polling. We will need to ask the server for new messages from time to time to see if there is something new for us. That sucks. So, we will have to add a mechanism that allows us to receive the messages as soon as they are created, like SSE or WebSockets.

There is one more XMPP advantage over a custom REST chat application. REST uses HTTP, an application level protocol that is built on top of a transport level protocol: TCP. So everytime you want to use your REST solution, you will need HTTP, a protocol that is not always available everywhere (maybe you need to embed this in a cheap piece of hardware?). Besides, we have XMPP built on top of TCP that's going to be always available.

What's the basic stuff I need to know to get started?

 

Well, you know a lot already but let's make a list. Lists are always good:

  • XMPP is built on top of TCP. It keeps an open connection all the time.
  • Client/Server architecture. Messages always go through a server.
  • Everything we send and receive is going to be XML and it's called Stanza.
  • We have three different types of stanzas: iq, message and presence.
  • Every individual on the XMPP network is univocally identified by a JID (Jabber ID).
  • All the stanzas are cointained in a Stream. Let's imagine the Stream as a white canvas where you and the server write the stanzas.
  • Stream, iq, message and presence are the core of XMPP. You can find everything perfectly detailed in RFC6120
  • XMPP can be extended to accomplish different stuff. Each extension is called XEP (XMPP Extension Protocol).
  What's a JID?

Jabber ID (JID) is how we univocally identify each individual in XMPP. It is the address to where we are going to send our stanzas.

This is how a JID looks like:

  • localpart: This is your username.
  • domainpart: Server name where the localpart resides.
  • resourcepart: This is optional, and it identifies a particular client for the user. For example: I can be logged in with andres@erlang-solutions.com on my iPhone, on my Android and on my mac at the same time... So all these will be the same localpart + domainpart but different resourcepart

I'm sure you have already noticed how similar the JID looks to a standard email address. This is because you can connect multiple servers together and the messages are rooted to the right user in the right server, just as email works. Pretty cool, right?

Sometimes you will see we have a JID with just the domain part. Why?! Because it's also possible to send stanzas to a service instead of a user. A service? What's a service?! Services are different pieces of an XMPP server that offer you some special functionality, but don't worry about this right now, just remember: you can have JIDs without a localpart.

What's a Stanza?

Stanza is the name of the XML pieces that we are going to be sending and receiving. The defined stanzas are: <message/>, <presence/> and <iq/>.

  <message/>

This is a basic <message/> stanza. Everytime you want to send a message to someone (a JID), you will have to send this stanza:

<message from='andres@erlang-solutions.com/iphone' to='juana@erlang-solutions.com' type='chat'> <body>Hey there!</body> </message>   <iq/>

It stands for Info/Query. It's a query-action mechanism, you send an iq and you will get a response to that query. You can pair the iq-query with the iq-response using the stanza id.

For example, we send an iq to the server to do something (don't pay attention to what we want to do... you just need to know there is an iq stanza and how the mechanism works):

<iq to='erlang-solutions.com' type='get' id='1'> <query xmlns='http://jabber.org/protocol/disco#items'/> </iq>

And we get back another iq with the same id with the result of the previous query:

<iq from='erlang-solutions.com' to='ramabit@erlang-solutions.com/Andress-MacBook-Air' id='1' type='result'> <query xmlns='http://jabber.org/protocol/disco#items'> <item jid='muc.erlang-solutions.com'/> <item jid='muclight.erlang-solutions.com'/> <item jid='pubsub.erlang-solutions.com'/> </query> </iq>   <presence/>

Used to exchange presence information, as you could have imagined. Usually presences are sent from the client to the server and broadcasted by it. The most basic, yet valid presence, to indicate to the server that a user is avaiable is:

<presence/>

After a sucessfull connection, you are not going to receive any <message/> until you make yourself available sending the previous presence.

If you want to make yourself unavailable, you just have to send:

<presence type="unavailable"></presence>

If we want to make the presences more useful, we can send something like this:

<presence> <status>On vacation</status> </presence>   What's a Stream?

Before answering this, let's refresh our mind. What's a Unix socket? From Wikipedia: A socket is a special file used for inter-process communication. These allows communication between two processes. So a socket is a file that can be written by two processes (in the same computer or in different computers in the same network). So the client is going to write to this file and server too.

Ok, but how is a socket related to a Stream? Well, we are going to be connected to a server using a socket, therefore we are going to have a 'shared file' between the client and the server. This shared file is a white canvas where we are going to start writting our XML stanzas. The first thing we are going to write to this file is an opening <stream> tag! And there you go... that's our stream.

Perfect, I understand what a stream is, but I still don't understand how to send a message to the server. Well, the only thing we need to do to send a message is writting a <message/> stanza in our shared file. But what happens when the server wants to send me a message? Simple: it will write the message in the 'shared file'.

Are we ok so far?

 

I'm sure at this point you have questions like:

  • "What?! An active TCP connection open all the time? I'm used to REST! How am I going to do that?!" 

​           Easy, you don't have to care about that any more! That's why we are going to use the library, and it will take care of that.

  • "You said nothing about how to connect to the server!"

           Believe me, you don't have to care about this either. If we start adding all this info, we are going to get crazy. Trust me, I've been there.

  • "What about encrypted messages? We need security! How are we going to handle this?"

          Again, you don't have to care about this at this point. Baby steps!

 

You just need to be able to answer: "What's XMPP?", "How do you send a message?", "How do you change your status in XMPP?", "How do you ask something to the server?", "What's a Stream?". If you can answer all that, you are WAY better than me when I started.

All the concepts we described so far are the core of XMPP. In part 2 you will find out how to get started with XMPPFramework, connecting to the server and authenticating a user!

Peter Saint-Andre: I'm Just a Thing on the Internet

2 December, 2016 - 00:00
Most people have no idea how the web works and don't understand the role they play in the web ecosystem. Partly this is because our mental model of the web dates back to the early days of "Web 1.0", when folks used the web to read physics papers and personal weblogs and such, somewhat as they would visit their local public library and read books and magazines and newspapers sitting on the shelves. My, how times have changed!

Ignite Realtime Blog: Openfire 4.0.4 is Released

1 December, 2016 - 14:53

The Ignite Realtime Community is pleased to announce the 4.0.4 release of Openfire.  Openfire is a cross-platform real-time collaboration server based on the XMPP (Jabber) protocol. This release signifies our ongoing effort to produce a stable 4.0 series release while continuing to work toward a 4.1 release. A changelog of fixes is available and indicates 16 issues resolved.

 

As always, please report issues in the Community Forums and you can download this release from our website.  Here are the sha1sum's for the downloads available.

 

 

Platform
sha1sumFilenameMac

1d95db39719843ec731ddc69ffb622212b85feb9

openfire_4_0_4.dmgWindows3d342c6f45abb9fc6aff990cf25d879ee20470edopenfire_4_0_4.exeBinary (tar.gz)

5b395f2aedfa55e989730ee960a8c6cb7624bd0e

openfire_4_0_4.tar.gzBinary (zip)

dfb2860197e50858728e2532cd8f21d8f61a6658

openfire_4_0_4.zipSource (tar.gz)e947ffdf4746d3d0a7efb7cf25d198c05668ab54openfire_src_4_0_4.tar.gzSource (zip)fbd8e4c143b7c4841fcb231db2c5974d90137b4fopenfire_src_4_0_4RPM Installer

94d5e0f21f7cbef8d5986e29e3f2f2fc3bec7796

openfire-4.0.4-1.i386.rpmDEB Installerb16d334a90c86e7b15e9ec44162ec01ca700434dopenfire_4.0.4_all.deb

 

As a reminder, Openfire's source code and development can be found on Github and an active developer discussion occurs within our open_chat@conference.igniterealtime.org groupchat.  Please consider helping out with Openfire's development!  We can use help in about all areas of development, but especially:

  • experienced packaging developers to improve the install/upgrade experience on various platforms
  • developers interested in transitioning Openfire to Maven
  • testing of bamboo generated builds

 

Thank you for your interest in Openfire!

ProcessOne: ejabberd 16.12 beta1

30 November, 2016 - 15:04

In 2016, we made several major code refactors and improvements on ejabberd. From Elixir support, to test suite and code clean up and modernization.

After 16.09 which brings a lot of improvements, the last version for 2016 includes a big refactor we have been preparing for long time, and a couple of new features that pursue the central effort on API, commands and web integration started early this year.

ejabberd 16.12 includes: a new API permissions framework for commands, a new more reliable BOSH module, major code refactor, more integration and unit tests, improved support for Erlang/OTP R19, compatibility with rebar3 build system and many fixes and optimizations.

New BOSH module

This new BOSH implementation improves BOSH users experience, as it’s more robust and efficient and overall over lower latency.

New API permissions framework

This new framework is one of the big changes in this release. It makes our API very flexible, allowing administrators to fine grain access to some users to a group of API, depending on the path used to call a given command.

While it should make it possible to implement any kind of access control on top of ejabberd management API, it also makes it much easier to configure ejabberd API for simple needs.

ejabberd API operations are organised around the concept of commands. ejabberd modules provide many commands, but the mechanism is generic and any module can provide its own set of commands.

All commands can be exposed through interfaces. Available interfaces are: ejabberdctl command-line tool, ejabberd ReST API and ejabberd XML-RPC API. The XML-RPC API still works but is deprecated in favor of the ReST API. You should migrate to ReST if you are using it.

Finally, ReST API can be accessed through two authentication mechanisms: HTTP Basic Authentication and OAuth 2.

Each command interface can have different restrictions based on how exposed and sensitive the commands are.

Note that the following configuration snippets assume ejabberd API listeners are properly configured, as defined in API listener configuration

By default, when no api_permission option is provided, ejabberd would use the following default permissions:

api_permissions: "console commands": - from: - ejabberd_ctl - who: all - what: "*" "admin access": - who: - admin - oauth: - scope: "ejabberd:admin" - admin - what: - "*" - "!stop" - "!start"

This will grant access to all commands when ejabberdctl is used (this is what “console commands” group does), and additionally each command that is authenticated by user that match acl auth from any source will be able to call all commands except start and stop.

It’s possible to extend this by adding new section like this:

api_permissions: ... "allow statbot to get server stats" - who: - user: "statbot@server.example.com" - what: - connected_users_number - num_active_users

This will allow user statbot to execute commands connected_users_number and num_active_users.

You can even relax the need to authenticate on ReST API, for example, if you only want to restrict the API to localhost usage, for admin tasks:

api_permissions: - who: - ip: "127.0.0.1/8" - what: - "*" - "!stop" - "!start"

As you can see, for simple cases, the API permission management is now very simple. Yet, the system is powerful and can accommodate your most advanced needs.

For more details, see full api permission documentation.

You can now rely on ejabberd hundreds of commands to integrate in a precise and secure way with your back ends.

Major code refactor

Finally, the most important change (in terms of line of code and impact on ejabberd code base) is refactor of XMPP packet handling in the entire code base. This improvement makes code simpler, safer, and smaller. ejabberd now uses a dedicated XMPP library, which helps developers packing/unpacking XMPP packets.

Example is always better than words:

Don’t build entire XML element, but rely on common “templates”

Before:

#xmlel{name = <<"identity">>, attrs = [{<<"category">>, <<"pubsub">>}, {<<"type">>, <<"pep">>}]} IQ#iq{type = error, sub_el = [Error, SubEl]}

After:

#identity{category = <<"pubsub">>, type = <<"pep">>} xmpp:make_iq_result(IQ, Error); Match packets efficiently

Before:

normal_state({route, From, <<"">>, #xmlel{name = <<"message">>, attrs = Attrs, children = Els} = Packet}, StateData) -> case is_user_online(From, StateData) of true -> case fxml:get_attr_s(<<"type">>, Attrs) of <<"groupchat">> -> ...

After:

normal_state({route, From, <<"">>, #message{type = Type, lang = Lang} = Packet}, StateData) -> case is_user_online(From, StateData) of true when Type == groupchat -> ... Don’t use macros to build error response

Before:

jlib:make_error_reply(Packet, ?ERRT_BAD_REQUEST(Lang, Txt));

After:

xmpp:make_error(Packet, xmpp:err_bad_request(Txt, Lang));

The code is safer, as all XMPP processing in the core of ejabberd can be typed-checked through static analyzer like Dialyzer.

Overall, this change paves the way to improvements of ejabberd API and will make it much more pleasant for contributors to write new modules and extensions. Improving the contribution documentation is next on our list and you are welcome to join the effort.

This big refactor impacts the entire code base. Now that all tests are successful, we provide an early beta before the final 16.12 release. 16.12-beta1 is not yet intended for production use. We are waiting for your feedback to check that special cases did not sleep through our careful refactor. Please, test it and send us feedback in your use case, so that we can catch the tricky bugs before the final version is released.

Changes

This is just a summary of the most relevant ones:

API / integration
  • New API permissions framework
Commands
  • Add configurable weight for ejabberd commands
  • add_rosteritem: Support several groups separated by ;
  • create_rooms_file: Fix reading room jids from file
  • delete_old_messages: Fix command for SQL backends
  • send_message: Don’t duplicate the message
Core XMPP
  • New BOSH module
  • Use fxml_gen XML generator
  • Use our new stand-alone XMPP library instead of jlib.erl
  • Don’t let MAM messages go into offline storage
  • Add xdata generator for XMPP data form
  • Get rid of excessive (io)list_to_binary/1 calls
HTTP
  • Add authentication support to mod_http_fileserver
  • ejabberd_http: Handle missing POST data gracefully
  • Use inets instead of lhttpc in http_p1
  • Add http_p1.erl, rest.erl, and oauth2 ReST backend for OAuth2 tokens
MUC
  • Create room on configuration request as per XEP-0045, 10.1.3
  • Ensure that presence_broadcast room option is stored
  • Fix conference disco#items when running multiple virtual hosts
  • Fix Result Set Management (RSM) for conference disco#items
  • Introduce muc_invite hook
  • Make the constant MAX_ROOMS_DISCOITEMS configurable
  • mod_carboncopy: Don’t copy MUC private messages
MUC/Sub
  • Store the flag “Allow Subscription” room option in database
  • When getting list of subscribed rooms, also check temporary ones
Relational databases support
  • Append ; to privacy_list_data exporting lines
  • Improve relational database import
Build
  • Make build system compatible with rebar3
  • Produce ejabberd.service and fix for systemd usage
Miscelanea
  • Bugfix: Don’t let MAM messages go into offline storage
  • Delete obsolete module mod_configure2
  • Bugfix: Ignore offline sessions in ejabberd command statistics
  • Rename #error{} record to #stanza_error{}
Feedback

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

Ignite Realtime Blog: Spark 2.8.2 Released

25 November, 2016 - 19:00

The Ignite Realtime community has just released Spark 2.8.2 and it can be downloaded from Ignite Realtime: Downloads .

 

This is a regular update. But there is a change, which can affect new users. "Accept all certificates" option on the Adanced menu on the Login screen won't be enabled by default for new users from now on. This option makes Spark automatically accept expired, self-signed and other incorrect TLS certificates. Which makes it less secure. If you are using self-signed certificates, your new users will have to enable this option manually (once). Or you can add this option to a preconfigured spark.properties file, if you distribute it to new users. Old users won't be affected by this (those upgrading straight from 2.7 version will still be affected). We encourage server admins to use certificates from trusted authorities (e.g. Let's Encrypt). This version also includes 4.1.9 update for Smack, which has a fix for a critical security vulnerability.

 

UPDATE (2016/11/29): it looks like Spark is not saving settings on Advanced menu, if a user has never opened it. For such users "Accept all certificates" setting will be disabled after updating to 2.8.2 from any version. They will have to go to Advanced menu and enable it, if they encounter "Unable to verify a certificate" error.

 

Those who are using Client Control plugin will have to update it to version 2.1.0 as Spark 2.8.2 is already modified to work with that version and it will miss many menus and checkboxes if being used with an older version of that plugin. 2.1.0 version requires Openfire 4.0.0 at least. If you are using older version of Openfire and want to keep using Client Control, you might want to hold of updating to Spark 2.8.2.

 

We also would like to mention SparkMeet plugin provided by our long time contributor Dele Olajide. Hopefully we will find a way to bundle this plugin with Spark or provide other options to use it. You can also vote here and tell what you think about it. Initial commit is done and its in Spark's source code sparkmeet plugin - initial release by deleolajide · Pull Request #253 · igniterealtime/Spark · GitHub.

 

For a complete list of changes please check Spark Changelog

 

As usually we encourage new developers to join Spark project and provide patches. Those familiar with Smack can join the development easier, as we are now using the latest stable version (4.1.9). Patches can be attached in the forums or submitted as PRs on GitHub.

 

Here are the contributors to this release (besides myself):

freeseawind · GitHub  contributed new skin for Spark - LittleLuck

Alexander198961 (Aleksander Kovtunenko) · GitHub  fixed the issue with Spark freezing on exit because of a faulty plugin

Michael  added more settings to default.properties and Client Control plugin, fixed showing of an empty Unfiled group and a few other issues

Martini26  updated Polish translation

speedy  fixed an issue with systray icon switching to regular one when it should be DND or Away and a few issues with invisible mode

nicoben (Nico Ben) · GitHub  updated Italian translation

 

Here are sha1 checksums for the downloads:

7a45319a723368b6c830a1c1a7287fb207140b4f spark_2_8_2.dmg

dd9c533868d42fc0ef651d544f463ff1fba5d21c  spark_2_8_2.exe

fb324011e7f1a61b5e793b2c6b1778306bf19102  spark_2_8_2_online.exe

20dbc5940a450ab3176f1dcf4d4dfd01f90b4b06  spark_2_8_2.tar.gz

813e110f6715f7f72af28e2ee90ed2bd9fa11005  spark-2.8.2.rpm

1409189a0877a203ef3fe16410a520056fe214d7  spark-2.8.2.src.rpm

565ff2947b4cca69f73f6818f18d49571db6bf78  spark_2.8.2.deb

Peter Saint-Andre: Living Deliberately

25 November, 2016 - 00:00
The other day I finished reading Nicholas Carr's book The Glass Cage: Automation and Us. Carr's argument about automation, and technology more generally, is that it is not an unmitigated good. If technology is not made to serve human ends, then humans will end up serving technology. We can see this clearly with social networking services. As the saying goes, if you're not paying then you're the product. Yet it goes much deeper than that. Tools such as the ubiquitous "like" button give you the pleasant feeling that you are interacting with your 500 friends on Facebook or Twitter or Instagram, but that feeling is illusory. Aristotle observed over 2300 years ago that true friendship requires spending time together; yet when clicking buttons on a website you are merely engaged in the simulacra of friendship, not the real thing. (I gather that Jacob Silverman makes this point quite forcefully in his book Terms of Service, which I have not yet read.)

Ignite Realtime Blog: Smack Security Advisory 2016-11-22

22 November, 2016 - 07:23

A critical security vulnerability has been found in Smack. Please upgrade immediately to Smack 4.1.9. Like all Smack releases with the same major and minor version numbers, 4.1.9 is a drop in replacement for all Smack 4.1 releases. Smack 4.1.9 is available on Maven Central.

 

The Ignite Realtime community would like to thank Sylvain Sarméjeanne for discovering and reporting the vulnerability to security@igniterealtime.org.

Fanout Blog: Moving from polling to long-polling

21 November, 2016 - 18:42

If you’ve built a REST API that clients poll for updates, you’ve probably considered adding a realtime push mechanism. Maybe you’ve been putting it off due to the added complexity, or the impact it might have on your API contract. These are valid concerns, but push doesn’t have to be that complicated.

In this article we’ll discuss how to update an API to use long-polling. It assumes:

  1. You have an existing REST API.
  2. You have clients repeatedly polling this API.

Long-polling is not the same as “plain” polling. With long-polling, the server delays the response to the client if there is no new data yet. This enables the server to respond instantly whenever the data does change. Aside from providing actual realtime updates, what’s great about long-polling is that technically it’s still RESTful, requiring hardly any changes to your API contract or client code.

Of course, long-polling may not be as efficient as streaming mechanisms like Server-Sent Events or WebSockets, but it’s inarguably more efficient than plain-polling. Let’s compare:

Mechanism Latency Load Plain-polling As high as the polling interval (e.g. 5 second interval means updates will be up to 5 seconds late) High Long-polling Zero Order of magnitude reduction


Long-polling is a great way to dip your feet in the realtime waters without having to dramatically change your API contract and client code.

...

Jack Moffitt: Servo Interview on The Changelog

21 November, 2016 - 00:00

The Changelog has just published an episode about Servo. It covers the motivations and goals of the project, some aspects of Servo performance and use of the Rust language, and even has a bit about our wonderful community. If your curious about why Servo exists, how we plan to ship it to real users, or what it was like to use Rust before it was stable, I recommend giving it a listen.

Valerian Saliou: Announcing Providence: A Spam Filter For XMPP

20 November, 2016 - 15:15

Fighting spam on decentralized & federated messaging networks (ie. XMPP) is "essentially war with a multi-headed hydra, when 3 new heads are instantly grown up when you cut off just one." — A. on the XSF mailing lists.

After going through an extensive search on the available ways to fight XMPP spam and reading a lot of emails on the XMPP Standards Foundation mailing lists, I came to a sad conclusion: for now, neither users nor server operators have satisfying ways to alleviate the problem of spam on XMPP. This is huge, considering some users receive tens of spammy messages a day.

As I was once the operator of Jappix.com (a public XMPP server that has grown very popular), I became aware of the problem of spam very early. First, with automated @jappix.com accounts which got created from our once-open In-Band Registration (IBR) and were used to attack our legitimate users, and then from external accounts from the federation (from both legitimate and illegitimate XMPP servers). Spam attacks add a tremendous load to your infrastructure, by overloading user offline message database, and bother your users to a point they'd want to stop using your service and move to another server, or even consider stopping using XMPP at all. Thus, I decided to take action and contribute further to the XMPP community, following my work on Jappix and Giggle.

This post announces the inception of an open-source project that aims to solve the problem of spam on the XMPP network. The following post will go through explaining what is considered as spam in the XMPP world, what are the odds if we do nothing to fix it and how other protocols cope (successfully) with spam. I will then suggest a set of solutions that may allow to prevent spam at the server-level, that can be plugged to existing XMPP servers on the market. Then, I will set a timeline for the project. This post is subject to change, as I collect more data and ideas from the community.

If you have any comment, or you would like to discuss the post, article, as well as contribute to the Providence project, feel free to chat with me on XMPP, or send me an email.

XMPP SPIM A Brief Description Of XMPP SPIM

In this post, Spam over XMPP will be referred to as SPIM (Spam + IM).

XMPP, as a federated, Extensible Messaging and Presence Protocol, may allow the following kinds of SPIM (vectors of attack):

  • Messaging SPIM: SPIM received from XMPP messages (with types chat, groupchat, or even normal or headline)
  • Presence SPIM: SPIM received via roster + presence subscribtion requests from spammy JIDs (mostly with type subscribe)
  • XMPP Protocol Extension SPIM: SPIM that exploits weaknesses / features of largely implemented XEPs, such as VJUD (vCard directory).

This post and subsequent project will only cover Messaging SPIM and Presence SPIM. Considering that no SPIM on any XMPP Protocol Extension has been reported so far on the XSF mailing lists, and being aware of the complexity of hardening the wide range of available XMPP Protocol Extensions, this will be deferred to later (if it becomes needed in the future).

All XMPP servers participating in the federated network are likely to be subject to SPIM. Not only outgoing SPIM, which can be easily prevented by restricting and controlling registrations (accounts hosted on a server, used to send SPIM to accounts to local/remote users). Indeed, incoming SPIM is much harder to control and alleviate as it may come from anywhere (legitimate and illegitimate servers - illegitimate servers being setup and used for the sole purpose of sending SPIMs), under any form (Message stanzas, Presence stanzas, IQ stanzas), and varying among XMPP users (the langage used, the content it promotes). The problem is very similar to email SPAM.

Example Of SPIMs

To give some emphasis to this post, I collected some SPIM messages, that I anonymized. I maintain a non-exhaustive corpus of them contained in the Providence project repository, for test and training purposes.

You can find one of them there (Russian SPIM):

Приглашаем Вас на RAMP (marketplace), это Огромный русский (Silk Road), посвященный продаже ПАВ c 2012 года. Вас ждет самый огромный выбор исключительно НАТУРАЛЬНЫХ веществ от проверенных продавцов со всей России. Полный каталог магазинов (работает только через TOR браузер): А здесь вы можете почитать отзывы и пообщаться с людьми, найти себе работу, открыть свой магазин и узнать массу полезной информации по веществам. (работает только через TOR браузер): [LINKS OBFUSCATED]

The SPIM above basically tries to promote a TOR-based marketplace, as an alternative to the long-defunct Silk Road. I obfuscated the Onion links that followed the message to avoid promoting it there.

It has been reported on the XSF Mailing Lists that some (if not all) of SPIMs are originating from the Matanga botnet. See discussion thread here.

Some well-known XMPP servers are known to host spammy JIDs, among which:

  • im.flosoft.biz
  • tigase.im (I got an email from Tigase founder: they are aware of this and working on mitigating outgoing spam)
  • sure.im
  • jabber.piratenpartei.de
  • pandion.im

In order to know which JID to target for SPIM, I suspect that spammers made up a list of JIDs to spam by cross-linking it to lists of known emails. Indeed, I received spam the day after setting up a new XMPP server on a previously unused XMPP address, which has however been used as my main email address for a long time. Another technique they may use is the following: they list all users registered on an XMPP server though the VJUD (Virtual Jabber User Directory) service that some XMPP servers expose (mostly ejabberd).

State Of The Market (2016)

As of 2016, some measures have been taken to provide operators ability to filter spam:

Server modules have been / are being developed:
  • Prosody modules: mod_spam_reporting
  • ejabberd modules: (upcoming Advanced Blocking module)
Some interesting talks have been given: Ideas are being discussed: Things have been improved in the defaults of XMPP servers:

Good practices have been implemented in the way XMPP server are shipped. Default configurations come with disabled registration and disabled public user search (ejabberd & Prosody)

But so far, operators have no "no-brainer" way to start blocking SPIM efficiently. Email server operators have a a decent number of open-source filters to help them block spam, and it's working decently well.

The Future If We Do Nothing

If no elegant solution is proposed / implemented to help block / help reduce the amount of SPIM XMPP users receive, we may soon see the following happening:

  • Users stopping launching their XMPP client: to avoid the burden of filtering spam.
  • Users stopping using XMPP at all: there's too much spam out there.
  • Operators applying unproper emergency solutions, among which: blocking IPs (or even blocks), blocking domains, blocking alphabets (I start seeing operators doing those)
  • Server offline message databases collapsing: under the amount of SPIM they retain (which should be delivered to the users once they come back online)

When I say that an elegant solution needs to be designed, I mean it in a way it will stop operators implementing SPIM blocking practices that break the fundamentals principles of a truly open XMPP network. Solving a problem via dirty configuration tricks is not sustainable on the long run.

SPIM/SPAM Prevention Techniques Community Ideas

Users on the XSF mailing lists have proposed the following solutions. Here are my comments on those ideas:

  • Use client-based anti-spams: not portable, waste of time for client developers as everyone will implement its own code. Some popular clients will remain unprotected and may not want to implement such a system.
  • Automated CAPTCHA responses to unknown message senders: would generate a lot of outgoing traffic to perform checks under large SPIM attacks, definitely not user-friendly.
  • Block IPs: block remote IPs, as a single IP can host multiple XMPP domains (more efficient than blocking domains). Though, it is very easy to switch a server to a new IP today using popular cloud hosting providers (eg: DigitalOcean allows this very easily).
  • Refuse servers with open In-Band Registration (IBR): would break the federation in its current state, and would not be very efficient as you'd need servers to proceed checks against each other by trying to connect to other's IBR services. Plus, a closed IBR doesn't state that the server don't contain spammy accounts. Take for instance a server that was setup for the sole purpose of sending SPIM can register accounts using internal server commands, and yet not expose an open IBR: it would be detected as non-spammy as its IBR appears closed, although it hosts spammy JIDs.
  • Enforcing servers to implement an anti-spam: how do you ensure remote servers implement an anti-spam solution? The only way I see it is to perform a remote check, for which a "yes, I implement an anti-spam system" response can be spoofed by spammy servers.
  • Deny all messages from JIDs not in roster: too strict, as people may want to speak without being subscribed to each other. Plus, it only prevents Message SPIMs, but don't prevent at all Presence Subscription SPIMs.
  • Block alphabets: as most SPIM messages are written in Russian (Cyrillic), some have suggested to block any message containing characters from the Cyrillic alphabet. Note that this is an horrifying solution, as we should not discriminate users by their language. Even if the current SPIM is mostly Russian in Cyrillic, I would definitely not like to see federated servers start blocking Cyrillic as they may discriminate what some of their users may say. As some said on the XSF ML, this is very Nazi, and I couldn't agree more with that.
Proposed Solutions

Given my positive experience of using SPAM filters for other protocols (email), I would suggest:

  1. Host the filter on the server, implement SPIM reporting XEPs on the clients: keep the complexity on the server, show simple SPIM reporting buttons to the user. Possibly, give the user the ability to view messages that have been retained on the server because categorized as SPIM, and provide a way to report them as non-SPIM. Clients which don't implement SPIM reporting features still benefit from server-side blocking. Reduces c2s network overhead on clients (critical for XMPP on mobile devices).
  2. Give a score to each incoming message from users who are not in roster: bypass the SPIM filters if the message is coming from a JID which is in the user's roster. JIDs don't need to be authenticated as spoofing is not possible on XMPP.
  3. Calculate message score as an aggregate of multiple filters: Bayesian Filter, Karma Filter, Authentication Filter (more on that below).
  4. Run a public DNSBL service, used by federated servers: will short-circuit the filter and block IPs that are being very spammy.

I came up with some more advanced ideas to alleviate SPIM:

  1. Proof-of-work mechanism on XMPP servers: given that sending a message has a large cost (eg. computer resources) relative to receiving a message, the economics would be no more in the favor of spammers, which need to send a massive amount of messages cheaply. See Hashcash as a starting point, which is used to fight email SPAM. As XMPP is extensible, we could slowly roll over such a protocol for Message and Presence stanzas, to end up enforcing it on the network (as the federation did with encryption in 2014).

The latter idea, which is, getting economics at work against spammers, would make Providence obsolete on the long-run. This should be considered as a long-term solution, that still needs approval from the community, as well as a standard (XEP). Then, both clients and servers will need to implement it, until we reach a point where it is wildly deployed. Only then the XSF can flip the switch and enforce the proof-of-work mechanism in the protocol. Finally defeating those pesky spammers.

More on what I propose is being described below.

A (Short) Study On How Email Fight SPAM

There are always benefits to reviewing how others do similar things. As XMPP don't have any past with fighting SPIM, and given the similarities between how the XMPP and Email/SMTP network is operated, it's worth giving a shot to how SPAM is being fougth with email:

1) As email addresses can be spoofed and email content can be altered, senders address/message authenticity needs to be verified using SPF and/or DKIM/DMARC.

Though XMPP doesn't have the spoofing problem, DMARC is still interesting for XMPP for its automated abuse reporting capabilities. Indeed, DMARC provides a way to define a target abuse email in DNS, used by federated servers to report incoming emails that failed DMARC validation for the sender server. If we put this in parallel to XMPP, a feature similar to DMARC could be used to automatically report messages that were marked as SPIM, in a distributed way. Using this mechanism securely implies enforcing DNSSEC. This would provide a way for server operators to be aware of outgoing SPIM activity on their server, and take action in to removing spammy accounts. We can already register some server administrator/abuse addresses using XEP-0157.

2) Email Bayesian filters are very efficient at categorizing HAM from SPAM alone, indistinctively of any sender reputation. I receive a lot of email SPAM and I use a Bayesian filter on my laptop (SpamSieve), which is able to filter SPAM with an accuracy of 98%, and generates very few false positives; less than 2-3 legitimate emails go to the Junk folder per month. SpamAssassin implements a Bayesian filter on the server, and is very popular on email servers. However, some Bayesian filters are not so efficient; their accuracy relies on fine-tuning of Bayesian hyperparameters.

The SPAM emails I receive look, in some ways, very similar to the XMPP SPIM I collected in Providence test corpus. Thus, I predict that using a Bayesian filter on XMPP messages from unknown senders (ie. not in roster) would be as efficient. The Bayesian database has to be user-specific, as it stores information on the words the user is used to send/receive in legitimate messages, and the words that have been reported as spam. It uses a probabilistic model that is very simple and very efficient. As everyone don't use the same words or speak about the same topics, a Bayesian filter is very efficient at detecting anomalies in messages that drift from the word dictionary the user is used to employ. If the Bayesian filter fails at predicting the spamicity of a message, we can still rely on the adjustment filters described right below.

3) Email blocking lists are terribly efficient at preventing spammy SMTP servers from delivering emails (ie. servers that only/mostly send SPAM messages). Public DNSBL are maintained by centralized authorities and used by the federated network of SMTP servers.

DNS-based blocking lists can also work very well for XMPP. The technique relies on the DNS infrastructure and is very efficient, but requires central authorities to exist, and be trusted by federated servers as a reliable source of trust. DNSBL should be used as a last-resort, circuit-breaker solution to prevent very spammy hosts that are involved in large SPIM attacks to trigger more elaborate filters, such as a Bayesian filter. Indeed, DNSBL requires minimal resources, while other filters may use more (if not a lot of) CPU / RAM / disk resources to return a score.

4) Email greylisting is good at minimizing the impact of a sudden SPAM attack on an email server, but it may add some latency for some legitimate emails in being accepted by the SMTP server. Though, it gives the idea of a reputation system with temporary throttling of unknown sending servers.

A similar, fine-grained reputation system can be used for XMPP. A reputation filter will help throttle a sender at a level depending on its past reputation. If the server is known to be less likely to be spammy (given an aggregate result of multiple reputation scoring methods), then it is less likely to be throttled when sending many messages. The other way, if it is unknown to the reputation database, its throttle level should be lower. And finally, if it is known to the reputation database as more likely to be spammy, its throttle level should be very low (ie. it cannot send a lot of messages per unit of time).

If you would like to learn more on the topic of email SPAM, I would recommend giving a look at those essays from Paul Graham, notably A Plan for Spam. The latter explains the fundamentals of detecting SPAM with the Bayesian approach.

A Proper Solution: Providence

To alleviate the problem of SPIM on the federated XMPP network - in a proper way - I propose Providence. Providence is a SPIM filter, dedicated to XMPP, that runs on the server. It is a complex aggregate of simple filters, that are known to work well with generalizing.

What Providence stands for: Providence aims at being a lightweight spam/ham classifier for XMPP servers, and will be built in Rust - a robust systems programming language backed by Mozilla. Providence would run as a daemon on the XMPP server and would be reachable either via an UNIX socket or via a TCP socket (thus, it can also be hosted on a dedicated server, and connected to the XMPP server through a LAN/VLAN; which is useful for large server deployments). A TCP-based Providence protocol will be built (named Providence Network Protocol; aKa PNP), inspired by the SpamAssassin Network Protocol.

Providence will be implementation-independant: meaning that it won't rely on the specifics of any XMPP server / software on the market. Full-fledged XMPP servers, as well as dedicated components, or even advanced XMPP clients, will be able to consume Providence services in a simple way.

What Providence means: the "providence" word is defined by the Oxford British Dictionary as "timely preparation for future eventualities" and "the protective care of God or of nature as a spiritual power". Though I am not a believer, I actually like the latter definition, as one can picture what the system does: it protects you from bad people.

Where Providence is hosted: the Providence project is available at: https://github.com/valeriansaliou/providence. Its development should start by the end of the first semester of 2017.

Providence makes things easy and elegant: the Grand Goal is to provide a well-designed, long-term solution to the problem of SPIM on XMPP. I expect to make it easy for XMPP server operators to install it. It should be pre-configured by default and won't require too much maintenance. Therefore, I plan to maintain both DEB, RPM and Pacman packages for popular Linux distributions.

As well, from the perspective of an operator, it should be easily pluggable to any popular XMPP server. Providence will ship with mod_providence modules for ejabberd, Prosody and (probably) Openfire. Those modules will implement the PNP and provide some additional filtering logic on the top of it. We can even imagine Providence binary coming as part of XMPP servers and auto-configured, so that when an operator installs, eg. Prosody, he doesn't even have to consider installing the Providence standalone package. Although, running Providence as a standalone package is definitely better for portability, scalability and upgrade purposes.

Filter Features

Below, you can find an exhaustive list of all filters Providence will implement, which make up what I call the Spam/Ham Decisional Pipeline.

Each filter unit returns a score, that is aggregated in a weighed sum. Providence gives more importance to the Bayesian Filter, over Karma Filter and Authentication Filter. Though, the Karma Filter can negatively affect the total score given the sender server has a reputation for spamming users. The weight for each filter has not yet been specified, as I still need to give some thought to it to find the Providence Magic Formula.

  • Bayesian Filter: most accurate filter for spam detection, is tied down to user activity and learns / adapts (the messages the user is used to receive / not used to)
    • Server-wide Bayes Database: local training database, linked to domain/virtualhost (is this really needed / safe?)
    • User-wide Bayes Database: local training database, linked to users JIDs
  • Karma Filter: reputation filter, weaker than Bayesian filter but still useful to improve accuracy
    • JID Karma Database: holds the reputation of a remote (offending/or not) JID - ponderated w/ spam/ham reports + number of messages this user sends to people not in roster (in local server) + if the user has a vcard w/ basic info (especially: avatar)
    • Sender Client Crawler: crawl the sender client and discover its capabilities (given the message origin resource, this can help detect clients that don't reply to CAPS requests and thus may be spam bots)
    • Sender User Rate Limit: apply a per sending user rate-limit threshold for sending parties that are not in recipient user roster
    • Sender User Blacklist: holds status on the number of spam reports for the sending user (using spam reporting XEP on recipient clients)
    • IP Karma Database: holds the reputation of a remote IP (prevents multiple spammy hosts attacks on same IP)
    • Sender IP Rate Limit: apply a per sending IP rate-limit threshold for sending parties that are not in recipient user roster (an IP may hold multiple spamming XMPP hosts)
    • Sender AS Greylist: some Autonomous Systems are less responsive to abuse requests, and those are know to hold more spammers (eg: AS from Russia and USA)
    • Sender IP Blacklist: IPs known to be only related to spam can be safely blocked by a public mechanism similar to DNSBL
  • Authentication Filter: gives a technical quality score to a server (the high complexity of a setup suggests the server is less likely to be spammy)
    • TLS Certificate Hostname Match Checker: does the remote server certificate matches server hostname?
    • TLS Certificate Valid Authority Checker: is the remote server certificate validated by a trusted authority?
    • TLS Certificate Expire Checker: is the remote server certificate still valid? (not expired)
    • SRV Records Validity Checker: advanced setups may have valid SRV records active; a spammer won't bother with this
    • DNSSEC Authentication Checker: advanced setups may have valid DNSSEC records active; a spammer won't bother with this
    • DANE/TLSA Checker: given the server has DNSSEC active, attempt to perform a DANE verification if there is a TLSA record
    • Reverse DNS Checker: proper server setups have a proper reverse DNS that resolves back to the server IP
    • Number Of XMPP Virtual Hosts On IP: approximate how many XMPP hosts/domains are active on the server IP (a spammy server may have more than normal)
    • SMTP Checker: checks if the XMPP server domain holds a valid SMTP MX record (checks if the domain also handles email, and thus more likely to be a long-term setup)
    • Website Checker: checks if the XMPP server has a website for users to land on (hammy XMPP servers are most likely to have a website)
    • In-Band Registration Checker: registrations must not be open to the wild (Web redirect or CAPTCHA-protected)
    • XMPP CAPS Checker: checks XMPP server capabilities (the more services the server provides, the more likely it is to be legit)
  • Granular Filter Cache: retain a cache of each filter score result to speed up further checks (with pre-defined granular TTL values)

Filters are grouped in nested pipelines, as shown on the following (simplified) schema:

This list of filters can also be found in the project's README.md file. It may be subject to change as I refine the theories behind the filters.

Training Features

Providence keeps efficient over time thanks to client training of the filters. Clients that implement some SPIM reporting XEPs will be able to make the Providence filters more efficient at SPIM filtering. I call this the Training Feedback Loop:

  • Bayesian Training: training of Bayesian filter
    • Server-wide Bayes Training: adjust probabilistic rules for spammy/hammy words server-wide (beware of database flood attacks) (is this really needed / safe?)
    • User-wide Bayes Training: adjust probabilistic rules for spammy/hammy words user-wide
  • Karma Filter: training of karma filter
    • JID Karma Training: adjust the reputation of a remote (offending/or not) JID
    • Host Karma Training: adjust the reputation of a remote host (+ sub-hosts)
    • IP Karma Training: adjust the reputation of a remote IP

This list of training features can also be found in the project's README.md file. It may be subject to changes.

User Privacy & Bayesian Databases

As Providence will maintain a Bayesian database for each account on the server (this is needed as each user use a different word dictionnary), user privacy needs to be taken into account very seriously.

Some XMPP users are self-hosted and thus, have total trust in the server they rely on. Some others are using public servers which they also trust, but for which they are less likely to accept seeing words extracted from the corpus of their messages and being collected in a database.

Considering the worst-case scenario of a server hack, a leaked Bayesian database could potentially reveal, among words, private information that has been exchanged at any point in time (eg. passwords). Thus, Providence will need to remove sensitive information from corpuses before storing words in the database. Such a "message cleaning" system cannot be 100% reliable, though.

I still need to think about a proper, elegant way to do it. I can picture a system where the XMPP server holds an user-specific Providence database key that's encrypted with the user XMPP account's password, and transmitted to Providence for each operation related to the user, in such a way the server operator cannot see it, thus decrypt any of Providence user databases. Though, I can also picture weaknesses to such a system.

Avoiding False Positives

Well-trained Bayesian filters can still "leak" a few legitimate messages and categorize them as SPIM.

Thus, we need a way for the user to be able to fetch all undelivered SPIM messages at any point in time, similarly to how email works with Junk / Spam folders. We still need a XEP to formalize it. Servers need to implement a Junk message box. Clients also need to implement such a XEP to provide the user a way to "list, delete or train Messages as non-SPIM", for those that have been categorized as SPIM.

Similarly, Presence Subscription requests that have not been delivered to the user may still be found in the Junk folder, and the user should be able to train the filters and mark a Presence Subscription stanza as legitimate. This would trigger a feedback on Providence Bayesian and/or Reputation databases.

Interoperability With Existing XEPs

Some already-existing XEPs may be amended or implemented as part of a working Providence implementation:

Though, a proper implementation of Providence on the XMPP server may involve new XEPs for features such as the retrieval and management of Messages & Presences classified as SPIM (the Junk box).

A Schedule For Initial Tests

Providence should not be used in "full mode" when the project is still in Alpha phase, as it may still prevent too many legitimate messages from being delivered to users if the filters are not well-tuned, and the Providence Magic Formula isn't yet perfect.

Thus, I propose a plan for Alpha tests, allowing the community to report any weakness in the filters, and benchmark how Providence does in real-world situations:

  • Build test fixtures of SPIM corpus: Providence will build a database of plaintext SPIM corpus, used to train the test filters and improve our development process. You are more than welcome to submit all the SPIM messages you receive (more on that below). This will help train the Providence Magic Formula and perform automated unit and acceptance tests on the filters.
  • Call for community tests: Providence will provide a simple framework for initial testing in "passive mode" on real / production XMPP servers, where the server operator get logs of all Providence decisions, containing the corpus of each message. A Providence toolkit will provide a way to report wrong decisions to the Providence project. This can be useful on small servers where it is okay to log all non-OTR user messages. Regarding larger public servers, I need to think of a way end users can enroll in the Providence test program and get reports of blocked messages directly to their JID via chat message. Then, the user can reply whether the decision is valid or invalid, without the server operator seeing the corpus of any message.
What Server Operators Should Do Now [Or Not Do] (Before Providence Is Available)

If you are an XMPP server operator, there are some actions you should take now to ensure your server is at least safe from Outgoing SPIM:

  1. Disable In-Band Registration (IBR): open servers are being targeted by spammers to register user accounts and use them as a relay for their spam. If you host a public server, you'd better enable a CAPTCHA or allowing users to register from a Web form only (the latter is not very XMPP standard though).
  2. Do not block servers: avoid blocking servers suspected to be spammy, as some may also host legitimate accounts. On the long run, operators will forget about those configuration rules being still active and this will hurt the federated network.
  3. Do not blacklist words: avoid blocking words that you know are spammy, as they may also be used by some of your users (and anyway, this is not a proper way to deal with the issue). Also, avoid blocking alphabets (eg. Cyrillic alphabet).

Also, ensure of the following to avoid Incoming SPIM (ie. SPIM targeting your users):

  1. Disable VJUD services and public directory services: they may help a spammer collect a list of your user's JIDs.
  2. Rate limit (suspect) incoming messages: if the Message / Presence sender JID is not in user's roster and is sending stanzas to different users in a short period of time, you may rate limit the sending server.

You can also help build the Providence test database by submitting the anonymized XML stanzas of SPIM messages your users receive. Ensure to remove the to attribute on stanzas, but keep the from (ie. the sender JID). You can either email-me an archive of the XML files, or fork the Providence repository on GitHub, add them to the fixtures folder and submit a Pull Request (PR). I will be happy to accept it!

I will keep the community posted on any substantial progress on Providence. I will come up with a technical paper on the Providence Magic Formula as well as an early Alpha version by the end of the 1st semester of 2017.

Peter Saint-Andre: Freedom is Messy

18 November, 2016 - 00:00
The recent statement by Elon Musk that skeptics of autonomous vehicles are effectively killing people has set me to thinking, especially in conjunction with my current reading of The Glass Cage by Nicholas Carr. Given that 40,000 or more Americans die each year in car accidents, Musk certainly has a point. Yet when I extrapolate this line of argument forward to a wide range of human activities, I start to wonder where our freedom and humanity will go. Eventually we might hear similar arguments against human doctors and nurses, human teachers, even human parents. We humans are messy - we get tired, we make mistakes, we go astray, we are imperfect. Yet we can be ethical, we can be caring, we can learn from our mistakes, we can inspire each other to live up to our ideals. To what extent will we lose what makes us human if we cede all activities to machines? Will we become mere pets, doted over (we hope!) but never taking initiative, never doing anything dangerous, never doing anything interesting? Last year I heard Steve Wozniak argue that becoming pets of the machines would be an ideal life; I strongly disagree.

Erlang Solutions: MongooseIM platform 2.0.0: higher value and lower friction

16 November, 2016 - 11:10

If you haven’t heard of MongooseIM before, it is a mobile messaging platform that provides massive scalability and allows mobile and web app builders to add chat features to their new or existing apps.

In the past months we worked hard on improving MongooseIM for all, and we managed to significantly lower the pain of integration by making it easier and faster; and lowering the barrier of entry for developers. The new features allow the addition of innovative real-time, social features to existing apps. Globally, the MongooseIM platform enables higher acquisition, retention, and engagement, by leveraging high-density, real-time network effects.

In this post we will walk you through all the changes that differentiate the MongooseIM platform on the market. You can also join our webinar on 24 November at 16:00, where myself and my colleague Ludwik Bukovski will talk about the most important features and answer your questions. Enjoy the read and let others read it too, so share and comment below!

Executive summary

 

The purpose of this article is to update MongooseIM enthusiasts who are already familiar with the previous versions of MongooseIM.

MongooseIM has made a huge jump forward with version 2.0.0 and its platform. Here are the major changes:

  • Pivot from standalone server to a platform: you now benefit from the consistency of a platform, as we provide validated and consistent components, both on the client-side and the server-side so that you can integrate faster
  • A REST API, both on the client-side and the server-side, for a more natural and logical integration with the modern developer culture
  • A modern and simple group chat called “MUC light” (Multi-User Chat light), highly useful from fast prototyping to massively scalable production systems
  • PubSub, or Publish-and-Subscribe, for social innovation, in order to increase the value of your network, thus accelerating your growth
  • Android and iOS libraries allow you to move at light-speed and deliver incredibly fast iterations with full confidence
  • WombatOAM plugins, for high transparency of running systems, both for optimisation purposes and metrics for business visibility
  • Useful documentation improvements that will help you plan and anticipate better
  • Larger range of tests for better trust in every single code change
  From server to platform, for consistency

 

This change is important for both those in tech roles (developer, devops, sysadmin) and those in executive positions (CTO, COO, founder, owner).

Integrating a discrete, standalone open source server is often a difficulty: although client/server interoperability is guaranteed by the strict use of open standards, the available third party software is not always consistent with the server in terms of features: some may lack in one side or the other.

The consequences are that you still have to code for yourself, and potentially contribute the missing pieces. This slows you down and is quite costly.

We now offer feature parity with the server. Features that have been designed, tested, and validated in real life together with the client and the server. Either we used our own backend software components, or we contributed to third party open source software components.. MongooseIM platform is made of:

  • MongooseIM server
  • WombatOAM system
  • XMPPFramework and Jayme for iOS
  • Smack and Retrofit for Android
  • Escalus, an XMPP client written in Erlang, for functional testing

For all components in the platform, we consistently support features: the same as those available on the server. It allows you to integrate much faster, with significantly lower costs, and bring your products to market much faster.

REST API: lowering the barrier of entry

 

Maybe the most important of the new features on the platform is the REST API on two sides of the server: server-server for backend integration, and client/server for client development.

Frontend, client-side: light speed development

This improvement will make a lot of client developers happy, regardless if they are mobile or web developers.

The problem lies in the fact that XMPP and XML are losing some love these days, as most developers use REST APIs with JSON everywhere, and it is difficult to learn XMPP and async programming.

Consequently, most developers are reluctant to use XMPP/XML, thus it becomes difficult to find good developers and manage the hiring process and the turnover.

Our solution is a very simple REST API, for all the most basic use cases. We will slowly extend it overtime, according to your needs.

With this REST API, we are lowering down the barrier of entry for developers, because it is easier and more natural to code and test. Your direct benefit is that hiring becomes much easier.

Note: We are definitely keeping XMPP at the core of MongooseIM, and are fully committed to the XMPP community. We are simply offering a new, parallel REST API, that starts simply and will improve overtime.

Backend, server-side: natural and logical technology for integration

A REST API also exists for backend developers and sysadmin, CTOs and architects.

Until today, you could only use the hook system and the command line to integrate MongooseIM.

The consequence was minimalistic integration, and thus MongooseIM remained almost standalone in an infrastructure.

Here comes the server-side REST API.

The benefits are obvious: easy and automatable integration and testing, as it is a natural/logical choice in the backend side of an infrastructure.

MUC light: mobile to the core, massive simplicity, monster scalability

 

Client-side developers, and UX designers will finally be thrilled by what this feature brings for their users.

The XMPP standard for group chat is MUC, which was designed in the 90s is fully based on presence, and has lots and lots of features.

The outcome is that, as a user, when you lose your network connection (switch to/from wifi, tunnel, uncovered zone), you then lose any link to the MUC group chats you are present in. You lose all messages that get exchanged meanwhile and you have to join again when your client recovers the network. The great number of features is also an overall complaint, being too complex for users and developers.

Our solution is MUC light, a presence-less, and much simpler group chat. You still are in your group chats when you lose your network connection. And you get all your messages, thanks to archiving. It has a very simple and basic set of use cases.

Your developers, UX designers, and ultimately your users are now able to use a modern group chat, independent of the connectivity. MUC light is far simpler for users and developers, and has greater robustness and fault-tolerance, and lower bandwidth consumption on the server-side.

PubSub: real-time non-chat applications

 

Another huge feature will make UX designers and client-developers love MongooseIM.

The issue with a chat system is that you can design real-time features, but all are based on chat philosophy. All real-time apps that are not chat, like social features, lack some concrete specificities.

The consequence is that everything looks like chat user experience.

The solution is the publish-and-subscribe design pattern, as Wikipedia puts it nicely. “[...] senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead characterize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.“ It is a very flexible and quite generic design approach.

This allows significant space for innovation, in areas such as real-time non-chat apps, and social features, helping you to deepen your user network and thus achieve massive value increase, for higher acquisition, retention, and engagement.

Android and iOS: consistency, trust, lower costs

 

Here, we have targeted the mobile developers, mainly iOS and Android.

Some third-party open source libraries had features that were not present in MongooseIM, and the other way around.

The direct effects were that there was a high need for more coding and testing, and consequently a general slow down in your project.

So we contributed to the libraries the same features that were present in the server. Of course, testing passed, as we designed the platform with consistency by default.

It is now much faster and easier to integrate the MongooseIM platform in your apps, providing a lower time-to-market, and far lower costs of development and maintenance.

iOS

On iOS, we use and contribute to:

  Android

On iOS, we use and contribute to:

  WombatOAM plugins

 

WombatOAM is an awesome tool for sysadmins and erlang developers.

Sometimes it is difficult to see and understand what’s happening in a live system written in Erlang/Elixir (as MongooseIM is).

Developers and sysadmins fall short of capacity to analyse and fix.

WombatOAM helps you with full visibility on metrics and over your live system.

You then have a very high capacity to maintain and optimise it.

Documentation

 

Documentation is an improvement for all.

We had a few issues: lack of clarity or visibility, sometimes people didn’t even know some features and concepts existed.

Thus some people failed to see the value, and the benefits from our awesome features and capabilities.

So we added some precisions to the existing docs, and added some new pages.

Now you can have a far clearer vision, and higher know-how on the MongooseIM platform.

  Tests

 

Tests are much needed by all developers, and also sysadmins and architects.

The problem is always the same, with functional tests coverage, and their repetitions.

A lack of tests generates a lot of uncertainty and doubt among product teams and their users, which in turn ends up consuming a lot of time.

Our solutions are automated tests and continuous deployments.

We inject a fair amount of determinism, as we catch our issues very early on. We consequently generate trust with reliable code, and some guarantee on non-regressions of all kinds.

Functional tests, continuous integration and deployment

Our platform works out of the box, and the functional test results are available publicly.

Continuous load testing

As we have to fulfill our commitment to advanced scalability, we are working on running continuous load testing.

Summary

 

MongooseIM 2.0.0 brings a massive change, and thus a small revolution in the world of messaging platforms!

We have greatly increased MongooseIM’s value, as we brought consistency thanks to the platform approach and design, and our contributions to the ecosystem. We also brought straightforward integration in infrastructures, which makes development on the platform way faster and easier. You benefit from higher trust in our codebase, with our continuous efforts on our testing strategy. Also now you have a clearer vision with our documentation and our roadmap.

We have also highly reduced the main frictions by lowering costs, lowering time to market, lower the barrier of entry, and lowering maintenance.

MongooseIM platform 2.0.0 is enabling you to differentiate with our technical advances and optimisations, and is opening a large range of potential innovations.

Call to action!

 

We encourage you to show some simple support and spread the word. But only if you agree and if you do so voluntarily. Please feel free to:

  • Star our repository on GitHub
  • Follow our account on Twitter
  • Star our Docker image on DockerHub

Visit the MongooseIM platform product page, and subscribe to mailing list!

This massive acceleration of MongooseIM platform is just a beginning. To be continued...

Fanout Blog: Five years of Fanout

14 November, 2016 - 19:10

A note from our founder, Justin.

Holy cow, it’s been five years. Five years ago today, Fanout was registered as a company. Back then, all we had were some scribbles in a text file. These days we have a production-ready product and many paying customers. I want to give a big thanks to our investors, advisors, mentors, early adopters, friends, and everyone else who helped get us here.

I also want to take this moment to share some of our experiences over the years.

...

Peter Saint-Andre: Below the Surface

11 November, 2016 - 00:00
This week, while America was losing its mind over the elections, I decided to unplug even further from the emotionally and mentally frenzied approach to life that is becoming more and more the norm. Among other things, I stopped using Twitter. More important, I finally dove into Nicholas Carr's book The Shallows, which describes the damaging effects of how we typically interact with computers and the Internet. As a result, I've started reforming my work habits: engaging in more "deep work" (I highly recommend Cal Newport's book by that name), shutting down all apps but the one or two I'm using to complete my current task, refraining from constant checks of email or chatrooms (Twitter isn't even on that list anymore!), and in general focusing on one thing at a time instead of fooling myself about my nonexistent ability to multitask.

ProcessOne: Elixir Paris Meetup #8 Summary

10 November, 2016 - 14:38

We add a great Paris.ex meetup in last july and video of the talks (in French) are now available.

We had three different talks:

Rails to Phoenix: Feedback coming from the rewrite of a SaaS application: Thibaut Barrère, Rails developer, gave us his feedback and tips discovered while rewriting his Rail application in Elixir in an incremental way.

Here is the video of the talk:

Slides De Rails à Phoenix – retour d’expérience sur une réécriture d’application SaaS

Background Processing (Exq, Sidekiq): Bryan Frimin explains how and why to use background processing tools and libraries in Elixir.

Here is the recording of the talk:

Phoenix Presence: the real time component of Phoenix: Mickaël Rémond demoed a basic chat server developed with Phoenix Presence, along with a step-by-step tutorial to build it. In the process, he had to migrate his application from Phoenix 1.0 to Phoenix 1.2 and describes the needed changes on the application code.

Here is the video of the talk:

Slides: Phoenix Presence: Le service temps réel de Phoenix

If you live in Paris and enjoy Elixir, don’t hesitate to watch the previous videos: Paris.ex videos

See you soon in a meetup !

Monal IM: Don’t upgrade to Monal 2.9.2

9 November, 2016 - 03:06

Xcode 8 and iOS 10 have introduced a change that breaks a deprecated feature  Monal uses to maintain connectivity. Do not update to 2.9.2, the app does not run in the background anymore. I will submit a build soon that will hopefully get the app working again.

Tigase Blog: Tigase on Raspberry Pi

8 November, 2016 - 20:34

Tigase XMPP Server is a Java-based XMPP communications software. Because Java is nearly ubiquitous, and not platform specific, the hardware that Tigase can run on is wide and varied.

JC Brand: Customizing and whitelisting SASL authentication mechanisms in Strophe.js

5 November, 2016 - 12:30
Introduction

If you've decided to read this fairly technical blogpost, then you probably have at least a rough idea what SASL is about and why one would want to create custom SASL auth mechanisms or whitelist the supported mechanisms.

I'll therefore provide just a very brief recap of the topics involved:

The Simple Authentication and Security Layer or SASL RFC 4422 is a framework for adding authentication support to connection-based protocols.

It provides an abstraction layer for authentication mechanisms, so that protocols, such as XMPP don't have to deal with the intricacies and complexities of supporting multiple authentication mechanisms.

It therefore makes auth mechanisms pluggable (if they are SASL compatible).

Strophe.js has supported SASL since a long time, but it didn't provide an easy way add custom SASL mechanisms, or to whitelist the mechanisms to be used.

Until now... or rather, since the 1.2.9 release.

Creating a custom SASL auth mechanism

To create a custom SASL authentication mechanism is fairly simple.

You can glean what's required by simply looking at how the default mechanisms are created.

See for example how the SASLPlain mechanism is defined.

And look as the SASLMechanism prototype to see the interface that the mechanism supports.

Perty much it boils down to creating constructor, settings its prototype to an invoked Strophe.SASLMechanism instance, providing its name, a boolean to indicate whether it should proactively respond without an initial server challenge, and an integer value specifying its priority amongst the supported mechanisms.

The default mechanisms and their respective priorities are:

  • EXTERNAL - 60
  • OAUTHBEARER - 50
  • SCRAM-SHA1 - 40
  • DIGEST-MD5 - 30
  • PLAIN - 20
  • ANONYMOUS - 10

Then it's a matter of implementing onChallenge and any of the other methods provided by the SASLMechanism prototype.

onChallenge is called once the server challenges the client to authenticate itself or proactively if the mechanism requires that the client initiates authentication (configured with the isClientFirst parameter of Strophe.SASLMechanism).

So, lets create a fictional auth mechanism called SASL-FOO which works similarly to SASL-PLAIN, except that the password is encrypted with double-encoding ROT13 (hint: this is a joke).

We would then create the authentication mechanism like so:

Strophe.SASLFoo = function() {}; Strophe.SASLFoo.prototype = new Strophe.SASLMechanism("FOO", true, 60); Strophe.SASLFoo.prototype.onChallenge = function(connection) { var auth_str = connection.authzid; auth_str = auth_str + "\u0000"; auth_str = auth_str + connection.authcid; auth_str = auth_str + "\u0000"; auth_str = auth_str + DoubleROT13(connection.pass); return utils.utf16to8(auth_str); }; Whitelisting the support SASL auth mechanisms

Now with SASL-FOO in hand, we can whitelist the supported authentication mechanisms by specifying a list of mechanisms in the options map passed in when we instantiate a new Strohe.Connection.

var service = 'chat.example.org'; var options = { 'mechanisms': [ SASLFoo, Strophe.SASLPlain ] }; var conn = new Strophe.Connection(service, options); Bonus: Whitelisting SASL auth mechanisms in Converse.js

Due to the above changes it'll also be possible to whitelist SASL mechanisms in Converse.js (version 2.0.1 and upwards).

This is done via the connection_options configuration setting:

converse.initialize({ connection_options: { 'mechanisms': [ converse.env.Strophe.SASLMD5, converse.env.Strophe.SASLPlain ] }, });

Prosodical Thoughts: Prosody 0.9.11 released

3 November, 2016 - 20:26

We are pleased to announce a new minor release from our stable branch.

This release contains a whole bunch of bug fixes, a brief summary of them follows:

  • HTTP parser: Improve buffering of incoming HTTP data and add size limits (#603)
  • sessionmanager: Fix for an issue which caused people to be kicked from conferences if mod_smacks was enabled (#648)
  • Dependencies: Workaround for compatibility with LuaSec 0.6 (#749)
  • MUC: Accept missing form as "instant room" request (#377)
  • C2S: Fix issues with destroying disconnected connections (#590, #641)
  • mod_privacy: Fix selection of the top resource(s) (#694)
  • mod_presence: Make sure both users get each others presence after adding each other (#673)
  • mod_http_files: Fix traceback when serving a non-wildcard path (#611)
  • mod_http_files: Preserve a trailing slash in paths (#639)
  • util.datamanager: Fix error handling (#632)
  • net.server_event: Fix internal socket API to allow writing from socket.ondrain callback (#661)
  • net.server_event: Fix timeout (commit)
  • net.server_event: Fix traceback due to write during TLS handshake (commit)
  • net.server_event: Fix buffer length check (commit)
Download

As usual, download instructions for many platforms can be found on our download page

If you have any questions, comments or other issues with this release, let us know!

yaxim: Registration Repaired

2 November, 2016 - 11:28

In the last three months, it was not possible to register new accounts on yax.im. This issue has been resolved now.

On 2016-07-27, the service was upgraded to the experimental prosody 0.10 branch, to allow support for XEP-0313: Message Archive Management and for its better MUC implementation.

Unfortunately, the upgrade also introduced prosody bug #724, where a logic error causes new user registrations to fail, except for spammer accounts. Because the yax.im admin was not aware of this and failed to see the signs of stagnation as well as angry user reports, it took over three months to find and fix the issue. Sorry!

The good news is: yax.im is accepting new users again! If you are running yaxim and your registration attempt has failed, please uninstall & reinstall the app or clear the app data to perform a new registration.