Planet Jabber

Distribuir contenido
Planet Jabber -
Actualizado: hace 2 horas 6 mins

Jérôme Poisson: Salut à Toi v0.5.1

18 September, 2014 - 22:28

We are happy to announce the release of Salut à Toi v0.5.1! This version focuses on security and code refactorisation to ease the add of new features and maintenance.

We remind you that SàT is a multi-purposes and multi-frontends XMPP client, mainly developed in Python. The more advanced frontends are Primitivus (console) and Libervia (web). Jp (command line) can be used for administrative tasks. Wix (desktop/WxWidgets) will be deprecated and replaced with Bellaciao (desktop/Qt). An Android frontend is also planned.


A new parameter has been added to define a password for the SàT profile, it is stored hashed in the database. Its raw version is used to cipher the other passwords - including the one for the XMPP account - which are encrypted. A scheme on the wiki explains how this all works: encryption.

Libervia now handles HTTPS. The administrator can choose the service(s) to enable: HTTP, HTTPS or both.

You can uses OTR for instant messaging end-to-end encryption. Primitivus console interface uses the Python library potr while Libervia is based on the Javascript implementation otr.js. Thus, your encrypted discussions on Libervia are really private because the encryption is done directly by your browser; you may encounter some slowdowns, especially when starting OTR.

Other additions

You will notice:

  • the add of chat rooms bookmarks;
  • the display of the composing states in chat rooms;
  • a better integration of the ad-hoc commands, e.g. to allow the administration of the server from Primitivus or Libervia;
  • the possibility to erase all your blog posts, change your password or delete your account from Libervia;
  • contextual menus on the roster contacts and the discussion panels;
  • a couple of new stuff concerning the static blog pages.


We care to redesign the code when some conception issues are found, or if we imagine a new mechanism which would work better. It is especially important for a project like SàT which is multi-interfaces and still in development. For this version we made several refactorisations concerning:

  • XMPP service discovery;
  • messages sending and reception;
  • textual commands management;
  • contact list management;
  • hierarchical organization of the constants;
  • Libervia's source files hierachy;
  • Primitivus keyboard shortcuts.

These modifications are not of a big interest for the users, but they ease our life, and maybe those of the people who would like to give us a hand! We see them as required first steps to initiate the development of the mobile phone frontend and the add of new features.

We extended the usage of XMLUI, our internal micro-format for describing user interfaces. We use it to manage frontend's dialogs from the backend. The user actions are now better integrated and we will keep on improving it for the next versions.

To be mentioned

The backend is now launched as a Twisted plugin and starts by default in daemon mode, just as Libervia. The initialisation sequence backend / frontends has been improved; this fixes the issues that could occur when SàT and Libervia were launched from a script within a short time. Moreover, we added a .service file for D-Bus to automatically launch the backend when a frontend needs it.

The default paths for the user files are now compliant to the XDG recommendations: configuration file in ~/.config/sat, database and the rest in ~/.local/share/sat. Any previously existing default configuration file will be retrieved and eventually updated.

If the XMPP server address and port are left empty in the connection parameters, the actual values can be retrieved from a DNS SRV record - if one is set for the "domain" part of your JID.

There's a new log system, fully customisable and managing the colors, formatting, filtering and outputs (files, memory...).

Administrative aspects

We submitted the file for the creation of the association "Salut à Toi"... which has been accepted without any trouble, positively surprising us. Our working mode is indeed a bit special for a French "1901-law" association: a collegial board with no president, secretary nor treasurer but two co-administrators. We remind you that there is behind this project a desire to fully involve ourselves, and this is not compatible with the pursue of another professional activity. This means, for the developers, the necessity to find a funding source. We will start with testing / adapting our idea of a good funding model for SàT, of course meeting the ethical and moral commitments that are defined by our social contract.

The association memberships are our favorite source of income! We defined in the internal rules several amounts for the annual subscription, between 0 and 100 euros and every one is free to choose. There's no typo! For the persons who would like to support us without being able or willing to contribute financially, that's possible - because moral support is important too. So there's no reason not to join ;-)

We are unfortunately not ready yet to offer you the possibility to make it online. We actually plan to open a bank account for the association at the end of this month, then we will prepare the online form to manage the subscriptions.


We attended this year the "Free Software Days" (aka JDLL in French) in Lyon, "Pas Sage en Seine" in Paris and the "Libre Software Meeting" (aka RMLL) in Montpellier - see the links to watch the recorded conferences (in French). We met or saw again some nice people at the stands and during the speeches Goffi made. Many thanks to the organizers of these events, and to which wrote an article about SàT following the last version's release, allowing a larger public to get to know about the project.

We also participated last week to the "XMPP Summit" and its hackaton in Berlin. We met there some XMPP developers, including two with whom we have regular contact, such as Edhelas from Movim. We presented together our issues with the protocol and will suggest new XEPs to standardize some practices (especially regarding Publish-Subscribe and blogging), and push their implementations.

Saturday, September the 27th between 2pm and 5:30pm, Goffi will participate to a radio program from "Ici et maintenant", recorded in Paris and about self-hosting. There will be many guests including two from the Jappix project.

We would like to organize some gatherings via the association, at least once per year as a general assembly and maybe more often. The date and the location haven't been discussed yet.

Please also notice the recent creation of a "users" mailing list for SàT, it completes the chat room and the "dev" mailing list.

And then?

We would like to migrate our own blogs to SàT. The version 0.6 will focus on blogging (in SàT, it is based on a fine access permission system, so you only write to the people you want), pictures storage and tags implementation. These are essentials features and maybe the last important works before the publication of the first public release, which has been a bit delayed and should be stamped 0.7 or 0.8.

ProcessOne: Sea Beyond 2014: ProcessOne Tech Event for Mobile Realtime

16 September, 2014 - 19:54

I am very pleased to announce date and venue for our third iteration of our tech event on mobile and realtime.

Sea Beyond event will happen on the 4th of december 2014. This year focus is “Real-time Technologies for Push and Chat on Mobile”.

As during previous edition, we have split the schedule in two distinct parts.

  • The Sandbox: a programme for developers to encourage innovation and the production of new software features
  • The Lighthouse: an interactive conference for users that illuminates the latest trends in real time communication


People loved that approach during the previous editions, as the event is thus open to all kind of skills and is focused on makers.

The programme is still to be defined, but you can expect to play, experiment and learn about exciting and innovative use cases around mobile, chat and push services.

We still have a few more weeks to put the final touch on the programme and prepare to surprise you. Stay tuned !

Read more on the official Sea Beyond page: SeaBeyond 2014


It the meantime, you can watch last edition summary video:

Alexander Gnauck: Unity development with MatriX

15 September, 2014 - 14:56

A question which comes up very often is the following:

Can I use MatriX with Unity 3d?
Of course you can. Many of our customers are using it with great success in their Unity projects.

Unity is based on Mono. MatriX is designed for cross platform and works on all major .NET platforms, including Mono.

Many Unity developers have some problems to get started with MatriX on Unity.
This post should be a small tutorial and address some of the issues you may be facing during your Unity development.

1) Download the correct MatriX dll (Mono build). MatriX builds are not universal, which means you need to download and reference the MatriX library for Mono.
The Mono build can be found at the latest binary downloads here:
when you browse to MatriX => Mono

2) The System.Xml.Linq.dll seems to be not a standard Unity reference. Download it as well and add it to your assets. Otherwise Unity may throw an exception when you run your project.

3) Depending on your Unity version you may get compiler errors on iOS. When you use the latest Unity version this should be no problem. On older Unity versions this problem is related to obfuscation. To resolve this please contact us directly.

4) The iOS stripping feature causes problems. It does not work when reflection gets used. The core Xml serializing engine in MatriX is build on reflection using a factory pattern. You either have to remove stripping from your complete project, or exclude the MatriX classes.
Here is an example of a linker.xml file excluding all MatriX classes:

<linker> <assembly fullname="Matrix"> <type fullname="Matrix.*" preserve="all"/> </assembly> <assembly fullname="mscorlib"> <namespace fullname="System.Security.Cryptography" preserve="all" /> </assembly> </linker>

5) Some of our users reported that connecting to an XMPP server does not work inside the Unity editor because of some networking problems of the IDE. When the code gets executed or published on a device it works fine. This is a restriction of Unity and not related to MatriX.

If you run into any other problems please contact us directly. We do our best to solve them.

Ignite Realtime Blog: Smack 4.1.0-alpha1 available

13 September, 2014 - 17:34

After months of hard work it's at the time to release the first alpha version of Smack 4.1, the open source Java XMPP client library, for testing purposes.


Smack 4.1 marks a milestone in the development history of Smack, as it's the first version that runs native on Android. This means that aSmack is no longer required and will be phased out in the future. Future aSmack releases will be solely form the stable 4.0 branch.


Together with support for Android, Smack 4.1 also adds support for XEP-198 "Stream Management" in smack-tcp. XMPP connections with enabled Stream Management provide acknowledgments of sent stanzas (and acknowledges received stanzas to the server) and allow transparent stream resumption in case of a network outage (for example because of a WiFi ↔ GSM switch on Android).


Smack 4.1.0-alpha1 is now available from Maven Central and we would welcome interested and adventurous users to try this early alpha release and provide feedback. For more information about using Smack 4.1 and how to include it in your Android project, consult the "Smack 4.1 Readme and Upgrade Guide".

Ignite Realtime Blog: Smack 4.1.0-alpha1 available

13 September, 2014 - 17:34

After months of hard work it's at the time to release the first alpha version of Smack 4.1, the open source Java XMPP client library, for testing purposes.


Smack 4.1 marks a milestone in the development history of Smack, as it's the first version that runs native on Android. This means that aSmack is no longer required and will be phased out in the future. Future aSmack releases will be solely form the stable 4.0 branch.


Together with support for Android, Smack 4.1 also adds support for XEP-198 "Stream Management" in smack-tcp. XMPP connections with enabled Stream Management provide acknowledgments of sent stanzas (and acknowledges received stanzas to the server) and allow transparent stream resumption in case of a network outage (for example because of a WiFi ↔ GSM switch on Android).


Smack 4.1.0-alpha1 is now available from Maven Central and we would welcome interested and adventurous users to try this early alpha release and provide feedback. For more information about using Smack 4.1 and how to include it in your Android project, consult the "Smack 4.1 Readme and Upgrade Guide".

Peter Saint-Andre: The Internet Is Dead, Long Live the Internet

4 September, 2014 - 00:00
The Internet is in trouble. In fact, the Internet as we knew and loved it is dead and gone. I gave a talk about that in July at TriConf in Richland, Washington. Thanks to the folks at JK Productions, you can now watch the talk for free on Vimeo.

Peter Saint-Andre: An Aristotelian Tangent

3 September, 2014 - 00:00
Although I am deep into absorbing Thoreau (currently reading Wild Fruits and his Journal), I continue to ponder some of the other writing projects I have on the back burner. Of late I've had a few stray thoughts about Aristotle. Most simply, what form shall I impart to my book about his ethical philosophy, and what shall I entitle it? Having explored, by then, the literary-philosophical forms of manifesto (even sermon?), dialogue, journal, and poetry cycle for my books about Rand, Epicurus, Thoreau, and Nietzsche, what will be left? For those aware of the Aristotelian tradition, the commentary form comes quickly to mind. Not that I (slight scholar that I am) can hope to approach the great commentaries of Averroes or Aquinas; but perhaps an epitome is within my reach, especially since that's consistent with my appreciation for short books (none of the volumes in my lifelong philosophy project will be more than 60 pages or so).

Thijs Alkema: HTTPS Attacks and XMPP 2: CRIME & BREACH

7 August, 2014 - 08:04

In part 1 I looked at BEAST and concluded that it would not be possible for XMPP. In this part, I’ll look at compression based attacks, similar to CRIME and BREACH for HTTPS.

CRIME: TLS can optionally use compression, which means it compresses the application data before encrypting it. The danger of compression is that it can make the size of the payload change by a variable amount, which depends on how much similarity exists within the content. If an attacker can convince the client or server to compress some secret data (like a cookie) together with some data chosen by the attacker, then the attacker can observe how similar his data was to the secret data. By repeating this process, the attacker can guess the cookie character by character.

BREACH: BREACH used the same principle, but applied to HTTP compression. Because HTTP only compresses the content, not the headers, this means this attack can not obtain cookies, but other secret data in the page can be obtained (like email-addresses).


Just like HTTPS, we have two ways of compression here: TLS compression and XEP-0138: Stream Compression. They also differ in what data they compress: TLS compression compresses the entire stream after TLS started, XEP-0138 doesn’t kick in until after the user has successfully authenticated (in c2s).

What is this compression thing anyway?

Both TLS compression and XEP-0138 compression can use zlib (also known as the DEFLATE algorithm), and this appears to be the most common compression method used.

The main method used for compression is by removing duplicated blocks of data and replacing them by back-references. If the compression algorithm sees </message> in the data, it checks whether it has occurred before. If it has, it replaces it by a reference to that previous point. It’s not required to insert a reference when it is possible, and references are only allowed to data in the last 32 kiB. This means that the decompressor must always buffer the last 32 kiB (the compression window), but the compressor may choose to keep as much as it likes (to save on memory, for example).

The data generated by zlib is no longer byte-aligned, to make sure you have an integer number of bytes, it’s necessary to do a sync flush. This means zlib inserts the required padding to make it fit in a whole number of bytes, so you can write it to a socket, for example. A full flush is a sync flush, but the compressor also throws away the data buffered in its compression window. This means the compression can not create back-references to data before the full flush.

TLS compression

With TLS compression, the user’s authentication is included in the compression, which makes it a great target. Assuming, of course, that the authentication mechanism used is PLAIN, because otherwise capturing the data exchanged during the login is worthless. Inserting data to be compressed together with the password is easy: for example, a client will reply to iqs with the same id as the original message had. If the id contained a guess for the user’s password, then the attacker can observe whether their guess was correct by checking the length after compression and encryption.

I managed to get this to work pretty easily: one client logs in using PLAIN authentication repeatedly, while another client sends it iqs and observes the length of the captured TLS encrypted packets. With some extra effort, I made it possible to do multiple guesses per session, around 8 seemed to work reliably. It’s possible to try to guess as long as the password is in the compression window, but you have to ensure the previous guesses don’t affect the compression of your next guess. It is somewhat simplified scenario, as the modified client I exploited doesn’t do anything after logging in (not even retrieving its roster), so I get more guesses than I would get in a real-world scenario, but I do think you would at least get a couple.

Due to the base64 encoding, 8 guesses per login works out to about 5 logins per character of the password, on average. For a typical user with a strong 8 character password who logs in every day, this means the password can be obtained in 1.5 month. If the attacker starts randomly closing the user’s connection (and the user automatically signs in again), this could be done in less than an hour.


The key to implementing this was to split the data into 3 categories:

  1. Fully compressible data. This will always be replaced by a single back-reference.
  2. Incompressible data. Data that is very unlikely to have occurred before in the stream.
  3. The part of the secret that is known, followed by a single character (the guess).

This looks like:

<iq to="" type="get" id=","><ping xmlns="urn:xmpp:ping" /></iq> <iq to="" type="get" id="}a}b}c}e}AHVzZXIAa}f}g}"><ping xmlns="urn:xmpp:ping" /></iq> <iq to="" type="get" id="|a|b|c|e|AHVzZXIAb|f|g|"><ping xmlns="urn:xmpp:ping" /></iq>

Which the client will reply to with:

<iq to="" type="result" id="," from="" /> <iq to="" type="result" id="}a}b}c}e}AHVzZXIAa}f}g}" from="" /> <iq to="" type="result" id="|a|b|c|e|AHVzZXIAb|f|g|" from="" />

  1. The first ping isn’t a guess, but it ensures <iq to="" type="result" id=" and " from="" /> occur in the compression window, therefore will be fully compressible in the next stanzas.

  2. }a}b}c}e}, }f}g}, |a|b|c|e| and |f|g| are used as incompressible data: } and | are not valid base64 characters, so won’t match with the password, and they don’t look like anything included in normal XMPP stanzas or anything a user would likely write. They ensure different guesses can’t influence each other, because they separate the compressible data and the guess.

  3. AHVzZXIA is what’s known so far, and a and b are the next guesses.

This method even works when used with block ciphers, where the exact length is unknown: by using the right amount of incompressible data, it’s possible to make the stanza compress to n blocks when the guess was wrong, but n-1 when the guess was correct.

XEP-0138: Stream Compression

For XEP-0138, the user’s password is safe. However, there is still other private data an attacker could try to guess. For example:

  • Retrieving the password to a MUC.
  • Obtaining your roster (as long as you don’t change it too often).
  • Determining whether you have recently received a message from a specific JID.
  • Determining whether a specific string occurs in a recent message (if it’s not too common and occurs elsewhere in the XMPP protocol).
  • Determining whether you have joined a specific MUC.

If you want to use compression: don’t.

If you absolutely have to use compression, disable TLS compression and use XEP-0138 and do a full flush after every stanza. This will be bad for your compression ratio, but the only way to be (somewhat) safe from these attacks. But keep in mind that you have to ensure both sides do this.

yaxim: yaxim 0.8.8 - Important Security Update

6 August, 2014 - 11:37

yaxim’s stated first goal is security. Unfortunately, there are days when you realize you failed hard at reaching that goal. All versions of yaxim before 0.8.8 are vulnerable to a Man-in-the-Middle attack, where an active attacker can redirect and read all your traffic by using a valid SSL certificate for his own server.

Please update immediately to 0.8.8 (ChangeLog, commit history)!

Man-in-the-Middle Vulnerability

yaxim has fallen victim to the problem known as CVE-2014-5075, allowing attackers to hook into the connection between yaxim and your server, and to read all your data (including your XMPP password!).

It is not known if this vulnerability has ever been exploited in the wild. Thereforeyou should immediately upgrade yaxim and change your XMPP server password. The latter can also be accomplished using yaxim 0.8.8.

New Password Change Dialog

Previous versions of yaxim only allowed changing the password stored in yaxim, not the one on the server.

Because changing your XMPP password is an important security element, and the embarrassing security vulnerability that existed in yaxim over the last years forces the author to make it possible, the password change dialog now incorporates the ability to change your password locally and on the server:

If you only want to change the password in yaxim, but not on the server, just untick the box:

Because the XMPP password is stored in yaxim’s (unencrypted) preferences, and most users do not remember their XMPP password anyway, this mechanism does not require entering the old password. If you consinder this a security problem, please do not give your smartphone to potential pranksters.

On related news, the startup wizard also requires to re-enter the password when registering a new account.

Other Security Improvements

As of now, yaxim will try to use TLSv1.2 or TLSv1.1 first, on Android 4.0 and later. Older Android devices are limited to TLSv1 unfortunately. SSLv3 will not be accepted any more, in accordance with current recommendations.

Furthermore, with this release it is no longer possible for third-party apps on your phone to access yaxim’s contact list or chat history. This issue was reported by Tarek Saier and is incorporated in 0.8.8.

Improvements have been made to the packet queue handling with XEP-0198, helping with very laggy connections.

User Interface Changes

The status dialog has been revamped by Joerg Mensmann. Now it features colorful icons in the status drop down menu as well as auto-completion for your old status messages:

A new setting allows to disable contact group display, moving all your contacts into one large group.

Furthermore, the “OK” button to send chat messages has been finally aligned to the bottom of the window, so it no longer moves up if you type a long chat message, and there is a new “huge” setting for the chat window font.

ProcessOne: TextOne 4.0.0 released

28 July, 2014 - 14:16

We just released a new version of TextOne iOS messaging client.

This is a version improving compliance with iOS 7 (and introduce a new and lighter design). It also improves networking stack and time to connect / retrieve messages.

It also fixes a long standing crash issue on launch that could sometime happen.

It can be downloaded from Apple App Store: TextOne on Apple AppStore.

ProcessOne: ejabberd Community 14.07

23 July, 2014 - 10:40

Two months after the previous huge release, we are keeping the pace with a new feature packed version. We added major improvements regarding to SIP support, Carbon copy, stream management and added Riak as a new supported database.

ejabberd Community 14.07 includes several improvements and many bugfixes over the previous 14.05 release:


  • SIP Outbound (RFC 5626) support
  • New option always_record_route
  • New options record_route and routes

Carbon Copy (XEP-0280)

  • Don’t log MUC messages with hint
  • Don’t carbon copy messages with hint
  • Let is_carbon_copy/1 recognize carbons
  • Don’t send XEP-0280 v1 copies back to sender

Stream Management (XEP-0198)

  • XEP-0198: Terminate session if stanza queue becomes too large
  • XEP-0198: Don’t exit on socket send failure
  • XEP-0198: Don’t drop session on failed resume
  • XEP-0198: Check whether routed packets are stanzas


  • Riak support

Install and config

  • New options log_rotate_count, log_rotate_size and log_rate_limit
  • html guide is now generated when building source tarball
  • Use p1_utils, and move treap.erl to p1_utils
  • Get rid of p1_mnesia file
  • old release notes are not installed anymore
  • Don’t “forget” listener options
  • Always enable STUN at compile time
  • Do not check for Erlang apps at configure time
  • Add –enable-riak configure flag


  • Add tests for stream management
  • Add tests for mod_carboncopy
  • Add tests for mod_caps
  • Improve MUC test cases
  • Travis CI: Enable Riak tests
  • Add Riak backend to the testing suit
  • The test suite no longer fails without –enable-transient_supervisors.


  • MUC messages with ~ were not logged (EJAB-1696).

And many bugfixes all over the source code.

This release requires at least Erlang/OTP R15, and works perfectly with R16B03, and will work with R17 if Riak support is disabled (–disable-riak).

When building from Git, “make install” now fails if you do not run “make doc” before due to doc/guide.html no longer being in the repo. If you don’t have necessary tools to build the guide, just pick the one from ejabberd sources tarball.

As usual, the release is tagged in the Git source code repository on github

The source package and binary installers are available at ProcessOne

If you suspect you found a bug, search or fill a bug report in Jira

Fanout Blog: Mongrel2 HTTP server now in Debian/Ubuntu

18 July, 2014 - 19:58

Mongrel2 is a fast and simple HTTP & WebSocket server that communicates to backend workers via ZeroMQ. It does one thing and does it very well, making it an ideal part of a componentized architecture. The code is event-driven, allowing it to support thousands of concurrent connections and also asynchronous behaviors. These properties are especially important to realtime applications.

Fanout has been one of the most active contributors to the Mongrel2 project over the past year, adding features such as TLS SNI and improved streaming capability. We've also been working on making the server easier for people to get started with. And with that, we are proud to announce official packages for Debian and Ubuntu!


ProcessOne: GigaOM releases report on brand engagement through in-app communication

17 July, 2014 - 08:22

Mobile apps now allow businesses to be more responsive to users than ever before, but this interaction has created new customer expectations. While a mobile app must fulfill its core function, it must also continue a dialog with its users to remain relevant. Doing so requires ongoing, intelligent, targeted outreach to customers and an extension of customer-service strategy into the app itself.

Brands and businesses that develop mobile applications must be aware of the demands and limits of an increasingly sophisticated mobile audience, build a communications strategy that spans the appropriate communications channels, and play to the strength of each channel.

In-App communication gathers as a whole three mobile oriented channels:

  • Native Push Notification services.
  • In-app notifications.
  • In-app chat.

Together, they form the three main components of brand realtime user relationship management on mobile. Today, those three approaches are your best bet to establish a strong communication channel that provides a real value to your users.

The field is quite new and we are glad to be able to share with GigaOM the first research on brand engagement through in-app communication.

The document gives examples, insights and best practices to tighten the links with your mobile application user base and provide them the best value at the right time.

We are pleased to make this report available for free to people interested in Boxcar Push Notifications Service. Join our mailing list focused on leveraging in-app communication for brands to receive report download link »

(Do not worry, it’s low traffic, no spam, we only care about the value we give you).

Report is also available to GigaOM subscribers.

Isode: New “Military forms using XMPP” whitepaper and access to our Military forms Demo

7 July, 2014 - 14:11

Forms are important for military operations, and there is often a need to handle forms quickly and share with a large number of users, such as Medical Evacuation (MEDEVAC) alerts.

XMPP based open standard instant messaging is widely used by military organizations and is a sensible framework for sharing forms. Our new whitepaper [Military Forms using XMPP], published on the Isode website today, looks at the requirements for military forms and how the XEP-0346 “Forms Display and Publishing”(FDP) can be used to provide real time military forms. It looks at how capabilities provided by M-Link support military forms using FDP, and how gateways can enable integration with other services. FDP is supported in the most recent R16.2 release along with FDP Management in M-Link Console.

We have created a military forms demonstration web site using FDP and Isode’s demonstration Web FDP client. To use this demonstration you will require information on the demonstration accounts. Please contact for access to this information.

If you wish to set up an FDP system, this is explained in our FDP Evaluation Guide which will show you how to set up a basic XMPP Form Display and Publishing (FDP) configuration using Isode’s M-Link server, Isode’s Demonstration web client and Isode’s FDP Demonstration Desktop Client.

Peter Saint-Andre: Thoreau and the Seasons of Man

5 July, 2014 - 00:00
This evening I finished reading Thoreau's first book, A Week on the Concord and Merrimack Rivers. Toward the very end, he writes as follows:

Fanout Blog: You might not need a WebSocket

1 July, 2014 - 01:32

Before I begin, I want to say that WebSockets are great. I've even implemented RFC 6455 myself in Zurl and Pushpin, which are used by the service. also supports WebSockets via its XMPP-FTW interface using Primus.

However, after spending quite some time working on large distributed applications and gaining a greater appreciation of REST and messaging patterns, I feel that much of what typical web applications want to accomplish with WebSockets (or with socket-like abstractions) is perhaps better solved by other means.


ProcessOne: Cardinality Estimation

30 June, 2014 - 08:47

In server world, we always need to maintain some metrics; We need to measure to improve. A very common one being “unique active user” per unit of time. While this is really easy to describe, it’s complex when it comes to implementation.

Naive implementation logs all events (let say, user connection), either on memory or disk, and count number of unique entries in the log on a given time frame by removing duplicates.
Well, this is really consuming and can not scale on real server handling millions of user. Better take the probabilistic estimation way… and here comes a very impressive algorithm.

Back in 2007, a team at INRIA published a paper about an efficient algorithm for estimating the number of distinct elements, known as the cardinality, of large data ensembles. This paper by Philippe Flajolet, Éric Fusy, Olivier Gandouet and Frédéric Meunier can be found here.
There are many blogposts about it already, but this paper is worth reading to make your day !

The HyperLogLog algorithm is just genius thanks to uncommon approach that make it require only few Kilobytes of data to give an accurate cardinality estimation on large sets. Basically, the idea is to rely on probability of distribution of user id numeric representation (hashes).

The HyperLogLog algorithm works for maximal cardinalities in the range [0..109] while handling a number of registers (m) in the range [24..216].
Of course, using less registers reduce computing time and memory use, at cost of cardinality estimation precision.

So, what is best value of m (number of registers) for my need ?
Well, that all depends ! Of course, we will run it on huge production services, so it must be fast. It also must be accurate in most cases.

Let’s assess errors margin and approximations

From the original paper, we have an estimation of the standard error of the algorithm:

Let σ ≈ 1.04/√m represent the standard error; the estimates provided by HYPERLOGLOG
are expected to be within σ, 2σ, 3σ of the exact count in respectively 65%, 95%, 99%
of all the cases.

bits m σ 2σ 3σ 10 1024 ±3.25% ±6.50% ±9.75% 11 2048 ±2.30% ±4.60% ±6.90% 12 4096 ±1.62% ±3.26% ±4.89% 13 8192 ±1.15% ±2.30% ±3.45% 14 16384 ±0.81% ±1.62% ±2.43% 15 32768 ±0.57% ±1.14% ±1.71% 16 65536 ±0.40% ±0.81% ±1.23%

Let say one need cardinality approximation ±~3%, with a trust level of 95%.
One can get cardinality ±3.26% using only 4096 registers.
Now if i want error <2% with a trust level of 99%, I must use 215 or 216 registers.

Now, what about execution runtime and memory consumption ?

Let’s play with the erlang implementation available on github thanks to Shyun Yeoh (vaxelfel).

First, let’s check how much Erlang VM memory is needed to store one hyperloglog record, given number of registers m

m memory in bytes 2048 2845 4096 6173 8192 13341 16384 28701 32768 62469 65536 131101

The memory consumption is approximative, and depends on default heap size and other low level memory parameters. But it gives the big picture anyway.

At last, let’s try to update an HyperLogLog record one million time with unique values, and report time and error margin:

m error ms 2048 +3.41% 2178 4096 +2.31% 2414 8192 +0.71% 6343 16384 -0.49% 11100 32768 +0.16% 25667 65536 +0.10% 9584

Without surprise, the consumed time in HyperLogLog update is almost proportional to data structure size in memory, at one exception.
When using m=216, there is an approximate 5x speedup on this implementation. I guess this is related to some binary matching optimization on Erlang VM but I could not get measures clearly stating this.


A production server never run on its own. The magic with HyperLogLog algorithm is that merging data from different servers is possible as long at they run with the same hash function and the same number of registers.
The resulting estimate of merged data gives cardinality for whole cluster. Bravo !


For rapid cardinality estimation using Erlang implementation, there is two main use of HyperLogLog:

  • For fast operation and minor resources consumption, use of 212 registers.
  • For precise estimation, if memory is a problem, use 214 registers, if CPU is a problem use 216 registers.

Alexander Gnauck: BOSH prebind in MatriX and attach to Strophe

29 June, 2014 - 10:13
Why prebind and attach?

When your web application is already logged in and needs to establish a XMPP connection you have a seamless and secure solution with prebind and attach. You don’t have to ask the user to provide the credentials again, and you don’t have to render the credentials insecure in your HTML output.

Prebind with MatriX

This code shows how to prebind a session with MatriX. On success MatriX raises the OnPrebind event and provides the Jid, Rid and Sid in the PrebindEventArgs. The Rid is the last one used by MatriX and must be incremented before getting passed to Strophe.
You can render this params in a new page and attach the session with Strophe.

var xmppClient = new XmppClient { Username = "testuser", XmppDomain = "", Password = "secret", Transport = Matrix.Net.Transport.BOSH, Uri = new System.Uri("") }; // subscribe to the prebind event xmppClient.OnPrebind += client_OnPrebind; void client_OnPrebind(object sender, PrebindEventArgs e) { prebindArgs = e; are.Set(); } // start prebinding with MatriX now xmppClient.Prebind();

You can download a complete ASP .NET example project here:
Bosh Prebind Example

Alexander Gnauck: Await MatriX

29 June, 2014 - 10:12

C# 5.0 brought us the great new Async and Await feature. It helps to avoid bottlenecks when it comes to performance or to improve the responsiveness of UI applications.

Both of them were never a problem in MatriX, because its completely asynchronous internal. We helped GUI developers a lot by handling all multi threading and invoking to GUI threads inside of MatriX.

But the traditional techniques for writing asynchronous code based on callbacks can be complicated, difficult to maintain, hard to debug, error prone, and often ends in the callback hell.

Here is an for requesting a vcard with the new async await API

var viq = new VcardIq { To = "", Type = IqType.get }; var vcardResult = await xmppClient.IqFilter.SendIqAsync(viq);

Here is a more complex example of creating a MUC room containing the following steps.

  • request our reserved nickname
  • create the room with the requested nickname
  • request the room configuration
  • submit the room configuration

With the old async callback based API this simple task requires you to setup multiple callbacks or listen to multiple events. This splits your code in many different functions, or you use many inline lambdas. Error handling gets complicated and the control flow is hard to follow.

So look at the code with the new async await API. It looks like simple synchronous code. Easy to read with simple try/catch error handling, fully asynchronous. The compiler is doing all the complicated work for us.

try { Jid roomJid = ""; var nick = await mucManager.DiscoverReservedNicknameAsync(roomJid); var createResult = await mucManager.CreateRoomAsync(roomJid, nick); // check from createResult if the room requires configuration if (createResult.Error != null && createResult.Error.Condition == ErrorCondition.ItemNotFound) { var roomConfig = await mucManager.RequestRoomConfigurationAsync(roomJid); var data = ProcessRoomConfig(roomConfig); var submitResult = await mucManager.SubmitRoomConfigurationAsync(roomJid, data); } } catch (TimeoutException ex) { // handle exceptions here } catch (Exception ex) { // handle exceptions here }

Many of our customers are tied to older .NET versions and not able to use the latest and greatest features. This is the reason why we will offer separate builds for .NET 4.5 with the async API extensions, and .NET 3.5 builds.

We will upload new .NET 4.5 builds soon, so stay tuned. If you can’t await then contact us directly ;-)

Alexander Gnauck: NuGet package available

29 June, 2014 - 10:12

Some of our customers asked for NuGet packages. The roadmap of our development is heavily impacted by our customer requests. We are always receptive to your criticism, ideas and praise

When you prefer NuGet for managing your references then you can start using it for MatriX as well now.

You can find the MatriX NuGet package here: