MongooseIM 2.1.1 — More than a patch!

by Piotr Nosek

About 3 months have passed since the final MongooseIM 2.1.0 release.

Over that time we’ve gathered feedback from community and picked out improvements we really wanted to include in 2.1.0 (but the schedule is merciless!). The outcome is a release that closes the 2.x series.

This is the last stop before the leap into 3.0.0, when you may expect significant changes and features.

You might ask, “why do they call it >>More than a patch<<?“ as there are many catchier titles we could have used.

The reason for that title is honesty, and us admitting our miscalculation. What started as a small revision, sort of closure after 2.1.0, turned into quite an impressive set on changes.

What are these? Keep reading! Oh, and by the way: welcome to MongooseIM 2.1.1!

Secure connections to databases

Security and connection encryption have always been important topics, not only in chat applications.

To always stay ahead of the game, we’ve ensured that you can integrate MongooseIM with external (i.e.: not Mnesia) databases with TLS enabled.

We’ve updated our dependencies where it was necessary and made sure everything is well-documented. With 2.1.1 you may add extra hardening to your platform, no matter if your MySQL/Postgres/etc. is running in the same local network, or in a separate location. For example, you could have a MongooseIM cluster running on bare metal in your data center, while calling MySQL deployed in a cloud (for extra flexibility).

Obviously, it does not apply to Mnesia, which runs alongside MongooseIM and uses Erlang distribution for data exchange.

If you’d like to add a similar level of security to it, you may enable TLS for Erlang distribution protocol. This improvement also does not apply to Redis, as there is no official support for connection encryption in this database yet. We may add some recommendations on enabling it in an unofficial way in the future.

Deumbrellification

A word of explanation: Deumbrellification is a process of transforming an umbrella structure (an application that contains more applications) into a flat one. It makes developers’ life easier, as it means more intuitive code hierarchy and better integration with certain build systems.

How does it apply to MongooseIM?

For a large part of its lifetime, MongooseIM has been bundled with MySQL and PostgreSQL client libraries. It required a nested structure — the main application was placed in apps/ directory alongside these drivers. In the meantime, two events took place: we’ve begun fetching these drivers as dependencies and we’ve switched from Rebar 2 to Rebar 3.

The former change has left us with more up-to-date libraries downloaded by Rebar, and with a somewhat awkward project structure with only one application remaining in apps/ directory. While we were certain that we want to improve it at some point, it had a low priority because it simply kept working. In order to make MongooseIM usable as a dependency, there was a separate src/ folder with a mongooseim.app.src file inside. It means that, while we had MongooseIM’s application file in src/, the rest of the code was in apps/ejabberd/. Again - not the prettiest solution, but it did its job.

The latter improvement required us to temporarily disable MongooseIM’s application file, because Rebar 3 no longer accepted such a structure. With the release of 2.1.0, community feedback strengthened our resolve to keep going and complete the journey. The full transition to OTP-compliance had to be done sooner rather than later; 2.1.1 is what we aimed for. All of the code is in src/ now, and all dependencies are fetched from other repositories. And, what is important, MongooseIM can be used as a dependency in other Rebar projects again!

For those who forked MongooseIM and modified it for their projects, we’ve prepared a comprehensive guide to what has been changed and how you should migrate your repository in order to sync with the current master.

Event Pusher

At some point a requirement arose for MongooseIM to be able to “publish” data to external endpoints. A fine example of such an event is sending a push notification request with message content to MongoosePush. Over time, it turned out that we need to deliver notifications not only to mobile devices, but also to other channels like Amazon SNS or a chosen HTTP endpoint. These three extensions were developed independently and existed as separate modules in MongooseIM.

In order to reduce maintenance costs and make the development of additional channels easier, we’ve introduced Event Pusher — a unified framework that combines the aforementioned extensions.

It’s an important change for everyone using mod_push, mod_http_notification and mod_aws_sns - these are now deprecated. Please check mod_event_pusher’s documentation to see how you can migrate.

Warning! Highly experimental!

2.1.1 includes a very important extension, which will be a subject of intensive development in upcoming version: the Global Distribution module.

Classic cluster intercommunitation in XMPP involves using distinct domains, and message routing between them requires interlocutors to be aware of the other party’s domain. While being a stable and well-tested solution, it is not very intuitive for many users and not so convenient for modern applications.

Global Distribution extension allows deploying multiple clusters that share a global routing table. It’s a solution for massive-scale projects that need MongooseIM instances in multiple data centers, while remaining transparent to the clients.

GD is highly experimental and verified mostly for 1-1 messaging. We haven’t ensured its proper interaction with other MongooseIM extensions yet. If you consider it a good fit for your use case — by all means try it and give us your feedback. Nevertheless, GD’s main “job” in 2.1.1 is to start maturing with the master branch.

Miscellaneous technical improvements & changes

Warning! Highly technical details ahead! Non-geeks may keep reading at their own risk!

Message Archive Management — moving forward

MAM is a commonly used extension in XMPP community. It has been published in 2012 (when 0.1 version was published) and undergone several important changes since then. First MAM implementation in MongooseIM was based on v0.2 of the XEP-0313. Subsequent updates to this XEP brought more or less significant modifications to the protocol (which involved XML namespace changes). MongooseIM’s mod_mam is compatible with multiple versions by default.

In 2.1.1 it is time to face the fact that v0.2 will soon be 5 years old and most (or perhaps all of them by now) of the client libraries are using v0.3 or newer protocol. From now on, v0.2 support is considered deprecated and will be removed in MongooseIM 3.0.0.

MySQL & UTF8 support

UTF8 support in MySQL has its surprises. One of them is the range of characters that utf8 encoding offers. It turns out that it doesn’t cover the full set. For example, it can’t store 4-byte emojis. In order to fix it, we’ve decided to set utf8mb4 encoding for all tables in our MySQL schema.

But it has created another issue — we’ve hit a limit for index key size because some of the varchar columns used in indices are pretty large. Changing their encoding from utf8 to utf8mb4 added an extra byte to every character, so indices based on them suddenly exceeded the 767-byte limit. We’ve decided to avoid extensive modification of column sizes in all affected tables, so the team has chosen to drop the official support for MySQL versions that do not have the “large prefixes” option implemented. As a part of this step, ROW_TYPE is now set to DYNAMIC.

In 2.1.1 the oldest supported MySQL version is 5.5.14. What is more, all servers of version 5.7.8 or earlier require additional configuration.

Important: No action is required for existing deployments, as we’ve modified only the schema file. All SQL backends in 2.1.1 are still 100% compatible with schema from 2.1.0 but obviously won’t benefit from this improvement, unless migrated.

Build fixes

MongooseIM 2.1.0 introduced an improved Message Archive ID generation method but it actually turned out that new C++ module could not be built on 32-bit systems. This issue has been resolved in 2.1.1 and the server can now be compiled both on 32 and 64-bit architectures.

The second fix helps developers using macOS High Sierra, which uses BoringSSL as default library for secure connections. MongooseIM’s dependency, fast_tls, relies on OpenSSL. Since it was compiled with a flat namespace, it resulted in mysterious runtime errors. 2.1.1 depends on a recent fast_tls version, which is not affected by this problem.

Changelog

Please feel free to read the detailed changelog where you can find a full list of source code changes and useful links.

Contributors

Special thanks to our contributors: @andrewvmail @igors!

What’s next?

MongooseIM 3.0.0 will feature several important changes and novelties. One of them will be a standard implementation of Inbox, which should cover most (or hopefully — all) of our community’s needs. Second is further improvements to Global Distribution. We are going to ensure that it is compatible with other MongooseIM extensions. The third goal is restructuring the documentation — we want to significantly improve usability and to introduce a more intuitive hierarchy of topics.

You may expect MongooseIM 3.0.0 release on April 17th!

Test our work on MongooseIM 2.1.1 and share your feedback

Help us improve the MongooseIM platform:

• Star our repo: esl/MongooseIM
• Report issues: esl/MongooseIM/issues
• Share your thoughts via Twitter: twitter.com/MongooseIM
• Download: Docker image with new release
• Sign up to our dedicated mailing list to stay up to date about MongooseIM, messaging innovations and industry news.
• Check out our MongooseIM product page for more information on the MongooseIM platform.

Originally published at www.erlang-solutions.com.