MnemoQ Remembers, So You Don’t Have To

By Matteo Brichese Wednesday, July 9, 2025

Rethinking inter-process communication with a lightweight, backup-friendly MQTT client

For the past seven years, I’ve been using D-Bus as the primary form of communication between processes in a very complex project. It was clear to all of us on the team that D-Bus wasn’t the right choice for transporting large volumes of data quickly. But, as often happens in projects like this, we couldn’t replace it. Not yet, at least. So we tried to fill the gaps when time allowed it. It wasn’t optimal, but it did its job.

In all my projects, I feel like I’ve never had the perfect way to transport data between processes. That got me thinking: what would my ideal solution look like? Here’s the wish list I came up with:

My Inter-Process Communication Wishlist

1. Broker-less Communication

I want to just start a process and have it communicate immediately — no waiting for a central server to come online. A tall order, I know.

2. Only Receive What I Need

I don’t want to deal with messages that aren’t meant for me. No spamming, no broadcasting.

3. Backup

When I bring a process online, I want it to be able to request all the data it missed — so it can catch up seamlessly and I don’t have to worry about timing all services bring up or restarting one.

First Attempt: ZeroMQ and Zyre

Because of my not-so-secret love for ZeroMQ, I tried implementing my needs with it. I immediately hit a roadblock. I had to design a network architecture from scratch, and I don’t have the time for that.

I knew about Zyre, a project for true P2P messaging using ZMQ. But after a quick test, I ran into more roadblocks than I cared for. I don’t need all the features that Zyre offers and I feel it would be definitely overkill for my needs. You should check out Zyre before designing your own!

Exploring MQTT

Plenty of my coworkers swear by MQTT and I never had the chance to get to know it, so why not give it a try?

The catch: MQTT is a broker-centric message system, which means I’d have to drop my first wish list item. But maybe, if I used a standard broker that could be installed easily and auto-start at boot, I could overlook that part and still be a happy camper.

Items #2 and #3 would need some custom code. Overall, it felt doable.

Design Concept: A Memory-Backed Broker

The idea: write an MQTT broker that includes a storage table. Each new outgoing message is logged with the topic/name as the key and the data as the value. The map would grow, but not exponentially — it would only store the last message for each topic.

A client could connect and request all data from a certain channel, and the broker would send it immediately. If a client crashes and restarts, it would be repopulated with the latest data from every other client.

But I quickly found out that writing a broker is no small task. I looked into modifying existing open-source brokers — but again, it felt wrong. Doing that would require compiling, installing and maintaining custom versions across every board or machine I wanted to use. No thanks.

Client-Side Backup Instead

So I thought, could a client subscribe to all topics and act as a backup?

The main issue with that is timing. It might miss messages depending on when it’s brought up. You’d have to remember to bring it up on time — which, in a system with many processes, could get annoying. I think we can do better.

What if each client keeps a small database of its latest sent messages?

Then, when a client starts, it can ask for the latest data from the topics it cares about. This way, we could use a standard broker, write just some small client-side code, and avoid maintaining anything special.

Time to Write Some Code

There are plenty of MQTT client libraries out there. For simplicity, I chose Qt6 and its MQTT library for this first draft. The code is easy enough to convert to pure C++ later if we want to build a library out of it — and I already had it installed in my virtual machine — so it just felt easy enough to get going. I prefer to spend my time coding than installing frameworks and tools!

To start, we needed a name for our class. Because I wanted to be able to say I used AI, I asked ChatGPT for a fitting name. After a few queries we landed on MnemoQ, a play on Mnemosyne, the Greek goddess of memory. It’s an MQTT client that remembers things. I thought it was fitting.

Introducing MnemoQ

class Mnemoq : public QObject
{
    Q_OBJECT
public:
    explicit Mnemoq(QObject *parent = nullptr);

    void connect();
    void disconnect();
    void send(QString topic, QByteArray message);
    void subscribe(QString topic);
    void unsubscribe(QString topic);
};

This is the bare minimum: a way to connect and disconnect from the broker, subscribe and unsubscribe from topics, and send messages.

Mnemoq::Mnemoq(QObject *parent)
    : QObject{parent}, m_client(this)
{
    m_client.setClientId(QUuid::createUuid().toString());
    m_client.setHostname("127.0.0.1");
    m_client.setPort(8586);
}
void Mnemoq::connect()
{
    m_client.connectToHost();
}
void Mnemoq::disconnect()
{
    m_client.disconnectFromHost();
}
void Mnemoq::send(QString topic, QByteArray message)
{
    m_client.publish(topic, message);
}

void Mnemoq::subscribe(QString topic)
{
    if (m_client.state() == QMqttClient::Connected) {
        m_client.subscribe(topic);
    }
}

void Mnemoq::unsubscribe(QString topic)
{
    if (m_client.state() == QMqttClient::Connected) {
        m_client.unsubscribe(topic);
    }
}

This basic code lets us connect to a broker, subscribe to a topic, and send messages. We still need to receive messages, so let’s forward any inbound messages to a signal that can be handled by the class using MnemoQ.

I only care for communication between processes on the same physical machine so, for now, we'll use a locally installed broker. In my tests I used Apache Mosquitto, the reason being that I can install it via APT in my development machine and run it immediately. By now you should know I value simplicity, a lot!

Handling Incoming Messages

signals:
    void connected();
    void disconnected();
    void received(QString channel, QByteArray message);

Then connect those signals:

QObject::connect(&m_client, &QMqttClient::connected, this, &Mnemoq::connected);
QObject::connect(&m_client, &QMqttClient::disconnected, this, &Mnemoq::disconnected);
QObject::connect(&m_client, &QMqttClient::messageReceived, this, [=](const QByteArray &message, const QMqttTopicName &topic) {
    emit received(topic.name(), message);
});

Now we have a basic MQTT client that connects, subscribes, sends, and receives. So far, so good, pretty standard stuff.

Let’s Add Our Twist: Local Backup

Originally I thought about using an external key-value database for the storage, maybe Redis? I do think it would be a neat feature. But, something else to install… It’s a sample, a proof of concept, so let’s stick to the basics. We’ll store all outgoing messages in a QMap.

void Mnemoq::send(QString topic, QByteArray message)
{
    m_client.publish(topic, message);
    m_store.insert(topic, message);
}

Now we can respond to a "BACKUP" message with the saved data:

QObject::connect(&m_client, &QMqttClient::messageReceived, this, [=](const QByteArray &message, const QMqttTopicName &topic) {
    if (message.message.compare("BACKUP") == 0) {
        sendBackup(topic);
    } else
        emit received(topic.name(), message);
});

void Mnemoq::sendBackup(QMqttTopicName topicName)
{
    m_client.publish(topicName, m_store.value(topicName.name()));
}

First problem: MQTT doesn’t allow wildcards when sending messages, so we can’t ask for “all” data in a group like temperature/#. The fix? Loop through the map and return all keys that start with the requested topic.

void Mnemoq::sendBackup(QMqttTopicName topicName)
{
    // Sending data from the backup value
    for (auto it = m_store.begin(); it != m_store.end(); ++it) {
        if (it.key().startsWith(topicName.name())) {
            m_client.publish(it.key(), it.value());
        }
    }
}

Filtering Out Our Own Messages

Now for wish list item #2—we don’t want to receive our own messages.

In MQTT v5, there’s a built-in NoLocal option for this. But since I’m not using v5, I added a manual solution: tag each message with an identity, then ignore anything coming from myself.

typedef struct {
    QString identity;
    QByteArray message;
} Message;

Instead of changing the send() signature, we serialize on the fly:

QByteArray Mnemoq::serialize(QByteArray message)
{
    Message mess;
    mess.identity = m_client.clientId();
    mess.message = message;
    return mess.identity.toUtf8() + mess.message;
}

Mnemoq::Message Mnemoq::deserialize(QByteArray message)
{
    Message ret;
    ret.identity = message.first(m_client.clientId().size());
    ret.message = message.last(message.size() - ret.identity.size());
    return ret;
}

Then update our handlers:

QObject::connect(&m_client, &QMqttClient::messageReceived, this, [=](const QByteArray &message, const QMqttTopicName &topic) {
    // Discard messages from ourselves
    Message msg = deserialize(message);
    if (msg.identity.compare(m_client.clientId()) == 0) {
        // I received my own message - IGNORING
        return;
    }
    if (msg.message.compare("BACKUP") == 0) {
        sendBackup(topic);
    } else
        emit received(topic.name(), msg.message);
});

void Mnemoq::send(QString topic, QByteArray message)
{
    m_client.publish(topic, serialize(message));
    m_store.insert(topic, message);
}

Pitfalls, Workarounds and Future Improvements

This code works, but it’s a quick-and-dirty prototype. Here are some caveats:

Clients must subscribe to a topic before issuing a BACKUP request
Without NoLocal, loops can form so filtering is required
A client shouldn’t send messages to topics it isn’t subscribed to, or it won’t receive the BACKUP if needed

Workarounds:

Automatically subscribe to a topic before sending if we are not subscribed already, then filter out messages from that topic. This requires keeping track of which topics the user wants to subscribe to and the ones we need to subscribe to in order for the system to work correctly.
Use a shared topic for BACKUP messages, all clients are automatically subscribed to. This also requires a better message structure to be able to better handle administrative communications.

A more scalable option? Use an external in-memory DB for key-value pairs. If you’re already using Redis or something similar, it might be a better fit — especially with a large topic list. And given the features that Redis already offers, it could even notify other systems of incoming messages and be saved to disk when needed.

Final Thoughts

This is a deceptively simple problem with a few sharp corners. But I hope this approach — building a client that “remembers”—gives you some ideas for your own system. if you’ve been stuck for years using a messaging system that’s not quite right, a better solution might just be a day of coding away.

Find the improved code here: https://github.com/bricke/MnemoQ