Category: MQTT

SIMCom Y7080 with MQTT (and MQTTS) – Success!

Posted on by Adam

Following on from my earlier article describing use of the SIMCom y7080 CAT-NB (NB-IoT) module, which ended with me “running into the sand” with MQTT and having to update the firmware, here is the report of success!

For reference, see the SIMCom Y70XX Series MQTT(S) Application note.

In this article, I will be using the mosquitto test broker, which has listeners on several ports according to different scenarios for SSL/TLS and authentication.

Preliminaries

I’m assuming a mobile/cellular connection has already been made (see earlier article). I’m also going to jump straight in with encrypted communications, i.e. MQTTS aka MQTT with SSL/TLS. In the cases given below I am not, however, concerned with either: a) verification of the server certificate or b) using a client certificate to authenticate to the server. Since the server is one I control and trust, (a) is not necessary, and conventional user/password authentication will be sufficient instead of (b) since the connection is encrypted. Only using encryption makes the setup of the modem somewhat easier but it still involves the modem and server undertaking some behind-the-scenes communications to transfer the encryption keys. This means using our limited bandwidth and takes quite a bite out of a typical NB-IoT subscription data allowance; the payload for a single simple MQTT message is likely to be very much smaller than the connection data cost.

Initial set-up is straight-forward, involving starting the modem’s MQTT service and one client within it (the 0 is the handle for the MQTT service):

AT+CMQTTSTART
AT+CMQTTACCQ=0,"client_123",1

Note that client id in AT+CMQTTACCQ is what will appear to the broker and that the final parameter must be 1 for a MQTTS (TLS) connection. Note also that for the encryption-only use case, it is not necessary to do any additional SSL configuration, as described in the Y70XX SSL Application Note. That 1 is all you need!

DNS

Using domain name servers to look up an IP address is nice but it does introduce an additional communication overhead; while this is rarely noticeable in most scenarios, the low bandwidth of NB-IoT makes using numerical IP addresses an attractive option. Things get a bit messy if you use a cloud service; I’ve noticed that the IP address for my Azure IoT Hub has changed (see my other article on talking MQTT with Azure IoT Hub).

It is easy to look up the IP address using nslookup (same command on linux and windows – yay!). For test.mosquitto.org it is 91.121.93.94 .

The Y7080 can also query DNS, using either servers whose addresses are provided by the mobile network operator or falling back on factory-defined servers (in China?).

Check the servers with:

AT+QIDNSCFG?

I got two Class A private addresses, so these must belong to the mobile network operator:
PrimaryDns:10.105.16.254
SecondaryDns:10.105.144.254

Querying DNS is as simple as:

AT+CMDNS=test.mosquitto.org

This is an asynchronous operation, so it returns “OK” very quickly and then provides the IP address as a separate message +CMDNS:91.121.93.94. Bad news comes in the form of +CMDNS:QUERY_DNS_FAILED; on the Vodafone platform this will happen if you query a domain name which is not listed in your APN Access List (see my previous article).

Repeating the CMDNS command straight-away will give a much faster response, so there must be caching somewhere (given the speed, I suppose this is internal to the modem). I’ve not found the cache duration documented anywhere but it is only of the order of a few minutes.

Making a Connection

Although using authentication will be essential for any in-service data logging, it makes sense to get a more simple situation working first. The connection to the simple MQTTS service of test.mosquitto.org without authentication uses port 8883:

AT+CMQTTCONNECT=0,"tcp://91.121.93.94:8883",60,1

Note that the Y7080 does require the “tcp://” and that the port is not a separate parameter.

This is another asynchronous command. Be prepared to wait several seconds, hoping for the successful message: +CMQTTCONNECT: 0,0. If you forget to make the last parameter of AT+CMQTTACCQ be 1 then the modem will very quickly follow that with +CMQTTCONNLOST: 0,2. This can also happen with test.mosquitto.org at quite frequent and seemingly-random times; the price of a free public service.

You can check whether the modem is connected with:

AT+CMQTTCONNECT?

A bare +CMQTTCONNECT: 0 tells of no connection, whereas a live connection will give a response containing the server URL etc.

Now the simple case works, it is a simple matter to add a username and password, using port 8885. The test.mosquitto.org documentation indicates we can use username=”rw” and password “readwrite” for a user with read and write access, so simply connect using:

AT+CMQTTCONNECT=0,"tcp://91.121.93.94:8885",60,1,rw,readwrite

Publishing Some Data

For each message, three steps are required (even if the topic stays the same). First declare the topic in two states, with the first stage declaring the length of the topic, e.g. for a five character long topic:

AT+CMQTTTOPIC=0,5

The Y7080 will respond with a prompt of “>” and wait for the topic string. I found that providing too few characters caused a broker disconnect when publishing. Conversely, providing too many characters simply caused the modem to use the first 5. Unlike the AT+MQPUB described in my previous article, the topic is provided using normal characters, without the need to convert to hexadecimal.

The payload is declared in similar fashion, with a “>” prompt to enter the payload string:

AT+CMQTTPAYLOAD=0,8

The payload string can be either shorter or longer than the length declared in AT+CMQTTPAYLOAD; the modem quietly does the right thing, sending shorter if shorter and truncating if longer.

And finally send it with (I am using QoS of 1 here):

AT+CMQTTPUB=0,1,60

The final command is, as you might by now expect, asynchronous. A short time after “OK”, success is indicated by +CMQTTPUB=0,0.

Tidy-up

Tidy-up is a three-stage process. These must be done in the correct order otherwise error responses are given:

AT+CMQTTDISC=0,120
AT+CMQTTREL=0
AT+CMQTTSTOP

 

 

Using Websockets with EmonPi MQTT Broker to Create a “Live Feed” Dashboard

Posted on by Adam

This is not a complete description of the background tech; there is plenty of info on the web about websockets, mqtt, and the javascript libraries.

The motivating idea behind this experiment is to be able to have a live-updating dashboard with the minimum set of dependencies of the “install X” kind. Since the EmonPi aleady has a MQTT broker, it provides the basis for feeding data to the dashboard. Websockets allows a “web page” (it can be a file viewed in your web browser) to send and receive data, and update the page, using JavaScript. This is not particularly difficult, but there are several steps which took a couple of evenings to research and put into practice, so here are some notes (for myself in the future, and anyone else who finds it!).

If you are looking for some “homework reading”, a decent place to start is Steve’s Internet Guide.

Make Mosquitto Listen for Websockets Connections

Mosquitto is the MQTT broker on the EmonPi. It is configured to listen for connections which employ the “mqtt:” protocol. It is possible to add a websockets listener (“ws:” protocol), with the conventional port 9001 assignment as follows:

  • SSH onto the EmonPi and navigate to /etc/mosquitto.
  • Modify the mosquitto.conf file to add “listener 9001” and “protocol websockets”, see below. I have also added explicit lines for the default mqtt listener on port 1883 in the interest of clarity, although I believe they are not required. You will need to use “sudo”. Alternatively, it should be possible to add a file to the “include_dir”.
  • Restart mosquitto using “sudo systemctl restart mosquitto.service”
pid_file /var/run/mosquitto.pid
persistence false
persistence_location /var/lib/mosquitto/
log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d

listener 1883
protocol mqtt

listener 9001
protocol websockets

allow_anonymous false
password_file /etc/mosquitto/passwd
log_type error

It should now be possible to test two things: that the existing mqtt protocol listener, which is relied on to service data to EmonCMS, and that the websockets listener us “up”. I used “MQTT Explorer”, a free and simple client, which should be set up to not validate a certificate and to have an empty “Basepath” (it defaults to “ws”).

Create the Websockets Dashboard with HTML and JavaScript

My primary aim for this experiment is to be able to co-opt EmonPi to broker air quality data from a home-brew particulate matter, VOC, NOx, CO2 sensor combo, but I’m using the existing emon data to demonstrate the concept, which boils down to “guages” using the Google Charts toolkit, and scrolling line charts using Chart.js.

Here is the code to hack about with, based on snippets from various places, with modifications and updating to a recent. It should just live in a plain text file with a “.html” extension, and can be opened in your web browser. It is not beautiful but demonstrates the concept. There is some logging to the “console”, which is where error messages will also appear. Hit F12 on Firefox or Edge (or Chrome too I think) to find the console.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <!-- This helps with viewing on mobile devices -->
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>Real-Time Charts</title>
    <!-- Google charts -->
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <!-- Chart.js -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.9.1/chart.min.js" integrity="sha512-ElRFoEQdI5Ht6kZvyzXhYG9NqjtkmlkfYk0wr6wHxU9JEHakS7UJZNeml5ALk+8IKlU6jDgMabC3vkumRokgJA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
    <!-- The Paho Javascript for MQTT over Websockets -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.min.js" type="text/javascript"></script>
    <!-- Bootstrap CSS - can be removed but will help with the styling -->
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.2.3/css/bootstrap.min.css" integrity="sha512-SbiR/eusphKoMVVXysTKG/7VseWii+Y3FdHrt0EpKgpToZeemhqHeZeLWLhJutz/2ut2Vw1uQEj2MbRF+TVBUA==" crossorigin="anonymous" referrerpolicy="no-referrer" />
    <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='main.css') }}">
</head>
<body>
<script type="text/javascript">
    // load the google charts stuff and only after it is loaded should we start to set up the data acquisition
    // otherwise, we find the charts library is being called upon before it exists!
    google.charts.load('current', {'packages':['gauge']});
    google.charts.setOnLoadCallback(doWhenReady);

    function doWhenReady() {
        // Create a client instance. NB clientid SHOULD BE DIFFERENT between browser clients; the following should work fine in a home environment.
        var clientId = "client-" + Date.now().toString();
        client = new Paho.MQTT.Client("192.168.1.1", 9001, clientId);
        // set callback handlers
        client.onConnectionLost = onConnectionLost;
        client.onMessageArrived = onMessageArrived;
        // connect the client
        client.connect({onSuccess:onConnect, userName: "emonpi", password: "emonpimqtt2016"});

        // called when the client connects
        function onConnect() {
          // Once a connection has been made, make subscriptions
          console.log("onConnect");
          client.subscribe("emon/emonpi/vrms");
          client.subscribe("emon/emonpi/power1");
        }

        // called when the client loses its connection
        function onConnectionLost(responseObject) {
          if (responseObject.errorCode !== 0) {
            console.log("onConnectionLost: " + responseObject.errorMessage);
            document.getElementById("status").innerText = "Lost connection due to: " + responseObject.errorMessage;
          }
        }

        // called when a message arrives. Note that the full topic name is aka "destinationName"
        function onMessageArrived(message) {
          document.getElementById("status").innerHTML = "";
          console.log("onMessageArrived: " + message.destinationName + " : " + message.payloadString);
          if (message.destinationName == "emon/emonpi/vrms") {
            // "+" converts string to numeric, which is then rounded by toFixed(), which returns a string!
            var vrms = (+message.payloadString).toFixed(2);
            // Google chart guage
            vrmsGaugeData.setValue(0, 1, +vrms);
            vrms_gauge.draw(vrmsGaugeData, vrmsOptions);
            // scrolling line chart.js
            if (vrmsChartData.data.labels.length === 20) {
                vrmsChartData.data.labels.shift();
                vrmsChartData.data.datasets[0].data.shift();
            }
            // we don't have the time in the mqtt message!
            var timestamp = new Date().toLocaleTimeString();
            vrmsChartData.data.labels.push(timestamp);
            vrmsChartData.data.datasets[0].data.push(vrms);
            lineChartVrms.update();
          } else if (message.destinationName == "emon/emonpi/power1") {
            var power1 = message.payloadString;
            // Google
            power1GaugeData.setValue(0, 1, +power1);
            power1_gauge.draw(power1GaugeData, power1Options);
          }
        }

        // ------------- Google Charts -----------
        var vrmsGaugeData = new google.visualization.arrayToDataTable([
            ['Label', 'Value'],
            ['Vrms', 240]
        ]);
        var power1GaugeData = google.visualization.arrayToDataTable([
            ['Label', 'Value'],
            ['Power', 0]
        ]);
        var vrmsOptions = {
            min        : 200,
            max        : 260,
            minorTicks : 5,
            greenFrom  : 230,
            greenTo    : 250,
            majorTicks : ['200', '210', '220', '230', '240', '250', '260']
        };
        var power1Options = {
            min        : 0,
            max        : 5000,
            minorTicks : 5,
            majorTicks : ['0', '1kW', '2kW', '3kW', '4kW', '5kW']
        };

        var vrms_gauge = new google.visualization.Gauge(document.getElementById('vrms-gauge'));
        var power1_gauge = new google.visualization.Gauge(document.getElementById('power1-gauge'));
        vrms_gauge.draw(vrmsGaugeData, vrmsOptions);
        power1_gauge.draw(power1GaugeData, power1Options);

        // chart.js
        var chartOptions = {
            responsive: true,
            tooltips: {
                mode: 'index',
                intersect: false,
            },
            hover: {
                mode: 'nearest',
                intersect: true
            },
            scales: {
                xAxes: {
                    display: true,
                    scaleLabel: {
                        display: true,
                        labelString: 'Time'
                    }
                },
                yAxes: {
                    display: true,
                    scaleLabel: {
                        display: true,
                        labelString: 'Value'
                    }
                }
            }
        };

        // this allows us to have common chart options, while varying the scale in each chart
        var vrmsChartOptions = structuredClone(chartOptions);
        // use sensible range, which will be expanded if the data goes outside "suggested"
        vrmsChartOptions.scales.yAxes.suggestedMin = 230;
        vrmsChartOptions.scales.yAxes.suggestedMax = 250;

        var vrmsChartData = {
        type : 'line',
        data : {
            labels: [],
            datasets : [{
            label : 'Vrms',
            backgroundColor : 'rgba(255, 136, 0, 0.5)',
            borderColor : 'rgba(255, 136, 0, 1.0)',
            fill: true,
            data : []
            }]
        },
        options : vrmsChartOptions
        };

        const lineChartVrms = new Chart('vrmsChart', vrmsChartData);
    }
</script>

<h1>EmonPi Live Feed</h1>
<div class="container">
    <div class="row">
        <div id="vrms-gauge" style="width: 120px; height: 120px;"></div>
        <div id="power1-gauge" style="width: 120px; height: 120px;"></div>
    </div>
    <div class="row" id="status">loading data...</div>
    <div class="col-10">
    <div class="card">
        <div class="card-body">
            <canvas id="vrmsChart"></canvas>
        </div>
    </div>
</div>
</div>
</body>
</html>

Don’t forget to change the IP address for the MQTT broker!

Serving the Dashboard

While the above HTML + JavaScript works fine as a file on your PC (so long as you have a network connection to acquire all those JavaScript libraries from), it can also be placed on the EmonPi.

I have opted to create a separate area from EmonCMS in the interest of avoiding too much risk of cock-up, future muddle, etc… The files in this separate area will all be “static” in the sense that they are just served to users as they are (no PHP etc). The dashboard page above IS static in this sense, since the JavaScript executes on your PC inside the web browser; all the EmonPi does is send it.

EmonCMS uses the Apache2 web server, so it is easy to make it listen on a different port (EmonCMS uses port 80, and I have chosen port 8001) using the following commands (in a SSH).

Create the place where the HTML file will live

cd /var/www
mkdir static

Check the directory ownership is correct using “ls -la”, which should show “drwxr-xr-x 2 pi pi 4096 Nov 27 23:02 static”.

If necessary “chown pi:pi static”.

Place the HTML + JavaScript file here (I suggest using the scp command, for example “scp dashboard.html pi@192.168.1.1:/var/www/static”).

Change the Apache2 config

There are two things to change, first to make apache2 listen on port 8001, and secondly to tell it to use the files in /var/www/static in connection with port 8001 requests.

cd /etc/apache2
sudo nano ports.conf

Add a single line “Listen 8001”, then save and exit.

For the second step, enter the “sites-enabled” directory. I chose to copy the existing emoncms.conf file, naming the copy “static.conf” and editing it to contain:

<VirtualHost *:8001>
  ServerName localhost
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/static

  # Virtual Host specific error log
  ErrorLog /var/log/static/apache2-error.log

  <Directory /var/www/static>
    Options FollowSymLinks Indexes
    AllowOverride All
    DirectoryIndex index.html
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

The changes are fairly obvious, but note that I have also added “Indexes” against “Options”. This makes apache give a listing of all the files in “static” when I use the URL “http://192.168.1.1:8001”. If you only have one dashboard, then simply name it “index.html” and it will appear when that URL is used.

Make a directory for logs

Otherwise apache2 will not restart.

cd /var/log
mkdir static

Restart Apache2

To load the new config.

sudo systemctl restart apache2

Make sure EmonCMS is still working, then visit your new site! It should work adequately on a mobile phone, once rotated to landscape format.

 

Using MQTT in Open Energy Monitor to Capture External Device Data

Posted on by Adam

I struggled to find clear and well-organised information about this using web searches, so here is a condensed “how to”. The scenario I have is using a home-brew ESP8266 based device attached to my solar PV inverter which I want to relay definitive power output to my Open Energy Monitor via MQTT. This seems quite simple in principle, and is simple in practice, but seemingly not well documented.

First thing is to publish the data to an MQTT broker on the emonpi with a topic which starts “emon/{source}/{key}”, replacing {source} with a suitable name for the data source and {key} with the attribute name for the data being sent. In my case, I used “emon/solis/power” as the data source is the output power for my Solis PV inverter. The message payload is simply the data value. This immediately makes the published data appear on the “Inputs” screen of EmonCMS.

Two refinements/possibilities:

Send JSON

Rather than just a single value, send several in the same message. There are two ways to do this:

a) use a topic of form “emon/{source}”

b) use a topic of form “emon/{source}/{key}”

If the message payload sent to the MQTT topic “emon/solis/power” looks like {“ac”: 90, “dc”: 120}, option (a) creates EmonCMS inputs “ac” and “dc” under “solis”, while option (b) creates inputs called “power_ac” and “power_dc”.

Include a timestamp

To do this, simply include an extra field in the JSON called “time”, with a value which is the Unix time. If you are testing, the Unix time needs to be fairly close to the actual time (ignoring summer time) otherwise EmonCMS will indicate “inactive”, but it still captures the data.

Aside: I used the VSMqtt plugin for VSCode as the MQTT client, as I’m using the PlatformIO plugin to develop my ESP8266 code (using the Arduino libraries). Update: these days I’ve changed to using MQTT Explorer as my go-to software for viewing MQTT messages.