An Internet of Things system – how to build it faster

By: Matteo Collina

A revolution is happening now, as you read this. It’s called the Internet of Things (IoT). An IoT system is an application that collects data from the real world, usually through sensors, and transforms the data into insights for its users. Humans are in control; we are in charge of communicating our desires to the system, which converts our desires into commands for actuators in the real world.

Any application that can be written in JavaScript, will eventually be written in JavaScript.”

-Jeff Atwood @codinghorror, Founder of StackOverflow

Internet of Things

An Internet of Things system, collecting data from sensors and generating insights for the users. These insights generate desires that are translated into commands that affect the real world.

The Internet of Things is equivalent in scale and impact to the industrial revolution. The increasing ubiquity of sensors and actuators is enabling ordinary people to interact digitally with their environment, and is enabling businesses and institutions to provide on-demand real-world services through digital means. Companies and engineers around the world are building IoT systems to disrupt industries that have not yet been touched by the digital revolution.

A new generation of devices running JavaScript

Year after year, IoT devices are getting cheaper, more powerful, and more power-efficient. For example, the Intel Edison is as tiny as a stamp, while delivering amazing performance with its dual-core atom system on a chip (SoC). The Intel Edison can fully run the Node.js framework, allowing developers to write IoT programs using Node. Devices like the Tessel are pushing the boundaries of the development paradigm by supporting JavaScript on barebone hardware. The Raspberry PI has been one of the cheapest platforms for quite a while now, and it supports Node.js. I believe in using JavaScript as a full-stack technology, from tiny devices to the cloud, and back to people’s laptops and phones. We are just scratching the surface of what a full stack architecture can achieve, reducing the development time and allowing us to develop new Internet of Things applications in a fraction of the time needed before. In particular, isomorphic libraries allow developers to reuse code between devices, in the server and in the browser.

An initial Internet of Things system architecture

 

IoT

A simple architecture of an IoT System, as any software architect would design it in the first step; this architecture is usually built from scratch.

The image above outlines a sample architecture for an Internet of Things application. A standard divide-and-conquer approach would break up the system into the following four major components:

  • A sensor API, which is called by the sensors to deliver data readings and receive commands.
  • A public API, which is called by the sensors to retrieve real-time data, historical data, and to manage the devices.
  • A set of operational services, which are responsible for authentication and authorization, among other things; these services manage their own database.
  • A set of data services, which are responsible for storing and analyzing the data, either in real time or offline.
Internet of Things

How the sensor data and insights flow in the simple architecture.

 

Internet of Things

How the users desires and commands flow in the simple architecture.

In this architecture, sensor data flows through all services before reaching the users. Likewise, commands move through the system in a similar way, touching all internal services before reaching the actuators. However, users want immediate feedback from the real world; real-time delivery of sensor data and commands is a key feature of the Internet of Things. Our simple divide-and-conquer architecture is not well suited to this data flow, as the entire system is involved in each piece of data or command. Another problematic aspect of this approach is its cost. We are handling real-time data in both our APIs, often in two different ways. Could we leverage this to achieve a better architecture?

API

Behavior of any publish/subscribe system.

A more effective architecture for Internet of Things systems

The core idea of this new architecture (see image above) is to expose a publish/subscribe broker up front to the devices and the users. Usually, these components are part of an enterprise service bus, and are used internally by all sort of organizations. Even popular NoSQL databases like Redis and MongoDB (via capped collection and tailable cursor) offer publish/subscribe functionalities.

IoT - Internet of-Things

How the sensor data and insight flow in a pub-sub driven architecture.

IoT

How the users’ desires and commands flow in a pub-sub driven architecture.

Because we have put a publish-subscribe broker in front of the users, we can have it direct the interaction between the sensors and the users, both for data and commands. This approach reduces latency, and allows for offline handling of time-series data and analytics. Moreover, this approach greatly reduces the implementation costs by using the same technology for all real-time communication.

MQTT.js

The architecture outlined in the paragraph above can be implemented using the MQTT protocol. MQTT (originally called MQ Telemetry Transport) is a publish-subscribe protocol with an extremely low data overhead (two bytes). It has built-in support for offline messaging and multiple qualities of service levels. Moreover, it can run on both standard TCP/TLS and over WebSocket (ideally HTTPS). It also offers a simple key/value store in the form of retain messages. All these features make it a good protocol to implement IoT systems. I maintain the MQTT.js module. This module allows you to interact with an MQTT broker both from Node.js and from the browser, thanks to Browserify. MQTT.js recently hit version 1.0.0, featuring a massive refactoring and a 2x performance improvement, and the library is now able to receive and send 50k msg/second on my Macbook Pro 2013 (70k msg/second with iojs). Let’s see how we can use MQTT.js to connect to a broker, on any device that can run Node.js. MQTT is also supported on the Arduino, the CC3200, and many more embedded operating systems and development board. Check the related documentation to port these examples to the different development boards. Receiving messages on devices is straightforward: just run “npm install mqtt”, and use the following code:

var mqtt = require(‘mqtt’)
var broker = ‘test.mosca.io’
var client = mqtt.connect(broker)
client.subscribe(‘mydevice/#)
client.handleMessage = function(packet, cb) {
 console.log(packet.payload.toString())
 cb()
}

Publishing is easy too:

var mqtt = require(‘mqtt’)
var broker = ‘test.mosca.io’
var client = mqtt.connect(broker)

client.publish(‘mydevice/hello’, ‘world’)
client.end()

Thanks to libraries like mraa, it is easy to control the Intel Edison or Raspberry PI hardware too:

var mqtt = require(‘mqtt’)
var broker = ‘test.mosca.io’
var client = mqtt.connect(broker)
var mraa = require(‘mraa')
var myOnboardLed = new mraa.Gpio(13)
myOnboardLed.dir(mraa.DIR_OUT)
client.subscribe(‘mydevice/#)
client.handleMessage = function(packet, cb) {
 var payload = packet.payload.toString()
 if (payload === 'on') {
 myOnboardLed.write(1)
 } else {
 myOnboardLed.write(0)
 }
 cb()
}

Keeping our devices configured correctly is usually tricky, because a device can be offline at any time. We therefore use retained messages. A retained message is delivered to anybody when they subscribe, such as after a reconnect. You can send a retained message as follows:

var config = JSON.stringify({
 enabled: "false"
})
client.publish("mydevice/config", config, {
 retain: true
})

Then you can process your messages as usual.   Mosca I am also the author of Mosca, a pure-node MQTT broker, that anybody can use to build an Internet of Things system. Mosca is embeddable within any node application, allowing complete customization and the implementation of the pub-sub architecture described above. Now we come to embedding specific behavior inside the broker, for storing each data into a database, having a custom authentication solution, and authorizing each publish and subscribe to the broker. As shown before, every IoT system needs this feature, and adding them to Mosca is extremely easy, as we will see in the following. This is how you process messages to store in a time series database such as InfluxDB:

var mosca = require(‘mosca’);
var server = new mosca.Server(settings);
var timeseries = require(‘./timeseries’);
function published(packet, client, callback) {
 timeseries.store(packet, callback);
}
server.published = published;
This is a custom authentication solution:
var mosca = require(‘mosca’);
var server = new mosca.Server(settings);
// Accepts the connection if the username
// and password are valid
function authClient(client, user, pass, cb) {
 var authorized = (
 user=== 'alice' &&
 pass.toString() ==) 'secret' );
 if (authorized) client.user = user;
 callback(null, authorized);
}
server.authenticate = authClient;

Below is an example of custom authorization to specific topics, both on publish and on subscribe:

var mosca = require(‘mosca’);
var server = new mosca.Server(settings);
function authPub(client, topic, payload, cb) {
 var ok = client.user == topic.split('/')[1];
 // we can alter the message here
 if (ok) callback(null, payload);
 else callback(null, false);
}
server.authorizePublish = authPub;
function authSub(client, topic, cb) {
 var ok = client.user === topic.split('/')[1];
 cb(null, ok);
}
server.authorizeSubscribe = authSub;

Lastly, Mosca supports a wide range of configuration options, from clustering on top of Redis or MongoDB to WebSocket support. Check out it’s wiki here for documentation. In this article, we dug deep into how to build an effective IoT System leveraging Node.js, MQTT and Mosca. If you want to know how to deploy a high availability cluster using nscale, our deployer product, check out this article. Credits: Icons from FontAwesome

This blog post is a write-up of my talk “Enterprise IoT Systems with Node and MQTT” at NodeSummit (San Francisco, 10-11 February 2015).

Want to work for nearForm? We’re hiring.


Email hello@nearform.com

Twitter @nearform

Phone +353-1-514 3545

Check out nearForm at www.nearform.com.


 

By: Matteo Collina

Matteo is a software engineer and Internet of Things (IoT) expert with a passion for coding, distributed architectures and agile methodologies. He has worked with a wide range of technologies (Java, Ruby, Javascript, Node.js, Objective-C) in a variety of fields. Matteo is the author of the Node.js MQTT Broker, Mosca, the LevelGraph database, and co-author of the book "Javascript: Best Practices" (FAG, Milan). In 2014, Matteo completed his PhD with a thesis entitled 'Application Platforms for the Internet of Things: Theory, Architecture, Protocols, Data Formats and Privacy'. Matteo is also an experienced conference speaker on the above topics. In his spare time, he builds and contributes to open source software (see his github profile https://github.com/mcollina) and sails the Sirocco.
  • Gaurav Mishra

    I was indulged in web development management for quite some time now and was in development in 2002-05. I would like to work with open source software, hardware and language to work in the field of IoT. we zeroed-in on Node.JS or Python for programming (they are easy to learn and understand so hopefully will be easy to implement too), I heard alot abt Node.JS’s advantages in past few days now.
    I have one problem with the architecture. I can make the webpage to display the data and can use one engine to analyse data before displaying. But,

    – Which database to be used? (NoSQL, MongoDB or that time stamped database (are they both different? does data goes to the MongoDB and to the time-stamp database separately at the same time and then is fetched by the analyzing system and then will be displayed to the webpage?

    –Will not the calls to the database engines be increased in this case?

    Moreover, we want to:
    – create the engine first, the one which can accept the data from sensors, put it into the database and can attach the analysis engine with it to analyse the data which could then be displayed to one tab of the webpage. How could this be possible with the minimal support. Do we need OS (linux (yocto) preferably) to do so? or the software which we are going to make in Node.JS will run on Google JS engine V8 and we don’t need any OS to run this engine? I am new to this technology and still trying to figure-it out how it works.

    Please help me regarding this.
    Thanks and Regards,
    Gaurav Mishra
    (MSc Computer Applications (System Administration), Symbiosis, Pune)

    +91-8800955890
    Delhi-NCR, India

    • http://gk.palem.in/ Gopalakrishna Palem

      Gaurav, The selection of DB and the other architectural pieces are typically governed by your problem domain, nature of the data and the type of analysis you are interested in performing on top of it (real-time vs batch) etc.

      For example, here is one sample M2M Predictive Maintenance application (Automotive) we built using realtime data collection from sensors and analyzing them in realtime for failure rate prediction and condition-based management: https://www.youtube.com/watch?v=GUg4FjhTRS0

      The above is for Automotive domain. Similarly, recommendation engines for retail industry, Tele-medicine for healthcare etc. each have their own architectural requirements.

      Ping me on Skype (id: Gopalakrishna.Palem) when you are free – we can discuss.

      Thanks,
      GK
      Big Data, IOT Consultant
      http://gopalakrishna.palem.in/

  • Shubha Sudhir

    Is there a generic service that subscribes to MQTT broker and saves messages to LMDB. I am aware of a similar service for MongoDB. https://github.com/izmailoff/mqtt-mongo

    • Matteo Collina

      You can easily add that behavior with the on(‘published’) event.

  • http://BigSonata.com/ Big Sonata

    @disqus_zpVIN4eojD:disqus Great works! Thanks for sharing.