sesam_flow
How you can use Sesam to visualize your data with IOT tech - Bouvet Norge blogmask-mobile

How you can use Sesam to visualize your data with IOT tech

Simen Sommerfeldt 
16. okt. 2016

Sesam is ideal for Internet-of-things: A typical IOT project requires you to gather data from many sources. This little example illustrates how you can make a visual alert when data changes - using a Raspberry Pi LED Matrix. And we get to meet a cousin of LISP

Imagine you are asked to install a "snitching tool" in the CEO's office so that he can see whenever people are overpaid, or if his favourites are underpaid! Like so...

Of course, I am underpaid!

What is this? It is a FLOW

A flow involves systems, datasets and pipes
A flow involves systems, datasets and pipes

The employee DATASET is fetched from the database SYSTEM using a PIPE. Another pipe adds a judgement, creating another dataset. Whenever this changes, a third pipe sends it off with JSON to the node-red system. Finally, a flow in Node-RED concatenates the first name with the judgement, and displays it.

The whole thing took me 15 minutes to make. Like Node-RED, Sesam is ideal to experiment with data and flows.

Getting Sesam

Sesam is available as a downloadable appliance and as a cloud service. These are in private beta and access can be requested by contacting info@sesam.io.

Defining the Database and Node-RED systems

Sesam has a lot of security features, and you can hide passwords and such using secrets. Since this is running locally, I go for the sharing policy :-)

Modell over med Sesam som Datahub

The systems as displayed in the overview pane. They are defined thus:

{
  "_id": "mysql",
  "type": "system:mysql",
  "charset": "utf8",
  "database": "emptest",
  "host": "mysql",
  "name": "Database",
  "password": "simen",
  "pool_max_overflow": 10,
  "pool_size": 10,
  "pool_timeout": 30,
  "port": 3306,
  "timezone": "UTC",
  "username": "root"
}
{
  "_id": "node-red",
  "type": "system:url",
  "base_url": "192.168.1.20:1880"
}

Getting the data

The mysql->employee pipe is shown here, with the latest updated record of the employee dataset

Mysql pil employee

This pipe definition is really easy: Just what to get and where to put it. From now on, Sesam will check the table regularly, and update the dataset. Pipes that use the employee dataset as source will only react on changes.

{
  "_id": "mysql->employee",
  "type": "pipe",
  "source": {
    "type": "sql",
    "system": "mysql",
    "table": "employee"
  },
  "sink": {
    "type": "dataset",
    "dataset": "employee"
  },
  "pump": {
    "schedule_interval": 10
  }
}

A pipe that decides if you are overpaid

Here comes the logic that makes the text you saw...

Sequence #278 updated 3 hours ago

This pipe copies the source dataset, and uses the Data Transformation Language transform to add a judgement:

{
  "_id": "judge_employees",
  "type": "pipe",
  "source": {
    "type": "dataset",
    "dataset": "employee"
  },
  "sink": {
    "type": "dataset",
    "dataset": "employee_judged"
  },
  "transform": {
    "type": "dtl",
    "dataset": "dataset-id",
    "name": "Some name",
    "rules": {
      "default": [
        ["copy", "*"],
        ["add", "judgement",
          ["if",
            ["gt", "_S.salary", 800], "overpaid", "underpaid"]
        ]
      ]
    }
  },
  "pump": {
    "schedule_interval": 10
  }
}

DTL is a functional language, and it took me some time to wrap my head around it in a recent project. I guess that's common for all of us with a background in imperative programming. It is a lot of fun once you get the hang of it!

What the entity looks like

Let's look what a judged employee looks like after a couple of changes:

Here we see the system attributes also - used for book-keeping
Here we see the system attributes also - used for book-keeping

Based on the key-value store RocksDB, each dataset keeps previous versions. As you can see, my salaray has been CUT!

The snitch: Sending it off to Node-RED

Employee_tell boss_node_red

The Final Pipe adds a component to the base URL, and sends it off to the HTTP listener. In a real-life scenario you can fine-tune how Sesam will behave if the target system does not answer, set a number of retries, and eventually specify a dead letter dataset, so you can catch which updates it missed. dataset, so you can catch which updates it missed.

The "tell_boss" pipe:

{
  "_id": "tell_boss",
  "type": "pipe",
  "source": {
    "type": "dataset",
    "dataset": "employee_judged"
  },
  "sink": {
    "type": "json",
    "system": "node-red",
    "url": "/employee"
  },
  "pump": {
    "schedule_interval": 10
  },
  "batch_size": 1
}

The batch_size allows us to control how many updates will be sent in each chunk.

Enter Node-RED and Sense-HAT

The Sense-HAT allows me to do a number of things, here I just use it as a marquee showing the text that it receives. What you see below is a Node-RED flow on my Raspberry.

Flow diagram
Node-RED with Sense HAT. The delay node in front of the http response makes Sesam wait for each update, so that the text gets a chance to be displayed

Now let's look at the data it receives and the code in the "Check & Set" node:

Skjermbilde-2016-10-16-kl.-21.59.19
The message as it arrives in Node-RED

Depending on the values of the _deleted and _previous flags, the recieving system gets a heads-up on what has happened with the data - and can react accordingly. If the _deleted flag is set, it was - yes you guessed it - deleted. If the _previous value is null, it is a new entry, otherwise this is an update.          

A little Javascript in the end

Now, let's look at the code in the "Check & Set" function node. Sesam always sends data in arrays, and as you saw in the pipe deifintion, I have already made sure that it only receives one element in each HTTP request. So all I do now, is to concatenate the values that the boss wants to see:

if((Object.prototype.toString.call( msg.payload ) === '[object Array]' )) {
    if (msg.payload.length>0){
        entity=msg.payload[0];
        msg.payload=entity.first_name+": "+entity.judgement
        return [msg,null];
    }
}
return [null,msg];

That's all there is to it...

This whole thing took me FIFTEEN MINUTES to make! As you can see, Sesam allows you to rapidly prototype stuff - and put it in production - perhaps replacing the Node-RED part with something more robust. If you want to know more about Sesam and Internet-of-Things, please have a look at the NDC presentation I gave last year (Slideshare)

 

Temaer