PART 3. CONNECTING DEVICES TO THE EVERYWARE CLOUD
Communication Data Model

 

Introduction to EC Data Model

·         Topic Structure

o   Publish Topic

o   Control Topic

·         Payload Structure

·         Rules-based Data Analysis

·         Message Life-Cycle

o   Birth Certificate

o   Death Certificate

o   Last Will and Testament

 

 

 

Introduction to EC Data Model

 

The Everyware™ Communication Data Model builds on the MQTT protocol by adding an open structure to the topic namespace and the message payload.  This addition enables real-time analysis of the data stream published by the devices and establishes a control channel to remotely manage the devices in a command and control pattern.  See the MQTT Introduction for more general details on the protocol.

 

 

Topic Structure

In the Everyware Cloud, there are two types of topics:

 

·         Publishing topic

·         Control topic

 

 

Publish Topic

A Publishing topic is used by a device (or application) to publish data into the Everyware Cloud platform.  The Control topic is used by the application or the platform (or another device) to send data to a device, such as for real-time control of physical control outputs.

 

A generic Publishing topic can be represented as

 

     accountName/assetId/semanticTopic

 

where

      accountName is the name of the Everyware Cloud account, and it acts as an unique identifier that represents a group of devices and users.

      assetId is a unique identifier within an account that represents a particular asset.  Eurotech recommends associating the assetId to a single gateway device and mapping it to the Client Identifier (Client ID) as defined in the MQTT specifications.  For a gateway, the MAC address of its primary network interface is generally used as the Client ID and Asset ID of that gateway.

      semanticTopic is the section of the topic used to further specify information about the application or sensors from which the data has been gathered, using an hierarchical namespace representation.

 

Eurotech suggests that the semanticTopic be further expanded, such as appId/resourceId/additionalTopic, to provide further levels of topic space that can be subscribed to and queried based on data content.  These suggested topic levels are described below:

 

      app_id is an identifier to denote an application running on the gateway device.  Eurotech suggests identifying versions of the application identifier in order to allow multiple versions of the application to coexist, e.g. CONF-V1, CONF-V2, DEPLOY-V1, DEPLOY-V2, etc.

      resource_id allows each application to own and manage one or more resources.  Examples of resources are sensors, actuators, local files, or configuration options.  Applications manage resources by being able to list them, read the latest value, or update them to a new value.  Each resource could be identified by a resource_id as part of the hierarchical topic.  For example, "sensors/temp" or "sensors/hum" can act respectively as resource IDs for a temperature sensor or a humidity sensor.

 

A given gateway identified by assetId (client ID) and belonging to accountName can have one of more applications – “app_id1” and “app_id2” – running.  Each application can manage one or more resources identified by distinct resource_id(s).

 

 

Control Topic

The Control topic is split into two different topics:

 

·         One that represents a control channel to the specific device

·         One that represents a control channel to all the devices belonging to that account

 

These two topics are represented as

 

     $EDC/accountName/assetId/semanticTopic

or

     $EDC/accountName/+/semanticTopic

 

where

 

      $EDC is the default part of the control topic and cannot be changed or altered.  This prefix will distinguish control topics from data topics used in unsolicited reports, and it marks the associated control messages as transient, not stored in the historical data archive if one is present.

      accountName is the name of the Everyware Cloud account owner.

      assetId is a unique ID representing a particular asset (either the application or the sensors from which the data has been gathered).

      semanticTopic is the section of the topic used to further specify information about the device or data, using an hierarchical name space representation.

      + is an operator that aggregates all the asset IDs belonging to a specific account (essentially a wildcard character used in a middle element of the topic namespace).

 

 

Another Control topic in Everyware Cloud that may be used in Rules is:

 

     $accountName/RulesAssistant/semanticTopic

 

where

 

      $accountName is the name of the Everyware Cloud account owner.

      RulesAssistant is specific to the Everyware Cloud rules processing

      semanticTopic is the section of the topic used to further specify information about the device or data, using an hierarchical name space representation.

 

See Using Cloud Rules for more information on rules-based analysis.

 

When creating a new account in the Everyware Cloud, the account holder is given publish and subscribe rights to the Publish and Control topics for that account.  By default, when establishing a new Cloud connection, the Everyware Cloud Client for Java will subscribe to the aforementioned control topics listing for messages that may be addressed to that device.  For more information on connecting to the Everyware Cloud in Java, please see the section Connecting to the Cloud in Java.

 

 

 

Payload Structure

Although the MQTT protocol is agnostic to the payload format, Eurotech has designed an open payload format that is flexible yet efficient when it comes to bandwidth conservation.  The EC Data Model suggests a data structure that is imposed on the MQTT payload.  It is free, is open, and can be adopted by third parties in order to gain real-time analysis as well as discovery on the device data through the Everyware Cloud platform.

 

The EC data model defines a message payload composed by

 

      Timestamp,

      EC Metrics,

      EC Position,

      Body

 

where

 

sentOn Timestamp is the time when the data was captured and sent to the Everyware Cloud platform.

 

A Metric is a data structure composed of the name of the variable, the value of the variable, and the type of the variable.  Each payload can have zero or more metrics.

 

          (Variable_name, Variable_value, Variable_type)

 

The following example represents a temperature (floating value) called temp1 with a value of 43.31:

 

          temp1, 43.31, float

 

Another example follows representing a digital input called din with a value of zero:

 

          din, 0, bool

 

The allowed variable types for EC Metrics are:

·         double – Double precision (64-bit) IEEE 754

·         float – Floating point (32-bit) IEEE 754

·         int64 – 64-bit integer value

·         int32 – 32-bit integer value

·         bool – “true” or “false”

·         string – ASCII string of text

·         byte array – Set of hexadecimal byte values

 

 

EC Position is a predefined data model for Geo location applications, optimizing the amount of data sent.  The EC Position consists of the following fields:

 

 

 

Body is a non-predefined part of the payload that allows additional information to be transmitted in any format determined by the user.  This field will be stored into the platform database, but the Everyware Cloud cannot apply any rules-based statistical analysis on it.

 

 

 

Rules-based Data Analysis

Most of the EC data payload is available for the real-time analysis through the platform rule engine, including the topic, the sentOn timestamp, all the fields of the EC Metrics, and the EC Position (but excluding the optional message Body).  See Using Cloud Rules for more information on the real-time statistical rule engine.

 

Example:

Let’s say that your application captures and sends the temperature and the status of a digital input, but it has no need for reporting a GPS position.  In a human-readable format, your application data model would read as follows:

 

 

When sending messages to the Everyware Cloud, the payload of a message is serialized.  For the serialization process, the Everyware Cloud leverages an open and very efficient algorithm that has been developed by Google and is currently in use by Google’s applications.  The serialization is transparent for the customer and has the goal to reduce the amount of data sent over the network for bandwidth conservation without incurring high serialization and deserialization costs.

 

MQTT and EC Data Communication Model are natively implemented into ESF.  Hence, connecting an ESF-enabled device is straight forward.  Non-ESF-based devices would have to implement MQTT and (by choice of the system designer) implement the EDC message structure and serialize the data using the Google algorithm.

 

 

 

Message Life-Cycle

The message life-cycle for an Everyware Cloud client session is:

 

1.    Connect to broker with MQTT (CONNECT)

2.    Send Birth Certificate (PUBLISH with EdcBirthPayload; if omitted, device will not be shown as active in Everyware Cloud console)

3.    Issue subscription for topics (SUBSCRIBE; optional)

4.    Publish data (PUBLISH; optional)

5.    Send Death Certificate (PUBLISH with EdcDisconnectPayload; if omitted, device will not be shown as cleanly disconnected from Everyware Cloud)

6.    Disconnect from broker (DISCONNECT; if omitted, MQTT Keep-Alive will be used to detect the missing client, and the Last Will and Testament will be sent to notify other subscribers of the broken connection)

 

 

Birth Certificate

One specific published message used in the Everyware Cloud is a “birth certificate.”  The birth certificate is published automatically by Eurotech’s EDC client code immediately after connection and includes information about the client, such as firmware version, model information, operation system (OS) version, latitude, longitude, etc.  The Everyware Cloud platform uses the birth certificate information to display details about the client and to show it active in the Everyware Cloud console.

 

The application can also publish an updated birth certificate at any time, by publishing a message using the EdcBirthPayload() class as the message payload.  This might be used, for instance, to update the GPS position reported in the Everyware Cloud.

 

The Birth Certificate is published on the topic:

 

     •$EDC/accountname/assetId/MQTT/BC

 

The following keys are required in the Birth Certificate payload:

 

 

The EC Position data model for a Birth Certificate is optional, but if used it contains the same fields as the EC Position for the message payload, described in Payload Structure.

 

 

Death Certificate

Another specific published message used in the Everyware Cloud is a “death certificate.”  The death certificate is published automatically by Eurotech’s EDC client code prior to a clean disconnection, using the EdcDisconnectPayload() class.  The Everyware Cloud platform uses the death certificate to update its console and show the device as inactive in the Everyware Cloud console.

 

The Death Certificate is published on the topic:

 

     •$EDC/accountname/assetId/MQTT/DC

 

 

Last Will and Testament

If the Last Will and Testament option was used in the MQTT connection to the broker, and if a device disappears from the network without issuing an MQTT DISCONNECT message, the Last Will and Testament message will be sent by the broker to the topic:

 

     •$EDC/accountname/assetId/MQTT/LWT

 

Any client who wishes to be informed of the broken connection may subscribe to this topic.