Payload Parser

Overview

The Payload Parser is a crucial component in IoT device integration, as sensors or external data sources transmit data in various formats such as binary, hexadecimal, JSON, and others. Its primary function is to decode raw data payloads and convert them into a Valeiot Data Point. This ensures that data from any sensor or external component can be easily interpreted, stored, and utilized within the platform as illustrated in the diagram below.

The Payload Parser is a JavaScript script that processes the data from the sensors and converts it into human-readable values. The script follows the protocol specifications provided in the manufacturer's harware documentation, to accurately decode the data fields.

Key Characteristics

• Preprocess: Payload Parsers allow you to transform arbitrary incoming payloads (e.g., custom JSON, binary, or vendor-specific formats) into standardized Valeiot Data Points before they are stored.
• Filtering: Selectively filter or discard specific fields from the payload to prevent unwanted data from being inserted into the device.
• Current Supported Languages: Javascript

Use Cases

• Data Decoding: The Payload Parser decodes the raw data payloads transmitted by sensors or external data sources. This includes converting binary, hexadecimal data into meaningful values such as latitude, longitude, battery voltage, speed, and other device statistics.
• Valeiot Data Point Normalization: Adjusting customized JSON to conform to the Valeiot Data Point structure, ensuring successful insertion into Data Sources.
• Flexibility: Handling various types of uplink messages and can be extended for custom processing before insertion into Data Sources.
• Error Handling: Handling errors or unexpected data formats, ensuring robust operation even in cases of corrupted or incomplete data.
• Customization: The parser can be customized to meet specific application requirements, such as filtering out certain data points or adding additional metadata to the decoded data.

Payload Parser Structure

The Payload Parser has a mandatory entry function called parse, which is executed whenever a request is sent. It should always follow the structure below, with its parameter being the payload, which contains the message sent in the request. Finally, it must return an array of Valeiot Data Points.

function parse(payload) {
  let dataList = [];

  //your logic goes here

  return dataList;
}

Code Validation

A valid parser consists of one or more function declarations, with parse as the entry point that the runtime will call with the incoming payload.

The parser code must satisfy the following rules:

1. No this. references
   The code must not contain the substring this.. Accessing the global object using this is not allowed.

2. Valid JavaScript
   The code must be valid JavaScript. Any parsing error (for example, a syntax error) will cause the validation to fail.

3. Only function declarations at the top level
   The top-level scope of the script must contain only function declarations (e.g., function parse(payload) { ... }).
   Any other statement type at the top level — such as expressions, var, let, const, or assignments — will be rejected.

4. Code Size Limit
   The maximum allowed size for the parser code is 10240 characters.

Native Functions in the Runtime

The following functions are injected into the JavaScript runtime and are available globally.
For security reasons, eval is removed (set to undefined).

base64ToHex(base64String)

Converts a Base64-encoded string into its hexadecimal representation.

• Argument: string (Base64-encoded).
• Returns: string (lowercase hexadecimal).
• Errors:
  • Type error if the argument is missing or not a string.
  • Type error if the string is not valid Base64.

hexToBytes(hexStringOrBytes)

Converts a hexadecimal string into a byte array.

• Argument: string or byte array (hex-encoded).
• Returns: byte array.
• Errors:
  • Type error if the argument is missing or not a string/byte array.
  • Type error if the value is not valid hexadecimal.

bytesToString(bytes)

Converts a byte array into a UTF-8 string.

• Argument: byte array.
• Returns: string (UTF-8 interpretation of the bytes).
• Errors:
  • Type error if the argument is missing or not a byte array.

toDatapoints(obj, [timestamp])

Builds an array of parser datapoints from a key–value object.

• Arguments:

  • obj: object — each key becomes a variable name and each value becomes the value of a datapoint.
  • timestamp (optional): number representing Unix time in seconds. If provided, it will be used as the time for all generated datapoints.

• Returns: array of { time, variable, value } objects.
  If timestamp is not provided, the current server time (RFC3339) will be used.

• Errors:

  • Type error if the first argument is missing or not an object.
  • Type error if the second argument is provided but is not a number (Unix seconds).

This function is useful when the payload is a flat object (for example: { temperature: 22.5, humidity: 80 }) and you want to generate one datapoint for each key.

unixToRFC3339(unixSeconds)

Converts Unix time in seconds to RFC3339 format.

• Argument: number (Unix time in seconds).
• Returns: string in RFC3339 format.
• Errors:
  • Type error if the argument is missing or not a number.

unixMilliToRFC3339(unixMilliseconds)

Converts Unix time in milliseconds to RFC3339 format.

• Argument: number (Unix time in milliseconds).
• Returns: string in RFC3339 format.
• Errors:
  • Type error if the argument is missing or not a number.

Debugging Payload Issues

If the parser is not behaving as expected, inspect the device using the Live Inspector tool.

Navigate to the device page, open the Live Inspector tab, and trigger or wait for an incoming payload request. The Live Inspector captures the raw request and the parser execution results, allowing you to analyze the payload processing step by step and identify potential errors.

Example 1 (simple)

Scenario

A device sends multiple variables in a single uplink. One of the variables is OUTPUT. We want to make the following changes before inserting the data in the data points:

• Change the ouput values from 1 and 0 to ONand OFF respectively
• Change the variable name to out

Solution

Step 1: Create a payload parser with the necessary logic for changing the value and variable from OUTPUT

Two logics has been implemented in the script below, understanding if the output variable exists and if positive, change the data. Then, remove the old data and insert the new one. (ChatGPT enhance this description)

parserExample.js

function parse(payload) {
  let dataList = [];

  dataList = [...payload];

  const output = payload.find((x) => x?.variable === "output");

  if (output) {
    let outputValue = "OFF";

    if (output?.value === 1) {
      outputValue = "ON";
    }

    const outputData = { variable: "out", value: outputValue };

    dataList = dataList.filter((x) => x.variable !== "output");
    dataList.push(outputData);
  }

  return dataList;
}

Copy the script example below and create a new Payload Parser. Then paste the script in Parser code section.

Step 2: Test by inserting a data point on a device using the payload parser created

Go to a device that has the payload parser created assigned to it. Try to create a new data point as "OUTPUT". Then see the new data point registered:

{ "variable": "out", "value": "ON" }

{ "variable": "out", "value": "OFF" }

Example 2 (advanced)

Below is an example of a JavaScript-based Payload Parser, which decodes the payload sent from the device and converts it into Valeiot Data Points:

parserExample.js

function parse(payload) {
  let dataList = [];

  if (typeof payload === "string") {
    const buffer = hexToUint8Array(payload);

    if (buffer[0]) {
      const batteryValue = buffer[0];
      dataList.push({
        variable: "battery",
        value: batteryValue,
      });
    }

    if (buffer[1]) {
      const temperatureValue = buffer[1];
      dataList.push({
        variable: "temperature",
        value: temperatureValue,
      });
    }

    if (buffer[3]) {
      const doorBit = buffer[3] & 0x08;
      let doorState = "";
      if (doorBit > 0x00) {
        doorState = "Closed";
      } else {
        doorState = "Open";
      }
      dataList.push({
        variable: "door_state",
        value: doorState,
      });
    }

    dataList = dataList.concat({ variable: "payload", value: payload });
  } else {
    dataList = dataList.concat(payload);
  }

  const ignoreVars = ["snr", "rssi", "gateway_eui", "fcnt", "fport"];
  dataList = dataList.filter((x) => !ignoreVars.includes(x.variable));

  return dataList;
}

The parser expects a payload as input, using the following example of a payload:

"621b6310";

The expected output is the result of data points:

[
  { variable: "battery", value: 98 },
  { variable: "temperature", value: 27 },
  { variable: "position_state", value: "Open" },
  { variable: "payload", value: "621b6310" },
];

Explanation

The decoding process is free, you can do basically whatever you want with javascript. For the example mentioned, the script decodes the battery, temperature and door state from the payload. It gets the equivalent from hexidacimal to decimal and perform a simple bitmask. It's also trimming a few unwanted variables such as snr, rssi, gateway_eui, fcnt and fport.

function parse(payload) {
  let dataList = [];

  if (typeof payload === "string") {
    const buffer = hexToUint8Array(payload);

    if (buffer[0]) {
      const batteryValue = buffer[0];
      dataList.push({
        variable: "battery",
        value: batteryValue,
      });
    }

    if (buffer[1]) {
      const temperatureValue = buffer[1];
      dataList.push({
        variable: "temperature",
        value: temperatureValue,
      });
    }

    if (buffer[3]) {
      const doorBit = buffer[3] & 0x08;
      let doorState = "";
      if (doorBit > 0x00) {
        doorState = "Closed";
      } else {
        doorState = "Open";
      }
      dataList.push({
        variable: "door_state",
        value: doorState,
      });
    }

    dataList = dataList.concat({ variable: "payload", value: payload });
  } else {
    dataList = dataList.concat(payload);
  }

  const ignoreVars = ["snr", "rssi", "gateway_eui", "fcnt", "fport"];
  dataList = dataList.filter((x) => !ignoreVars.includes(x.variable));

  return dataList;
}

The calculator below shows hexadecimal and binary values.

Battery: 0110 0010 or 0x62 or 98

Temperature: 0001 1011 or 0x1B or 27

Door Status: 0001 0000 which is 0x10 when operating 0x10 & 0x08 the result is 0x00

  00010000   (0x10)
& 00001000   (0x08)
  --------
  00000000   (Result)

This example demonstrates how the Payload Parser can be used to decode data from the device and convert it into Valeiot Data Point format.