Pinnacle™ 100 Modem Integration Guide

In this guide we will walk through the steps for integrating the Memfault Firmware SDK into any project using Ezurio's Pinnacle™ 100 Cellular Modem (i.e Sentrius™ MG100).

tip

The Pinnacle-100-Firmware has built-in support for the Memfault Firmware SDK. Ezurio has released some excellent documentation on how to integrate Memfault and the AWS Out-of-Box demo includes a full Memfault integration that can be tried out. We recommend following both this documentation page as well as Ezurio's.

Upon completion of the integration, the following subcomponents will be added to your system!

• Reboot Reasons
• Trace Events
• RAM-backed stack dump Collection

Integration Steps

This tutorial assumes you have a working environment for building the Pinnacle 100 Firmware (such as an application based on the AWS Out-of-Box Demo

Create a Memfault Account

The integration requires a Memfault cloud account to process data collected by the Memfault Firmware SDK. You can create one for free from here!

Update Kconfig options

Add or update the appropriate options in your system's prj.conf:

# Project-specific configuration settings

CONFIG_MEMFAULT_NCS_PROJECT_KEY="YOUR_PROJECT_KEY"
CONFIG_MEMFAULT_NCS_FW_TYPE="pinnacle-fw"

Upload Symbol File

At this point, you should be able to generate test events and crashes and push the data to the Memfault UI.

You can confirm the error traces and crashes have arrived successfully by navigating to the "Issues" page- there should be a new issue with the "Symbols Missing" label:

Clicking into the Issue will show the trace and a message that the symbol file is missing. It can be uploaded by clicking the button highlighted below:

After this step, you will see the processed trace in the list of issues!

note

The Issue created when the symbol file is missing can now be set to resolved. It's good practice to always upload a symbol file before devices send any data.

Symbol files can also be uploaded from the Software → Symbol Files tab (follow this deep link to be brought to the symbol file upload point in the UI):

In addition to supporting decoding trace/coredump data, symbol files are also required for decoding Metrics.

tip

You can programmatically upload symbol files with the Memfault CLI tool.

Symbol files should be uploaded to Memfault before devices send any data, to ensure all device data can be decoded.

Publish data to the Memfault cloud

Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.

HTTPS

The integration will automatically push data to Memfault over HTTPS when an LTE connection becomes available prior to connecting to AWS IoT allowing one to immediately get information about unexpected resets as well as debug any connectivity issues with AWS IoT.

MQTT

After a connection with AWS IoT is established, Memfault data is published to the following topic.

prod/<board>/<device_id>/memfault/<memfault_project_key>/chunk

Forwarding data to Memfault from AWS IoT is very easy! You just need to configure an AWS IoT Rule for the topic and connect it to an AWS Lambda function. This can either be done manually in the AWS console or programmatically using CloudFormation.

CloudFormation

caution

These steps assume you have installed the AWS CLI and have set up credentials to access your AWS instance.

1. Copy and paste the contents below into cfnTemplate_memfault.yml
2. Deploy the configuration

$ aws cloudformation deploy --template cfnTemplate_memfault.yml --stack-name memfault-pinnacle-100-chunk-proxy --capabilities CAPABILITY_NAMED_IAM

Waiting for changeset to be created..
Waiting for stack create/update to complete

cfnTemplate_memfault.yml file contents

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Memfault Pinnacle 100 Chunk Proxy
Parameters:
  ApplicationName:
    Type: String
    Default: memfault-chunk-proxy

Resources:
  LambdaIotRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${ApplicationName}-lambda-iot-role
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: !Sub /${ApplicationName}/
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AWSIoTFullAccess
      Policies:
        - PolicyName: LambdaIotPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource:
                  - arn:aws:logs:*:*:log-group:/aws/lambda/*

  #
  # Proxy Memfault "chunks" to Memfault cloud for processing
  # For more details about data transport see https://mflt.io/data-to-cloud
  #
  MemfaultProxyFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: "MemfaultPinnacle100ProxyFunction"
      Code:
        ZipFile: |
          const https = require('https')
          exports.https = https

          exports.handler = (event, context, callback) => {
            const data = Buffer.from(event.data, 'base64');
            // Topic Format: prod/<board>/<device_id>/memfault/<memfault_project_key>/chunk
            const topicParams = event.topic.split('/')
            const deviceSerial = topicParams[2]

            const options = {
              hostname: 'chunks.memfault.com',
              port: 443,
              path: `/api/v0/chunks/${deviceSerial}`,
              method: 'POST',
              headers: {
                'Content-Type': 'application/octet-stream',
                'Content-Length': data.length,
                'Memfault-Project-Key': topicParams[4]
              }
            }

            const req = https.request(options, res => {
              const response = {
                statusCode: res.statusCode
              };
              callback(null, response);
            })

            req.on('error', error => {
              console.error(error)
            })

            req.write(data)
            req.end()
          }
      Handler: "index.handler"
      Role: !GetAtt LambdaIotRole.Arn
      Runtime: nodejs12.x
      Timeout: 15
      Environment:
        Variables:
          ApplicationName: !Ref ApplicationName

  MemfaultProxyFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    DependsOn: MemfaultProxyFunction
    Properties:
      RetentionInDays: 14
      LogGroupName: !Join ["", ["/aws/lambda/", !Ref MemfaultProxyFunction]]

  MemfaultProxyFunctionPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt
        - MemfaultProxyFunction
        - Arn
      Principal: iot.amazonaws.com
      SourceArn: !Sub arn:aws:iot:${AWS::Region}:${AWS::AccountId}:rule/${MemfaultProxyRule}

  MemfaultProxyRule:
    Type: AWS::IoT::TopicRule
    Properties:
      RuleName: "MemfaultPinnacle100ProxyRule"
      TopicRulePayload:
        AwsIotSqlVersion: "2016-03-23"
        RuleDisabled: false
        Sql:
          !Sub SELECT encode(*, 'base64') AS data, topic() AS topic FROM
          'prod/+/+/memfault/#'
        Actions:
          - Lambda:
              FunctionArn: !GetAtt MemfaultProxyFunction.Arn

Manually

Create AWS Lambda

First you will need to create an AWS Lambda to forward data to Memfault.

const https = require("https");
exports.https = https;

exports.handler = (event, context, callback) => {
  const data = Buffer.from(event.data, "base64");
  // Topic Format: prod/<board>/<device_id>/memfault/<memfault_project_key>/chunk
  const topicParams = event.topic.split("/");
  const deviceSerial = topicParams[2];

  const options = {
    hostname: "chunks.memfault.com",
    port: 443,
    path: `/api/v0/chunks/${deviceSerial}`,
    method: "POST",
    headers: {
      "Content-Type": "application/octet-stream",
      "Content-Length": data.length,
      "Memfault-Project-Key": topicParams[4],
    },
  };

  const req = https.request(options, (res) => {
    const response = {
      statusCode: res.statusCode,
    };
    callback(null, response);
  });

  req.on("error", (error) => {
    console.error(error);
  });

  req.write(data);
  req.end();
};

Create AWS IoT Rule

Next you will need to create a new AWS IoT Rule:

SELECT encode(*, 'base64') AS data, topic() AS topic FROM 'prod/+/+/memfault/#'

Finally, as part of the setup you will need to attach the Lambda you just created to the AWS IoT rule so that the lambda gets triggered each time new data arrives.

Troubleshooting

If you encounter any issues in your data transfer implementation, Memfault has tools to help debug!

• To troubleshoot data not getting uploaded or processed correctly by the Memfault cloud, take a look at the Integration Hub → Processing Log view. This provides a filterable, chronological view of recent errors that have occurred while processing received data. See this documentation page for more information on the Integration Hub.
• A view you can use toview the raw "Chunk" data payloadsthat have arrived for your project.
• Precanned Data Payloads you can pass through your `user_transport_send_chunk_data()` implementation to test data transfer in isolation.
• Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.