Skip to main content

Data from Firmware to the Cloud

Introduction​

The Memfault SDK will collect data from your devices in the field such as coredumps, Heartbeats and events, which need to be sent to the Memfault cloud for analysis.

The ways in which devices get data back to the internet varies a lot. Some devices have a direct internet connection, for example, through an LTE or Wi-Fi modem. Others are indirectly connected and send data back through a "gateway", for example by connecting over Bluetooth to a phone app that relays the data back to the internet.

To make the integration as easy as possible while catering to as many different connectivity paths, the Memfault Firmware SDK contains a "data packetizer". The data packetizer breaks up all data that the SDK needs to send out (heartbeats, coredumps, events, etc.) into smaller pieces called chunks. The chunks can be sized as large or small as required to match the capabilities and constraints of the device and its connectivity stack.

Each of these chunks need to be posted to Memfault's chunks HTTP API. The buffering, reassembly and interpretation of the chunks is done by the Memfault cloud.

Building the path between getting chunks from the SDK to posting them to the HTTP API is the only integration work which needs to be done to get the data.

Implementation Notes​

Reliability​

The mechanism to send the chunks back to the Memfault cloud will need to be reliable. By that we mean that data integrity is checked and that chunks are not dropped unknowingly (and are retransmitted in case of data corruption or drops). Missing data and corrupt data errors will be detected by the Memfault cloud, but those chunks will be discarded.

Frequency​

The device firmware is expected to periodically check whether there is data available and send the chunks out. See the data_packetizer.h header file for the C API.

The Memfault cloud buffers chunks, until the sequence of related chunks are received. However, if it takes a prolonged period of time to post the remainder of the related chunks, the chunks may be dropped. Because of this and to minimize reporting latencies, it is recommended to drain the data packetizer at least daily.

Size​

The smallest allowed chunk size is 9 bytes. That said, it is recommended to use the largest possible chunk size that your transport path allows. Smaller chunk sizes generally equate to slower transfers. The (maximum) chunk size can be changed from chunk to chunk (see the memfault_packetizer_get_next() C API).

Order​

Chunks from a given device need to be posted to the chunks HTTP API sequentially, in the order in which the Firmware SDK's packetizer created the chunks. When posting chunks concurrently, ensure that requests for the same device cannot happen concurrently, to avoid violating the ordering requirement. Out of order chunks will be best-effort reordered on the Memfault server.

  • HTTP requests should not be pipelined. That is, the HTTP response for a POST must be received before the next HTTP request can be sent.
  • In case the Memfault server responds with HTTP status code 503 (Service Unavailable ) or 429 (Too Many Requests), the client must back off and retry the request after the delay specified in the Retry-After response header.

Batching​

To minimize overhead and optimize throughput, batch-upload chunks to the chunks HTTP API using multipart/mixed requests and re-use HTTP connections to Memfault's servers. Similarly, you may want to set the MEMFAULT_EVENT_STORAGE_READ_BATCHING_ENABLED flag. (see memfault_event_storage.c).

HTTP Utilities​

For devices using LTE and Wi-Fi and therefore are able to make the HTTP request directly, the Memfault SDK provides HTTP utilities to help construct the request.

  • Define a global HTTP configuration struct (see http_client.h for the full definition)

    #include "memfault/components.h"

    sMfltHttpClientConfig g_mflt_http_client_config = {
      // retrieve Project Key from: https://mflt.io/project-key
      .api_key = YOUR_PROJECT_KEY,
    }

  • Provide implementations for the platform HTTP C APIs

  • Use the http/util.h header to construct the request headers

  • Invoke the http/http_client.h C API when ready to drain chunks

The SDK provides several implementations of this pattern for common platforms. Refer to these for example usage:

Firmware SDK Example Usage​

Normal Mode​

In this mode a call to memfault_packetizer_get_chunk() always returns a complete "chunk". The size of the "chunk" is completely up to you (just needs to be ≥9 bytes). It is your responsibility to get the "chunk" reliably to the Memfault cloud. Typically, the size of the chunk will align with the MTU size of the underlying transport. Some size examples:

  • For BLE, the "chunk" size may be close to 20 bytes to align with the minimal MTU size (23 bytes)
  • For a network stack, the "chunk" size may be closer to 1500 bytes to align with the size of an ethernet frame

Example Code​

#include "memfault/core/data_packetizer.h"

//! Example usage of "Normal Mode" of Packetizer
//! return true if there is more data to send, false otherwise
bool try_send_memfault_data(void) {
  // buffer to copy chunk data into
  uint8_t buf[USER_CHUNK_SIZE];
  size_t buf_len = sizeof(buf);

  bool data_available = memfault_packetizer_get_chunk(buf, &buf_len);
  if (!data_available ) {
    return false; // no more data to send
  }

  // send payload collected to chunks/ endpoint
  return user_transport_send_chunk_data(buf, buf_len);
}

Parameters​

The memfault_packetizer_get_chunk() API takes 2 parameters:

  • buf a pointer to a buffer
  • buf_len a pointer to a size variable

The buf_len parameter is both an input and an output: it sets the maximum size to be copied into buf, and is set by memfault_packetizer_get_chunk() to the number of bytes copied into buf.

If calling memfault_packetizer_get_chunk() in a loop, be sure to reset the buf_len value on subsequent calls:

uint8_t buf[128];
size_t buf_len;
bool data_available = true;

while (data_available) {
  // always reset buf_len to the size of the output buffer before calling
  // memfault_packetizer_get_chunk
  buf_len = sizeof(buf);
  data_available = memfault_packetizer_get_chunk(buf, &buf_len);

  if (data_available) {
    bool send_ok = user_transport_send_chunk_data(buf, buf_len);
    if (send_ok == false) {
      break;
    }
  }
}

Data Recovery​

If there is a failure when sending data, you can abort the packetization with memfault_packetizer_abort(). This will allow any partially read pieces of data to be re-read in their entirety, resulting in no data loss:

uint8_t buf[128]
size_t buf_len = sizeof(buf);
bool data_available = memfault_packetizer_get_chunk(buf, &buf_len);

if (data_available) {
  bool send_ok = user_transport_send_chunk_data(buf, buf_len);
  if (send_ok == false) {
    // unexpected failure, abort in-flight transaction
    memfault_packetizer_abort();
    return -1;
  }
}

Note, however, that if the chunk that failed to send is:

  • From a data source that only spanned one chunk
  • The last chunk in a series of chunks from a data source

the data cannot be re-read, and memfault_packetizer_abort() will be a no-op.

Detecting Complete Drain​

The memfault_packetizer_get_chunk() API may not completely fill the output buffer. To drain all chunks, it should be called until it returns false. An output buf_len value less than the passed buf_len value should not be used to assume the staged data is all drained:

uint8_t buf[128]
size_t buf_len = sizeof(buf);
bool data_available = memfault_packetizer_get_chunk(buf, &buf_len);

if (buf_len < sizeof(buf)) {
  #error Do not use this as a check for last chunk!
}

Streaming Mode​

note

This mode is only necessary for transports that have a maximum message size smaller than the desired uploaded chunk size. For example, a Zigbee transport with a 20 byte packet size, and a gateway that can handle 1500 byte HTTP POST requests.

It can also be used with an HTTP client that has a small buffer for the request payload but supports a callback for filling more data during the request operation, operating in a "streaming" mode.

This is therefore an optimization for systems with tight connectivity requirements, and is not necessary or recommended for most implementations.

The Memfault packetizer has two API calls that operate as a pair, memfault_packetizer_begin() and memfault_packetizer_get_next():

  • memfault_packetizer_begin(...) lets you configure the operation mode of the packetizer and returns true if there is more data to send
  • memfault_packetizer_get_next(...) fills a user provided buffer with the next "chunk" of data to send out over the transport

In this mode, the packetizer is capable of building "chunks" which span multiple calls to memfault_packetizer_get_next(). This mode can be used as an optimization when a transport is capable of sending messages of an arbitrarily large size.

For example, some use case examples include a raw TCP socket or a serial streaming abstraction such as Bluetooth Classic SPP.

In these situations it's unlikely the entire message could be read into RAM all at once so the API can be configured to split the read of the chunk across multiple memfault_packetizer_get_next() calls.

Memfault requires that a chunk be uploaded in one piece, therefore for this approach, the gateway device will have to reassemble chunks that span multiple calls to memfault_packetizer_get_next().

Example Code​

#include "memfault/core/data_packetizer.h"

//! Example Usage of "Advanced Operation Mode" where a single "chunk" spans multiple calls
//! to `memfault_packetizer_get_next()`. This lets us keep the RAM footprint low while
//! sending large payloads in a single HTTP request
bool send_memfault_data_multi_part(void) {
  const sPacketizerConfig cfg = {
    // Enable multi packet chunking. This means a chunk may span multiple calls to
    // memfault_packetizer_get_next().
    .enable_multi_packet_chunk = true,
  };

  sPacketizerMetadata metadata;
  bool data_available = memfault_packetizer_begin(&cfg, &metadata);
  if (!data_available) {
    // There are no more chunks to send
    MEMFAULT_LOG_INFO("All data has been sent!");
    return false;
  }

  // Note: metadata.single_chunk_message_length contains the entire length of the chunk
  //
  // This is the "Content-Length" for the HTTP POST request to the Memfault "chunks" endpoint
  //
  // When using HTTP directly on your device, this is where you would start the http request and
  // build the headers. Or, tell a gateway device doing the HTTP request how much data will be
  // posted:
  send_data_start_over_socket(metadata.single_chunk_message_length);

  while (1) {
    uint8_t buffer[20];
    size_t read_size = sizeof(buffer);

    // We pass in the buffer to fill and the size of the buffer.
    // On return, read_size will be populated with how much data was actually written
    eMemfaultPacketizerStatus packetizer_status = memfault_packetizer_get_next(buffer, &read_size);
    if (packetizer_status == kMemfaultPacketizerStatus_NoMoreData) {
      // We know data is available from the memfault_packetizer_begin() call above
      // so _NoMoreData is an unexpected result
      MEMFAULT_LOG_ERROR("Unexpected packetizer status: %d", (int)packetizer_status);
      break;
    }
    // This is the call to a system specific function for sending data over the transport
    send_data_over_socket(buffer, read_size);

    if (packetizer_status == kMemfaultPacketizerStatus_EndOfChunk) {
      // We have reached the end of the chunk. In this call, let the gateway know that this is the end
      // of the chunk, so it knows to package all the data received since last time as a single chunk.
      send_data_end_over_socket();
      break;
    }
  }

  return true;
}

Forwarding Chunks for a Downstream Device​

In some cases, a device may not have direct internet connectivity and needs to route chunks through a gateway device.

Two options are available when forwarding chunks:

  1. Set the uploading Device Serial to match that of the downstream device.

    When using the HTTP client provided by the Memfault SDK, this can be done by setting the g_mflt_http_client_config.get_device_info to a function that returns the downstream device's serial.

  2. Enable #define MEMFAULT_EVENT_INCLUDE_DEVICE_SERIAL 1 in the downstream device's Memfault platform configuration. This will include the device serial in the chunked messages generated by that device, which overrides the uploading Device Serial used by the gateway device. This adds the overhead of including the device serial in the chunked messages.

    When uploading this data, instead of using the Device Serial in the request, a UUID should be used for all chunks in the same stream, eg:

    https://chunks.memfault.com/api/v0/chunks/6f2bd2e3-99b1-4bc4-8394-5233b4b54a05

    Memfault's backend uses the UUID to correlate chunks for reassembly.

Troubleshooting​

If you encounter any issues in your data transfer implementation, Memfault has tools to help debug!

  • To troubleshoot data not getting uploaded or processed correctly by the Memfault cloud, take a look at the Integration Hub → Processing Log view. This provides a filterable, chronological view of recent errors that have occurred while processing received data. See this documentation page for more information on the Integration Hub.
  • A view you can use toview the raw "Chunk" data payloadsthat have arrived for your project.
  • Precanned Data Payloads you can pass through your `user_transport_send_chunk_data()` implementation to test data transfer in isolation.
  • Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.
Tags:
Last updated on
This page is human-authored. If you see any errors, outdated information, or any other suggestions for improving the contents, please contact us!