OTA - Updating Multiple Components

This guide is intended for users who have already integrated the Memfault SDK into their firmware and require more complex OTA functionality than the "single device image" strategy provides.

note

This guide should be used in conjunction with the Web App → Over-the-Air Updates (OTA) platform guide, which describes in general how Memfault's OTA system operates.

For standard single-image MCU OTA, please see the guide at MCU → Over-the-Air Updates (OTA).

Background

Memfault's OTA solution is designed to be flexible enough to support a wide range of use cases, from simple single-image OTA to complex multi-component OTA.

OTA configuration depends heavily on the device's OTA requirements. For example, a device with a single MCU and a single image will have different requirements than a device with multiple MCUs and multiple images, and a device that requires delta upgrades will have different requirements as well.

Multi-Component OTA

Some devices have multiple components that require OTA, for example a device containing 2 MCUs:

• an application processor
• a modem chip

In this case, Memfault recommends using a single OTA payload containing the updates for all updatable components in the system (_note: URL below is not representative of a real OTA payload URL):

As compared to updating components individually, using a combined OTA bundle reduces the number of configurations devices can run in the field, making testing easier and reducing complexity.

An additional constraint on multi-component systems is interoperability between the updatable components. Often these systems will not support downgrading due to the complexity of managing forwards/backwards compatibility in the interfaces between components, as well on each components' on-device configuration:

When representing such a device in Memfault, the software_version reported by the primary device (in the above example, it's the "Main MCU" device) should match the "combined" bundle software version.

It can also be useful to track individual component versions in Memfault, if it's inconvenient to extract the versions from the OTA bundle (or if the individual components are built separately before being combined into a single bundle). These values can be tracked as string attributes in Memfault, for example:

Separate Update Payloads: Combined Bundle and Manifest

If a single monolithic bundle is not practical (due to on-device storage limitations or bandwidth constraints), one strategy is to prepend a manifest to the OTA bundle payload:

The device fetches the manifest section of the OTA payload using HTTP Range Requests, then fetches each subcomponent payload using range requests, based on the offsets in the manifest.

A very simple example implementation of this scheme is as follows (for reference only):

#include <stdint.h>
#include <stdio.h>
#include <unistd.h>

// supported manifest version, in case the structure needs to change
#define BUNDLE_MANIFEST_VERSION 1

enum bundle_manifest_entry_type {
  BUNDLE_MANIFEST_ENTRY_TYPE_MAIN_FW,
  BUNDLE_MANIFEST_ENTRY_TYPE_MODEM_FW,
  BUNDLE_MANIFEST_ENTRY_TYPE_MAX,
};

// note: set to packed for consistent field spacing compared to
struct __attribute__((packed)) bundle_manifest {
  // should be BUNDLE_MANIFEST_VERSION
  uint8_t version;
  // number of entries in the manifest. should always be =
  // BUNDLE_MANIFEST_ENTRY_TYPE_MAX
  uint8_t num_entries;
  // crc32 of the entire manifest
  uint32_t crc;
  struct bundle_manifest_entry {
    // should match enum bundle_manifest_entry_type
    uint8_t type;
    struct version {
      uint8_t major;
      uint8_t minor;
      uint8_t patch;
    } version;
    // offset from the start of the bundle
    uint32_t offset;
    // size in bytes
    uint32_t size;
  } entries[BUNDLE_MANIFEST_ENTRY_TYPE_MAX];
};

int fetch_bundle_manifest(struct bundle_manifest *manifest, const int sock_fd) {
  // fetch the bundle manifest from the remote server
  // return 0 on success, -1 on failure
  // on success, the manifest will be stored in the manifest struct
  int rv = read(sock_fd, manifest, sizeof(struct bundle_manifest));
  if (rv != sizeof(struct bundle_manifest)) {
    LOG_ERROR("unexpected manifest size when fetching: %d", rv);
    return -1;
  }

  // check the version
  if (manifest->version != BUNDLE_MANIFEST_VERSION) {
    LOG_ERROR("unexpected manifest version: %d / %d", BUNDLE_MANIFEST_VERSION,
              manifest->version);
    return -1;
  }

  // verify manifest integrity
  const uint32_t expected_crc32 = manifest->crc;
  manifest->crc = 0;
  const uint32_t computed_crc32 = crc32(manifest, sizeof(*manifest));
  if (expected_crc32 != computed_crc32) {
    LOG_ERROR("manifest has bad crc32: 0x%x / 0x%x", expected_crc32,
              computed_crc32);
    return -1;
  }

  // check the entry count
  if (manifest->num_entries != BUNDLE_MANIFEST_ENTRY_TYPE_MAX) {
    LOG_ERROR("manifest invalid entry count: %d / %d",
              BUNDLE_MANIFEST_ENTRY_TYPE_MAX, manifest->num_entries);
    return -1;
  }

  // confirm there's a matching entry for each type. this also enforces ordering
  // of entries, which is not mandatory for the implementation but makes things
  // simpler.
  for (int i = 0; i < BUNDLE_MANIFEST_ENTRY_TYPE_MAX; i++) {
    if (manifest->entries[i].type != i) {
      LOG_ERROR("mismatched entry: %d/%d", i, manifest->entries[i].type);
      return -1;
    }
  }

  return 0;
}

Separate Update Payloads: Using Hardware Version

Memfault uses a hardware_version identifier to select the correct OTA payload. This value is passed as part of the OTA request, and is intended for cases when multiple hardware revisions of a device are in the field, each needing a unique OTA payload (for example, to support a GPIO pinout change).

This identifier can also be used to select payloads in a multi-MCU system, but it has some caveats; the component OTA release versions need to be synchronized.

For example:

• 1.0.0 is deployed to the field, with a payload for each component:
  • 1.0.0-app.bin
  • 1.0.0-gps.bin

The device can execute component-based OTA following this procedure:

Note that this can get a little more complicated if you need actual hardware revision specific payloads:

• dvt = Design Verification Test hardware revision
• mp = Mass Production hardware revision

Required payloads:

• 1.0.0-dvt-app.bin
• 1.0.0-dvt-gps.bin
• 1.0.0-mp-app.bin
• 1.0.0-mp-gps.bin

In the case where components do not update on every release, this strategy works nicely.

For example:

• 1.0.0:
  • 1.0.0-app.bin
• 1.0.1:
  • 1.0.1-app.bin
  • 1.0.1-gps.bin
• 1.0.2:
  • 1.0.2-app.bin

Memfault's OTA will return the highest active matching release for a device. So for these example payloads + deployed OTA versions, the following requests yield the following response:

current_version	hardware_version	Payload Returned
1.0.0	app	1.0.2-app.bin
1.0.0	gps	1.0.1-gps.bin
1.0.1	app	1.0.2-app.bin
1.0.1	gps	none- up to date
1.0.2	app	none- up to date
1.0.2	gps	none- up to date

This requires the device knowing each subcomponent's installed current_version value.

Separate Update Payloads: Using Memfault Projects

Another strategy for multi-component OTA is to use separate Memfault projects, one for deploying each component OTA payload.

The device makes requests to each project, using the appropriate project key.

This has a substantial downside of requiring duplicate Cohort management on each project, but can simplify the on-device OTA client implementation. And each component can have an independent version sequence, which can be useful.

Not recommended, but useful in some cases.

OTA with Downstream Devices

Some devices don't have a direct connection to the internet, and instead rely on a gateway device for report Memfault data and fetching OTA payloads. In this case, the gateway device can send the OTA request on behalf of the downstream device like so:

The gateway device will have to pass the downstream device's metadata in the OTA GET request:

# downstream device metadata:
$ export DEVICE_SERIAL=DEMOSERIAL
$ export HARDWARE_VERSION=mp
$ export SOFTWARE_TYPE=stm32-fw
$ export CURRENT_VERSION=0.0.0
# now form the OTA request:
$ export MEMFAULT_LATEST_URL_API_ROUTE=https://device.memfault.com/api/v0/releases/latest/url
$ curl -i -X GET "${MEMFAULT_LATEST_URL_API_ROUTE}\
?device_serial=${DEVICE_SERIAL}\
&hardware_version=${HARDWARE_VERSION}\
&software_type=${SOFTWARE_TYPE}\
&current_version=${CURRENT_VERSION}"
  --header "Memfault-Project-Key: ${YOUR_PROJECT_KEY}"

Delta Firmware Updates

In the context of MCU devices, Delta Firmware Updates can be pre-computed before uploading to Memfault.

This enables specific OTA paths, for example a Delta Release that upgrades from 1.0.0 to 1.0.1:

Memfault recommends that the device support both Delta and Full OTA, in the case where a Delta payload is not practical.

Due the nature of pre-computed Delta payloads, each release can generate n Delta Releases:

Release Version	Delta Releases
1.0.0	1.0.0-1.0.1
1.0.1	1.0.0-1.0.2 1.0.1-1.0.2
1.0.2	1.0.0-1.0.3 1.0.1-1.0.3 1.0.2-1.0.3
...	...

If it's necessary (or desirable) to limit the number of Delta Releases, you can use Memfault's data on the number of devices on each release to determine which Delta update paths should be generated:

For example, only generate Delta Releases for the 5 releases with the most devices, and apply a Full Release for all devices on less popular releases.