Over-the-Air Updates (OTA)

Memfault provides secure and scalable updates for all IoT devices (MCU, Android, Linux, or a combination of them). With real-time monitoring and controlled releases, you can debug and ship new releases confidently.

tip

Memfault uses a globally distributed CDN for low latency and high speed downloads regardless of geolocation. It supports features such as HTTP/2, Brotli and GZip, and byte range requests, among others.

Introduction and Concepts

To deliver OTA Updates:

• Memfault users have different ways to configure which OTA payloads to distribute to which devices.
• Each device polls Memfault's Device OTA Endpoint.
• Memfault uses various inputs, predominantly the current Software Version of the device, to decide what to return.
• Each device may receive information about a new available payload.
• Ultimately, this process is meant to replace or update the software running on a device.

Read about related Memfault-specific terminology that you may need to refer to to better grasp the contents of this document:

• Software Versions and Hardware Versions.
• Version Schemes to understand Software Version ordering.
• Generic Memfault terminology documentation.

tip

Memfault always recommends signing OTA artifacts, to protect devices from installing counterfeit/invalid firmware. Securing OTA artifacts varies by platform.

Android

AOSP builds are always signed as part of the image creation process.

Linux

SWUpdate images for Linux OTA should always be signed.

MCU

Platforms using MCUBoot will have image verification support (including Zephyr-based projects, which has built-in support).

Other platforms may have built-in support for signing and verifying OTA images, or you may need to implement this yourself.

Release Types

Memfault provides two Release entities to solve the challenges of over-the-air updates: Full Releases and Delta Releases.

Full Releases

Full Releases provide a way to upgrade device Software Versions regardless of the version the device is currently on.

If we define the version of the most current Release as 2.0 and every version lower than 2.0 as *, we can describe the behavior of a Full Release as: [*→2.0].

See the updating behavior section for more details.

Must-pass-through Full Releases

Full Releases can be marked as must-pass-through, or MPT for short. When an MPT Release is activated, a device on a lower version will be forced to "pass through" this Release before an update is allowed to a Full Release with a version greater than that of the MPT Release. Typically this setting is not needed.

Delta Releases

A Delta Release describes a transition from exactly one From Version to exactly one To Version. The behavior can be described as: [1.0→2.0].

Note that Delta Releases are referred to as "incremental updates" in Android documentation, or "differential updates" in other places.

A Delta Release is only applied when the device's current version matches the Release's From Version.

Delta Releases allow for more fine-grained control over the update paths of devices. Additionally, Delta Releases generally contain smaller payloads, reducing required bandwidth and costs associated with rolling out updates.

See the updating behavior section for more details on Delta Releases.

Another good place to learn more about delta updates for MCU-based devices in our Interrupt blog post Saving bandwidth with delta firmware updates.

Memfault recommends that the device support both Delta and Full OTA, in the case where a Delta payload is not practical.

Due the nature of pre-computed Delta payloads, each release can generate n Delta Releases:

Release Version	Delta Releases
1.0.0	1.0.0-1.0.1
1.0.1	1.0.0-1.0.2 1.0.1-1.0.2
1.0.2	1.0.0-1.0.3 1.0.1-1.0.3 1.0.2-1.0.3
...	...

If it's necessary (or desirable) to limit the number of Delta Releases, you can use Memfault's data on the number of devices on each release to determine which Delta update paths should be generated:

For example, only generate Delta Releases for the 5 releases with the most devices, and apply a Full Release for all devices on less popular releases.

Activating a Release

You can activate a Release for a Cohort of your choice. While Releases are shared between all Cohorts, activating them happens on a per-Cohort basis.

When activating a Release for a Cohort, you can choose between two behaviors:

Normal Activation of a Release

Releases activated normally are available immediately to all devices in a Cohort.

Staged Rollout of a Release

Staged-rollout Release Activations can be used to make a Release available to a Cohort incrementally.

When a staged-rollout Release Activation is created for a Cohort, a random subset of its devices (amounting to the selected rollout percentage) will be marked as staged in this Cohort. Staged-rollout release activations are only applicable to devices that are staged in the Cohort.

info

The number of devices marked as staged is calculated only when this percentage is configured, and not continuously adjusted. If devices are added to or removed from a Cohort, the actual percentage of devices marked staged may differ from the percentage previously configured.

Best Practices When Activating Releases

Memfault enables risk mitigation through different mechanisms.

Monitoring the Effects of Releases

As part of your OTA management workflow, make sure to set up Metric and Issue Charts to monitor the health of your fleet. You can use Comparison Mode to see how devices in a different Cohort or in a newer Software Version are doing relative to other devices.

Activating Releases in a test Cohort first

Since Releases are shared among all Cohorts, you can use the Release activation mechanism to test a Release in (for example) a beta Cohort before activating the same Release in a production Cohort. Once you've gained confidence through monitoring that the software in a new Release is working as expected, you can activate the Release for a bigger Cohort.

Using staged-rollout Releases

Releases activated using a staged rollout can be used to limit the amount of devices that will download a specific Release, reducing the risk of shipping bugs to a greater portion of the fleet. Once you've gained confidence through monitoring that the software in a new staged-rollout Release is working as expected, you can increase its rollout percentage.

Deleting a Release

Releases can be deleted when they are no longer needed. Before a Release is deleted, all Release Activations must be marked as Pulled.

Follow the steps below to ensure that all Activations are marked as Pulled and to ultimately delete the Release.

1. Navigate to the specific Release that should be deleted.

2. In the top right, under Release Activations, for any entries that are not Pulled, click the Edit button and Abort them.

   

3. Once all Activations have been marked as Pulled, click Delete on the Release page and proceed through the confirmation.

Note that if all Activations are not marked as Pulled on a Release, an error will occur when trying to delete the Release.

Updating Behavior

When a device queries the Device OTA Endpoint, the Memfault platform needs to decide which OTA payload to send it. This section covers the behavior of our backend when receiving one such query.

In this section, we'll sometimes refer to one Software Version being higher or lower than another. To understand how Memfault resolves Software Version ordering, refer to the Version Schemes documentation section.

info

If you wish to better understand Memfault's behavior from the point of view of a specific device, you can use the "Inspect details" feature in the device details view.

Applicability of an OTA Release

Along with the query to the Device OTA Endpoint, a device sends:

• the Device ID (often the device serial number) to determine its Cohort,
• the Software Type to determine the set of Releases against which to apply these rules,
• its current Software Version,
• device-specific attributes such as Hardware Version.

A Release must be active for the Cohort a device is in to be applicable.

To determine if an active Release is applicable to a device, its payload attributes are tested against the parameters sent by the device. If they don't match, a Release is considered to be incompatible and treated as if it did not exist. Any existing device in a given Cohort or new device that joins a Cohort (incl. devices first activated and auto-joining the default Cohort) will be evaluated against the current set of active releases in the Cohort and receive updates accordingly.

An Release activated as a staged-rollout will only be applicable to the staged portion of the Cohort it targets. In other words, if the device is not staged in its Cohort, such Releases will not be applicable.

Choosing Among Multiple Applicable OTA Releases

If a device has multiple possible update paths, the following rules of precedence apply to all active Releases per Cohort to determine which release is selected:

1. Use a Delta Release whose From Version matches the current version of the device.
2. Otherwise, use a must-pass-through (MPT) Release with the lowest Target Version that's greater than the current version of the device.
3. Otherwise, use a Full Release with the highest To Version greater than the current version of the device.

important

The precedence can be roughly summarized as: Delta Release > MPT Release > Full Release.

Due to this precedence, devices can "bypass" must-pass-through Releases via a Delta Release. Consequently, a Delta Release must include all information to get from version A→B.

This is especially useful for defining upgrade paths from factory versions to the latest major version (or similar).

An example:

Allowing Downgrades

For Full Releases, downgrades are disallowed by default. This is to prevent a device from downgrading to a version that is older than the one it is currently running, which may cause issues.

But there are times when a device or set of devices do need to be downgraded to a previous OTA Release. There are two ways to accomplish this. The first one is to "downgrade by upgrading" and the second is to use the Disable Version Checking feature of Memfault's Cohorts.

Let's work through an example, where a device is running Release 1.0.1 but needs to be downgraded to the OTA payload served in Release 1.0.0. Pulling the 1.0.1 Release by itself does not start returning the 1.0.0 as the latest Release for devices, as this would potentially cause a downgrade situation!

Downgrading by Upgrading

The recommended way to downgrade devices is to rebuild the old OTA payload exactly as it was, but also update the version that is used with the OTA image and during the creation within Memfault.

Following our example, an engineer would check out the 1.0.0 codebase, update the version "1.0.0" to "1.0.2", and build the OTA image. They would then upload this payload under a "1.0.2" Release, and Activate the Release.

With this in place, device will check in and report that they are on 1.0.1, be served the 1.0.2 OTA, and install it. Success!

Disabling Version Checking on a Cohort

The other way to downgrade a set of devices is to use Memfault's feature called Disable Version Checking. This will bypass Memfault's built-in protection from returning a version 1.0.0 when the device reports that it is already running 1.0.1. Note that even though 1.0.0 will be returned from Memfault, the underlying operating system installing the OTA payload might reject it due to it being a downgrade.

The steps that are typically taken to accomplish this are:

• Move devices that need to be downgraded from 1.0.1 to 1.0.0 to a new Cohort.
• Deploy 1.0.0 to this new Cohort.
• Enable the setting Disable Version Checking.

Devices in this new Cohort that report that they are running 1.0.1 will now be given the 1.0.0 Release.

Note that the system installing this Release must still be capable of performing this downgrade. It can potentially be a dangerous operation as many persistent storage systems are not both backward and forward compatible.

info

For Delta Releases, downgrades are currently not supported.

Overriding the effective Cohort for a specific Project Key

You can specify an OTA Cohort Override for a specific Project Key in the Project Key settings

Some potential use cases for this feature are:

• Configuring a different Project Key for devices enrolled in a beta program. By setting a Cohort override on the default, non-beta Project Key, you can ensure that reset devices will receive stable releases regardless of the stored device Cohort.
• Downloading recovery firmware using a different Project Key
• Configuring a different Project Key when a device boots up in recovery mode

Note that this setting only affects OTA update behaviour, and has no effect on the device's stored Cohort at this time.

important

If a device was previously marked as staged in its actual Cohort as part of "configuring a staged rollout", it will still be marked as such when using an OTA Cohort override. Take note that it would have access to staged releases in the overriding Cohort as well.

Managing OTA Updates with Memfault

The Memfault application contains OTA-related features in several of its views.

OTA Releases

The OTA Releases section of the Memfault application can be accessed from the main menu on the left side. In it, you'll be able to access a list view of all Full Releases and the equivalent list view for Delta Releases by selecting one of the tabs under the title of the page.

Let's take a look at how to create Releases using the Memfault application.

important

Please note that the Release version you choose is expected to match the Software Version a device will report once it has installed the OTA payload for this Release.

caution

If you're on Android, we recommend following the Android-specific OTA guide in order to create Releases.

Creating a Full Release

To create a Full Release, click on the Create Release button on the top-right area of the OTA Releases view. Among other information, you'll be prompted for a version and whether the Release is a must-pass-through (MPT) Release.

Creating a Delta Release

To create a Delta Release, click on the drop-down icon next to the Create Release button on the top-right area of the OTA Releases view, and select Create Delta Release. Among other information, you'll be prompted for a From Version and a To Version.

Adding an OTA payload to a Release

Once you've created a Release, you'll be redirected to the Release detail view. To add an OTA payload, click on the Add OTA payload to Release button on the top-right corner of the OTA payloads card.

Alongside the actual payload file, you'll be prompted for different inputs that a device querying the Device OTA Endpoint must match for this Release to be applicable to it.

caution

If you're on Android, you'll need to follow the Android-specific OTA guide to upload an OTA payload, since it can only be done using the Memfault CLI.

See the updating behavior section for more details.

Cohort Version Management

To dive into your OTA configuration for a specific Cohort, you can access the Cohort detail view by first opening the Cohorts list view from the main menu, and then selecting the Cohort you're interested in.

Under the Version Management tab, you'll find the version matrix: a live view into the very next OTA update that each device in this Cohort will receive.

The version matrix consists of rows representing current versions that are currently reported by devices in the field. Its columns represent future versions devices of the Cohort will report (after having applied OTA releases). What follows is a tour of the different interface elements of the version matrix.

In each cell of the version matrix you'll find a count of devices. If, for example, a device count of 10K showed up on a cell in the intersection of row version 1.1 and the column 2.0, that would mean 10K devices are currently on version 1.1 and the immediate next step they're going to take is to download an OTA payload for a Release with version 2.0.

Conversely, if 5K devices were to show up on a cell in a row with version 1.0 and under the column for version 1.0, that would mean those 5K devices will stay on version 1.0.

All device counts on the version matrix are clickable, and will take you to a complete list of the devices represented under that count.

Figure 1: Full Releases highlighted on the version matrix.

Figure 1 highlights the area of the version matrix that shows future versions. In this example, three Full Releases exist (for versions 1.0, 2.0 and 1.1), but only Full Releases 1.0 and 2.0 are active. This is represented by the blue checkbox icon . The dotted line around the icon for the 2.2 version means that there is no Full Release with the same version.

The icon (which conventionally means "final state") on top of the 2.2 version shows us that 2.2 is a final target version for this Cohort. Eventually, all devices in this Cohort will end up on this version 2.2.

Figure 2: Delta Releases highlighted on the version matrix.

Figure 2 highlights the area of the version matrix that represents Delta Releases. The Current Version column is where you may see the From Version of a Delta Release, and the Target column is where the Target Version of a Delta Release can be found. An empty drop-down icon indicates that there is no active Delta Release from the row's version. In the picture, only a Delta Release (marked with a ) from 2.0 to 2.1 is active .

Figure 3: Full Releases and Delta Releases on the version matrix.

Figure 3 shows the following scenario:

• Two active Full Releases exist (1.0 and 2.0) and there's one active Delta Release from 2.0 to 2.1.
• Full Release 1.0 is currently not applicable to any device.
• 21K devices currently on 0.9 are going to download a Full Release to 2.0.
• 36K devices currently on 1.1 are going to download a Full Release to 2.0.
• 132K devices currently on 2.0 are going to download a Delta Release from 2.0 to 2.1.
• 738K devices are already on 2.1. They will not receive any updates and will stay on version 2.1.

Figure 4: a staged rollout on the version matrix.

Figure 4 shows the addition of a staged rollout of a Delta Release from version 2.1 to version 2.2. The percentage icon indicates a staged rollout. In this scenario, only 5.9K devices out of a total of 54K that are on version 2.1 will download the payload from Delta Release from 2.1 to 2.2.

Managing OTA Updates

The version matrix in the Cohort Version Management tab can also be used to manage your OTA Releases.

Figure 5: highlighted actions on the version matrix.

Figure 5 highlights the location of buttons on the version matrix that allow for fine-grained control of OTA Releases in a specific Cohort.

You can manage Full Releases by using the controls arranged horizontally on the top side of the version matrix:

• Click on the plus icon on the right end to create a new Full Release.
• Use the highlighted buttons under each future version to create a Full Release targeting this version, or to create a Release Activation, abort it, or configure it.

Similarly, you can manage Delta Releases using the controls arranged vertically on the left side of the matrix:

• Click on the plus icon at the bottom to create a new Delta Release.
• Use the buttons next to each current version to create a Delta Release with this From Version, or to activate an existing Delta Release.

Protected Cohorts

In the Cohort detail view, you can also find the Protected Cohort feature:

All Cohorts are Protected by default, which means only Project Managers can edit Release Activations for them (see user roles).

Organization Auth Tokens can be set to permit Release Activations for Protected Cohorts on creation:

Cohorts View

In the Cohorts list view, accessible from the main menu on the left side of the application, some OTA-related information is available for each Cohort:

• Whether a Release is active for the Cohort (under the Release Activation column).
• A software distribution widget (under the Software Distribution column).

The distribution of Software Versions in this Cohort is shown in different sections of the software distribution widget, with the Software Version representing the ongoing active Release highlighted and placed at the start.

Release Activations

You can see a chronological list of all Release Activations, by navigating to OTA Releases and clicking on N releases beside the Create Release button.

This will bring you a page showing a list of every Release Activation, with the user that deployed that release and the status of the Release. This page does not show Release Deactivations or Modifications. Note that the Activated By user will be blank if deployment was activated with an Organization Auth Token.

Requiring Approvals for Release Activations

Cohorts can be configured to require an additional Team member with the right permissions to review Release Activations before they're executed. To configure this, you find the Cohort in the Cohort list page, open the Settings tab and select a Team in the "Require Approvals for Release Activations" section.

If the Cohort is protected, then only Project Managers will be allowed to request Release Activations and approve them.

We recommend you learn about Teams in order to make the best use of this feature.

When activating a Release targeting a Cohort that requires approval, a Release Activation Request will be created instead and another team needs to approve it in order to perform the Release Activation.

Debugging Your Configuration

If a device in your fleet is not downloading the OTA payload you anticipated from our description of update behavior, you can use the device details view to find out about the Software Version history of each particular device, and to inspect the result of it calling the Latest Release endpoint.

The Inspect details feature allows you to see the result of calling the Latest Release endpoint as this device.

Device OTA Endpoint

Memfault provides different types of Device OTA Endpoints for different device families and existing setups:

Device Family	Typical OTA Update Endpoint
OTA Updates for MCU devices	The MCU SDK uses the Latest Release Endpoint.
OTA Updates for Android (AOSP)	The Android SDK Latest Release Endpoint.
OTA Updates for Embedded Linux	SWUpdate users typically use our hawkBit DDI Compatible API (recommended!), or anyone can choose to make generic HTTP requests to our Latest Release Endpoint.

OTA & Device Version Specifiers

note

More information on Memfault versioning can be found at Software Versioning.

Here are Memfault's recommendations around versioning:

• Device software_version identifiers should use SemVer 2.0 versioning when possible
• Reported software_version should exactly match the OTA Release Version name

Note that SemVer 2.0 "build metadata" (1.0.0+abcd) is not supported by default, but needs to be opted into by contacting Memfault Support.

For tying Source Control versions to OTA Releases, Memfault recommends using the following strategies:

• Recommended: use a tag in your Source Control system that matches the OTA Release Version name (eg git tag -a -m=1.0.0)
• upload the Source Control commit hash as the OTA Release revision field (eg git rev-parse HEAD), either from the Web UI or the Memfault CLI.