Over-the-Air Updates (OTA)
Memfault provides secure and scalable updates for all IoT devices (MCU, Android, Linux, or a combination of them). With real-time monitoring and controlled releases, you can debug and ship new releases confidently.
Memfault uses a globally distributed CDN for low latency and high speed downloads regardless of geolocation. It supports features such as HTTP/2, Brotli and GZip, and byte range requests, among others.
Introduction and Concepts
To deliver OTA Updates:
- Memfault users have different ways to configure which OTA payloads to distribute to which devices.
- Each device polls Memfault's Device OTA Endpoint.
- Memfault uses various inputs, predominantly the current Software Version of the device, to decide what to return.
- Each device may receive information about a new available payload.
- Ultimately, this process is meant to replace or update the software running on a device.
Read about related Memfault-specific terminology that you may need to refer to to better grasp the contents of this document:
- Software Versions and Hardware Versions.
- Version Schemes to understand Software Version ordering.
- Generic Memfault terminology documentation.
Memfault provides two Release entities to solve the challenges of over-the-air updates: Full Releases and Delta Releases.
Full Releases provide a way to upgrade device Software Versions regardless of the version the device is currently on.
If we define the version of the most current Release as
2.0 and every version
*, we can describe the behavior of a Full Release as:
See the updating behavior section for more details.
Must-pass-through Full Releases
Full Releases can be marked as must-pass-through, or MPT for short. When an MPT Release is activated, a device on a lower version will be forced to "pass through" this Release before an update is allowed to a Full Release with a version greater than that of the MPT Release. Typically this setting is not needed.
A Delta Release describes a transition from exactly one From Version to exactly
one To Version. The behavior can be described as:
Note that Delta Releases are referred to as "incremental updates" in Android documentation, or "differential updates" in other places.
A Delta Release is only applied when the device’s current version matches the Release’s From Version.
Delta Releases allow for more fine-grained control over the update paths of devices. Additionally, Delta Releases generally contain smaller payloads, reducing required bandwidth and costs associated with rolling out updates.
See the updating behavior section for more details on Delta Releases.
Another good place to learn more about delta updates for MCU-based devices in our Interrupt blog post Saving bandwidth with delta firmware updates.
Activating a Release
You can activate a Release for a Cohort of your choice. While Releases are shared between all Cohorts, activating them happens on a per-Cohort basis.
When activating a Release for a Cohort, you can choose between two behaviors:
Normal Activation of a Release
Releases activated normally are available immediately to all devices in a Cohort.
Staged Rollout of a Release
Staged-rollout Release Activations can be used to make a Release available to a Cohort incrementally.
When a staged-rollout Release Activation is created for a Cohort, a random subset of its devices (amounting to the selected rollout percentage) will be marked as staged in this Cohort. Staged-rollout release activations are only applicable to devices that are staged in the Cohort.
Best Practices When Activating Releases
Memfault enables risk mitigation through different mechanisms.
Monitoring the effect of Releases
As part of your OTA management workflow, make sure to set up Metric and Issue Charts to monitor the health of your fleet. You can use Comparison Mode to see how devices in a different Cohort or in a newer Software Version are doing relative to other devices.
Activating Releases in a test Cohort first
Since Releases are shared among all Cohorts, you can use the Release activation
mechanism to test a Release in (for example) a
beta Cohort before activating
the same Release in a
production Cohort. Once you've gained confidence through
monitoring that the software in a new
Release is working as expected, you can activate the Release for a bigger
Using staged-rollout Releases
Releases activated using a staged rollout can be used to limit the amount of devices that will download a specific Release, reducing the risk of shipping bugs to a greater portion of the fleet. Once you've gained confidence through monitoring that the software in a new staged-rollout Release is working as expected, you can increase its rollout percentage.
When a device queries the Device OTA Endpoint, the Memfault platform needs to decide which OTA payload to send it. This section covers the behavior of our backend when receiving one such query.
In this section, we'll sometimes refer to one Software Version being higher or lower than another. To understand how Memfault resolves Software Version ordering, refer to the Version Schemes documentation section.
If you wish to better understand Memfault's behavior from the point of view of a specific device, you can use the "Inspect details" feature in the device details view.
Applicability of an OTA Release
Along with the query to the Device OTA Endpoint, a device sends:
- the Device ID (often the device serial number) to determine its Cohort,
- the Software Type to determine the set of Releases against which to apply these rules,
- its current Software Version,
- device-specific attributes such as Hardware Version.
A Release must be active for the Cohort a device is in to be applicable.
To determine if an active Release is applicable to a device, its payload attributes are tested against the parameters sent by the device. If they don’t match, a Release is considered to be incompatible and treated as if it did not exist.
An Release activated as a staged-rollout will only be applicable to the staged portion of the Cohort it targets. In other words, if the device is not staged in its Cohort, such Releases will not be applicable.
Choosing Among Multiple Applicable OTA Releases
If a device has multiple possible update paths, the following rules of precedence apply to determine which release is selected.
The following rules apply to all active Releases per Cohort:
- Use a Delta Release whose From Version matches the current version of the device.
- Otherwise, use a must-pass-through (MPT) Release with the lowest Target Version that’s greater than the current version of the device.
- Otherwise, use a Full Release with the highest To Version greater than the current version of the device.
In short, the precedence can be summarized like so: Delta Release > MPT Release > Full Release.
Managing OTA Updates with Memfault
The Memfault application contains OTA-related features in several of its views.
The OTA Releases section of the Memfault application can be accessed from the main menu on the left side. In it, you’ll be able to access a list view of all Full Releases and the equivalent list view for Delta Releases by selecting one of the tabs under the title of the page.
Let's take a look at how to create Releases using the Memfault application.
Please note that the Release version you choose is expected to match the Software Version a device will report once it has installed the OTA payload for this Release.
If you're on Android, we recommend following the Android-specific OTA guide in order to create Releases.
Creating a Full Release
To create a Full Release, click on the Create Release button on the top-right area of the OTA Releases view. Among other information, you'll be prompted for a version and whether the Release is a must-pass-through (MPT) Release.
Creating a Delta Release
To create a Delta Release, click on the drop-down icon next to the Create Release button on the top-right area of the OTA Releases view, and select Create Delta Release. Among other information, you'll be prompted for a From Version and a To Version.
Adding an OTA payload to a Release
Once you've created a Release, you'll be redirected to the Release detail view. To add an OTA payload, click on the Add OTA payload to Release button on the top-right corner of the OTA payloads card.
Alongside the actual payload file, you'll be prompted for different inputs that a device querying the Device OTA Endpoint must match for this Release to be applicable to it.
See the updating behavior section for more details.
Cohort Version Management
To dive into your OTA configuration for a specific Cohort, you can access the Cohort detail view by first opening the Cohorts list view from the main menu, and then selecting the Cohort you're interested in.
Under the Version Management tab, you'll find the version matrix: a live view into the very next OTA update that each device in this Cohort will receive.
The version matrix consists of rows representing current versions that are currently reported by devices in the field. Its columns represent future versions devices of the Cohort will report (after having applied OTA releases). What follows is a tour of the different interface elements of the version matrix.
In each cell of the version matrix you'll find a count of devices. If, for example, a device count of 10K showed up on a cell in the intersection of row version 1.1 and the column 2.0, that would mean 10K devices are currently on version 1.1 and the immediate next step they're going to take is to download an OTA payload for a Release with version 2.0.
Conversely, if 5K devices were to show up on a cell in a row with version 1.0 and under the column for version 1.0, that would mean those 5K devices will stay on version 1.0.
All device counts on the version matrix are clickable, and will take you to a complete list of the devices represented under that count.
Figure 1 highlights the area of the version matrix that shows future versions. In this example, three Full Releases exist (for versions 1.0, 2.0 and 1.1), but only Full Releases 1.0 and 2.0 are active. This is represented by the blue checkbox icon . The dotted line around the icon for the 2.2 version means that there is no Full Release with the same version.
The icon (which conventionally means "final state") on top of the 2.2 version shows us that 2.2 is a final target version for this Cohort. Eventually, all devices in this Cohort will end up on this version 2.2.
Figure 2 highlights the area of the version matrix that represents Delta Releases. The Current Version column is where you may see the From Version of a Delta Release, and the Target column is where the Target Version of a Delta Release can be found. An empty drop-down icon indicates that there is no active Delta Release from the row's version. In the picture, only a Delta Release (marked with a ) from 2.0 to 2.1 is active .
Figure 3 shows the following scenario:
- Two active Full Releases exist (1.0 and 2.0) and there's one active Delta Release from 2.0 to 2.1.
- Full Release 1.0 is currently not applicable to any device.
- 21K devices currently on 0.9 are going to download a Full Release to 2.0.
- 36K devices currently on 1.1 are going to download a Full Release to 2.0.
- 132K devices currently on 2.0 are going to download a Delta Release from 2.0 to 2.1.
- 738K devices are already on 2.1. They will not receive any updates and will stay on version 2.1.
Figure 4 shows the addition of a staged rollout of a Delta Release from version 2.1 to version 2.2. The percentage icon indicates a staged rollout. In this scenario, only 5.9K devices out of a total of 54K that are on version 2.1 will download the payload from Delta Release from 2.1 to 2.2.
Managing OTA Updates
The version matrix in the Cohort Version Management tab can also be used to manage your OTA Releases.
Figure 5 highlights the location of buttons on the version matrix that allow for fine-grained control of OTA Releases in a specific Cohort.
You can manage Full Releases by using the controls arranged horizontally on the top side of the version matrix:
- Click on the plus icon on the right end to create a new Full Release.
- Use the highlighted buttons under each future version to create a Full Release targeting this version, or to create a Release Activation, abort it, or configure it.
Similarly, you can manage Delta Releases using the controls arranged vertically on the left side of the matrix:
- Click on the plus icon at the bottom to create a new Delta Release.
- Use the buttons next to each current version to create a Delta Release with this From Version, or to activate an existing Delta Release.
In the Cohorts list view, accessible from the main menu on the left side of the application, some OTA-related information is available for each Cohort:
- Whether a Release is active for the Cohort (under the Release Activation column).
- A software distribution widget (under the Software Distribution column).
The distribution of Software Versions in this Cohort is shown in different sections of the software distribution widget, with the Software Version representing the ongoing active Release highlighted and placed at the start.
Debugging Your Configuration
If a device in your fleet is not downloading the OTA payload you anticipated from our description of update behavior, you can use the device details view to find out about the Software Version history of each particular device, and to inspect the result of it calling the Latest Release endpoint.
The Inspect details feature allows you to see the result of calling the Latest Release endpoint as this device.
Device OTA Endpoint
Memfault provides different types of Device OTA Endpoints; which gets uses depends on your requirements and existing setup.
|Device Family||Typical OTA Update Endpoint|
|OTA Updates for MCU devices||The MCU SDK uses the Latest Release Endpoint.|
|OTA Updates for Android (AOSP)||The Android Bort SDK Latest Release Endpoint.|
|OTA Updates for Embedded Linux||SWUpdate users typically use our hawkBit DDI Compatable API (recommended!), or anyone can choose to make generic HTTP requests to our Latest Release Endpoint.|