More Documentation​

Memfault’s documentation at docs.memfault.com received a new landing page and various other additions.

The updated landing page outlines content across MCU, Android, and Linux. It also links to additional learning resources such as video content and source code examples. Other improvements include the new page Introduction to Memfault, more information on Memfault’s Linux Support, and many smaller new sections such as a comparison between Android Bug Reports and Caliper.

Improvements on Charts​

Memfault surfaces relevant insights about your fleet via charts. They receive updates regularly to be more comprehensive and useful.

The dashboard chart “Active Devices” was redesigned to be a bar chart to communicate its underlying data more clearly: “Devices that communicated with Memfault in a given time period”. Tooltips on several charts better highlight the numerical values their lines and bars represent. The chart “Reboot Reasons” also allows for drill-down on any day by clicking on the respective labels.

Improved Device Search​

Memfault organizes your data around the concept of devices. An investigative search for matching devices is a key activity when analyzing data around device attributes and time-series metrics. Memfault’s device search also acts as a hub when clicking on details on charts, OTA management, or as part of crash analysis. It allows you to precisely describe a population of devices before performing follow-up tasks such as assigning them to a specific Cohort or describing a new Device Set.

We made significant improvements to the device search! Most fields now accept multiple values (”OR”) and can be repeated where applicable (”AND”). You can search for device attributes and time-series data at the same time and it is even possible to search for historical data across different time ranges at once.

When combined, you can now conveniently describe (and store as Device Set) populations such as

• “Devices on v1.2 that were charged >90% earlier this week but rebooted due to low power yesterday” – to validate presumed bug fixes via operational data, or
• “Devices with attributes was_reworked: true, shipment_state: "back to us" and assignee: "John"” – to use custom device attributes for inventory management

Android SDK Bort 4.0​

Memfault’s AOSP SDK “Bort” received a major update 4.0 with many improvements such as additional support for Android 12 and multi-user compatibility.

We added support for Custom Metrics so that devices can report product-specific numerical values, strings, and state transitions on regular intervals. Together with a growing set of built-in Metrics, this leads to a powerful combination of per-device debugging (e.g. correlation of CPU activity and battery voltage) and fleet-wide insights (e.g. “how many devices exceed 80% of their storage” or “What is the average battery discharge rate?”). The reported values contribute to the recently introduced Time-Series Metrics and Device Attributes.

Another rather advanced feature enables devices without direct Internet connection to report crashes and metrics to Memfault. Data is packaged as .mar (Memfault Archive) files and vendors may upload them at a later time or upon request (e.g. downloaded periodically via USB or in a local network by an auxiliary device). This allows vendors to use Memfault in scenarios with strict security requirements.

Dashboard changes to charts​

Memfault’s Dashboard provides an overview of your fleet at a glance. We have updated the charts “Active Devices” (more sources used as signal) and “Software Versions” (only active devices considered) to better compare apples to apples.

The visible time range for “Software Versions” can now be changed from “2 months” all the way down to just “24 hours”. Since the same chart is now being used on the Cohort details page, it not only allows you to see long-term trends, but also acts as timely signal to observe the effect of an ongoing OTA software rollout.

The charts “Seen Devices” and “Received Events” have been removed.

Improved OTA with Version Matrix and Delta Releases​

Memfault steps up OTA observability and release management. Use the new Version Matrix (Fleet → Cohorts → Cohort Details) to learn about the version distribution of your devices (rows) and which version your devices will be updated to via OTA (columns).

Changes to your software rollout (including percentages for staged rollouts) are reflected in the matrix immediately. This is especially helpful when using the newly introduced Delta Releases that describe a path between specific versions. Every software version, OTA release, and number in the matrix is clickable to get to more details if needed.

Even complex and unusual scenarios are visible at a glance: devices with no compatible OTA payload, devices with a higher version than the cohort’s target release, must-pass-through releases and their effect, as well as many other edge cases are represented with data that is always live.

The Version Matrix gives you confidence both while preparing your software rollout and during the ongoing rollout, and it still helps you understand the version distribution of your fleet when no further updates are planned.

Device Sets​

You can now observe the development of your device fleet even better via fully customizable Device Sets (Fleet→Device Sets). Based on your project-specific Device Attributes and Time-Series Metrics each set describes devices that match unique criteria such as “devices that exceeded 40MB of daily network traffic” or “devices in the UK”.

You can create a Device Set from any existing device search and use them to track key performance indicators over time.

Issue Charts​

The new chart type Issue Chart complements the custom Metrics Chart (Dashboards→Metrics) and plots the number of occurrences of a specific group of issues over time. When combined with the recently introduced Chart Comparison Mode, Issue Charts allow for sophisticated tracking of ongoing software updates to answer the relevant question “does my software update fix the bugs it claims to fix?”

Comparison Mode for Metric Charts​

Since the introduction of Custom Metrics, Memfault provided various ways to define charts for plotting fleet-wide aggregations (sum, count, min, max, mean). These charts are often used to visualize trends over time (e.g. "hours between recharge") or to detect anomalies (e.g. "attempt until successful connect") for a subset of the fleet. The controls in the top-right corner allow for ad-hoc filtering by cohort and/or software version.

With the advent of the Chart Comparison Mode, users can now apply up to 4 such filters simultaneously to plot the resulting data series side-by-side inside each custom chart.

Use this feature to visually compare your beta cohort against the production fleet or to see at a glance if there are any regressions for key metrics between the last software versions.

Linked Devices​

Sometimes, products span across multiple devices: Each pair of wireless earplugs has a left and right version, for each smart home installation there may be multiple components in a network, or some related device information lives outside of Memfault, and your workflow benefits from quick navigation to another software system.

Memfault's Linked Devices provide convenient navigation in these scenarios. It builds these connections by looking at Device Attributes that can be edited in the UI, updated via API, or sent from devices at any time.

MCU Heap Visualization​

Memfault can now track and visualize heap allocations with any coredump on a growing number of MCU Architectures starting with ARM Cortex-M, nRF Connect SDK, ESP32 ESP-IDF, ESP8266, and Dialog DA1469x.

This optional debugging feature tags all heap allocations with their respective size, address, and location of code that allocated it. Use it to understand memory distribution, heap fragmentation, or to find memory leaks in an out-of-memory situation in the field.

Managing Rate Limits​

Memfault's frontend is now more upfront with the visualization of quotas and rate limits that may affect large projects.

When managing Timeseries Metrics and Device Attributes, Memfault informs about the number of remaining metrics for fleet-wide analytics. The dedicated page at Settings→Quotas presents an overview.

Compact Logs in Memfault's Firmware SDK​

Memfault's Firmware SDK now includes support for what we are calling "Compact Logs". It is similar to Zephyr's dictionary-based-logging and a few other implementations in other RTOS's, but we think we've hit the perfect balance of performance, features, ease-of-use, and size compression.

Let's walk through a quick example of what compact logs are. Here is a log line that a user of the Memfault SDK could write:

MEMFAULT_TRACE_EVENT_WITH_LOG("QSPI Flash Erase Failure: error code: 0x%x", 0x5);

With a generic formatted string implementation such as printf, we would need to include the entire string contents in our log buffer or storage. In total, this requires 41 bytes.

00000000: 5153 5049 2046 6c61 7368 2045 7261 7365  QSPI Flash Erase
00000010: 2046 6169 6c75 7265 3a20 6572 726f 7220   Failure: error
00000020: 636f 6465 3a20 3078 35                   code: 0x5

Once developers run out of firmware code space, a common pattern is to start shortening log lines or removing them entirely. We believe this is a bad practice! We should be able to keep our useful log lines and make the messages as long as necessary!

This is why we created compact logs. With compact logs, every log "string" is placed in a special section in the ELF file and we just need to reference each log line with an index, or ID.

For a compact log, we just need to encode an integer ID for the string portion and the arguments to be serialized. In this example, this information requires just 5 bytes.

00000000: 8218 8805                                ....

In this example, we can reduce the storage occupied by 88%! For a firmware that contains 100-1000 log lines, these savings are dramatic.

Check out our documentation for more information on how to enable compact logs in your firmware.

Along with compact logs, we've been busy adding more ways to filter and slice the data in our application and building up some exciting features related to Custom Events and State. More news to come next month.

Nordic nRF Connect SDK Memfault Integration​

Nordic Semiconductor's nRF Connect SDK now has Memfault's SDK built-in, allowing for customers and users of Nordic's chips and tools to integrate and use Memfault in minutes.

Check out our press release for more information!

As for new Memfault features, we made improvements to symbol-file uploading and downloading, added favorite projects, and sped everything up a bit.

Heartbeat Metric Improvements​

One of Memfault's most capable features is Metrics. With Metrics, you can determine how specific metrics on a single device correlate with each other to root cause performance and power regressions, or aggregate them on a fleet level and determine whether the health of your fleet of devices is improving over time!

We're announcing new features today which should help make our metrics feature more usable.

First, users can now view all the Metric names that have been received from heartbeat events in the Settings -> Metrics page. This allows for easy auditing of metric names and configrations.

If you find any that are stale or shouldn't be tracked anymore, you can archive them.

When Edit is clicked, users can configure the valid ranges for the metric, which will improve the displayed charts by normalizing the Y-axis to possible values. An example of where this could be useful is battery life, where the min and max values are 0-100.

Users can also specify a single value for Memfault to ignore, such as 0xffffffff, which can represent the absence of a value.

These new features will enable developers and customer support representatives to more easily dig into the health of a single device and the entire fleet.

That's not all though! We've made more improvements to symbol file uploading, bugreport filtering, and the archiving of software versions.

Memfault Now Offering Free Trial​

The day you have all (maybe) been waiting for is here! Anyone can now sign up for a free trial of Memfault! There is no sales team in your way of getting your devices integrated with Memfault and monitoring your fleet of devices.

Check out our press release to learn more.