Skip to main content

Integration Guide

This guide will cover integrating the Memfault Linux SDK into systems built with Yocto.

When you're done, you will have a basic integration setup that can talk to the Memfault cloud, enabling one or more of the following features:

Assumptions and dependencies​

Optionally:

  • For OTA:
    • An A/B partition scheme (two partitions for root filesystems, one active and one to receive the next update). Systems will typically also have a data partition which is persisted accross updates.
    • SWUpdate as an OTA update client.
    • A bootloader, configured to collaborate with SWUpdate. U-Boot is often used. The bootloader selects which root partition to boot from and will switch the active partition after an update. This setup is very system dependent and beyond the scope of this guide but our example layer includes SWUpdate and U-Boot configurations for QEMU and RaspberryPis.
  • For Custom Metrics:
  • For Log Collection:

Create a Project and get a Project Key

Go to app.memfault.com and from the "Select A Project" dropdown, click on "Create Project" to setup your first project such as "smart-sink-dev".

Once you've created your project, you'll be automatically taken to an page that includes your project key. Copy the key and follow the rest of this guide.

Add memfaultd to your system​

tip

Keep meta-memfault-example open as a reference implementation. Your integration should look similar to it once you're done following the steps in this tutorial.

The memfaultd service is the main orchestrator of the subsystems that conform and are used by the Memfault Linux SDK. As with all our SDKs, its source code is available on GitHub.

Among its responsibilities are:

  • Keeping a queue of items to be uploaded to the Memfault cloud.
  • Uploading that data at a specified interval, and recovering from network failures using an exponential back-off system, independent of the data collection interval.
  • Controlling whether data is allowed to be collected (e.g. by user consent) from a device or not.
  • Adjusting which data is collected based on the fleet sampling configuration of this device.
  • Configuring SWUpdate dynamically to perform OTA updates.
  • Receiving data from Fluent-bit and CollectD, aggregating it in memory or on disk.
  • Providing configuration access via memfaultctl (memfaultctl is a symbolic link to memfaultd).

memfaultd is required for integration.

Add meta-memfault layer and dependencies​

Include the meta-memfault layer in your bblayers.conf file, alongside dependencies:

# Apart from other layers you may already depend on:
BBLAYERS ?= " \
    ${YOCTOROOT}/sources/memfault-linux-sdk/meta-memfault \
    ${YOCTOROOT}/sources/memfault-linux-sdk/meta-rust-bin \
    # Provides collectd (recommended) - Also a dependency of the swupdate layer:
    ${YOCTOROOT}/sources/meta-openembedded/meta-oe \
"
Rust in Yocto for the Memfault SDK

memfaultd requires the Rust compiler version 1.65.0 or higher. Dunfell does not ship with Rust support and Kirkstone ships with 1.59.0 so we provide Rust recipes with the SDK.

The meta-rust-bin layer included in the memfault-linux-sdk is a slightly modified version of rust-embedded/meta-rust-bin . This version differs from upstream by providing a new class cargo_bin instead of overriding the cargo class provided by meta-oe.

We found that overriding cargo often caused problems with some Poky packages, notably python3-cryptography-native. The modified version that we provide does not have this issue.

Use systemd as your init system​

memfaultd currently depends on systemd to:

  • restart services it cooperates with, for example, collectd and swupdate;
  • detect when a reboot was requested by the user;

To use systemd as an init system (using poky), add to your conf/local.conf file:

DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES:remove = "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED:append = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

Authorize Memfault License​

Since memfaultd has a commercial license, you'll need to add an exception for it, for example in your local.conf:

LICENSE_FLAGS_ACCEPTED:append = " commercial_memfaultd"

(Optional) Add meta-swupdate for OTA​

If you are planning on using swupdate for OTA, you will also need to add the meta-swupdate layer:

# Apart from other layers you may already depend on:
BBLAYERS ?= " \
    ${YOCTOROOT}/sources/memfault-linux-sdk/meta-memfault \
    # Provides collectd (recommended) - Also a dependency of the swupdate layer:
    ${YOCTOROOT}/sources/meta-openembedded/meta-oe \
    ${YOCTOROOT}/sources/meta-swupdate \
"

You can grab a copy of meta-swupdate from the source repository. Note that meta-oe is a dependency of meta-swupdate.

And add the necessary dependencies to your image file (swupdate will be required automatically by the memfaultd recipe as long as the swupdate feature is active):

IMAGE_INSTALL:append = " \
    u-boot-env \
    u-boot-fw-utils \
"

Minimal Configuration of memfaultd​

(Optional) Opt out of memfaultd default features​

The memfaultd daemon is modular and you can control which features are built-in to reduce code size, dependencies and resource usage. By default, all features are enabled.

The available features are:

Set PACKAGECONFIG for the memfaultd package to control which features are included in the memfaultd build. For example, you can add a memfaultd.bbappend recipe with the following contents:

PACKAGECONFIG := "swupdate collectd coredump logging"

Alternatively, in your local.conf file:

PACKAGECONFIG:pn-memfaultd := "swupdate collectd coredump logging"

Add a memfault-device-info executable to your build​

tip

You may want to copy over the entire memfault-device-info recipe from our examples to get started. Make sure to edit its contents following the steps in this section.

The memfaultd daemon expects to find a memfault-device-info executable in $PATH. The memfault-device-info executable is created by you during integration and is meant to provide device-specific information in order to identify it in Memfault services. The expected output is in the following format:

MEMFAULT_DEVICE_ID=<device-id>
MEMFAULT_HARDWARE_VERSION=<hardware-version>

Read more about Device ID, Software Versions, and Hardware Versions to better understand these core Memfault platform concepts.

Here's an example memfault-device-info shell script one might add to /usr/bin/memfault-device-info:

#!/bin/sh
echo MEMFAULT_DEVICE_ID=$(cat /etc/machine-id)
echo MEMFAULT_HARDWARE_VERSION=$(dmidecode -s system-product-name)

The script would output something like this:

$ memfault-device-info
MEMFAULT_DEVICE_ID=4a7d5d74-e8f0-471f-9f8c-23e3dd5ce18c
MEMFAULT_HARDWARE_VERSION=some-board-rev-1.3

If you added a recipe named memfault-device-info, don't forget to add it to your image dependencies:

IMAGE_INSTALL:append = " \
    memfault-device-info \
    u-boot-env \
    u-boot-fw-utils \
"

Add /etc/memfaultd.conf to your build​

tip

Find a full reference on /etc/memfaultd.conf here. You may want to copy over the entire memfaultd.bbappend recipe from our examples to get started. Make sure to edit its contents following the steps in this section.

A default memfaultd.conf (called builtin.conf in the source files) is embedded within the memfaultd binary, and the daemon will fall back to it if /etc/memfaultd.conf does not provide an override for a specific key. All values in it can be overwritten by adding an /etc/memfaultd.conf file to your build, e.g. using a bbappends recipe for memfaultd (see an example). You may copy parts of the built-in config (or the whole file) to customize configuration on your target device.

Note that while the sample file has reasonable defaults for most keys, it cannot guess a value for some fields such as software_version, software_type or project_key. Hence, a minimal /etc/memfaultd.conf file should look like this:

{
  "software_version": "1.0.0",
  "software_type": "main",
  "project_key": "<YOUR_PROJECT_KEY>"
}

Note that /etc/memfaultd.conf is designed to be in a read-only filesystem.

Read more about Software Versions and Hardware Versions to better understand these core Memfault platform concepts.

Configure memfaultd storage​

Persistent storage​

In /etc/memfaultd.conf:

{
  ...
  "persist_dir": "<YOUR_PERSISTENT_STORAGE_DIR>"
}

By default it will be a directory in /media. You will most likely need to change this to wherever your device stores data in a persistent manner. Requirements for this directory are:

  • That it is writable to by memfaultd.
  • That its data persists after firmware upgrades.

Your persist_dir will be written to for the following reasons:

  • Save persistent configuration options (enable_data_collection and enable_dev_mode).
  • Save the internal reboot reason file when memfaultd restarts.
  • Save the Fleet Sampling configuration (only when it changes).

Temporary storage​

You can configure memfaultd to use a different folder for data that will be written more frequently and does not necessarily needs to be persisted across reboots. You can even decide to store this temporary storage directory on a temporary filesystem in RAM (tmpfs), although in this case, the data will be lost if a reboot or crash occurs before the data was uploaded.

{
  ...
  "tmp_dir": "<YOUR_TEMPORARY_STORAGE_DIR>"
}

By default the temporary storage directory is the persistent storage directory. This is the recommended configuration.

This directory will be used for:

  • Coredumps
  • Heartbeats (metrics)
  • Logs
  • Storing the boot id of the device across memfaultd restarts
  • Storing the Memfault reboot events queue

Note that the collectd.interval_seconds does not affect reads or writes to tmp_dir, unless you've configured collectd to write there, too. By default, collectd will store metrics in memory.

You can control how much space memfaultd will use as temporary storage. See our configuration reference for more context.

Set enable_data_collection​

By default, enable_data_collection is false (see the default configuration). This is to enable asking end users for consent before collecting or transmitting any data to Memfault services.

Once the end user has given their consent, you can enable data collection like so:

$ memfaultctl enable-data-collection

To disable it:

$ memfaultctl disable-data-collection

The memfaultd service will restart automatically whenever you run either of those commands if called with a value different from the current configuration.

Take a look at the /etc/memfaultd.conf reference for more information.

tip

To always enable data collection you can include the key enable_data_collection with the value true in your configuration file.

Test the integration​

You can test your integration by rebooting your test device. Then, on the Memfault app, open Fleet -> Devices and check that your device shows up. If you open the specific device, you should be able to find its reboot events under the Reboots tab.

Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.

Integration next-steps​

Congratulations! 🎉 Memfault is running in your product. You will start getting useful insights automatically.

To get more value from Memfault, we recommend the following steps.

Reboot reasons​

Memfault will automatically classify some reboots such as kernel panics, user triggered reboot and OTA updates.

For other reboot reasons that are specific to your product, you can inform memfaultd before restarting the system. Memfault Dashboard will then give you an accurate picture of why your devices are restarting.

Our Reboot Reason Tracking Guide presents several mechanisms to do this.

Coredumps​

memfaultd will automatically configure the kernel to capture coredumps and upload them to Memfault.

Follow our Coredumps Guide to upload debug symbols to Memfault during your build process. This is required for Memfault coredump analysis to work.

Metrics​

By default, collectd will capture system metrics and provide an overview of how the system is doing.

Our Metrics Guide explains how to collect custom application-specific metrics from your system.

Logging​

If you have enabled the logging feature, memfaultd will expect network connections from fluent-bit. fluent-bit will capture kernel and system logs and upload them to Memfault.

You can customize which logs are captured, filter out some lines and capture data from other sources (file on disk, serial port, etc). Refer to our Linux Logging Guide.

OTA​

A working OTA setup will require collaboration between the bootloader, the updater (for example swupdate) and memfaultd. You will also need to configure your build process to generate update images in a format compatible with the updater.

Follow our Embedded Linux OTA Integration Guide to configure and use OTA.