Skip to main content

Nordic nRF Connect SDK Integration Guide

In this guide we will walk through the steps for integrating the Memfault Firmware SDK into a project using Nordic Semiconductor's nRF Connect SDK (NCS). All versions of the nRF Connect SDK since v1.6.0 include Memfault when downloaded from Nordic's website, so we suggest using the nRF Connect SDK >= v1.6.0 even though integration with Memfault has been tested as far back as nRF Connect SDK v1.4.0.

This integration is written for the nrf9160-DK, but is similar when targeting other boards with the nRF Connect SDK.

tip

nRF Connect SDK v1.6.0, and all versions since, has built-in support for the Memfault Firmware SDK on nRF9160-based targets. Nordic Semiconductor provides some excellent documentation on how to integrate Memfault in your existing nRF Connect SDK project, together with a sample integration project (source available here). We recommend following both this documentation page as well as Nordic's.

Upon completion of the integration, the following subcomponents will be added to your system!

/img/docs/mcu/reboot-reason-chart.png
/img/docs/mcu/trace-reason-example.png
/img/docs/mcu/logs-with-coredump.png

Integration Steps​

info

This tutorial assumes you have a working nRF Connect SDK Environment and are able to flash a board supported by the SDK (i.e nRF91, nRF53, nRF52, etc)

Create a Project and get a Project Key

Go to app.memfault.com and from the "Select A Project" dropdown, click on "Create Project" to setup your first project. Choose a name that reflects your product, such as "smart-sink-dev".

Once you've created your project, you'll be automatically taken to an page that includes your project key. Copy the key and follow the rest of this guide.

Include memfault-firmware-sdk in the build​

The SDK is part of the West manifest in nRF Connect SDK and is automatically downloaded when running west update. By default, it is downloaded to <nRF Connect SDK path>/modules/lib/memfault-firmware-sdk/.

Update Kconfig options​

Add or update the appropriate options in your system's prj.conf:

# Project-specific configuration settings

CONFIG_MEMFAULT=y
CONFIG_MEMFAULT_NCS_PROJECT_KEY="YOUR_PROJECT_KEY"

# On nRF52 or nRF53 projects, and nRF-Connect SDK <2.1.0 (Memfault SDK <
# 0.32.0), this setting is required to build the project (it de-selects the
# built-in Memfault HTTPS root certificate storage, which is only compatible
# with the nRF9160):
CONFIG_MEMFAULT_ROOT_CERT_STORAGE_CUSTOM=y

If you'd like to update the Memfault SDK without updating the nRF-Connect SDK, see the instructions here. Memfault aspires to maintain full backwards compatibility with supported nRF-Connect SDK versions.

You can find more details on the available options using menuconfig, guiconfig, and in the Kconfig sources in modules/memfault-firmware-sdk/ports/zephyr/Kconfig.

You can skip this section.

Create trace reasons definition file​

The trace event module within the SDK makes it easy to track errors in a way that requires less storage than full coredump traces and also allows the system to keep running after capturing the event. The program counter, return address and a custom "reason" are saved.

The list of custom reasons is defined in a separate file that you need to create.

The file must be named memfault_trace_reason_user_config.def and placed in a directory included in your project.

$ cd $YOUR_PROJECT_ROOT
$ mkdir -p config
$ touch config/memfault_trace_reason_user_config.def

And then in your project's CMakeLists.txt add:

zephyr_include_directories(config)

To start, we recommend adding a couple reasons for error paths in your codebase (such as peripheral bus read/write failures, transport errors and unexpected timeouts).

Here is what the memfault_trace_reason_user_config.def file should look like:

// your_project/config/memfault_trace_reason_user_config.def
MEMFAULT_TRACE_REASON_DEFINE(your_custom_error_reason_1)
// i.e
MEMFAULT_TRACE_REASON_DEFINE(critical_log)
MEMFAULT_TRACE_REASON_DEFINE(flash_error)
// ...

Add Additional Config Files​

Two other config files are required to build with the Memfault SDK:

$ cd $YOUR_PROJECT_ROOT
$ mkdir -p config
$ touch config/memfault_platform_config.h
$ touch config/memfault_metrics_heartbeat_config.def

See the nRF Connect Docs for more information.

Generate Some Trace Events​

Next, we'll need to use the MEMFAULT_TRACE_EVENT macro to capture a trace event when an error occurs.

Note that it is perfectly fine to use the same reason in different places. Because the program counter and return address are captured in the trace event, you will be able to see the two topmost frames (function name, source file and line) in Memfault's Issue UI and distinguish between the two.

The nRF Connect SDK port also exposes a mflt test trace CLI command which can be used to generate events for test purposes:

uart:~$ mflt test trace
<dbg> <mflt>: Trace Event Generated!

You can also start to add trace events for error paths you are interested in tracking:

#include "memfault/core/trace_event.h"
// [ ...]
void ble_le_process_ll_pkt(...) {
  // ...
  if (invalid_msg_id) {
    MEMFAULT_TRACE_EVENT(bt_invalid_msg_id);
    // ...
  }
  // ..
}

Log metadata can also be added to a trace event to capture additional info:

void record_temperature(...) {
  int rv = spi_flash_erase(...);
  if (rv != 0) {
    MEMFAULT_TRACE_EVENT_WITH_LOG(flash_error, "Flash Erase Failure: rv=%d, spi_err=0x%x",
      spi_bus_get_status());
  }
}

Publish data to the Memfault cloud​

Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.

(nRF9160) HTTPS​

When HTTP is available, you can use the Memfault integration to post data from the CLI (mflt post_chunks) or by adding a periodic task:

#include "memfault/nrfconnect_port/http.h"

void some_periodic_task(void) {
  memfault_nrfconnect_port_post_data();
}

Bluetooth Low Energy​

Using the Memfault Diagnostic Service (MDS), nRF-Connect SDK v2.1.0+​

As of the v2.1.0 release of the nRF-Connect SDK, there is built-in support for the Memfault Diagnostic Service, which is a Bluetooth LE GATT service designed to enable exporting Memfault data in a standardized way.

Nordic has detailed documentation on the Memfault Diagnostic service here:

https://docs.nordicsemi.com/bundle/ncs-2.4.2/page/nrf/libraries/bluetooth_services/services/mds.html

There is also a sample application, "Bluetooth LE: Peripheral Memfault Diagnostic Service (MDS)", documented here:

https://docs.nordicsemi.com/bundle/ncs-2.4.2/page/nrf/samples/bluetooth/peripheral_mds/README.html

Nordic and Memfault provide the following Bluetooth LE gateways for testing the MDS implementation:

For a quick test, we recommend using the "Bluetooth LE: Peripheral Memfault Diagnostic Service (MDS)" sample application and one of the mobile apps with a compatible mobile device.

Without using the MDS​

Extensive details about how data from the Memfault SDK makes it to the cloud can be found here. In short, the Memfault SDK packetizes data into "chunks" that must be pushed to the Memfault cloud via the chunk REST endpoint.

A typical integration looks like this:

#include "memfault/core/data_packetizer.h"
// [...]

bool try_send_memfault_data(void) {
  // buffer to copy chunk data into
  uint8_t buf[USER_CHUNK_SIZE];
  size_t buf_len = sizeof(buf);

  bool data_available = memfault_packetizer_get_chunk(buf, &buf_len);
  if (!data_available ) {
    return false; // no more data to send
  }

  // send payload collected to chunks endpoint; this could for example send the
  // packet over a GATT characteristic to the peer device
  user_transport_send_chunk_data(buf, buf_len);
  return true;
}

void send_memfault_data(void) {
  // [... user specific logic deciding when & how much data to send
  while (try_send_memfault_data()) { }
}

Force a Crash and Upload to Memfault​

Example crashes can be generated using the mflt CLI:

uart:~$ mflt test assert
[00:01:20.910,217] <err> os: ***** USAGE FAULT *****
[00:01:20.915,893] <err> os:   Attempt to execute undefined instruction
[00:01:20.923,278] <err> os: r0/a1:  0x00008001  r1/a2:  0x200145b8  r2/a3:  0x2001c794
[00:01:20.932,067] <err> os: r3/a4:  0x00028415 r12/ip:  0x0003290e r14/lr:  0x0001a4bf
[00:01:20.940,856] <err> os:  xpsr:  0x21000000
[00:01:20.946,166] <err> os: s[ 0]:  0x000272e6  s[ 1]:  0x00028415  s[ 2]:  0x00008001  s[ 3]:  0x00027fab
[00:01:20.956,695] <err> os: s[ 4]:  0x00028415  s[ 5]:  0x000272e6  s[ 6]:  0x0003290e  s[ 7]:  0x000272eb
[00:01:20.967,254] <err> os: s[ 8]:  0x00000001  s[ 9]:  0x2001463c  s[10]:  0x2001463c  s[11]:  0x0002af18
[00:01:20.977,813] <err> os: s[12]:  0x00000001  s[13]:  0x00028415  s[14]:  0x0002840b  s[15]:  0x0000d699
[00:01:20.988,342] <err> os: fpscr:  0x0002d529
[00:01:20.993,621] <err> os: r4/v1:  0x00028415  r5/v2:  0x00000001  r6/v3:  0x0002c118
[00:01:21.002,410] <err> os: r7/v4:  0x200145c8  r8/v5:  0xfffffffc  r9/v6:  0x00000002
[00:01:21.011,199] <err> os: r10/v7: 0x0002c118  r11/v8: 0x20014640    psp:  0x20014598
[00:01:21.019,989] <err> os: EXC_RETURN: 0xffffffbc
[00:01:21.025,604] <err> os: Faulting instruction address (r15/pc): 0x0001a4be
# ...
*** Booting Zephyr OS build v2.4.0-ncs1  ***
<inf> <mflt>: GNU Build ID: ef460cafbcf67633d470c125a7cc8fd02ed97538
Memfault Demo App Started!
uart:~$ mflt get_core
<inf> <mflt>: Has coredump with size: 3764

On nRF91 you can post directly to the cloud:

uart:~$ mflt post_chunks
<dbg> <mflt>: Response Complete: Parse Status 0 HTTP Status 202!
<dbg> <mflt>: Body: Accepted
<dbg> <mflt>: No more data to send

Test Tip​

Data can also also be dumped out over the CLI using the mflt export_data CLI command.

uart:~$ mflt export_data
<inf> <mflt>: MC:SFQCpwIBAwEHalRFU1RTRVJJQUwKbXRlc3Qtc29mdHdhcmUJajEuMC4wLXRlcw==:
<inf> <mflt>: MC:gCx0Bm10ZXN0LWhhcmR3YXJlBKEBoXJjaHVua190ZXN0X3N1Y2Nlc3MBMeQ=:

The CLI output can then be saved to a file and the "chunk" data can be posted to the Memfault Cloud using the Memfault CLI tool or simply copy-pasted under the Chunks Debug view.

$ memfault --project-key ${YOUR_PROJECT_KEY} post-chunk --encoding sdk_data_export cli-output.txt
Found 2 Chunks. Sending Data ...
Success

Upload Symbol File​

At this point, you should be able to generate a test trace event and push it to the Memfault UI. You can confirm the issue has arrived successfully by navigating to the "Issues" page. Follow the link pointed to below and upload a symbol file (for Zephyr + nRF Connect SDK projects, you can find it in your build folder at build/zephyr/zephyr.elf).

After this step, you will see the trace in the list of issues!

tip

You can programmatically upload symbol files with the Memfault CLI tool.

Troubleshooting Data Transfer​

See the docs on data transfer troubleshooting.

Sending data to Memfault over UDP​

Memfault servers do not accept UDP transfers. To transfer data from a device that can send UDP datagrams (such as the nRF9160), an additional server is necessary to act as a relay between the device and Memfault servers.

A working sample of a complete set-up can be found here. It features a Python UDP server that relays data to Memfault, and a firmware implementation that includes a UDP client. The device and the Python server use a simple binary encoding format that includes the following sections, separated by a NULL byte:

  • A version section identifying the encoding format for forward compatibility.
  • A Project Key.
  • The device serial.
  • The Memfault chunk.

This configuration allows for a simple, stateless UDP server that just passes data over to Memfault.

Configuring and Invoking an OTA Update​

LTE/Internet Connected Devices (nRF9160)​

The NRF Connect sample integration project does not come with OTA support enabled by default. To enable this support a few things need to be added. The update process is invoked with a CLI command, this needs to be enabled. the memfault_platform_config.h file, add this line:

#define CONFIG_MEMFAULT_FOTA_CLI_CMD    1

Add or update the appropriate options in your system's prj.conf:

# The subsystems we need so OTA payloads can be written to
# flash and updated by MCUBoot
CONFIG_DFU_TARGET=y
CONFIG_DFU_TARGET_MCUBOOT=y
CONFIG_IMG_MANAGER=y
CONFIG_FLASH=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

# For Memfault FOTA, we will use the FOTA_DOWNLOAD API's
# from the nRF Connect SDK which depends on the DOWNLOAD_CLIENT
CONFIG_FOTA_DOWNLOAD=y
CONFIG_DOWNLOAD_CLIENT=y

# Enable printing of file download progress to console
CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y
CONFIG_MEMFAULT_FOTA=y
CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE=400
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=1600

The full prj.conf file can be viewed here.

Confirm you have already built and uploaded the OTA compatible app_update.bin to Memfault with the appropriate semantic version set. The device you wish to update should be running a lower version.

Finally to invoke the OTA process enter the following command:

uart:~$ mflt_nrf fota
<inf> <mflt>: Checking for FOTA
<inf> <mflt>: FOTA Update Available. Starting Download!
<inf> download_client: Setting up TLS credentials, tag 1003
<inf> download_client: Configuring socket timeout (30 s)
...
<inf> <mflt>: FOTA In Progress
<inf> download_client: Downloaded 2048/204467 bytes (1%)
<inf> download_client: Downloaded 4096/204467 bytes (2%)
...
<inf> download_client: Downloaded 204467/204467 bytes (100%)
<inf> download_client: Download complete
<inf> dfu_target_mcuboot: MCUBoot image upgrade scheduled. Reset device to apply
*** Booting Zephyr OS build v2.7.99-ncs1  *** to install update!
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3

The device resets automatically, running your new firmware.

BLE Connected Devices (nRF52/nRF53)​

For Nordic devices, FOTA can be performed using the normal Nordic-provided DFU tools, sample apps, and libraries.

Follow the below links for details on how to set that up:

The only Memfault-specific piece of that FOTA process is the fetching of the OTA binary payload, which is performed via an HTTP request against Memfault's server, passing the Memfault Project Key and the Device Serial for the target device:

Below is a brief diagram showing how an nRF52, mobile phone, and Memfault's server interact during an OTA update:

Built-in Metrics for the nRF9160​

The nRF Connect SDK optionally provides some pre-enabled metrics for the nRF9160 example:

  • Ncs_AtCmdUnusedStack
  • Ncs_ConnectionPollUnusedStack
  • Ncs_LteTimeToConnect, kMemfaultMetricType_Timer
  • Ncs_LteConnectionLossCount
  • Ncs_LtePsmTauSec
  • Ncs_LtePsmActiveTimeSec
  • Ncs_LteEdrxIntervalMsec
  • Ncs_LteEdrxPtwMsec
  • Ncs_LteMode
  • Ncs_LteModemFwVersion

These Kconfig flags control whether those metrics are enabled:

When setting up a Project in the Memfault app, selecting the nRF91 MCU will pre-populate a few fleet metrics charts matching the optional metrics above:

Image showing the pre-populated fleet metric charts

See the list in this file in the NCS repo:

Example projects are provided both by Memfault and Nordic:

Nordic Memfault Documentation:

Frequently Asked Questions (FAQ)​

Updating the Memfault SDK​

The nRF Connect SDK embeds the Memfault SDK as a module:

nrf/west.yml
loading...

If you would like to update the Memfault SDK without updating the nRF Connect SDK, you can do so by adding the Memfault SDK as a module in your application's west.yml:

west.yml

        manifest:
          remotes:
            - name: nrf-connect-sdk
              url-base: https://github.com/nrfconnect
            # Add the Memfault GitHub repo
            - name: memfault
              url-base: https://github.com/memfault

          projects:
            - name: sdk-nrf
              remote: nrf-connect-sdk
              path: nrf
              revision: 2.3.0
              import: true

            # Explicitly add the Memfault SDK, to override the version in the sdk-nrf manifest
            - name: memfault-firmware-sdk
              path: modules/lib/memfault-firmware-sdk
              revision: 0.43.3
              remote: memfault
          self:
            path: my_example_application

Zephyr's west tool will use the explicitly listed version of the memfault-firmware-sdk module instead of the one specified by the nrf-connect-sdk module. Reference: