Nordic nRF Connect SDK Integration Guide
In this guide we will walk through the steps for integrating the Memfault Firmware SDK into a project using Nordic Semiconductor's nRF Connect SDK. The integration has been tested against the releases ranging from v1.4.0 up to v2.3.0. All versions of the nRF Connect SDK since v1.6.0 include Memfault when downloaded from Nordic's website.
This integration is written for the nrf9160-DK, but is similar when targeting other boards with the nRF Connect SDK.
nRF Connect SDK v1.6.0, and all versions since, has built-in support for the Memfault Firmware SDK on nRF9160-based targets. Nordic Semiconductor provides some excellent documentation on how to integrate Memfault in your existing nRF Connect SDK project, together with a sample integration project. We recommend following both this documentation page as well as Nordic's.
Upon completion of the integration, the following subcomponents will be added to your system!
Integration Steps
This tutorial assumes you have a working nRF Connect SDK Environment and are able to flash a board supported by the SDK (i.e nRF91, nRF53, nRF52, etc)
Create a Project and get a Project Key
Go to app.memfault.com and from the "Select A Project" dropdown, click on "Create Project" to setup your first project such as "smart-sink-dev".
Once you've created your project, you'll be automatically taken to an page that includes your project key. Copy the key and follow the rest of this guide.
Include memfault-firmware-sdk in the build
- nRF Connect SDK 1.6.0
- nRF Connect SDK <=1.5.0
You will first need to add the memfault-firmware-sdk to your projects manifest.
By default, for the nRF Connect SDK this is located at nrf/west.yml
:
diff --git a/west.yml b/west.yml
index 17071f47..84bb97f0 100644
--- a/west.yml
+++ b/west.yml
@@ -112,6 +114,10 @@ manifest:
- name: Alexa-Gadgets-Embedded-Sample-Code
path: modules/alexa-embedded
revision: face92d8c62184832793f518bb1f19379538c5c1
remote: alexa
+ - name: memfault-firmware-sdk
+ url: https://github.com/memfault/memfault-firmware-sdk
+ path: modules/memfault-firmware-sdk
+ revision: master
There's a couple lines in this file that look similar. Make sure you have added this under the "projects:" level.
You will then need to run west update
to clone the memfault-firmware-sdk:
$ west update
# ...
=== updating memfault-firmware-sdk (modules/memfault-firmware-sdk):
--- memfault-firmware-sdk: initializing
Initialized empty Git repository in modules/memfault-firmware-sdk/.git/
--- memfault-firmware-sdk: fetching, need revision master
The SDK is part of the West manifest in nRF Connect SDK and is automatically
downloaded when running west update
. By default, it is downloaded to
<nRF Connect SDK path>/modules/lib/memfault-firmware-sdk/
.
Update Kconfig selections in prj.con
- nRF Connect SDK 1.6.0
- nRF Connect SDK <=1.5.0
Add the appropriate options to your system's prj.conf
:
# Project-specific configuration settings
CONFIG_MEMFAULT=y
# Add to pick up support for sending Memfault data over HTTP
CONFIG_MEMFAULT_HTTP_ENABLE=y
# nRF9160 only, uses modem to store root CA's
CONFIG_MEMFAULT_ROOT_CERT_STORAGE_NRF9160_MODEM=y
Add the appropriate options to your system's prj.conf
:
# Project-specific configuration settings
CONFIG_MEMFAULT=y
CONFIG_MEMFAULT_NCS_PROJECT_KEY="YOUR_PROJECT_KEY"
# On nRF52 or nRF53 projects, and nRF-Connect SDK <2.1.0 (Memfault SDK <
# 0.32.0), this setting is required to build the project (it de-selects the
# built-in Memfault HTTPS root certificate storage, which is only compatible
# with the nRF9160):
CONFIG_MEMFAULT_ROOT_CERT_STORAGE_CUSTOM=y
If you'd like to update the Memfault SDK without updating the nRF-Connect SDK, see the instructions here. Memfault aspires to maintain full backwards compatibility with supported nRF-Connect SDK versions.
You can find more details on the available options using menuconfig
,
guiconfig
, and in the Kconfig sources in
modules/memfault-firmware-sdk/ports/zephyr/Kconfig
.
- nRF Connect SDK 1.6.0
- nRF Connect SDK <=1.5.0
Zephyr Patch
nRF Connect SDK 1.4 and above include a Zephyr patch that enables collection of full register state when a crash takes place.
For older SDK versions, you need to apply the patch yourself. For convenience,
we have included the patch in the memfault-firmware-sdk, up through 0.31.5
of
the Memfault Firmware SDK:
$ cd ${ZEPHYR_BASE}
$ git apply ${MEMFAULT_FIRMWARE_SDK}/ports/zephyr/v2.x/coredump-support.patch
Zephyr 1.14 support was removed in 0.32.0
of the Memfault Firmware SDK;
projects using nRF Connect SDK < 1.4 will need to stay on version 0.31.5
or
older of the Memfault Firmware SDK.
Implement Platform Dependencies
In order to save traces, you will need to implement a dependency function that
tells memfault version information. You can add this to the main.c
for the app
being built or a file of its own.
#include "memfault/core/platform/device_info.h"
void memfault_platform_get_device_info(sMemfaultDeviceInfo *info) {
// platform specific version information
*info = (sMemfaultDeviceInfo) {
.device_serial = "DEMOSERIAL",
.software_type = "main-mcu-fw",
.software_version = "1.0.0",
.hardware_version = "pvt",
};
}
Implement nRF9160 Platform Dependencies
Add the Project Key and install the Root CA's used by Memfault:
#include "memfault/http/http_client.h"
sMfltHttpClientConfig g_mflt_http_client_config = {
.api_key = "<YOUR_PROJECT_KEY>",
};
And call memfault_nrfconnect_port_install_root_certs();
prior to calling
lte_lc_init_and_connect()
:
#include "memfault/nrfconnect_port/http.h"
void main(void) {
// ...
memfault_nrfconnect_port_install_root_certs();
lte_lc_init_and_connect();
// ...
You can skip this section.
Create trace reasons definition file
The trace event module within the SDK makes it easy to track errors in a way that requires less storage than full coredump traces and also allows the system to keep running after capturing the event. The program counter, return address and a custom "reason" are saved.
The list of custom reasons is defined in a separate file that you need to create.
The file must be named memfault_trace_reason_user_config.def
and placed in a
directory included in your project.
$ cd $YOUR_PROJECT_ROOT
$ mkdir -p config
$ touch config/memfault_trace_reason_user_config.def
And then in your project's CMakeLists.txt
add:
zephyr_include_directories(config)
To start, we recommend adding a couple reasons for error paths in your codebase (such as peripheral bus read/write failures, transport errors and unexpected timeouts).
Here is what the memfault_trace_reason_user_config.def
file should look like:
// your_project/config/memfault_trace_reason_user_config.def
MEMFAULT_TRACE_REASON_DEFINE(your_custom_error_reason_1)
// i.e
MEMFAULT_TRACE_REASON_DEFINE(critical_log)
MEMFAULT_TRACE_REASON_DEFINE(flash_error)
// ...
Add Additional Config Files
Two other config files are required to build with the Memfault SDK:
$ cd $YOUR_PROJECT_ROOT
$ mkdir -p config
$ touch config/memfault_platform_config.h
$ touch config/memfault_metrics_heartbeat_config.def
See the nRF Connect Docs for more information.
Generate Some Trace Events
Next, we'll need to use the MEMFAULT_TRACE_EVENT
macro to capture a trace
event when an error occurs.
Note that it is perfectly fine to use the same reason in different places. Because the program counter and return address are captured in the trace event, you will be able to see the two topmost frames (function name, source file and line) in Memfault's Issue UI and distinguish between the two.
The nRF Connect SDK port also exposes a mflt trace
CLI command which can be
used to generate events for test purposes:
uart:~$ mflt trace
<dbg> <mflt>: Trace Event Generated!
You can also start to add trace events for error paths you are interested in tracking:
#include "memfault/core/trace_event.h"
// [ ...]
void ble_le_process_ll_pkt(...) {
// ...
if (invalid_msg_id) {
MEMFAULT_TRACE_EVENT(bt_invalid_msg_id);
// ...
}
// ..
}
Log metadata can also be added to a trace event to capture additional info:
void record_temperature(...) {
int rv = spi_flash_erase(...);
if (rv != 0) {
MEMFAULT_TRACE_EVENT_WITH_LOG(flash_error, "Flash Erase Failure: rv=%d, spi_err=0x%x",
spi_bus_get_status());
}
}
Publish data to the Memfault cloud
Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.(nRF9160) HTTPS
When HTTP is available, you can use the Memfault integration to post data from
the CLI (mflt post_chunks
) or by adding a periodic task:
#include "memfault/nrfconnect_port/http.h"
void some_periodic_task(void) {
memfault_nrfconnect_port_post_data();
}
Bluetooth Low Energy
Using the Memfault Diagnostic Service (MDS), nRF-Connect SDK v2.1.0+
As of the v2.1.0 release of the nRF-Connect SDK, there is built-in support for the Memfault Diagnostic Service, which is a Bluetooth LE GATT service designed to enable exporting Memfault data in a standardized way.
Nordic has detailed documentation on the Memfault Diagnostic service here:
There is also a sample application, "Bluetooth LE: Peripheral Memfault Diagnostic Service (MDS)", documented here:
Nordic and Memfault provide the following Bluetooth LE gateways for testing the MDS implementation:
the Python MDS Bluetooth LE gateway script, from the nRF-Connect SDK: https://github.com/nrfconnect/sdk-nrf/tree/v2.1.0/scripts/memfault
a Web Bluetooth LE browser-based tool, from Memfault: https://mflt.io/mds
iOS and Android mobile applications and open-source libraries:
For a quick test, we recommend using the "Bluetooth LE: Peripheral Memfault Diagnostic Service (MDS)" sample application and one of the mobile apps with a compatible mobile device.
Without using the MDS
Extensive details about how data from the Memfault SDK makes it to the cloud can be found here. In short, the Memfault SDK packetizes data into "chunks" that must be pushed to the Memfault cloud via the chunk REST endpoint.
A typical integration looks like this:
#include "memfault/core/data_packetizer.h"
// [...]
bool try_send_memfault_data(void) {
// buffer to copy chunk data into
uint8_t buf[USER_CHUNK_SIZE];
size_t buf_len = sizeof(buf);
bool data_available = memfault_packetizer_get_chunk(buf, &buf_len);
if (!data_available ) {
return false; // no more data to send
}
// send payload collected to chunks endpoint; this could for example send the
// packet over a GATT characteristic to the peer device
user_transport_send_chunk_data(buf, buf_len);
return true;
}
void send_memfault_data(void) {
// [... user specific logic deciding when & how much data to send
while (try_send_memfault_data()) { }
}
Force a Crash and Upload to Memfault
Example crashes can be generated using the mflt
CLI:
uart:~$ mflt crash
# ...
*** Booting Zephyr OS build v2.4.0-ncs1 ***
<inf> <mflt>: GNU Build ID: ef460cafbcf67633d470c125a7cc8fd02ed97538
Memfault Demo App Started!
uart:~$ mflt get_core
<inf> <mflt>: Has coredump with size: 3764
On nRF91 you can post directly to the cloud:
uart:~$ mflt post_chunks
<dbg> <mflt>: Response Complete: Parse Status 0 HTTP Status 202!
<dbg> <mflt>: Body: Accepted
<dbg> <mflt>: No more data to send
Test Tip
Data can also also be dumped out over the CLI using the mflt export_data
CLI
command.
uart:~$ mflt export_data
<inf> <mflt>: MC:SFQCpwIBAwEHalRFU1RTRVJJQUwKbXRlc3Qtc29mdHdhcmUJajEuMC4wLXRlcw==:
<inf> <mflt>: MC:gCx0Bm10ZXN0LWhhcmR3YXJlBKEBoXJjaHVua190ZXN0X3N1Y2Nlc3MBMeQ=:
The CLI output can then be saved to a file and the "chunk" data can be posted to the Memfault Cloud using the Memfault CLI tool:
$ memfault --project-key ${YOUR_PROJECT_KEY} post-chunk --encoding sdk_data_export cli-output.txt
Found 2 Chunks. Sending Data ...
Success
Upload Symbol File
At this point, you should be able to generate a test trace event and push it to the Memfault UI. You can confirm the issue has arrived succesfully by navigating to the "Issues" page. Follow the link pointed to below and upload a symbol file.
After this step, you will see the trace in the list of issues!
You can programmatically upload symbol files with the Memfault CLI tool.
Troubleshooting Data Transfer
See the docs on data transfer troubleshooting.
Sending data to Memfault over UDP
Memfault servers do not accept UDP transfers. To transfer data from a device that can send UDP datagrams (such as the nRF9160), an additional server is necessary to act as a relay between the device and Memfault servers.
A
working sample of a complete set-up can be found here.
It features a Python UDP server that relays data to Memfault, and a firmware
implementation that includes a UDP client. The device and the Python server use
a simple binary encoding format that includes the following sections, separated
by a NULL
byte:
- A version section identifying the encoding format for forward compatibility.
- A Project Key.
- The device serial.
- The Memfault chunk.
This configuration allows for a simple, stateless UDP server that just passes data over to Memfault.
Configuring and Invoking an OTA Update
By default, the NRF Connect
sample integration project
does not come with OTA support enabled by default. To enable this support a few
things need to be added. The update process is invoked with a CLI command, this
needs to be enabled. the memfault_platform_config.h
file, add this line:
#define CONFIG_MEMFAULT_FOTA_CLI_CMD 1
In the prj.conf
file the following lines need to be added:
# The subsystems we need so OTA payloads can be written to
# flash and updated by MCUBoot
CONFIG_DFU_TARGET=y
CONFIG_DFU_TARGET_MCUBOOT=y
CONFIG_IMG_MANAGER=y
CONFIG_FLASH=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y
# For Memfault FOTA, we will use the FOTA_DOWNLOAD API's
# from the nRF Connect SDK which depends on the DOWNLOAD_CLIENT
CONFIG_FOTA_DOWNLOAD=y
CONFIG_DOWNLOAD_CLIENT=y
# Enable printing of file download progress to console
CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y
CONFIG_MEMFAULT_FOTA=y
CONFIG_DOWNLOAD_CLIENT_MAX_FILENAME_SIZE=400
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=1600
The full prj.conf
file can be viewed
here.
Confirm you have already built and uploaded the OTA compatible app_update.bin
to Memfault with the appropriate semantic version set. The device you wish to
update should be running a lower version.
Finally to invoke the OTA process enter the following command:
uart:~$ mflt_nrf fota
<inf> <mflt>: Checking for FOTA
<inf> <mflt>: FOTA Update Available. Starting Download!
<inf> download_client: Setting up TLS credentials, tag 1003
<inf> download_client: Configuring socket timeout (30 s)
...
<inf> <mflt>: FOTA In Progress
<inf> download_client: Downloaded 2048/204467 bytes (1%)
<inf> download_client: Downloaded 4096/204467 bytes (2%)
...
<inf> download_client: Downloaded 204467/204467 bytes (100%)
<inf> download_client: Download complete
<inf> dfu_target_mcuboot: MCUBoot image upgrade scheduled. Reset device to apply
*** Booting Zephyr OS build v2.7.99-ncs1 *** to install update!
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
The device resets automatically, running your new firmware.
Built-in Metrics for the nRF9160
The nRF Connect SDK optionally provides some pre-enabled metrics for the nRF9160 example:
Ncs_AtCmdUnusedStack
Ncs_ConnectionPollUnusedStack
Ncs_LteTimeToConnect, kMemfaultMetricType_Timer
Ncs_LteConnectionLossCount
Ncs_LtePsmTauSec
Ncs_LtePsmActiveTimeSec
Ncs_LteEdrxIntervalMsec
Ncs_LteEdrxPtwMsec
Ncs_LteMode
These Kconfig flags control whether those metrics are enabled:
When setting up a Project in the Memfault app, selecting the nRF91
MCU will
pre-populate a few fleet metrics charts matching the optional metrics above:
See the list in this file in the NCS repo:
Useful links
Example projects are provided both by Memfault and Nordic:
Nordic maintained Asset Tracker v2 Application for nRF9160
- see the documentation for how to build with Memfault enabled
Nordic Memfault Documentation:
- nRF Connect SDK documentation on built-in Memfault Firmware SDK integration (starting from v.1.6.0).
Frequently Asked Questions (FAQ)
Updating the Memfault SDK
The nRF Connect SDK embeds the Memfault SDK as a module:
If you would like to update the Memfault SDK without updating the nRF Connect SDK here are some strategies:
Add the Memfault SDK as a module in the application's west.yml
# Example application manifest
manifest:
remotes:
- name: nrf-connect-sdk
url-base: https://github.com/nrfconnect
projects:
- name: sdk-nrf
remote: nrf-connect-sdk
path: nrf
revision: v2.3.0
import: true
# list the Memfault SDK as a top-level module
- name: memfault-firmware-sdk
url: https://github.com/memfault/memfault-firmware-sdk
path: modules/lib/memfault-firmware-sdk
revision: 0.41.1
self:
path: my_example_application
Zephyr's west
tool will use the explicitly listed version of the
memfault-firmware-sdk
module instead of the one specified by the
nrf-connect-sdk
module. Reference:
Fork the nRF Connect SDK
Update the west.yml
in the forked copy to point to a different Memfault SDK
version. This strategy is only recommended if other modifications to the nRF
Connect SDK are required, since it means maintaining and updating the forked
repo.