Skip to main content

ARM Cortex-M Integration Guide

Overview#

The following guide will walk you through step-by-step how to integrate and test the Memfault SDK for a Cortex-M device using the GNU GCC, Clang, IAR, ARM MDK, or TI ARM Compiler.

Upon completion of the integration, the following subcomponents will be added to your system!

Pre-Integration#

(Optionally) Collect a coredump capture using GDB#

If you do not use GDB, skip ahead to the next step.

If you use GDB, a full coredump can be captured by just using the debugger! This can be useful during development to take advanatge of Memfault's analyzers when a device crashes or hangs.

To perform the capture, navigate to http://app.memfault.com/ and select the "Issues" page in the Memfault UI, click on the "Manual Upload" button, and click on "walk-through on how to upload your first coredump". From there you can follow the guided steps to perform a capture. At the end you will see an analysis of your system state.

Create a Project and get a Project Key

Go to app.memfault.com and from the "Select A Project" dropdown, click on "Create Project" to setup your first project such as "smart-sink-dev".

Once you've created your project, you'll be automatically taken to an integration guide which includes your project key. Copy it to follow the rest of the guide.

Integration Steps#

Prepare folder for memfault-firmware-sdk & port#

The memfault-firmware-sdk is a self-contained C SDK you will need to include into your project.

Create a folder to hold memfault-firmware-sdk as well as the configuration and porting files to your platform.

cd ${YOUR_PROJECT_DIR}
mkdir -p third_party/memfault
cd third_party/memfault
# Add memfault repo as submodule, subtree, or copy in as source directly
git submodule add https://github.com/memfault/memfault-firmware-sdk.git memfault-firmware-sdk
cp memfault-firmware-sdk/ports/templates/* .

When you are done, you should have the following directory structure in your project:

memfault/
//...
β”œβ”€β”€ memfault-firmware-sdk (submodule)
| # Files where port / glue layer to your platform will be implemented
β”œβ”€β”€ memfault_platform_port.c
|
| # Configuration Headers
β”œβ”€β”€ memfault_metrics_heartbeat_config.def
└── memfault_trace_reason_user_config.def
└── memfault_platform_log_config.h
└── memfault_platform_config.h

Add Sources to Build System#

Based on the build system you are using, expand the appropriate tab below and follow the steps to add the sources to your target

MEMFAULT_PORT_ROOT := <YOUR_PROJECT_ROOT>/third_party/memfault
MEMFAULT_SDK_ROOT := $(MEMFAULT_PORT_ROOT)/memfault-firmware-sdk
MEMFAULT_COMPONENTS := core util panics metrics
include $(MEMFAULT_SDK_ROOT)/makefiles/MemfaultWorker.mk
<YOUR_SRC_FILES> += \
$(MEMFAULT_COMPONENTS_SRCS) \
$(MEMFAULT_PORT_ROOT)/memfault_platform_port.c
<YOUR_INCLUDE_PATHS> += \
$(MEMFAULT_COMPONENTS_INC_FOLDERS) \
$(MEMFAULT_SDK_ROOT)/ports/include \
$(MEMFAULT_PORT_ROOT)
info

Be sure to update YOUR_SRC_FILES, YOUR_INCLUDE_PATHS, and YOUR_PROJECT_ROOT accordingly for your system above!

Core Dependencies#

Open the memfault_platform_port.c copied into your third_party/memfault folder and fill out the stub implementations accordingly.

Initialize Memfault Subsystem On Bootup#

From your main routine, add a call to memfault_platform_boot() prior to the startup of an RTOS or baremetal while loop.

#include "memfault/components.h"
// ...
int main(void) {
// ...
memfault_platform_boot();
// ...
}

Reboot Tracking Dependencies#

Memfault has a module for tracking and reporting what resets are taking place on your platform.

Implement the following function for your MCU. The ports listed in other tabs serve as a good reference.

//! @file memfault_platform_port.c
void memfault_reboot_reason_get(sResetBootupInfo *info) {
const uint32_t reset_cause = 0; // TODO: Populate with MCU reset reason
eMemfaultRebootReason reset_reason = kMfltRebootReason_Unknown;
// TODO: Convert MCU specific reboot reason to memfault enum
*info = (sResetBootupInfo) {
.reset_reason_reg = reset_cause,
.reset_reason = reset_reason,
};
}

Allocate noinit region & Collect Reboot Info#

Add the following to your memfault_platform_port.c

//! @file memfault_platform_port.c
MEMFAULT_PUT_IN_SECTION(".noinit.mflt_reboot_tracking")
static uint8_t s_reboot_tracking[MEMFAULT_REBOOT_TRACKING_REGION_SIZE];
void memfault_platform_reboot_tracking_boot(void) {
sResetBootupInfo reset_info = { 0 };
memfault_reboot_reason_get(&reset_info);
memfault_reboot_tracking_boot(s_reboot_tracking, &reset_info);
}
note

It's expected that s_reboot_tracking is placed in "noinit" RAM. That is, a region of RAM not initialized on bootup or used by your bootloaders. This can be achieved by adding a "noinit" section to your linker script.

Logging Dependency#

The Memfault SDK will (sparingly) emit diagnostic logs to alert of integration configuration problems. The logging subsystem can also easily serve as logging infrastructure for your own platform if you do not yet have one set up.

Based on your setup, choose one of the options below:

Platform does logging via Macros
  1. Add #define MEMFAULT_PLATFORM_HAS_LOG_CONFIG 1 to third_party/memfault/memfault_platform_config.h
  2. Remap Memfault logging macros to platform logging macros by adding the following macros to the memfault_platform_log_config.h file created earlier.
//! @file memfault_platform_log_config.h
#pragma once
#define MEMFAULT_LOG_DEBUG(fmt, ...) YOUR_PLATFORM_DEBUG_LOG( fmt, ## __VA_ARGS__)
#define MEMFAULT_LOG_INFO(fmt, ...) YOUR_PLATFORM_INFO_LOG( fmt, ## __VA_ARGS__)
#define MEMFAULT_LOG_WARN(fmt, ...) YOUR_PLATFORM_WARN_LOG( fmt, ## __VA_ARGS__)
#define MEMFAULT_LOG_ERROR(fmt, ...) YOUR_PLATFORM_ERROR_LOG( fmt, ## __VA_ARGS__)
Platform has no logging or uses printf()

Implement memfault_platform_log() in third_party/memfault/memfault_platform_port.c. For example,

void memfault_platform_log(eMemfaultPlatformLogLevel level, const char *fmt, ...) {
va_list args;
va_start(args, fmt);
char log_buf[128];
vsnprintf(log_buf, sizeof(log_buf), fmt, args);
const char *lvl_str;
switch (level) {
case kMemfaultPlatformLogLevel_Debug:
lvl_str = "D";
break;
case kMemfaultPlatformLogLevel_Info:
lvl_str = "I";
break;
case kMemfaultPlatformLogLevel_Warning:
lvl_str = "W";
break;
case kMemfaultPlatformLogLevel_Error:
lvl_str = "E";
break;
default:
break;
}
vsnprintf(log_buf, sizeof(log_buf), fmt, args);
your_platform_printf("[%s] MFLT: %s\n", lvl_str, log_buf);
}

Timer Dependencies#

The memfault metric subsystem requires a repeating timer in order to collect "heartbeats" and a time since boot

Implement the following function for your MCU. The ports listed in other tabs serve as a good reference.

//! @file memfault_platform_port.c
bool memfault_platform_metrics_timer_boot(uint32_t period_sec, MemfaultPlatformTimerCallback callback) {
// Schedule a timer to invoke callback() repeatedly after period_sec
return true;
}
uint64_t memfault_platform_get_time_since_boot_ms(void) {
// Return time since boot in ms, this is used for relative timings.
return 0;
}

RTOS Port Files#

The Memfault SDK includes pre-canned integrations for many RTOS environments. Expand the tab for the one applies to your system below in order to pick them up.

FreeRTOS Specific Sources

Add FreeRTOS Specific Sources To Compilation#

${MEMFAULT_SDK_ROOT}/ports/freertos/src/memfault_core_freertos.c
${MEMFAULT_SDK_ROOT}/ports/freertos/src/memfault_freertos_ram_regions.c
${MEMFAULT_SDK_ROOT}/ports/freertos/src/memfault_panics_freertos.c

Register Memfault Task Tracing in FreeRTOSConfig.h#

//! @file FreeRTOSConfig.h
#pragma once
#include "memfault/ports/freertos_trace.h"
tip

While in the file, we recommend using the following settings to catch issues:

#define configCHECK_FOR_STACK_OVERFLOW 2
void vAssertCalled(const char *file, int line);
#define configASSERT(x) if ((x) == 0) vAssertCalled( __FILE__, __LINE__ )

Initialize FreeRTOS Port on Boot#

#include "memfault/components.h"
+ #include "memfault/ports/freertos.h"
//...
int memfault_platform_boot(void) {
+ memfault_freertos_port_boot();
+
memfault_build_info_dump();

Register Assert Handler#

Memfault can automatically capture crash information on asserts by calling MEMFAULT_ASSERT() from your platforms assert handler:

#include "memfault/panics/assert.h"
#define YOUR_PLATFORM_ASSERT(x) MEMFAULT_ASSERT(x)

Implement Coredump Storage Dependency#

When the system faults or asserts crash information can be recorded and sent to Memfault after the device reboots. In order for this information to be saved, it must be persisted to a storage area that persists across a device reset.

Implement Coredump Storage Area#

Coredumps can be saved in a .noinit region of SRAM, internal flash, external flash, or even streamed out over a peripheral to another MCU when a coredump takes place.

Custom Port Template#

//! @file memfault_platform_port.c
void memfault_platform_coredump_storage_get_info(sMfltCoredumpStorageInfo *info) {
*info = (sMfltCoredumpStorageInfo) {
.size = /* size of coredump storage area */,
};
}
bool memfault_platform_coredump_storage_read(uint32_t offset, void *data,
size_t read_len) {
}
bool memfault_platform_coredump_storage_erase(uint32_t offset, size_t erase_size) {
}
bool memfault_platform_coredump_storage_write(uint32_t offset, const void *data,
size_t data_len) {
}
void memfault_platform_coredump_storage_clear(void) {
}

Add a Unique Identifier to target#

Memfault has facilities for capturing a unique identifier automatically as part of the build process. This information is used by Memfault's cloud infrastructure to ensure the symbol file uploaded matches the data reported in a coredump.

note

A longer discussion about firmware versioning best practices and build identifiers can be found here.

Memfault has two ways to add a Build ID to a target. Select the appropriate one below based on your toolchain setup.

GNU Build ID (Requires Use of GNU GCC Compiler)
  1. Add -Wl,--build-id to flags passed to linker via GCC
  2. Add #define MEMFAULT_USE_GNU_BUILD_ID 1 to third_party/memfault/memfault_platform_config.h
  3. Add the following snippet to your projects linker script (.ld file) where "FLASH" below will match the name of the MEMORY section read-only data and text is placed in.
.note.gnu.build-id :
{
__start_gnu_build_id_start = .;
KEEP(*(.note.gnu.build-id))
} > <YOUR_FLASH_SECTION>
info

Be sure to update \<YOUR_FLASH_SECTION\> to match the name of the section .text is placed in!

Python Script Run on ELF (Compiler Agnostic)

Update your build system to invoke the scripts/fw_build_id.py script on your ELF as part of a post-build step.

For example, using CMake, this can be achieved by adding a custom command to the target:

add_custom_command(TARGET ${YOUR_EXECUTABLE_NAME} POST_BUILD
COMMAND python ${MEMFAULT_SDK_ROOT}/scripts/fw_build_id.py ${YOUR_EXECUTABLE_NAME}
)
info

The script depends on pyelftools so make sure you have run pip install pyelftools or added to your requirements.txt

Confirm Integration Links#

At this point your project should compile and be able to boot. On startup you should see some messaging like the following print when memfault_platform_boot() is called:

[I] Memfault Build ID: cefaeb9407ab0dac7a5efe9fd510618ee21e9df7
[I] S/N: MFLT0123
[I] SW type: app-fw
[I] SW version: 1.0.0+cefaeb940
[I] HW version: dvt1
[I] Reset Reason, RESETREAS=0x4
[I] Reset Causes:
[I] Software
[I] Memfault Initialized!

Verification & Tuning Steps#

With Memfault compiling into your build, it's now time to add some initial instrumentation, and test the integration!

Define First Trace Event#

Trace events are used for generating alerts in the Memfault UI when unexpected events are encountered. Let's start by creating a generic type for critical errors that have been detected.

--- a/config/memfault_trace_reason_user_config.def
+++ b/config/memfault_trace_reason_user_config.def
@@ -0,0 +1 @@
+MEMFAULT_TRACE_REASON_DEFINE(critical_error)

Define First Heartbeat#

Heartbeat metrics allow you to easily monitor your platform and confirm it is operating as expected.

Typical Heartbeat Examples
  • investigate problems that didn't cause a reboot (bad connectivity, network or sensor error rates) and marginality that crops up in a fleet
  • providing leading indicators of problems (rapid battery drain, drop in activity, etc)
  • compare trends across releases (improved connectivity, data efficiency etc)

Best Practices around each metric type that we recommend:

  • Timers - These are used to track time spent in particular states and can be used for debugging connectivity, battery life, and performance issues.
  • Counters - Track counts of a particular event class taking place. Suggested use cases are:
    • Operations your system are performing (number of events serviced by a task, bytes sent over network, bytes written to flash). Alerts can be configured to identify devices operating outside normal thresholds.
    • Counts of events for events that should never happen (i2c bus write error count, flash write error). You can alert if any of these situations are seen.
  • Gauges - These are values sampled at the end of each heartbeat interval. Common items include
    • Battery Metrics (Drop over an hour, current percent)
    • Heap utilization / stack high watermark

Add Metric to memfault_metrics_heartbeat_config.def#

//! @file memfault_metrics_heartbeat_config.def
MEMFAULT_METRICS_KEY_DEFINE(MainTaskWakeups, kMemfaultMetricType_Unsigned)

Instrument Code to Update Heartbeat#

voit my_main_task(void) {
while (1) {
your_rtos_wait_for_event();
memfault_metrics_heartbeat_add(
MEMFAULT_METRICS_KEY(MainTaskWakeups), 1);
}
}

Add Test Commands to CLI#

The Memfault SDK functionality can be easily exercised via CLI test commands.

note

If your platform does not have a CLI, these commands can also be wired up to a button press or called from main() on bootup from a test image

#include "memfault/components.h"
//
// Test Platform Ports
//
int test_logging(int argc, char *argv[]) {
MEMFAULT_LOG_DEBUG("Debug log!");
MEMFAULT_LOG_INFO("Info log!");
MEMFAULT_LOG_WARN("Warning log!");
MEMFAULT_LOG_ERROR("Error log!");
return 0;
}
// Runs a sanity test to confirm coredump port is working as expected
int test_coredump_storage(int argc, char *argv[]) {
// Note: Coredump saving runs from an ISR prior to reboot so should
// be safe to call with interrupts disabled.
your_platform_disable_interrupts();
memfault_coredump_storage_debug_test_begin();
your_platform_enable_interrupts();
memfault_coredump_storage_debug_test_finish();
return 0;
}
//
// Test core SDK functionality
//
// Triggers an immediate heartbeat capture (instead of waiting for timer
// to expire)
int test_heartbeat(int argc, char *argv[]) {
memfault_metrics_heartbeat_debug_trigger();
return 0;
}
int test_trace(int argc, char *argv[]) {
MEMFAULT_TRACE_EVENT_WITH_LOG(critical_error, "A test error trace!");
return 0;
}
//! Trigger a user initiated reboot and confirm reason is persisted
int test_reboot(int argc, char *argv[]) {
memfault_reboot_tracking_mark_reset_imminent(kMfltRebootReason_UserReset, NULL);
memfault_platform_reboot();
}
//
// Test different crash types where a coredump should be captured
//
int test_assert(int argc, char *argv[]) {
MEMFAULT_ASSERT(0);
return -1; // should never get here
}
int test_fault(void) {
void (*bad_func)(void) = (void *)0xEEEEDEAD;
bad_func();
return -1; // should never get here
}
int test_hang(int argc, char *argv[]) {
while (1) {}
return -1; // should never get here
}
// Dump Memfault data collected to console
int test_export(int argc, char *argv[]) {
memfault_data_export_dump_chunks();
return 0;
}

Post Data to Cloud via Local Debug Setup#

Prior to having an end-to-end transport in place, Memfault data can be exported over any serial connection or via GDB directly.

Post Chunks To Memfault#

All data collected by Memfault can be exported as opaque packets called "chunks". To trigger an export, call the test_export command added to your device CLI.

If data has been collected, you will see strings with the format: MC:BASE64_ENCODED_CHUNK: in the dump.

note

It's perfectly fine for other logs to be interleaved with the exported data.

For example:

shell> reboot
[00:00:01] INFO: System Booting!
[00:00:02] INFO: Memfault Build ID: d8d6a047282f025fffa29fa767100f310bc40f80
shell> trace
# CLI command making a call to `memfault_data_export_dump_chunks`
shell> export
[00:00:10] INFO: MC:SFQCpwIBAwEHalRFU1RTRVJJQUwKbXRlc3Qtc29mdHdhcmUJajEuMC4wLXRlcw==:

Copy the logs from the console, navigate to the "Chunks Debug" view for your project, and paste the logs:

Select "Next" to review the chunks that were identified and (optionally) update the "Device Serial" being used to report data:

Select "Post" to submit the chunks to Memfault. The chunks will appear in the view directly below. Check and make sure there are no "Errors" to address.

(Optional) Automate Chunk Posting#

Posting chunks from a local setup can be automated using our CLI tool or GDB script.

note

This strategy can also be used even when an end-to-end transport is in place for local QA testing or in CI/CD test automation setups.

Automated Posting Options
Prerequisite: Project Key from Memfault UI#

A Project key will be needed in order to communicate with Memfault's web services. Go to https://app.memfault.com/, navigate to the project you want to use and select 'Settings'β†’'General'. The 'Project Key' listed is what you will want to use.

With Desktop CLI Tool

Install the Python Memfault CLI Tool#

The Memfault Desktop CLI tool. tool is written in Python and published publicly in the Python Package Index (pypi).

To install it, make sure you have a recent version of Python 3.x installed and run:

$ pip3 install memfault-cli
$ memfault --help

Save device logs to file#

Start your console with your favorite terminal client. Let the system run for a bit, periodically dumping data to the console by calling memfault_data_export_dump_chunks.

You should see strings with the format: MC:BASE64_ENCODED_CHUNK: in the dump.

note

It's perfectly fine for other logs to be interleaved with the exported data.

For example:

shell> reboot
[00:00:01] INFO: System Booting!
[00:00:02] INFO: Memfault Build ID: d8d6a047282f025fffa29fa767100f310bc40f80
shell> trace
# CLI command making a call to `memfault_data_export_dump_chunks`
shell> export
[00:00:10] INFO: MC:SFQCpwIBAwEHalRFU1RTRVJJQUwKbXRlc3Qtc29mdHdhcmUJajEuMC4wLXRlcw==:

Save the resulting logs to a file such as logs.txt

Post chunks from logs with Memfault CLI#

Run the desktop Memfault CLI tool on your saved log file. The utility will parse the logs, extract the Memfault data, and post it here for processing!

$ memfault --project-key ${YOUR_PROJECT_KEY} post-chunk --encoding sdk_data_export logs.txt
Found 2 Chunks. Sending Data ...
Success
With GDB

Prerequisites#

  • You need to have a way to debug your product with GDB as part of your development setup
  • The GDB version you are using needs to have the GDB Python API enabled. (It's generally the default or there is a -py version of GDB which has it included.)
  • You need to compile your firmware with debug symbol information (i.e -g CFLAG)
note

Even if you are using other compilers such as ARMCC or IAR, you can load the ELF (.out, .elf) generated and debug it in GDB as well.

Launch GDB and connect debugger#

For example:

$ arm-none-eabi-gdb-py my_app.elf --ex="target remote :3333"
(gdb)

Load Memfault GDB script#

Copy and paste and run the following in gdb:

python exec('try:\n from urllib2 import urlopen\nexcept:\n from urllib.request import urlopen'); exec(urlopen('https://mflt.io/memfault-gdb-script').read())
tip

Some GDB versions do not support Python scripting by default. If you see a message like "Python scripting is not supported in this copy of GDB", you might be able to run an alternative GDB binary with a -py suffix, for example arm-none-eabi-gdb-py.

Register Handler#

Copy the command below and paste it into GDB and hit enter. This will load a handler which automatically posts data to Memfault whenever memfault_data_export_dump_chunks is called.

memfault install_chunk_handler --verbose --project-key <YOUR_PROJECT_KEY>
note

memfault_data_export_dump_chunks() needs to be called by your port as part of a CLI command or periodic task in order for the data to be extracted. For example:

#include "memfault/components.h"
int export_data_cli_command(int argc, char *argv[]) {
memfault_data_export_dump_chunks();
return 0;
}
void some_periodic_task(void) {
memfault_data_export_dump_chunks();
}

Try It Out!#

Now, every time memfault_data_export_dump_chunks is called, the data passed as a parameter to the function will automatically be posted to the Memfault cloud. You don't have to do anything else! You will see logs print in the gdb CLI as data is posted:

(gdb) Continuing.
Successfully posted 45 bytes of chunk data
Successfully posted 53 bytes of chunk data
[...]

Upload Symbol File#

At this point you should be able to generate a test events and crashes and push the data to the Memfault UI.

You can confirm the error traces and crashes have arrived succesfully by navigating to the "Issues" page. Follow the link pointed to below and upload a symbol file.

After this step you will see the trace in the list of issues!

tip

You can programmatically upload symbol files with the Memfault CLI tool.

Troubleshooting Data Transfer#

If you encounter any issues in your data transfer implementation, Memfault has tools to help debug!

Data Transport Steps#

With data collection verified locally, the final step is to push data automatically to the Memfault cloud for analysis.

Post Data to Cloud via Device Transport#

Extensive details about how data from the Memfault SDK makes it to the cloud can be found here. In short, the Memfault SDK packetizes data into "chunks" that must be pushed to the Memfault cloud via the chunk REST endpoint.

The Memfault SDK exposes an API that packetizes Memfault data into "chunks". These "chunks" need to be forwarded to the Memfault Cloud (either directly via HTTPs or indirectly via a gateway device such as a computer or mobile application)

tip

If you cannot post data directly using HTTP, no problem at all. In fact, that is one of the most common configurations. In these setups, data can be proxied from the MCU to a device which can perform a HTTP request over transports such as UART, BLE, COAP, LoRa, Zigbee, etc). The size of a "chunk" is fully configurable by you can be tuned to best match your transport requirements. All message re-assembly is dealt with by Memfault in the cloud.

#include "memfault/components.h"
bool try_send_memfault_data(void) {
// buffer to copy chunk data into
uint8_t buf[USER_CHUNK_SIZE];
size_t buf_len = sizeof(buf);
bool data_available = memfault_packetizer_get_chunk(buf, &buf_len);
if (!data_available ) {
return false; // no more data to send
}
// send payload collected to chunks endpoint
user_transport_send_chunk_data(buf, buf_len);
return true;
}
void send_memfault_data(void) {
if (memfault_packetizer_data_available()) {
return false; // no new data to send
}
// [... user specific logic deciding when & how much data to send
while (try_send_memfault_data()) { }
}