Trace events make it easy to track the frequency at which unexpected errors are occurring on a running system.
Since all the data collected by the Memfault SDK is passed opaquely up to the Memfault cloud, after following this initial guide to send trace events no further changes will be needed on the transport side to send any of the other types of data which can be collected by the SDK (such as coredumps and metrics).
Here's an example where Trace Events are captured for Bluetooth protocol CRC errors and invalid message IDs:
Using a Git client, clone the
For Make or CMake based projects, the Memfault SDK provides a helper file which can be included by the build system to pick up the necessary dependencies.
For example, the Make integration looks like this:
If you are using another build system, sources can also be added to the projects source list manually:
$MEMFAULT_FIRMWARE_SDK/components/[core, util]/includeto the include paths provided as
-Iarguments to gcc
- pick up the source files located at
The exact list you need for this tutorial can be found here
Aside from the program counter and return address, a trace event also contains a
user-defined "error reason". The list of custom reasons is defined in a separate
configuration file named
memfault_trace_reason_user_config.def which you need
memfault_trace_reason_user_config.def file will get
#include-ed, so make
sure the directory in which you create the file is part of the header search
To start, we recommend adding a "test" trace error reason you can easily trigger (i.e via a CLI command) and a couple for error paths in your codebase (such as peripheral bus read/write failures, transport errors and unexpected timeouts).
Here is what the
memfault_trace_reason_user_config.def file should look like:
Next, we'll need to use the
MEMFAULT_TRACE_EVENT macro to capture a trace
event when an error occurs.
Note that it is perfectly fine to use the same reason in different places if that makes sense in the context of your code. Because the program counter and return address are captured in the trace event, you will be able to see the two topmost frames (function name, source file and line) in Memfault's Issue UI and distinguish between the two.
For test purposes, you can add a CLI command that logs a trace event:
You can also start to add trace events for error paths you are interested in tracking:
In order to save traces, you will need to implement the dependency functions below:
All events generated in the Memfault SDK are stored and transmitted using a
compressed format (CBOR). As they await to be sent, they are stored in the
"event storage" core component. The size of each trace event requires ~50 bytes
of storage. The exact size needed to store a single event can be determined with
memfault_trace_event_compute_worst_case_storage_size(). On bootup, initialize
trace event storage like this:
At this point data collection for trace events is fully implemented! Now we just need to push the data to the Memfault cloud ...
Create a Project and get a Project Key
Go to app.memfault.com and from the "Select A Project" dropdown, click on "Create Project" to setup your first project such as "smart-sink-dev".
Once you've created your project, you'll be automatically taken to an integration guide which includes your project key. Copy it to follow the rest of the guide.
Extensive details about how data from the Memfault SDK makes it to the cloud can be found here. In short, the Memfault SDK packetizes data into "chunks" that must be pushed to the Memfault cloud via the chunk REST endpoint
A typical integration looks like this:
At this point you should be able to generate a test trace event and push it to the Memfault UI. You can confirm the issue has arrived succesfully by navigating to the "Issues" page. Follow the link pointed to below and upload a symbol file.
After this step you will see the trace in the list of issues!
You can programatically upload symbol files with the Memfault CLI tool.
If you run into any issues with your data transfer implementation, we have tools to help!
- A UI you can use to view the raw "Chunk" data payloads that have arrived for your project.
- Precanned Data Payloads you can
pass through your
user_transport_send_chunk_data()implementation to test data transfer in isolation.
- A GDB Script which can be installed to send chunks programatically every time a breakpoint is hit.
With the transport path in place and a minimal set of data flowing, there's now a number of firmware-only changes which can be made to start collecting other diagnostic data from your system. You can find links to step-by-step guides for integrating each subsystem here.