PlatformIO + ESP-IDF Integration Guide

This tutorial will go over integrating the Memfault Firmware SDK into a system that is using Espressif's ESP-IDF for an ESP32 chip, and leveraging the PlatformIO integrated development environment.

tip

An example project demonstrating the result of this integration can be found here, for reference:

https://github.com/memfault/platformio-esp32-espidf

Prerequisites

This guide assumes that PlatformIO has been installed and you have already configured a PlatformIO-based project for ESP-IDF. If not, the PlatformIO docs on working with ESP-IDF as a framework is a great resource!

Before continuing, verify that you are able to flash your device with the pio, PlatformIO's CLI tool:

pio run -t upload

Create a Project and get a Project Key

Go to app.memfault.com and from the "Select A Project" dropdown, click on "Create Project" to setup your first project. Choose a name that reflects your product, such as "smart-sink-dev".

Once you've created your project, you'll be automatically taken to an page that includes your project key. Copy the key and follow the rest of this guide.

1. Clone Memfault SDK

Clone the Memfault SDK as a submodule in the third-party/ directory in your application:

git submodule add https://github.com/memfault/memfault-firmware-sdk.git third-party/memfault-firmware-sdk

2. Add Memfault SDK to the ESP-IDF CMake Build System

In your top level CMakeLists.txt, you will need to Register Memfault esp-idf as a "component":

# CMakeLists.txt (top-level)
cmake_minimum_required(VERSION 3.16)

# Uncomment the line below using the path to your source code if
# it is not ./main/ (e.g. src/).
#
# Later, you can add optionally add platform specific dependencies
# (e.g. memfault_platform_get_device_info()) that the Memfault SDK
# will search this folder for.
#
# esp-idf component registration uses directory naming; typically
# an esp-idf src folder is named 'main', so we need to change from
# the default to 'src'
# set(MEMFAULT_PLATFORM_PORT_COMPONENTS src)

set(MEMFAULT_FIRMWARE_SDK third-party/memfault-firmware-sdk)
include(${MEMFAULT_FIRMWARE_SDK}/ports/esp_idf/memfault.cmake)

# Make sure that including the ESP-IDF project.cmake comes after
# the Memfault additions above
include($ENV{IDF_PATH}/tools/cmake/project.cmake)

# [...]
project(<YOUR_PROJECT>)

info

If you are using compact logging, there is a slightly different form required, use the snippet found here. For more information on compact logging and why you might want to use it, see Compact Logs.

See the example ESP32 project in the Memfault SDK for a working example of this step, located at examples/esp32.

3. Add the Memfault Configuration Files

There are a few files that you must fill in that Memfault requires. There are template versions in <PATH_TO_MEMFAULT_FIRMWARE_SDK>/ports/templates/. Assuming a typical ESP32 project structure with a root directory and a main project sub-directory it is simplest to copy these files into main, e.g.

cd main
mkdir -p memfault
cd memfault
cp ../../third-party/memfault-firmware-sdk/ports/templates/*config* .

Note that only the *config* files are copied, not the memfault_platform_port.c file.

When you are done, you should have the following directory structure in your project:

project-root/
├── CMakeLists.txt  # Top-level
└── main
    ├── CMakeLists.txt  # Project-level
    └── memfault
        | # Configuration Headers
        ├── memfault_metrics_heartbeat_config.def
        ├── memfault_trace_reason_user_config.def
        ├── memfault_platform_log_config.h
        └── memfault_platform_config.h

The directory with the config files should be added to the includes in the project.

With the structure above, the Project-level CMakeLists.txt file should have the "." path added to the INCLUDE_DIRS variable:

# CMakeLists.txt (project-level)
set(COMPONENT_ADD_INCLUDEDIRS
   .
   memfault
   # [Other project include directories]
)

note

Quotes on filenames and paths without spaces are not required in CMake lists.

4. Enable Coredump Collection in sdkconfig

By default, the esp-idf uses the CONFIG_ESP32_ENABLE_COREDUMP_TO_NONE=y. You will need to update sdkconfig (either via idf.py menuconfig or manually) to CONFIG_ESP32_ENABLE_COREDUMP_TO_FLASH=y. The change you will see in the sdkconfig file will look something like:

diff --git a/sdkconfig b/sdkconfig
index a49446d..83a81a8 100644
--- a/sdkconfig
+++ b/sdkconfig
@@ -274,7 +274,7 @@ CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ=160
 CONFIG_TRACEMEM_RESERVE_DRAM=0x0
 # CONFIG_ESP32_ENABLE_COREDUMP_TO_FLASH is not set
 # CONFIG_ESP32_ENABLE_COREDUMP_TO_UART is not set
-CONFIG_ESP32_ENABLE_COREDUMP_TO_NONE=y
+CONFIG_ESP32_ENABLE_COREDUMP_TO_FLASH=y

This can also be done in sdkconfig.defaults if your project does not version control the sdkconfig file (will require a fullclean build to take effect).

Note: The remainder of the COREDUMP settings, including CONFIG_ESP_COREDUMP_DATA_FORMAT_ELF, do not affect the Memfault coredump. They are specific to the ESP-IDF coredump features.

5. Create a Coredump Flash Partition

You will need to allocate an internal flash partition to store panic data. Edit your partitions.csv file to add a coredump partition. If you do not have a partitions.csv file you can create one from the template below and place it in your top-level project directory where the sdkconfig file is located. Depending on your chip's Flash size you may need to modify the storage partition size in order to fit the coredump partition. Check the Flash chip size in menuconfig, see (Top) → Serial flasher config and check the "Flash Size" entry. Be sure to specify a custom partition file in the sdkconfig (see note below).

Depending on what data is included in the coredump, the size of the coredump partition might also need to be increased to a larger size, like 512K. The memfault_coredump_storage_compute_size_required() function in the Memfault SDK can be used to compute the maximum coredump size, which can vary depending on user configuration.

Note that the default Memfault coredump regions are defined by the memfault_platform_coredump_get_regions() function in the Memfault SDK's ESP-IDF port: ports/esp_idf/memfault/common/memfault_platform_coredump.c, but can be adjusted to suit the user's needs, either via Kconfig options or by implementing a separate memfault_platform_coredump_get_regions() to override the built-in weakly defined one.

# Name,   Type, SubType, Offset,  Size,  Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
storage,  data, fat,     ,        1M,
coredump, data, coredump,,        256K,

info

The coredump flash partition size must be aligned to 4096 bytes (4KB).

note

Be sure to indicate the actual name you chose for the partition file in the sdkconfig file. This can be done easily by running idf.py menuconfig, selecting (Top) → Partition Table, and setting the name of the "Custom partition CSV file". Optionally, you can edit the sdkconfig file directly (search for the variables prefixed with CONFIG_PARTITION_TABLE_):

CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partition-table.csv"

6. Initializing Memfault

You will need to initialize the SDK from your application. This can be done simply by adding a call to memfault_boot() from your main_app() during system boot. We recommend doing this as early as possible in your initialization process, so Memfault components can begin capturing data. Be sure to first initialize any dependencies used by memfault_platform_get_device_info() as this is called within memfault_boot.

Here's a code example:

#include "memfault/esp_port/core.h"

void app_main() {
  memfault_boot();

  // Remainder of your main task
}

7. Configure Memfault Project Key

The Project Key is set via a Kconfig variable, MEMFAULT_PROJECT_KEY.

In ESP-IDF this can be set in a few different places:

• Add a line to sdkconfig.defaults:

  CONFIG_MEMFAULT_PROJECT_KEY="your-project-key"

• Set it in your generated sdkconfig file, either via pio run -t menuconfig or by directly editing the file

8. Configure Environment

Next, activate the PlatformIO virtual environment and install Memfault's dependencies::

source ~/.platformio/penv/bin/activate

pip install -r third-party/memfault-firmware-sdk/requirements.txt
pip install memfault-cli
pip install mflt_build_id

9. Add Build ID Script

To enable Memfault to uniquely identify binaries, create a script add_build_id.py in the top-level directory that tags binaries with a Build ID:

"""
Small wrapper to hook the Memfault build id script into platform IO.
"""

Import("env")

env.AddPostAction(
    "$BUILD_DIR/${PROGNAME}.elf",
    env.VerboseAction(
        " ".join(
            [
                "$PYTHONEXE",
                "$PROJECT_DIR/third-party/memfault-firmware-sdk/scripts/fw_build_id.py",
                "$BUILD_DIR/${PROGNAME}.elf",
            ]
        ),
        "Adding build id to $BUILD_DIR/${PROGNAME}.elf",
    ),
)

This script will be hooked into PlatformIO in the next step.

10. Configure platformio.ini

In your platformio.ini file, you'll need to make a handful of changes:

1. Add a CMake argument to indicate to Memfault that this is a PlatformIO project
2. Add a setting to your platformio.ini file to pick up the partitions.csv file added previously
3. Add the add_build_id.py script extra_scripts
4. Add required include folders and one include file for the Memfault SDK to work with PlatformIO

The end result will be:

; needed to tell Memfault that we are using a platformIO project
board_build.cmake_extra_args =
	-DPLATFORMIO_ENABLED=TRUE

; add coredump flash partition
board_build.partitions = partitions_example.csv

; wrapper to run the memfault build id script on the output binary
extra_scripts =
    post:add_build_id.py

; manually include the freertos_trace.h
build_flags =
  -Imain/memfault
  -Ithird-party/memfault-firmware-sdk/third-party/include
  -include third-party/memfault-firmware-sdk/ports/include/memfault/ports/freertos_trace.h

11. Regenerate sdkconfig and recompile

Reset project after all changes have been applied. Note, your generated sdkconfig file may have a different name give environment being using from your platformio.ini file (e.g. [env:esp32-c3-devkitc-02] --> sdkconfig.esp32-c3-devkitc-02).

pio pkg update
pio run --target fullclean
rm sdkconfig.espidf

Build with the platformio CLI:

pio run

12. Upload symbol file to project

Upload the symbol file at ${PROJECT}/.pio/build/espidf/firmware.elf to Memfault using the Memfault CLI tool. We use this information to reconstruct stacktraces and decompress metrics submitted to Memfault.

memfault upload-mcu-symbols \
  --org-token ${MEMFAULT_AUTH_TOKEN} \
  --org ${MEMFAULT_ORG} \
  --project ${MEMFAULT_PROJECT} \
  upload-mcu-symbols \
  ${PROJECT}/.pio/build/espidf/firmware.elf

13. Testing things out

Flash the device and start the serial monitor:

pio run --target upload --target monitor

note

Add the following settings to your platformio.ini file to customize the serial monitor:

; disable filtering so the ANSI colors come through
monitor_filters = direct

; set baud rate
monitor_speed=115200

Now that the project builds, test that the important subsystems of Memfault are working.

In the application console, you can run help to see commands from Memfault

esp32> help

Trigger collection of Heartbeat data and push it to Memfault.

esp32> Heartbeat
esp32> post_chunks

The data should appear in the Processing Log as "Received Heartbeat:" when you navigate to the Processing Log under the Integration Hub.

Trigger a crash and coredump collection with:

esp32> crash

Post the data to Memfault once rebooted:

esp32> post_chunks

The coredump should appear under the Issues tab. If you don't see it, check Integration Hub > Processing Log to check if there were any processing errors.

14. Publish data to the Memfault cloud

At this point, the Memfault SDK is able to recognize crashes, track reboots, and store metrics. Most of this information is stored in RAM, except for coredumps which we keep in flash. To send that information to the cloud we need to call the memfault_esp_port_http_client_post_data() function.

We recommend running it at a periodic interval. If there is no new data to send, the call reduces to a no-op. The same function can also be invoked via the Memfault CLI with the mflt post_chunks command.

int rv = memfault_esp_port_http_client_post_data();
if (rv != 0) {
  MEMFAULT_LOG_ERROR("Unable to post Memfault data, error %d", rv);
}

15. ESP-IDF Metrics

The Memfault SDK provides a number of built-in metrics for ESP-IDF projects, including but not limited to the following:

• idle_task_run_time_percent : single core devices
• idle0_task_run_time_percent + idle1_task_run_time_percent : dual core devices
• mbedtls_mem_max_bytes
• mbedtls_mem_used_bytes
• tcp_drop_count
• tcp_rx_count
• tcp_tx_count
• udp_drop_count
• udp_rx_count
• udp_tx_count
• wifi_connected_time_ms
• wifi_disconnect_count
• wifi_sta_min_rssi
• wifi_ap_oui
• heap_free_bytes
• heap_largest_free_block_bytes
• heap_allocated_blocks_count
• heap_min_free_bytes
• cpu_temp (not supported on original ESP32)

For more information on how creating your own metrics, see the Metrics page.

Next Steps

For more information on logging, OTA, reboot reasons, and other pieces of Memfault, explore the Core Subsystems Guides!