Espressif ESP32 ESP-IDF Integration Guide

This tutorial will go over integrating the Memfault Firmware SDK into a system that is using Espressif's ESP-IDF for an ESP32 chip. It assumes you already have a working project/toolchain for the ESP32. If you do not, the official getting started guide is a great resource!

The integration works for all versions of ESP-IDF between 3.x and 5.0.

This integration is tested on the ESP32-WROVER-KIT V4.1, but applies similarly to any ESP32-based board.

note

Currently Memfault supports the following ESP32 part families. If it's not listed, we might be working on it already! Drop us a note at support@memfault.com and we'll let you know!

• ESP32 (Xtensa)
• ESP32-S2 (Xtensa)
• ESP32-S3 (Xtensa)
• ESP32-C3/C6 (RISC-V)

tip

An example project demonstrating the result of this integration can be found here, for reference:

https://github.com/memfault/esp32-standalone-example

Memfault also maintains a full demonstration project in the Memfault Firmware SDK, under examples/esp32

Clone Memfault SDK

Using a Git client, clone the memfault-firmware-sdk repository from:

https://github.com/memfault/memfault-firmware-sdk.git

The Memfault SDK does not need to be located within the esp directory, you may place it anywhere.

Add Memfault SDK to the ESP-IDF CMake Build System

In your top level CMakeLists.txt, you can register the Memfault esp-idf port as a "component" by simply including ports/esp_idf/memfault.cmake:

# CMakeLists.txt (top-level)
# [...]

set(MEMFAULT_FIRMWARE_SDK /path/to/memfault-firmware-sdk)
include(${MEMFAULT_FIRMWARE_SDK}/ports/esp_idf/memfault.cmake)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(<YOUR_PROJECT>)

Add Memfault Build ID to the ESP-IDF CMake Build System

It is recommended that you use a build ID to uniquely identify each build. Do this by including Memfault build ID generation by adding a custom post-build step to your top level CMakeLists.txt.

# CMakeLists.txt (top-level)
# [...]

# Be sure to set this variable appropriately
set(MEMFAULT_FIRMWARE_SDK /path/to/memfault-firmware-sdk)
include(${MEMFAULT_FIRMWARE_SDK}/ports/esp_idf/memfault.cmake)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(<YOUR_PROJECT>)

add_custom_command(TARGET ${CMAKE_PROJECT_NAME}.elf
  POST_BUILD
  COMMAND
  python ${MEMFAULT_FIRMWARE_SDK}/scripts/fw_build_id.py ${CMAKE_PROJECT_NAME}.elf)

The fw_build_id.py script adds a build ID that changes whenever the binary changes. You can use this build ID as part of your software version reported by memfault_platform_get_device_info(). See the demo app in <PATH_TO_MEMFAULT_FIRMWARE_SDK>/examples/esp32/apps/memfault_demo_app for an example of how to use the build ID in your code.

note

fw_build_id_py depends on pyelftools so you may need to install it by running pip install pyelftools.

Add the Memfault Porting Files

There are a few files that you must fill in that Memfault requires. There are template versions in <PATH_TO_MEMFAULT_FIRMWARE_SDK>/ports/templates/. Assuming a typical ESP32 project structure with a root directory and a main project sub-directory it is simplest to copy these files into main, e.g.

cp <PATH_TO_MEMFAULT_FIRMWARE_SDK>/ports/templates/* .

When you are done, you should have the following directory structure in your project:

project-root/
├── CMakeLists.txt  # Top-level
└── main
    | CMakeLists.txt  # Project-level
    //...
    | # Files where port / glue layer to your platform will be implemented
    ├── memfault_platform_port.c
    |
    | # Configuration Headers
    ├── memfault_metrics_heartbeat_config.def
    ├── memfault_trace_reason_user_config.def
    ├── memfault_platform_log_config.h
    └── memfault_platform_config.h

Add Sources to Build System

Edit the project-level CMakeLists.txt file in main and add memfault_platform_port.c to the SRCS variable and ${MEMFAULT_FIRMWARE_SDK}/ports/include to the INCLUDE_DIRS variable. It should look something like below. You defined the variable MEMFAULT_FIRMWARE_SDK in the top-level CMakeLists.txt file.

# CMakeLists.txt (project-level)
idf_component_register(
  SRCS
  # [Your project souces]
  memfault_platform_port.c

  INCLUDE_DIRS
  # [Other project include directories]
  .
  ${MEMFAULT_FIRMWARE_SDK}/ports/include
)

note

Quotes on filenames and paths without spaces are not required in CMake lists.

Enable Coredump Collection in sdkconfig

By default, the esp-idf uses the CONFIG_ESP32_ENABLE_COREDUMP_TO_NONE=y. You will need to update sdkconfig (either via idf.py menuconfig or manually) to CONFIG_ESP32_ENABLE_COREDUMP_TO_FLASH=y. The change you will see in the sdkconfig file will look something like:

Note: The remainder of the COREDUMP settings, including CONFIG_ESP_COREDUMP_DATA_FORMAT_ELF, do not affect the Memfault coredump. They are specific to the ESP-IDF coredump features.

diff --git a/sdkconfig b/sdkconfig
index a49446d..83a81a8 100644
--- a/sdkconfig
+++ b/sdkconfig
@@ -274,7 +274,7 @@ CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ=160
 CONFIG_TRACEMEM_RESERVE_DRAM=0x0
 # CONFIG_ESP32_ENABLE_COREDUMP_TO_FLASH is not set
 # CONFIG_ESP32_ENABLE_COREDUMP_TO_UART is not set
-CONFIG_ESP32_ENABLE_COREDUMP_TO_NONE=y
+CONFIG_ESP32_ENABLE_COREDUMP_TO_FLASH=y

Create a Coredump Flash Partition

You will need to allocate an internal flash partition to store panic data. Edit your partitions.csv file to add a coredump partion. If you do not have a partitions.csv file you can create one from the template below and place it in your top-level project directory where the sdkconfig file is located. Depending on your chip's Flash size you may need to modify the storage partition size in order to fit the coredump partition. Check the Flash chip size in menuconfig, see (Top) -> Serial flasher config and check the "Flash Size" entry. Be sure to specify a custom partition file in the sdkconfig (see note below).

Depending on what data is included in the coredump, the size of the coredump partition might also need to be increased to a larger size, like 512K. The memfault_coredump_storage_compute_size_required() function in the Memfault SDK can be used to compute the maximum coredump size, which can vary depending on user configuration.

Note that the default Memfault coredump regions are defined by the memfault_platform_coredump_get_regions() function in the Memfault SDK's ESP-IDF port: ports/esp_idf/memfault/common/memfault_platform_coredump.c, but can be adjusted to suit the user's needs, either via Kconfig options or by implementing a separate memfault_platform_coredump_get_regions() to override the built-in weakly defined one.

# Name,   Type, SubType, Offset,  Size,  Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
storage,  data, fat,     ,        1M,
coredump, data, coredump,,        256K,

note

Be sure to indicate the actual name you chose for the partion file in the sdkconfig file. This can be done easily by running idf.py menuconfig, selecting (Top) -> Partition Table, and setting the name of the "Custom partition CSV file". Optionally, you can edit the sdkconfig file directly (search for the variables prefixed with CONFIG_PARTITION_TABLE_).

Create a Project and get a Project Key

Go to app.memfault.com and from the "Select A Project" dropdown, click on "Create Project" to setup your first project such as "smart-sink-dev".

Once you've created your project, you'll be automatically taken to an page that includes your project key. Copy the key and follow the rest of this guide.

Implement Changes for Your Project to Incorporate Memfault

Metrics and Trace Definitions

The device metrics and trace event subsystems require configuration files to be added to your include path (memfault_metrics_heartbeat_config.def & memfault_trace_reason_user_config.def, respectively). Because you copied them to your main directory these files' locations should be known to CMake.

memfault_platform_get_device_info()

When a panic occurs, information is collected to uniquely identify the device and version of software running. Update the implementation of the function memfault_platform_get_device_info() in memfault_platform_port.c by setting the fields shown in the returned sMemfaultDeviceInfo struct.

// memfault_platform_port.c

// [... other code ...]

// NOTE: This needs to be safe to call from the exception handler so
// if you are reading from data from subsytems which uses locking esp_efuse_*
// you will need to pre-populate the data on bootup.
void memfault_platform_get_device_info(sMemfaultDeviceInfo *info) {
  // platform specific version information
  *info = (sMemfaultDeviceInfo) {
    .device_serial = "ESP32_DEMOSERIAL",
    .software_type = "esp32-main",

    // It is recommended to use the build ID script. Below is safe to
    // call regardless.
    .software_version = memfault_create_unique_version_string("1.0.0"),
    .hardware_version = "esp-wrover-kit",
  };
}

For ESP32 devices, the base MAC address is an option for generating device serial numbers for test purposes. It is accessible from the esp_read_mac() function included from esp_mac.h. Do not use the other MAC address loading functions until esp_read_mac has been called at least once, as they only return the in memory copy of the MAC address while esp_read_mac properly fetches it from flash if it has not been loaded yet.

See an example in the Memfault SDK esp32 example.

Note: there is a non-zero chance of overlapping addresses from Espressif, so it is not recommended for large production fleets of devices. As many as 20 duplicates in a set of 4000 chips have been reported in ESP-IDF user forums.

Initializing Memfault

When working with Memfault Firmware SDK version 0.40.0 and above, you will need to initialize the SDK from your application. This can be done simply by adding a call to memfault_boot from main_app. We recommend doing this as early as possible in your initialization process, so Memfault components can begin capturing data. Additionally, you should initialize any dependencies used by memfault_platform_get_device_info as this is called within memfault_boot.

Here's a code example:

#include "memfault/esp_port/core.h"

void app_main() {
  // Initialize any components that are dependencies of memfault_platform_get_device_info

  memfault_boot();

  // Remainder of your main task
}

When using SDK versions prior to 0.40.0, an additional step is required. You will need to explicitly set CONFIG_MEMFAULT_AUTOMATIC_INIT=n in your project.

Configure Memfault Project Key

The project key is set via a Kconfig variable, MEMFAULT_PROJECT_KEY.

In ESP-IDF this can be set in a few different places:

• add a line to sdkconfig.defaults:

  CONFIG_MEMFAULT_PROJECT_KEY="your-project-key"

• set it in sdkconfig, either via idf.py menuconfig or by directly editing the file

• set it into an extra sdkconfig file, for example sdkconfig.memfault_project_key, and include the file when building:

  $ idf.py build -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.memfault_project_key"

Regenerate sdkconfig and recompile

Assuming that your project has already been built before adding Memfault you will need to rebuild now that the Memfault SDK has been integrated into your ESP-IDF project.

$ idf.py clean
$ idf.py menuconfig  # Verify your settings
$ idf.py build

Testing things out

With UART console

If you built a project with a UART console the Memfault esp-idf port registers some extra CLI commands so you can easily try things out! Flash and connect to the esp32 dev board's UART console.

$ idf.py -p /dev/ttyUSB1 flash monitor

note

Your serial port specification may differ.

As a first step you can try collecting and sending a coredump with the following commands:

// force a crash
esp32> crash 3
// wait for reboot
// join network
esp32> join <SSID> <PASSWORD>
// Post data
esp32> post_chunks

No UART console

If you built a project without a UART console you can test a coredump by inserting an assert that will fail, e.g.

   // [...]
   MEMFAULT_ASSERT(0);

Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.

Publish data to the Memfault cloud

With the esp-idf, data can be posted anytime you are connected to the internet by calling the following code. We recommend running it at a periodic interval. If there is no new data to send, the call reduces to a no-op. The same logic can also be invoked via the memfault CLI with the mflt post_chunks mentioned above.

#include "memfault/core/debug_log.h"
#include "memfault/http/http_client.h"

int memfault_post_data(void) {
  sMfltHttpClient *http_client = memfault_http_client_create();
  if (!http_client) {
    MEMFAULT_LOG_ERROR("Failed to create HTTP client");
    return -1;
  }
  const eMfltPostDataStatus rv = memfault_http_client_post_data(http_client);
  if (rv < 0) {
    MEMFAULT_LOG_ERROR("%s error: %d", __func__, rv);
  }
  const uint32_t timeout_ms = 30 * 1000;
  memfault_http_client_wait_until_requests_completed(http_client, timeout_ms);
  memfault_http_client_destroy(http_client);
  return rv;
}

Frequently Asked Questions

Automatic Library Initialization

By default in Memfault Firmware SDK versions 0.40.0 and above, the Memfault library is not initialized automatically. See the previous section for more info. For SDK versions prior to 0.40.0, a Kconfig exists but is deprecated and not supported for use.