Memfault RTOS Support

Memfault automatically detects what RTOS your system is running. For uploaded coredumps, it will attempt to extract backtraces for all threads in the system and optionally determine what state each thread is in and the stack usage high water mark.

In order for this to work correctly, the coredump must capture the RTOS' thread state variables, all thread control blocks as well as the stack memory of each thread. The following sections list for each RTOS what needs to be captured. See the documentation on Coredump Collection for details on how to set up additional coredump capturing regions.

FreeRTOS

Required

These variables must be captured in the coredump. On top of this, the memory of all task control blocks (TCB_t) must be captured. Because FreeRTOS TCBs double as linked list nodes, all TCBs must be captured.

• uxCurrentNumberOfTasks
• pxCurrentTCB
• pxReadyTasksLists
• xDelayedTaskList1
• xDelayedTaskList2
• xPendingReadyList
• uxTopReadyPriority
• xSchedulerRunning

note

There were several releases of FreeRTOS that did not include uxTopUsedPriority, see this GitHub issue:

https://github.com/FreeRTOS/FreeRTOS-Kernel/issues/33#issue-583304006

If the version of FreeRTOS in use doesn't include the fix, add the sample file (for example, this copy) and relevant linker flags.

Optional

• xTasksWaitingTermination
• xSuspendedTaskList

To enable stack usage high water marking, the configCHECK_FOR_STACK_OVERFLOW configuration option must be enabled and the full stacks for each thread must be captured as well.

How to Capture the Required Variables

The memfault_freertos_ram_regions.c file contains details on selectively capturing the required variables.

note

Note that this is only necessary if all of RAM is not captured as part of the coredump (i.e. when coredump storage and bandwidth is limited).

The strategy has 3 parts:

1. update linker script to enclose the needed variables in a known region
2. Update memfault_platform_coredump_get_regions() to include that region's data in a coredump region
3. Use the FreeRTOS traceTASK_CREATE/DELETE() macros to track tasks during runtime

See example implementations below.

GCC

Update the linker script (typically a file that ends with the .ld extension) to enclose the necessary .bss symbols with some exported symbols:

.bss (NOLOAD) :
{
    _sbss = . ;
    __bss_start__ = _sbss;
    __memfault_capture_bss_start = .;
    /* Place all objects from the FreeRTOS timers and tasks modules here.
       Note that some build systems will use 'timers.o' as the object
       file name, and some may use variations of 'timers.c.o' or
       'timers.obj' etc. This pattern should capture all of them. */
     *tasks*.o*(.bss COMMON .bss*)
     *timers*.o*(.bss COMMON .bss*)
    __memfault_capture_bss_end = .;

And then update memfault_platform_coredump_get_regions() to include those variables:

const sMfltCoredumpRegion *memfault_platform_coredump_get_regions(
    const sCoredumpCrashInfo *crash_info, size_t *num_regions) {
  int region_idx = 0;

  // any higher priority regions would go here, i.e. active stack

  // collect the FreeRTOS timer and task variables required for RTOS decode
  extern uint32_t __memfault_capture_bss_start;
  extern uint32_t __memfault_capture_bss_end;
  const size_t memfault_region_size = (uint32_t)&__memfault_capture_bss_end -
      (uint32_t)&__memfault_capture_bss_start;

  s_coredump_regions[region_idx] = MEMFAULT_COREDUMP_MEMORY_REGION_INIT(
    &__memfault_capture_bss_start, memfault_region_size);
  region_idx++;

  // remaining regions would go here. typically the
  // 'memfault_freertos_get_task_regions()' helper would be used to capture the
  // task data.
  region_idx += memfault_freertos_get_task_regions(&s_coredump_regions[region_idx],
      MEMFAULT_ARRAY_SIZE(s_coredump_regions) - region_idx);
  // remaining regions

IAR

Update the linker script (typically a file with a .icf extension) to include this line:

define block memfault_freertos { section .bss* object *tasks*.o*, section .bss* object *timers*.o* };

And then update memfault_platform_coredump_get_regions() to include that block:

// In the source file where memfault_platform_coredump_get_regions is defined add the following snippet
// Be sure to include additional sMfltCoredumpRegion entries in your regions array for the FreeRTOS regions
// created by the linker script and helper function
const sMfltCoredumpRegion *memfault_platform_coredump_get_regions(
    const sCoredumpCrashInfo *crash_info, size_t *num_regions) {
  int region_idx = 0;

  // any higher priority regions would go here, i.e. active stack

  // collect the block defined in the linker script
  const void *memfault_freertos_region_start = __section_start("memfault_freertos");
  const size_t memfault_freertos_region_size = __section_size("memfault_freertos");
  s_coredump_regions[region_idx] = MEMFAULT_COREDUMP_MEMORY_REGION_INIT(
     memfault_freertos_region_start, memfault_freertos_region_size);
  region_idx++;

  // remaining regions would go here. typically the
  // 'memfault_freertos_get_task_regions()' helper would be used to capture the
  // task data.
  region_idx += memfault_freertos_get_task_regions(&s_coredump_regions[region_idx],
      MEMFAULT_ARRAY_SIZE(s_coredump_regions) - region_idx);
  // remaining regions

Finally, modify your FreeRTOSConfig.h to include this following snippet:

//! @file FreeRTOSConfig.h
#pragma once

#include "memfault/ports/freertos_trace.h"

Built-in FreeRTOS Metrics

The Memfault SDK has support for some built-in FreeRTOS Metrics:

• idle_task_run_time_percent (for both single- and dual-core systems)
• timer_task_stack_free_bytes

To enable these metrics, include the definition in the user Heartbeat configuration file:

memfault_metrics_heartbeat_config.def

loading...

View on GitHub

Be sure to add the Memfault FreeRTOS Port files to your project, located here in the SDK: sdk/embedded/ports/freertos/src/

Coredump Stack High Watermark

Memfault will automatically compute stack high watermarks (represented as High Water Mark: x Bytes Free) for each task captured in a coredump.

Memfault has two ways of computing the high watermarks:

1. If a task's entire stack region is captured in the coredump, the watermark is computed when the coredump is processed. This requires setting #define configRECORD_STACK_HIGH_ADDRESS 1 in FreeRTOSConfig.h.
2. If #define MEMFAULT_COREDUMP_COMPUTE_THREAD_STACK_USAGE 1 is set in memfault_platform_config.h, the high watermark is computed when the coredump is captured. The full task stack does not need to be present in the coredump for Memfault to show the watermark. This also requires setting #define INCLUDE_uxTaskGetStackHighWaterMark 1 in FreeRTOSConfig.h.

Zephyr

Required

These variables must be captured in the coredump:

• _kernel
• _kernel_openocd_offsets
• _kernel_openocd_size_t_size

Tracking Thread Stack Usage

Tracking a Single Thread's Stack Usage

To use a Memfault Metric to track a thread's max stack usage:

memfault_metrics_heartbeat_config.def

// Define the metric key
MEMFAULT_METRICS_KEY_DEFINE(my_thread_stack_min_free_bytes, kMemfaultMetricType_Unsigned)

Make sure the appropriate Kconfig settings are enabled:

prj.conf

CONFIG_INIT_STACKS=y
CONFIG_THREAD_STACK_INFO=y

Then, in the heartbeat collection function, get the stack usage:

#include <zephyr/kernel.h>
#include "memfault/components.h"

// The thread whose stack usage you want to track. This is stored when the
// thread is created.
static struct k_thread *my_thread;

void memfault_metrics_heartbeat_collect_data(void) {
  size_t stack_free_bytes;
  k_thread_stack_space_get(my_thread, &stack_free_bytes);
  MEMFAULT_METRIC_SET_UNSIGNED(my_thread_stack_min_free_bytes, stack_free_bytes);
}

Tracking Multiple Threads' Stack Usage

To track multiple threads' stack usage, the simplest approach is to make the thread handles exposed from the module where the threads are created. Then follow the same approach as above, once for each thread being tracked.

An alternative approach is to use the k_thread_foreach API to iterate over all threads in the system, and record data for the threads of interest:

memfault_metrics_heartbeat_config.def

// Define the metric keys
MEMFAULT_METRICS_KEY_DEFINE(my_thread_stack_min_free_bytes, kMemfaultMetricType_Unsigned)
MEMFAULT_METRICS_KEY_DEFINE(my_other_thread_stack_free_bytes, kMemfaultMetricType_Unsigned)

#include <zephyr/kernel.h>
#include "memfault/components.h"

//! Callback to collect stack usage for specific threads
static void prv_collect_thread_stack_usage_cb(const struct k_thread *thread, void *user_data) {
  // Table mapping thread names to metric IDs
  static const struct {
    const char *name;
    MemfaultMetricId id;
  } threads[] = {
    { "my_thread", MEMFAULT_METRICS_KEY(my_thread_stack_min_free_bytes) },
    { "my_other_thread", MEMFAULT_METRICS_KEY(my_other_thread_stack_free_bytes) },

  };

  // scan for a matching thread name
  for (size_t i = 0; i < ARRAY_SIZE(threads); i++) {
    if (strcmp(thread->name, threads[i].name) == 0) {
      // Get the stack usage and record in the matching metric id
      size_t stack_free_bytes;
      k_thread_stack_space_get(thread, &stack_free_bytes);
      memfault_metrics_heartbeat_set_unsigned(threads[i].id, stack_free_bytes);
    }
  }
}

void memfault_metrics_heartbeat_collect_data(void) {
  k_thread_foreach(prv_collect_thread_stack_usage_cb, NULL);
}

Eclipse ThreadX RTOS

Required

These variables must be captured in the coredump:

• _tx_thread_current_ptr
• _tx_thread_created_ptr
• _tx_thread_created_count
• _tx_thread_system_state

Capturing Threads

::: note

Memfault currently does not have a built-in integration for tracking ThreadX threads on creation. Please contact us if you're interesting in support for this feature!

:::

Capturing all of target RAM (.data and .bss) will capture any thread state, as well as the above ThreadX kernel variables, and will enable thread-aware coredumps.

If capturing all of target RAM is not possible, you can selectively capture the ThreadX kernel variables, thread control blocks, and thread stack regions.

This is done as follows:

1. Modify the linker script to place the required kernel variables in a specific region of .bss:

    .bss (NOLOAD) :
    {
        _sbss = . ;
        __bss_start__ = _sbss;
        __memfault_capture_bss_start = .;
        /* Place ThreadX global thread state variables in a single span */
        *tx_thread_initialize.o*(.bss COMMON .bss*)
        __memfault_capture_bss_end = .;
    }

2. Implement the memfault_platform_coredump_get_regions() function to include the ThreadX kernel variables:

    const sMfltCoredumpRegion *memfault_platform_coredump_get_regions(
        const sCoredumpCrashInfo *crash_info, size_t *num_regions) {
      int region_idx = 0;
      static sMfltCoredumpRegion s_coredump_regions[
        // active stack
        1
        // ThreadX kernel data structure
        + 1
        // ThreadX thread data structures (TCB + stack region)
        + MEMFAULT_PLATFORM_MAX_TASK_REGIONS * 2
      ];
   
      const size_t stack_size = memfault_platform_sanitize_address_range(
        crash_info->stack_address, MEMFAULT_PLATFORM_ACTIVE_STACK_SIZE_TO_COLLECT);
   
      s_coredump_regions[0] =
        MEMFAULT_COREDUMP_MEMORY_REGION_INIT(crash_info->stack_address, stack_size);
   
      // any higher priority regions would go here, i.e. active stack
   
      // collect the ThreadX kernel variables required for RTOS decode
      extern uint32_t __memfault_capture_bss_start;
      extern uint32_t __memfault_capture_bss_end;
      const size_t memfault_region_size = (uint32_t)&__memfault_capture_bss_end -
          (uint32_t)&__memfault_capture_bss_start;
   
      s_coredump_regions[region_idx] = MEMFAULT_COREDUMP_MEMORY_REGION_INIT(
        &__memfault_capture_bss_start, memfault_region_size);
      region_idx++;
   
      // See this example here for a strategy to capture ThreadX thread information:
      // https://github.com/memfault/memfault-firmware-sdk/compare/noahp/threadx-thread-capture
      region_idx += memfault_threadx_get_thread_regions(&s_coredump_regions[region_idx],
          MEMFAULT_ARRAY_SIZE(s_coredump_regions) - region_idx);
   
      *num_regions = region_idx;
   
      return s_coredump_regions;
    }

Other RTOSs

Memfault also supports these other RTOSs: Argon, ChibiOS, Mynewt, NuttX, Quantum Platform, and Keil RTX5 / ARM mbedOS.