Skip to main content

MCU Logging

When debugging an issue, it is often useful to inspect what the system was doing in the time leading up to the problem.

On embedded systems it's also desirable to buffer recent logs in RAM and periodically flush the logs out to a slower medium (i.e UART, Flash). This way logging does not impact any real time behavior of the system.

The Memfault SDK offers a simple RAM based logging buffer which can be used to accomplish both!

Logs in coredump​

Any logs present in the buffer get automatically decoded when a coredump is uploaded:

Independently collected logs​

Logs can also be uploaded independently, without being part of a coredump. These logs will appear on the Timeline of a Device. A tabular view of log files is also available under the Log Files tab:

Integration Steps​

In this guide we will walk through how to save logs using the Memfault core log utility and how the module can be used to queue up logs to flush out to a slower storage medium.

1. Initialize log storage​

First you will need to allocate some space for the RAM backed log storage. Any size works but at least a couple hundred bytes is recommended so at least a few logs can be stored at any given time.

#include "memfault/components.h"
// [...]

// static RAM storage where logs will be stored. Storage can be any size
// you want but you will want it to be able to hold at least a couple logs.
static uint8_t s_log_buf_storage[512];

int main(void) {
  // [...]
  memfault_log_boot(s_log_buf_storage, sizeof(s_log_buf_storage));
}

2. Create or Update Logging Macro​

If your project does not have any logging infrastructure yet, you can make use of the implementation in the SDK directly:

#include "memfault/core/log.h"
#include "memfault/core/platform/debug_log.h"

#define YOUR_PLATFORM_LOG_DEBUG(...) \
  MEMFAULT_SDK_LOG_SAVE(kMemfaultPlatformLogLevel_Debug, __VA_ARGS__)
#define YOUR_PLATFORM_LOG_INFO(...)  \
    MEMFAULT_SDK_LOG_SAVE(kMemfaultPlatformLogLevel_Info, __VA_ARGS__)
#define YOUR_PLATFORM_LOG_WARN(...)  \
    MEMFAULT_SDK_LOG_SAVE(kMemfaultPlatformLogLevel_Warning, __VA_ARGS__)
#define YOUR_PLATFORM_LOG_ERROR(...) \
    MEMFAULT_SDK_LOG_SAVE(kMemfaultPlatformLogLevel_Error, __VA_ARGS__)

Otherwise, you can wrap a preexisting implementation with a call to MEMFAULT_LOG_SAVE():

#define YOUR_PLATFORM_LOG_ERROR(...) \
  do { \
    MEMFAULT_LOG_SAVE(kMemfaultPlatformLogLevel_Error, __VA_ARGS__); \
    your_platform_log_error(__VA_ARGS__) \
  } while (0)
caution

By default, only logs greater than the Debug level will be saved but you can change the level either by calling memfault_log_set_min_save_level() or by using the MEMFAULT_RAM_LOGGER_DEFAULT_MIN_LOG_LEVEL compile time define.

3. [Optional] Flush Logs to Different Mediums​

The Memfault logging system can optionally be used as a general purpose location to store logs you plan to save to flash or flush out over UART.

In the example above this would mean removing the direct calls to your_platform_log_error(__VA_ARGS__) and leaving the call to MEMFAULT_LOG_SAVE():

// [...]
#define YOUR_PLATFORM_LOG_ERROR(...) \
  MEMFAULT_LOG_SAVE(kMemfaultPlatformLogLevel_Error, __VA_ARGS__);

Then as part of a RTOS task or bare-metal while loop you need to check if there are any logs available to flush:

void your_platform_log_flush_to_uart_task(void) {

  // .. code to wait for log read event ..

  // draining logs over a UART
  while (1) {
    sMemfaultLog log = { 0 };
    const bool log_found = memfault_log_read(&log);
    if (!log_found) {
      break;
    }
    // an implementation in your platform that can flush a log over a UART
    your_platform_uart_println(log.level, log, log.msg_len);
  }
}

Note that anytime MEMFAULT_LOG_SAVE is called, memfault_log_handle_saved_callback() is invoked by the memfault log module. You can use this callback to programmatically schedule log flush events when new logs have been generated by adding the following:

void memfault_log_handle_saved_callback(void) {
   your_rtos_schedule_log_flush();
}
note

If the RAM buffer fills, the oldest logs will be overwritten. Any time this happens when you next call memfault_log_read(), a log such as "... 5 messages dropped ...", will be emitted indicating how many messages were dropped.

4. Thread Safety (if applicable)​

If you are using an RTOS where tasks do not run to completion, you need to implement the memfault_lock and memfault_unlock APIs.

Note that locks are only held while copying data into the backing circular buffers so the durations the lock is held will always be very short.

#include "memfault/components.h"

static YourPlatformRecursiveMutexType s_memfault_lock;

void memfault_lock(void) {
   your_platform_recursive_mutex_lock(s_memfault_lock);
}

void memfault_unlock(void) {
  your_platform_recursive_mutex_unlock(s_memfault_lock);
}
note

Under some cases, locking during log saving is problematic (for example, if logging can occur from an ISR, where the normal memfault_lock() implementation would fail).

In this case, it may make sense to skip the build-in lock used by memfault_log_save_preformatted(), and instead use alternate locking primitives. Here is an example using C <stdatomic.h> utilities, to record preformatted log strings from the _write() system call:

#include <stdatomic.h>

static atomic_bool s_memfault_log_lock = ATOMIC_VAR_INIT(false);

// Implement _write() to call memfault_log_save_preformatted_nolock() after acquiring a lock
int _write(int file, char *ptr, int len) {
  (void)file;

  if (len <= 0) {
    return len;
  }

  if (!atomic_exchange(&s_memfault_log_lock, true)) {
    memfault_log_save_preformatted_nolock(kMemfaultPlatformLogLevel_Info, ptr, len);
    atomic_store(&s_memfault_log_lock, false);
  }

  return len;
}

5. [Optional] Collect Logs after Events of Interest​

By design, logs are not collected and uploaded automatically. To trigger collection of logs, your code will need to call the memfault_log_trigger_collection() API. This API "freezes" the current contents of the log buffer, marking those logs as pending upload to Memfault. We recommend using this API to capture additional information when significant events or unexpected errors take place in your platform.

Pending logs get uploaded just like any other type of data, through the data packetizer.


#include "memfault/components.h"

void your_collect_logs(void) {
    // Prepare all logs that are present in the log buffer for collection:
    memfault_log_trigger_collection();

    // When log collection is triggered, let's upload the data from the
    // packetizer to get the logs to the Memfault cloud:
    your_upload_packetizer_data();
}

Rate Limiting

Ingestion of Log Files may be rate-limited. Avoid calling memfault_log_trigger_collection() more than once per hour per device.

6. [Optional] Collect Logs as Part of Coredump​

caution

This step assumes you have integrated coredump collection in your project.

tip

If you are already collecting all of your bss RAM region in a coredump, logs will automatically be recovered and displayed in the Memfault UI alongside a crash and you can skip this step.

If you are only collecting select RAM regions, collection of the regions needed to decode logs can be automatically collected by adding the following to your memfault_platform_config.h.

//! @file memfault_platform_config.h

#define MEMFAULT_COREDUMP_COLLECT_LOG_REGIONS 1

You'll also want to double check that you have a sufficient amount of space to store the region in your coredump storage. This can be achieved by performing the following check after you have called memfault_log_boot():

#include "memfault/panics/coredump.h"
#include "memfault/panics/platform/coredump.h"

void your_platform_coredump_init(void) {
  sMfltCoredumpStorageInfo storage_info = { 0 };
  memfault_platform_coredump_storage_get_info(&storage_info);
  const size_t size_needed = memfault_coredump_storage_compute_size_required();
  if (size_needed > storage_info.size) {
    MEMFAULT_LOG_ERROR("Coredump storage too small. Got %d B, need %d B",
                       storage_info.size, size_needed);
  }
  MEMFAULT_ASSERT(size_needed <= storage_info.size);
}

Include Timestamp In Log Messages​

For systems using Memfault's Logging component, here's an example approach to add a timestamp to log entries:

#include "memfault/core/platform/core.h"
#define MEMFAULT_TIMESTAMPED_LOG_SAVE(level, fmt, ...)              \
  do {                                                              \
    uint64_t uptime = memfault_platform_get_time_since_boot_ms();   \
    MEMFAULT_LOG_SAVE(level, "[%llu] " fmt, uptime, ##__VA_ARGS__); \
  } while (0)
  ...
  MEMFAULT_TIMESTAMPED_LOG_SAVE(kMemfaultPlatformLogLevel_Info, "Log with a timestamp, reason=%d", 42);

The resulting log message will be formatted like this:

I [123456] Log with a timestamp, reason=42
Last updated on
This page is human-authored. If you see any errors, outdated information, or any other suggestions for improving the contents, please contact us!