Skip to main content

Linux Coredumps

Introduction​

By enabling coredump support with Memfault, the memfaultd daemon will automatically collect, preprocess and upload coredumps from your devices and send them to the Memfault platform. In turn, the Memfault platform acts as a remote analyzer for your coredump files and (using your project's debugging symbols) is able to display a rich view of a coredump, displaying backtraces for all threads, and allowing you to inspect state in full detail.

From man core:

The default action of certain signals is to cause a process to terminate and produce a core dump file, a file containing an image of the process's memory at the time of termination. This image can be used in a debugger (e.g., gdb(1)) to inspect the state of the program at the time that it terminated. A list of the signals which cause a process to dump core can be found in signal(7).

Additionally, Memfault takes care of grouping traces from coredumps into issues, managing and deduplicating issues, and providing metrics on issues and monitoring via notifications, granting you a clear image of how your fleet is behaving, as well as tight control over the success of your OTA updates.

A Linux coredump as an issue on the Memfault Web App
A Linux coredump as an issue on the Memfault Web App.
tip

Keep meta-memfault-example open as a reference implementation. Your integration should look similar to it once you're done following the steps in this tutorial.

Prerequisites​

The memfaultd daemon, built with plugin_coredump​

Follow the integration guide to learn how to set this up for your device. A key function of memfaultd is to preprocess and upload coredumps to the Memfault platform. It does this via its plugin_coredump.

plugin_coredump is enabled by default. Read more on how to configure which plugins memfaultd builds with.

Linux kernel configuration​

Ensure that your Linux kernel is built with the following options enabled:

  • CONFIG_COREDUMP=y in order to enable coredump creation by the kernel
  • CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y in order to enable default mappings. for processes. Specifically, Memfault requires:
    • bit 0: dump anonymous private mappings,
    • bit 1: dump anonymous shared mappings, and
    • bit 4: (available since Linux 2.6.24) dump ELF headers.

Read man core in order to learn more about these settings. We recommend reading the entirety of this man page if you're in need of basic understanding of how coredumps work in Linux.

In order to confirm that your kernel is correctly configured, check out your project's configuration file in tmp/work/[machine]/[kernelpackage]/[version]/build/.config. For example:

tmp/work/raspberrypi3-poky-linux/linux-yocto/5.15.62+gitAUTOINC+59c8898d45_7cb30c5e95-r0/linux-raspberrypi3-standard-build/.config`

core_pattern and core_pipe_limit​

The following files will be modified by memfaultd at runtime and must not be written to by any other process.

  • /proc/sys/kernel/core_pattern (see man core)
  • /proc/sys/kernel/core_pipe_limit (see man core)

Make sure your Linux image does not contain any other services that may write to these files. To do this, check for the inclusion of other coredump handlers such as systemd (which can act as a coredump handler when built with -Dcoredump=true), and check your /etc/sysctl.d drop-in directory for anything other than Memfault that may be setting kernel.core_pattern or kernel.core_pipe_limit.

To check whether you've succeeded in letting memfaultd take care of these files, confirm that the contents of the core_pattern file reference the memfault-core-handler binary (see Test your integration).

Note that memfaultd respects privacy settings and only sets core_pattern if data collection is enabled at runtime.

Extend LICENSE_FLAGS_ACCEPTED​

Since memfaultd and memfaultd-core-handler both have a commercial license, you'll need to add an exception for them, for example in your local.conf:

LICENSE_FLAGS_ACCEPTED:append = " commercial_memfaultd commercial_memfault-core-handler"

Make sure prelinking is disabled​

Prelinking is an operation which optimizes applications load times by resolving library symbols before launch. However most of the benefits of prelinking are lost unless PIE is disabled and both Glibc and Yocto are dropping support for it. It is disabled by default in Yocto since 3.4 - Honister.

To disable prelinking on an older version of Yocto where it is still enabled by default, you need to override USER_CLASSES in local.conf. On Yocto versions before 3.4 - Honister, USER_CLASSES included image-prelink by default.

If you already have USER_CLASSES overridden in local.conf, make sure image-prelink is not included in the list. If not, you can use USER_CLASSES:remove = "image-prelink" to keep the defaults, except for "image-prelink".

This is how we disable prelinking in our example image:

USER_CLASSES = "buildstats"

We strongly recommend against using Link Time Optimization (LTO) in the programs you want to be able to debug using coredumps. When LTO is enabled, the analysis of the coredumps can become lacking or even fail completely.

Upload debugging symbols​

After a coredump was collected on the device, memfaultd will upload it to the Memfault Web App. To allow the Memfault Web App to reconstruct all details from such coredumps, you'll need to upload debugging symbols for each of the binaries in your Linux image. We recommend doing this as part of your build process (since this needs to be done for every version), either locally or ideally as part of continuous integration, in order to keep Memfault up to date with your build's newest debugging symbols.

In order to facilitate this, we've added a Yocto-specific helper subcommand to the Memfault CLI (starting with version 0.11.0): memfault upload-yocto-symbols. Before you use it, you'll need to update your base image to include the following, and then run a build:

# Support memfault-cli upload-yocto-symbols command
DEPENDS:append = " elfutils-native"
IMAGE_GEN_DEBUGFS = "1"
IMAGE_FSTYPES_DEBUGFS = "tar.bz2"

Here's an example memfault upload-yocto-symbols invocation:

$ source oe-init-build-env
$ memfault \
    --org $YOUR_ORGANIZATION_SLUG \
    --org-token $ORGANIZATION_AUTH_TOKEN \
    --project $YOUR_PROJECT_SLUG upload-yocto-symbols \
    --image tmp/deploy/images/raspberrypi3/base-image-raspberrypi3.tar.bz2 \
    --dbg-image tmp/deploy/images/raspberrypi3/base-image-raspberrypi3-dbg.tar.bz2

Supported formats are .tar, .tar.bz2, .tar.gz and .tar.xz. The format of --image may differ from that of --dbg-image.

It's recommended that you run this command after having sourced the build environment script using source oe-init-build-env.

tip

IMAGE_GEN_DEBUGFS will cause Yocto to build an archive with separated debug info of all the binaries on the system. Your system image will not get larger but a separate -dbg.tar.gz file will be saved next to your system image.

This archive is used by the memfault upload-yocto-symbols command. It does not upload the complete file directly: it extracts it to a temporary directory and generates unstripped copies of the binaries and their symbols.

If you can't do this, then you can pass the information needed from the build environment as command-line arguments:
  • --eu-unstrip-path: path to a local eu-unstrip binary from elfutils. Note that one is available in tmp/sysroot-components/x86_64/elfutils-native if you've added DEPENDS:append = " elfutils-native" to your build.
  • --package-debug-split-style: your project's PACKAGE_DEBUG_SPLIT_STYLE. In Poky, it defaults to debug-with-srcpkg. Read more about it in the Yocto reference.

If you're not using Yocto, you can replicate its behavior by using our REST API to upload symbols.

You can always upload symbol files individually using the Memfault Web Application. Open Software -> Symbol Files and click on Upload Symbol File, or simply click on this deep link to the app. Note that the symbol files you upload here must be individual ELF files containing both the program text as well as debugging symbols. The Yocto-generated debug image (for example in .tar.bz2 format) contains stripped binaries and (separately) debug binaries. To upload them to Memfault, you'll need to use the Memfault CLI as shown above.

tip

Read docs on CI and authentication in order to obtain credentials that you can use in your build environment or in continuous integration.

Set enable_data_collection​

By default, enable_data_collection is false (see the default configuration). This is to enable asking end users for consent before collecting or transmitting any data to Memfault services.

Once the end user has given their consent, you can enable data collection like so:

$ memfaultctl enable-data-collection

To disable it:

$ memfaultctl disable-data-collection

The memfaultd service will restart automatically whenever you run either of those commands if called with a value different from the current configuration.

Take a look at the /etc/memfaultd.conf reference for more information.

(Optional) Configure what's included in a coredump​

Currently, Memfault support for coredumps does not include custom filtering or selection of processes, memory regions or variables. These features are planned for a future release.

We'd recommend waiting for a release of the Memfault Linux SDK with first-class support for coredump filtering, but if your team needs it now, the kernel does provide a certain degree of configurability:

  • Call prctl(PR_SET_DUMPABLE, 0) in order to declare a process as non-dumpable. By default, processes are dumpable. See man prctl for more information.
  • Set the appropiate flags in /proc/[pid]/coredump_filter (see man core).
    • Note that in order to work properly with Memfault, coredump_filter needs to include at least bits 0, 1 and 4 (see man core).
  • Call madvise(ptr, bytes, MADV_DONTDUMP) (with MADV_DONTDUMP or MADV_DODUMP) in order to declare certain memory regions as dumpable (or non-dumpable). The effects of this syscall take precedence over the settings in the coredump_filter for this file (see man madvise for more information).

Test your integration​

You can test your integration with memfaultctl. The trigger-coredump command will fork itself and crash.

# memfaultctl trigger-coredump

If developer mode is active, memfaultctl will immediately push the coredump to Memfault. Otherwise you can force memfaultd to sync immediately with:

# memfaultctl sync

If your integration is all set and you've enabled data collection, you'll be able to see a new issue pop up in your project's Issues page.

Debugging Issues​

You might want to check that the core_pattern file is being set correctly by memfaultd when it starts. Check that the output of this command includes memfault-core-handler:

$ cat /proc/sys/kernel/core_pattern

To see logs coming from memfaultd, run journalctl:

$ journalctl --unit memfaultd
Dec 21 10:45:20 qemuarm64 systemd[1]: Started memfaultd daemon.
Dec 21 10:45:24 qemuarm64 memfaultd[246]: coredump:: Received corefile for PID 268, process 'memfaultctl'
Dec 21 10:45:24 qemuarm64 memfaultd[246]: coredump:: writing coredump with max size: 98304000
Dec 21 10:45:24 qemuarm64 memfaultd[246]: coredump:: enqueued corefile for PID 268
Dec 21 10:45:30 qemuarm64 memfaultd[246]: network:: Successfully transmitted file '/media/memfault/core/corefile-fbf1b4c8-b184-4b6c-87c1-6229bd55afd7.gz'