By enabling coredump support with Memfault, the
memfaultd daemon will
automatically collect, preprocess and upload coredumps from your devices and
send them to the Memfault platform. In turn, the Memfault platform acts as a
remote analyzer for your coredump files and (using your project's debugging
symbols) is able to display a rich view of a coredump, displaying backtraces for
all threads, and allowing you to inspect state in full detail.
The default action of certain signals is to cause a process to terminate and produce a core dump file, a file containing an image of the process's memory at the time of termination. This image can be used in a debugger (e.g., gdb(1)) to inspect the state of the program at the time that it terminated. A list of the signals which cause a process to dump core can be found in signal(7).
Additionally, Memfault takes care of grouping traces from coredumps into issues, managing and deduplicating issues, and providing metrics on issues and monitoring via notifications, granting you a clear image of how your fleet is behaving, as well as tight control over the success of your OTA updates.
meta-memfault-example open as a reference
implementation. Your integration should look similar to it once you're done
following the steps in this tutorial.
memfaultd daemon, built with
Follow the integration guide to learn how to set this
up for your device. A key function of
memfaultd is to preprocess and upload
coredumps to the Memfault platform. It does this via its
plugin_coredump is enabled by default. Read more on how to configure which
memfaultd builds with.
Linux kernel configuration
Ensure that your Linux kernel is built with the following options enabled:
CONFIG_COREDUMP=yin order to enable coredump creation by the kernel
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=yin order to enable default mappings. for processes. Specifically, Memfault requires:
bit 0: dump anonymous private mappings,
bit 1: dump anonymous shared mappings, and
bit 4: (available since Linux 2.6.24) dump ELF headers.
man core in order to learn more about these settings. We
recommend reading the entirety of this
man page if you're in need of basic
understanding of how coredumps work in Linux.
In order to confirm that your kernel is correctly configured, check out your
project's configuration file in
tmp/work/[machine]/[kernelpackage]/[version]/build/.config. For example:
The following files will be modified by
memfaultd at runtime and must not be
written to by any other process.
Make sure your Linux image does not contain any other services that may write to
these files. To do this, check for the inclusion of other coredump handlers such
as systemd (which can act as a coredump handler when built with
-Dcoredump=true), and check your
directory for anything other than Memfault that may be setting
To check whether you've succeeded in letting
memfaultd take care of these
files, confirm that the contents of the
core_pattern file reference the
memfault-core-handler binary (see
Test your integration).
memfaultd respects privacy settings and only sets
if data collection is enabled at runtime.
memfaultd-core-handler both have a commercial license,
you'll need to add an exception for them, for example in your
LICENSE_FLAGS_ACCEPTED:append = " commercial_memfaultd commercial_memfault-core-handler"
Make sure prelinking is disabled
Prelinking is an operation which optimizes applications load times by resolving library symbols before launch. However most of the benefits of prelinking are lost unless PIE is disabled and both Glibc and Yocto are dropping support for it. It is disabled by default in Yocto since 3.4 - Honister.
To disable prelinking on an older version of Yocto where
it is still enabled by default, you need to override
local.conf. On Yocto versions before 3.4 - Honister,
image-prelink by default.
If you already have
USER_CLASSES overridden in
local.conf, make sure
image-prelink is not included in the list. If not, you can use
USER_CLASSES:remove = "image-prelink" to keep the defaults, except for
This is how we disable prelinking in our example image:
USER_CLASSES = "buildstats"
Link Time Optimization (LTO)
We strongly recommend against using Link Time Optimization (LTO) in the programs you want to be able to debug using coredumps. When LTO is enabled, the analysis of the coredumps can become lacking or even fail completely.
Upload debugging symbols
After a coredump was collected on the device,
memfaultd will upload it to the
Memfault Web App. To allow the Memfault Web App to reconstruct all details from
such coredumps, you'll need to upload debugging symbols for each of the binaries
in your Linux image. We recommend doing this as part of your build process
(since this needs to be done for every version), either locally or ideally as
part of continuous integration, in order to keep Memfault up to date with your
build's newest debugging symbols.
In order to facilitate this, we've added a Yocto-specific helper subcommand to
the Memfault CLI (starting with version
memfault upload-yocto-symbols. Before you use it, you'll need to update your
base image to include the following, and then run a build:
# Support memfault-cli upload-yocto-symbols command
DEPENDS:append = " elfutils-native"
IMAGE_GEN_DEBUGFS = "1"
IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
Here's an example
memfault upload-yocto-symbols invocation:
$ source oe-init-build-env
$ memfault \
--org $YOUR_ORGANIZATION_SLUG \
--org-token $ORGANIZATION_AUTH_TOKEN \
--project $YOUR_PROJECT_SLUG upload-yocto-symbols \
--image tmp/deploy/images/raspberrypi3/base-image-raspberrypi3.tar.bz2 \
Supported formats are
.tar.xz. The format of
--image may differ from that of
It's recommended that you run this command after having sourced the build
environment script using
IMAGE_GEN_DEBUGFS will cause Yocto to build an archive with separated debug
info of all the binaries on the system. Your system image will not get larger
but a separate
-dbg.tar.gz file will be saved next to your system image.
This archive is used by the
memfault upload-yocto-symbols command. It does not
upload the complete file directly: it extracts it to a temporary directory and
generates unstripped copies of the binaries and their symbols.
If you can't do this, then you can pass the information needed from the build environment as command-line arguments:
--eu-unstrip-path: path to a local
eu-unstripbinary from elfutils. Note that one is available in
tmp/sysroot-components/x86_64/elfutils-nativeif you've added
DEPENDS:append = " elfutils-native"to your build.
--package-debug-split-style: your project's
PACKAGE_DEBUG_SPLIT_STYLE. In Poky, it defaults to
debug-with-srcpkg. Read more about it in the Yocto reference.
If you're not using Yocto, you can replicate its behavior by using our REST API to upload symbols.
You can always upload symbol files individually using the Memfault Web Application. Open Software -> Symbol Files and click on Upload Symbol File, or simply click on this deep link to the app. Note that the symbol files you upload here must be individual ELF files containing both the program text as well as debugging symbols. The Yocto-generated debug image (for example in
.tar.bz2 format) contains stripped binaries and (separately) debug binaries. To upload them to Memfault, you'll need to use the Memfault CLI as shown above.
Read docs on CI and authentication in order to obtain credentials that you can use in your build environment or in continuous integration.
false (see the default
configuration). This is to enable asking end users for
consent before collecting or transmitting any data to Memfault services.
Once the end user has given their consent, you can enable data collection like so:
$ memfaultctl enable-data-collection
To disable it:
$ memfaultctl disable-data-collection
memfaultd service will restart automatically whenever you run either of
those commands if called with a value different from the current configuration.
Take a look at the
/etc/memfaultd.conf reference for
(Optional) Configure what's included in a coredump
Currently, Memfault support for coredumps does not include custom filtering or selection of processes, memory regions or variables. These features are planned for a future release.
We'd recommend waiting for a release of the Memfault Linux SDK with first-class support for coredump filtering, but if your team needs it now, the kernel does provide a certain degree of configurability:
prctl(PR_SET_DUMPABLE, 0)in order to declare a process as non-dumpable. By default, processes are dumpable. See
man prctlfor more information.
- Set the appropiate flags in
- Note that in order to work properly with Memfault,
coredump_filterneeds to include at least bits
- Note that in order to work properly with Memfault,
madvise(ptr, bytes, MADV_DONTDUMP)(with
MADV_DODUMP) in order to declare certain memory regions as dumpable (or non-dumpable). The effects of this syscall take precedence over the settings in the
coredump_filterfor this file (see
man madvisefor more information).
Test your integration
You can test your integration with
will fork itself and crash.
# memfaultctl trigger-coredump
If developer mode is active,
memfaultctl will immediately
push the coredump to Memfault. Otherwise you can force
memfaultd to sync
# memfaultctl sync
If your integration is all set and you've enabled data collection, you'll be able to see a new issue pop up in your project's Issues page.
You might want to check that the
core_pattern file is being set correctly by
memfaultd when it starts. Check that the output of this command includes
$ cat /proc/sys/kernel/core_pattern
To see logs coming from
$ journalctl --unit memfaultd
Dec 21 10:45:20 qemuarm64 systemd: Started memfaultd daemon.
Dec 21 10:45:24 qemuarm64 memfaultd: coredump:: Received corefile for PID 268, process 'memfaultctl'
Dec 21 10:45:24 qemuarm64 memfaultd: coredump:: writing coredump with max size: 98304000
Dec 21 10:45:24 qemuarm64 memfaultd: coredump:: enqueued corefile for PID 268
Dec 21 10:45:30 qemuarm64 memfaultd: network:: Successfully transmitted file '/media/memfault/core/corefile-fbf1b4c8-b184-4b6c-87c1-6229bd55afd7.gz'