Skip to main content

Reference: memfaultd Configuration

Useful links:

Basic Usage

Running memfaultd --help prints the following:

Usage: memfaultd [-c <config-file>] [-s] [-Z] [-v] [-V]

Memfault daemon.

Options:
-c, --config-file use configuration file
-s, --show-settings
show settings and exit immediately
-Z, --daemonize daemonize (fork to background)
-v, --version show version
-V, --verbose verbose output
--help display usage information

The --config-file path defaults to /etc/memfaultd.conf. The settings you add in /etc/memfaultd.conf extend the built-in configuration file.

Start/Stop scripts

The meta-memfault layer provides a systemd unit file, as well as a SysVinit start/stop script. The proper file is selected automatically based on your Yocto configuration. If you are not using Yocto, you can refer to these files to add your own start/stop scripts.

Note that for SysVinit systems, the init script is installed with sequence number 15 which places memfaultd after the networking stack (01) but before collectd and swupdate in the boot order.

/etc/memfaultd.conf

A full configuration example can be found in the source directory of memfaultd. Here's a copy:

{
"persist_dir": "/media/memfault",
"tmp_dir": null,
"tmp_dir_min_headroom_kib": 10240,
"tmp_dir_min_inodes": 100,
"tmp_dir_max_usage_kib": 102400,
"upload_interval_seconds": 3600,
"heartbeat_interval_seconds": 3600,
"enable_data_collection": false,
"enable_dev_mode": false,
"software_version": "<YOUR_SOFTWARE_VERSION>",
"software_type": "<YOUR_SOFTWARE_TYPE>",
"project_key": "<YOUR_PROJECT_KEY>",
"base_url": "https://device.memfault.com",
"swupdate": {
"input_file": "/etc/swupdate.cfg",
"output_file": "/tmp/swupdate.cfg"
},
"reboot": {
"last_reboot_reason_file": "/media/last_reboot_reason"
},
"coredump": {
"coredump_max_size_kib": 96000,
"compression": "gzip",
"rate_limit_count": 5,
"rate_limit_duration_seconds": 3600,
"capture_strategy": {
"type": "threads",
"max_thread_size_kib": 10240
}
},
"http_server": {
"bind_address": "127.0.0.1:8787"
},
"fluent-bit": {
"extra_fluentd_attributes": [],
"bind_address": "127.0.0.1:5170",
"max_buffered_lines": 1000,
"max_connections": 4
},
"logs": {
"compression_level": 1,
"max_lines_per_minute": 500,
"rotate_size_kib": 10240,
"rotate_after_seconds": 3600,
"log_to_metrics": null
},
"mar": {
"mar_file_max_size_kib": 10240,
"mar_entry_max_age_seconds": 604800
},
"battery_monitor": null,
"connectivity_monitor": null
}

The settings you add in /etc/memfaultd.conf extend the built-in configuration file.

Top-level /etc/memfaultd.conf configuration

FieldDescriptionRecommended value
upload_interval_seconds

The period (in seconds) to flush the queue and post it to Memfault. Note that a recovery system with exponential back-off is in place for network failures.



Setting this to 0 disables uploads. You can still manually trigger an upload with the memfaultctl sync command, the SIGUSR1 signal or export the data manually with memfaultctl export .



We recommend setting this to 3600 (1 hour) or more. Values lower than this will cause the uploads to get throttled by the Memfault cloud. See the Rate Limiting section for more information.

3600
heartbeat_interval_seconds

The period (in seconds) of metrics aggregation into a heartbeat event.

3600
enable_data_collection

Whether memfaultd should collect and post data to Memfault by default. Read more

here

.

false (ask for user consent first)

enable_dev_mode

Enable or disable developer mode. Read more here.

false (only use in development)

base_urlThe base URL to Memfault's device API.https://device.memfault.com
software_version

The current version of your software. Read more

here

.

Project-specific
software_type

The current type of your software. Read more

here

.

Project-specific
project_key

A write-only key for your Project. Find yours in

Project → Settings

in the app.

Project-specific
persist_dir

A directory where memfaultd can store application data persistently (needs to survive firmware upgrades). Read more

here

.

Project-specific
tmp_dir

A directory where memfaultd can store temporary data. This can be a temporary filesystem. Read more here

"" (will use persist_dir)

tmp_dir_min_headroom_kib

Minimum space to keep available on the tmp_dir filesystem. memfaultd will stop writing and will delete buffered data when free space goes below this value.

10% of the filesystem space - or less if your application also uses this filesystem.

tmp_dir_min_inodes

Minimum number of inodes to keep available on the tmp_dir filesystem. memfaultd will stop writing when free inode count goes below this value.

10% of the filesystem inodes.
tmp_dir_max_usage_kib

Maximum size of memfault data on the tmp_dir filesystem. Memfault will start deleting older data and stop writing when this limit is reached.

Project-specific.
http_server

Configuration values for the built-in HTTP server. Read more.

See

http_server
swupdate

Configuration values for the swupdate feature if enabled in memfaultd (default). Read more.

See

swupdate
reboot

Configuration values for the reboot feature. Read more.

See

reboot
coredump

Configuration values for the coredump feature if enabled in memfaultd (default). Read more.

See

coredump

http_server

{
"http_server": {
"bind_address": "127.0.0.1:8787"
}
}
FieldDescriptionRecommended value
bind_addressAddress (including port) that the server will bind to."127.0.0.1:8787"

The built-in http server is used to receive metrics from collectd and to manually export the data collected by Memfault.

swupdate

{
"swupdate": {
"input_file": "/etc/swupdate.cfg",
"output_file": "/tmp/swupdate.cfg"
}
}
FieldDescriptionRecommended value
input_file

Will be used as the base SWUpdate configuration when generating $output_file. May specify a suricatta section (gets merged with generated parameters). If an identify section exists it will get replaced. See

upstream SWUpdate docs

.

/etc/swupdate.cfg
output_file

Generated by memfaultd using $input_file as a base. Needs to be passed to SWUpdate as its config file. See

an example here

.

/tmp/swupdate.cfg

reboot

{
"reboot": {
"last_reboot_reason_file": "/media/last_reboot_reason"
}
}
FieldDescriptionRecommended value
last_reboot_reason_file

The path where memfaultd's reboot reason tracking feature will attempt to find the

device-specific reboot reason

. If the file does not exist, memfaultd will interpret this as "no device specific reason known".

Project-specific

coredump

{
"coredump": {
"compression": "gzip",
"coredump_max_size_kib": 96000,
"rate_limit_count": 5,
"rate_limit_duration_seconds": 3600,
"capture_strategy": {
"type": "threads"
}
}
}

To configure the location where coredumps are stored during processing, see tmp_dir.

FieldDescriptionRecommended value
capture_strategy

The strategy used to select which regions of memory to capture. See

capture_strategy
kernel_selection
compression

Compression to use when storing on disk and uploading to Memfault ( none or gzip).

gzip
coredump_max_size_kibThe maximum size of a coredump that can be processed.96000
rate_limit_count*

The maximum amount of coredumps to process in a given period of rate_limit_duration_seconds.

5*

rate_limit_duration_seconds*

A window in which a maximum of rate_limit_count coredumps can be processed.

3600*

* Please consult with the Memfault team if you need to change rate-limiting settings for your integration, as the Memfault Web App will further enforce rate limiting rules.

When a program crashes, the kernel will attempt to produce a coredump for the crashing process. When memfaultd receives the coredump, it will first apply the rate limiting policy, to limit the number of coredumps that can get generated within a period of time. If the rate limit is exceeded, the coredump is dropped.

Next, memfaultd will determine the maximum size that is allowed, based on the coredump_max_size_kib, storage_max_usage_kib and storage_min_headroom_kib configuration values and the amount of available storage space.

Unless there is no available storage, the coredump is written into a temporary holding area inside the tmp_dir.

Finally, the coredump is added to the upload queue. This queue is serviced periodically (see the top-level upload_interval_seconds).

Capture Strategy

memfaultd can use different strategies to capture the memory of the crashed program. Refer to our coredump guide for more information.

threads
{
"coredump": {
"capture_strategy": {
"type": "threads",
"max_thread_size_kib": 32
}
}
}
kernel_selection
{
"coredump": {
"capture_strategy": {
"type": "kernel_selection"
}
}
}

fluent-bit

{
"fluent-bit": {
"extra_fluentd_attributes": [],
"bind_address": "127.0.0.1:5170",
"max_buffered_lines": 1000,
"max_connections": 4
}
}
FieldDescriptionRecommended value
extra_fluentd_attributes

List of fluentd attributes to save beyond

the defaults

.

[]
bind_address

Address and port to bind the fluent-bit listener to.


Replacing 127.0.0.1 by 0.0.0.0 will open the log collection service to the network (not recommended).

127.0.0.1:5170
max_buffered_lines

Maximum number of lines to buffer in memory before applying backpressure to fluent-bit.

10000
max_connectionsMaximum number of simultaneous connected sockets with fluent-bit.4

logs

{
"logs": {
"compression_level": 1,
"log_to_metrics": null,
"max_lines_per_minute": 500,
"rotate_size_kib": 10240,
"rotate_after_seconds": 3600
}
}
FieldDescriptionRecommended value
compression_levelCompression level (0 - none, 1 - fast to 9 - best).1 - Fast
log_to_metrics

Configuration for

log to metrics

conversion.

null (disabled)

max_lines_per_minuteMaximum number of lines to write per minute.500
rotate_after_secondsLog files will be rotated when they reach this number of seconds.3600
rotate_size_kib

Log files will be rotated when they reach this size (in kibibytes).

10240 (10 MiB)

log_to_metrics

To configure the conversion of log into metrics, you define a key rules in the log_to_metrics section of the configuration file. This key must contain a list of valid rules.

Each rule is a JSON object with:

  • A type which most always be count_matching for now,
  • An optional filter which is a list of key-value pairs that must be present in the log line for the rule to apply. This is useful to quickly filter out logs that should be ignored without trying to apply the regular expression.
  • A pattern which is a regular expression that must match the log line for the rule to apply. Note that the regular expression must be a valid JSON string, so you need to escape the backslashes.
  • A metric_name which is a string which can contain $1, $2, etc. to reference the capture groups in the regular expression.
Example

The following configuration (included in our meta-memfault-example image) will count how many time SystemD services were restarted and how often the Linux Out Of Memory Killer decided to kill a process:

{
// ...
"logs": {
// ...
"log_to_metrics": {
"rules": [
{
"type": "count_matching",
"filter": {
"_SYSTEMD_UNIT": "init.scope"
},
"pattern": "(.*): Scheduled restart job, restart counter is at",
"metric_name": "systemd_restarts_$1"
},
{
"type": "count_matching",
"filter": {
},
"pattern": "Out of memory: Killed process \\d+ \\((.*)\\)",
"metric_name": "oomkill_$1"
}
]
}
}
}

mar

{
"mar": {
"mar_file_max_size_kib": 10240,
"mar_entry_max_age_seconds": 604800
}
}
FieldDescriptionRecommended value
mar_file_max_size_kibMaximum size of one MAR ZIP file for upload.10240 (10 MiB)
mar_entry_max_age_seconds

Maximum age of one MAR entry on disk.


Setting this to 0 disables age-based MAR cleanup

604800 (1 week)

memfaultd will transform log files and coredumps into MAR entries on disk and keep them in the MAR staging area until they are uploaded.

To upload them, a ZIP file is generated "on the fly", grouping multiple MAR entries together. The mar_file_max_size_kib controls how large this file is allowed to grow. If the individual entries are larger than this setting, they will be uploaded one by one. The mar_entry_max_age_seconds config controls how long an individual MAR entry is allowed to live on disk before it is deleted.

We recommend lowering this value if your Internet connection is not fast or reliable.

battery_monitor

The battery monitor is an optional feature to easily capture battery metrics from battery powered devices. It is disabled by default.

The battery monitor will run a command at fixed interval to get battery charging state and state of charge. You configure both the interval (in seconds) and the command to run.

{
"battery_monitor": {
"interval_seconds": 10,
"battery_info_command": "/usr/bin/battery_monitor.sh"
}
}

The output of the battery_info_command should follow the format <CHARGING STATE>:<STATE OF CHARGE>, just like the [battery readings of the memfaultctl add-battery-reading][memfaultctl-add-battery-reading] command.

connectivity_monitor

The connectivity monitor is an optional feature to easily capture connectivity metrics from devices. It is disabled by default.

To enable the connectivity monitor, add the following to your /etc/memfaultd.conf:

{
"connectivity_monitor": {
"interval_seconds": 10,
"targets": [{ "host": "8.8.8.8", "port": 443, "protocol": "tcp" }]
}
}

The only supported protocol is "tcp".

memfaultd will attempt to make a connection to the specified targets in the order they appear in, using the specified protocol. If the target is unreachable, a connection attempt will be made using the next target in the array. As soon as one reachable target is found, the device will be assumed to be connected during the current interval.