Reference: memfaultd Configuration

Useful links:

Basic Usage

Running memfaultd --help prints the following:

Usage: memfaultd [-c <config-file>] [-s] [-Z] [-v] [-V]

Memfault daemon.

Options:
  -c, --config-file use configuration file
  -s, --show-settings
                    show settings and exit immediately
  -Z, --daemonize   daemonize (fork to background)
  -v, --version     show version
  -V, --verbose     verbose output
  --help            display usage information

The --config-file path defaults to /etc/memfaultd.conf. The settings you add in /etc/memfaultd.conf extend the built-in configuration file.

Start/Stop scripts

The meta-memfault layer provides a systemd unit file, as well as a SysVinit start/stop script. The proper file is selected automatically based on your Yocto configuration. If you are not using Yocto, you can refer to these files to add your own start/stop scripts.

Note that for SysVinit systems, the init script is installed with sequence number 15 which places memfaultd after the networking stack (01) but before collectd and swupdate in the boot order.

`/etc/memfaultd.conf`

A full configuration example can be found in the source directory of memfaultd. Here's a copy:

{
  "persist_dir": "/media/memfault",
  "tmp_dir": null,
  "tmp_dir_min_headroom_kib": 10240,
  "tmp_dir_min_inodes": 100,
  "tmp_dir_max_usage_kib": 102400,
  "upload_interval_seconds": 3600,
  "heartbeat_interval_seconds": 3600,
  "enable_data_collection": false,
  "enable_dev_mode": false,
  "software_version": "<YOUR_SOFTWARE_VERSION>",
  "software_type": "<YOUR_SOFTWARE_TYPE>",
  "project_key": "<YOUR_PROJECT_KEY>",
  "base_url": "https://device.memfault.com",
  "swupdate": {
    "input_file": "/etc/swupdate.cfg",
    "output_file": "/tmp/swupdate.cfg"
  },
  "reboot": {
    "last_reboot_reason_file": "/media/last_reboot_reason",
    "capture_pstore": false
  },
  "coredump": {
    "coredump_max_size_kib": 96000,
    "compression": "gzip",
    "rate_limit_count": 5,
    "rate_limit_duration_seconds": 3600,
    "capture_strategy": {
      "type": "threads",
      "max_thread_size_kib": 32
    },
    "log_lines": 100
  },
  "http_server": {
    "bind_address": "127.0.0.1:8787"
  },
  "fluent-bit": {
    "extra_fluentd_attributes": [],
    "bind_address": "127.0.0.1:5170",
    "max_connections": 4
  },
  "logs": {
    "compression_level": 1,
    "max_lines_per_minute": 500,
    "rotate_size_kib": 10240,
    "rotate_after_seconds": 3600,
    "storage": "persist",
    "source": "journald",
    "level_mapping": {
      "enable": false
    },
    "extra_attributes": [],
    "max_buffered_lines": 1000
  },
  "mar": {
    "mar_file_max_size_kib": 10240,
    "mar_entry_max_age_seconds": 604800,
    "mar_entry_max_count": 1000
  },
  "battery_monitor": null,
  "connectivity_monitor": null,
  "metrics": {
    "high_resolution_telemetry": {
      "enable": true,
      "max_samples_per_minute": 750
    },
    "enable_daily_heartbeats": false,
    "system_metric_collection": {
      "poll_interval_seconds": 10,
      "enable": true
    },
    "statsd_server": {
      "bind_address": "127.0.0.1:8125",
      "legacy_gauge_aggregation": false
    }
  }
}

The settings you add in /etc/memfaultd.conf extend the built-in configuration file.

Top-level `/etc/memfaultd.conf` configuration

Field	Description	Recommended value
`upload_interval_seconds`	The period (in seconds) to flush the queue and post it to Memfault. Note that a recovery system with exponential back-off is in place for network failures. Setting this to `0` disables uploads. You can still manually trigger an upload with the `memfaultctl sync` command, the `SIGUSR1` signal or export the data manually with `memfaultctl export`. We recommend setting this to `3600` (1 hour) or more. Values lower than this will cause the uploads to get throttled by the Memfault cloud. See the Rate Limiting section for more information.	3600
`heartbeat_interval_seconds`	The period (in seconds) of metrics aggregation into a Heartbeat event.	3600
`enable_data_collection`	Whether `memfaultd` should collect and post data to Memfault by default. Read more in the Yocto integration guide.	`false` (ask for user consent first)
`enable_dev_mode`	Enable or disable developer mode. Read more in the `memfaultctl` CLI reference.	`false` (only use in development)
`base_url`	The base URL to Memfault's device API.	`https://device.memfault.com`
`software_version`	The current version of your software. Read more about software versions. If unset, the default is the value associated with `VERSION_ID` in `/etc/os-release` on your device.	Project-specific
`software_type`	The current type of your software. Read more about software types. If unset, the default is the value associated with `ID` in `/etc/os-release` on your device.	Project-specific
`project_key`	A write-only key for your Project. Find yours in Project → Settings in the app.	Project-specific
`persist_dir`	A directory where `memfaultd` can store application data persistently (needs to survive firmware upgrades). Read more about persistent storage.	Project-specific
`tmp_dir`	A directory where `memfaultd` can store temporary data. This can be a temporary filesystem. Read more about temporary storage	`""` (will use `persist_dir`)
`tmp_dir_min_headroom_kib`	Minimum space to keep available on the `tmp_dir` filesystem. `memfaultd` will stop writing and will delete buffered data when free space goes below this value.	10% of the filesystem space - or less if your application also uses this filesystem.
`tmp_dir_min_inodes`	Minimum number of inodes to keep available on the `tmp_dir` filesystem. `memfaultd` will stop writing when free inode count goes below this value.	10% of the filesystem inodes.
`tmp_dir_max_usage_kib`	Maximum size of memfault data on the `tmp_dir` filesystem. Memfault will start deleting older data and stop writing when this limit is reached.	Project-specific.
`http_server`	Configuration values for the built-in HTTP server. Read more.	See `http_server`
`swupdate`	Configuration values for the `swupdate` feature if enabled in `memfaultd` (default). Read more.	See `swupdate`
`reboot`	Configuration values for the `reboot` feature. Read more.	See `reboot`
`coredump`	Configuration values for the `coredump` feature if enabled in `memfaultd` (default). Read more	See `coredump`
`persist_storage`	Configuration for persistent storage of MAR entries. This configuration dictates of the persistent directory for MAR, as well as which data types will be stored there. Note this config will be ignored if `tmp_dir` is not also set.	See `persist_storage`

`http_server`

{
  "http_server": {
    "bind_address": "127.0.0.1:8787"
  }
}

Field	Description	Recommended value
`bind_address`	Address (including port) that the server will bind to.	`"127.0.0.1:8787"`

The built-in http server is used to receive metrics from collectd and to manually export the data collected by Memfault.

`swupdate`

{
  "swupdate": {
    "input_file": "/etc/swupdate.cfg",
    "output_file": "/tmp/swupdate.cfg"
  }
}

Field	Description	Recommended value
`input_file`	Will be used as the base SWUpdate configuration when generating `$output_file`. May specify a `suricatta` section (gets merged with generated parameters). If an `identify` section exists it will get replaced. See upstream SWUpdate docs.	`/etc/swupdate.cfg`
`output_file`	Generated by `memfaultd` using `$input_file` as a base. Needs to be passed to SWUpdate as its config file. See an example here.	`/tmp/swupdate.cfg`

`reboot`

{
  "reboot": {
    "last_reboot_reason_file": "/media/last_reboot_reason",
    "capture_pstore": false
  }
}

Field	Description	Recommended value
`last_reboot_reason_file`	The path where `memfaultd`'s reboot reason tracking feature will attempt to find the device-specific reboot reason . If the file does not exist, `memfaultd` will interpret this as "no device specific reason known".	Project-specific
`capture_pstore`	Whether or not to capture the contents of the pstore log buffer on reboot. This can be helpful for diagnosing the cause of kernel panics. Read more on kernel panics in the reboot reason tracking guide. Logs written to the pstore buffer by Ramoops will be uploaded as a custom data recording.	`false`

`coredump`

{
  "coredump": {
    "compression": "gzip",
    "coredump_max_size_kib": 96000,
    "rate_limit_count": 5,
    "rate_limit_duration_seconds": 3600,
    "capture_strategy": {
      "type": "threads"
    },
    "log_lines": 100
  }
}

To configure the location where coredumps are stored during processing, see tmp_dir.

Field	Description	Recommended value
`capture_strategy`	The strategy used to select which regions of memory to capture. See `capture_strategy`	`kernel_selection`
`compression`	Compression to use when storing on disk and uploading to Memfault ( `none` or `gzip`).	`gzip`
`coredump_max_size_kib`	The maximum size of a coredump that can be processed.	96000
`rate_limit_count`^*	The maximum amount of coredumps to process in a given period of `rate_limit_duration_seconds`.	5^*
`rate_limit_duration_seconds`^*	A window in which a maximum of `rate_limit_count` coredumps can be processed.	3600^*
`log_lines`	The maximum number of log lines to capture in the coredump. Setting to 0 will disable the capture of log lines in the coredump.	100
`filters`	A list of filters to ignore coredumps from specific executables. See Filters.	`[]` (no filters)

^* Please consult with the Memfault team if you need to change rate-limiting settings for your integration, as the Memfault Web App will further enforce rate limiting rules.

When a program crashes, the kernel will attempt to produce a coredump for the crashing process. When memfaultd receives the coredump, it will first apply the rate limiting policy, to limit the number of coredumps that can get generated within a period of time. If the rate limit is exceeded, the coredump is dropped.

Next, memfaultd will determine the maximum size that is allowed, based on the coredump_max_size_kib, storage_max_usage_kib and storage_min_headroom_kib configuration values and the amount of available storage space.

Unless there is no available storage, the coredump is written into a temporary holding area inside the tmp_dir.

Finally, the coredump is added to the upload queue. This queue is serviced periodically (see the top-level upload_interval_seconds).

Capture Strategy

memfaultd can use different strategies to capture the memory of the crashed program. Refer to our coredump guide for more information.

`threads`

{
  "coredump": {
    "capture_strategy": {
      "type": "threads",
      "max_thread_size_kib": 32
    }
  }
}

`stacktrace`

{
  "coredump": {
    "capture_strategy": {
      "type": "stacktrace"
    }
  }
}

`kernel_selection`

{
  "coredump": {
    "capture_strategy": {
      "type": "kernel_selection"
    }
  }
}

Filters

You can configure memfaultd to ignore coredumps originating in either a specific executable name or path. This can be helpful to reduce noise from background services that may crash frequently, or when working something in development.

`executable_name`

Filter coredumps based on the executable name.

{
  "coredump": {
    "filters": [
      {
        "type": "executable_name",
        "name": "/path/to/executable"
      },
      {
        "type": "executable_name",
        "name": "/another/path/to/executable"
      }
    ]
  }
}

`executable_path`

Filter coredumps based on the executable path.

{
  "coredump": {
    "filters": [
      {
        "type": "executable_path",
        "path": "/path/to/executables"
      },
      {
        "type": "executable_path",
        "path": "/another/path/to/executables"
      }
    ]
  }
}

`fluent-bit`

{
  "fluent-bit": {
    "extra_fluentd_attributes": [],
    "bind_address": "127.0.0.1:5170",
    "max_buffered_lines": 1000,
    "max_connections": 4
  }
}

Field	Description	Recommended value
`extra_fluentd_attributes`	List of fluentd attributes to save beyond the defaults. Note that using `logs.extra_attributes` is preferred. All values set here will be merged with `logs.extra_attributes`	`[]`
`bind_address`	Address and port to bind the fluent-bit listener to. Replacing `127.0.0.1` by `0.0.0.0` will open the log collection service to the network (not recommended).	`127.0.0.1:5170`
`max_buffered_lines`	Maximum number of lines to buffer in memory before applying backpressure to fluent-bit.	10000
`max_connections`	Maximum number of simultaneous connected sockets with fluent-bit.	4

`logs`

{
  "logs": {
    "compression_level": 1,
    "log_to_metrics": null,
    "max_lines_per_minute": 500,
    "rotate_size_kib": 10240,
    "rotate_after_seconds": 3600,
    "source": "fluent-bit",
    "storage": "persist",
    "extra_attributes": [],
    "max_buffered_lines": 1000
  }
}

Field	Description	Recommended value
`compression_level`	Compression level (0 - none, 1 - fast to 9 - best).	1 - Fast
`filtering`	Configuration for log line filtering conversion.	`null` (disabled)
`max_lines_per_minute`	Maximum number of lines to write per minute.	500
`rotate_after_seconds`	Log files will be rotated when they reach this number of seconds.	3600
`rotate_size_kib`	Log files will be rotated when they reach this size (in kibibytes).	10240 (10 MiB)
`source`	Whether `memfaultd` should pull logs from `fluent-bit`, `syslog`, or `journald`	`"fluent-bit"`, `"journald"`, or `syslog` configuration object
`storage`	Whether `memfaultd` should store processed logs on disk or not	`"persist"` or `"disabled"`
`extra_attributes`	List of log attributes to save beyond the defaults of `MESSAGE`, `_PID`, `PRIORITY`, and `_SYSTEMD_UNIT`	`[]`
`max_buffered_lines`	The maximum number of log lines that will be stored in memory during processing. Any more will be dropped	1000

`filtering`

To configure Log Filtering, you define a key rules in the filtering section of the configuration file. This key must contain a list of valid rules.

Each rule is a JSON object with:

A an optional pattern which is a regular expression that must match the log line for the rule to match. Note that the regular expression must be a valid JSON string, so you need to escape the backslashes.
A optional counter_name which is a string which can contain $1, $2, etc. to reference the capture groups in the regular expression. If a counter_name is defined, a counter within memfaultd will be incremented anytime a log line matches the rule, which will be included in any ongoing Metric Reports.
An optional service which is the name of the service or application that emitted the log line. For systemd services, the .service suffix must be included.
An optional level, which specifies a log level for the matching log lines.
An optional action which is one of exclude, include, or pass. This defines what memfaultd should do with matching log lines. If exclude or include, subsequent log filtering rules will not be evaluated for the matching log line, while if pass log rule evaluation will continue. If there is a not an action defined, the default_action will be used.

See our example Log Filtering configuration here.

`mar`

{
  "mar": {
    "mar_file_max_size_kib": 10240,
    "mar_entry_max_age_seconds": 604800,
    "mar_entry_max_count": 1000
  }
}

Field Description Recommended value

mar_file_max_size_kib Maximum size of one MAR ZIP file for upload. 10240 (10 MiB)

Field	Description	Recommended value
`mar_file_max_size_kib`	Maximum size of one MAR ZIP file for upload.	10240 (10 MiB)
`mar_entry_max_age_seconds`	Maximum age of one MAR entry on disk. Setting this to `0` disables age-based MAR cleanup	604800 (1 week)
`mar_entry_max_count`	The maximum number of MAR entries stored on disk. If the number of entries is exceeded entries will be deleted starting with the oldest.	1000

mar_entry_max_age_seconds

Maximum age of one MAR entry on disk.

Setting this to 0 disables age-based MAR cleanup

604800 (1 week)

mar_entry_max_count The maximum number of MAR entries stored on disk. If the number of entries is exceeded entries will be deleted starting with the oldest. 1000

memfaultd will transform log files and coredumps into MAR entries on disk and keep them in the MAR staging area until they are uploaded.

To upload them, a ZIP file is generated "on the fly", grouping multiple MAR entries together. The mar_file_max_size_kib controls how large this file is allowed to grow. If the individual entries are larger than this setting, they will be uploaded one by one. The mar_entry_max_age_seconds config controls how long an individual MAR entry is allowed to live on disk before it is deleted.

We recommend lowering this value if your Internet connection is not fast or reliable.

`battery_monitor`

The battery monitor is an optional feature to easily capture battery metrics from battery powered devices. It is disabled by default.

The battery monitor will run a command at fixed interval to get battery charging state and state of charge. You configure both the interval (in seconds) and the command to run.

{
  "battery_monitor": {
    "interval_seconds": 10,
    "battery_info_command": "/usr/bin/battery_monitor.sh"
  }
}

The output of the battery_info_command should follow the format <CHARGING STATE>:<STATE OF CHARGE>, just like the [battery readings of the memfaultctl add-battery-reading][memfaultctl-add-battery-reading] command.

`connectivity_monitor`

The connectivity monitor is an optional feature to easily capture connectivity metrics from devices. It is disabled by default.

To enable the connectivity monitor, add the following to your /etc/memfaultd.conf:

{
  "connectivity_monitor": {
    "interval_seconds": 10,
    "targets": [{ "host": "8.8.8.8", "port": 443, "protocol": "tcp" }]
  }
}

The only supported protocol is "tcp".

memfaultd will attempt to make a connection to the specified targets in the order they appear in, using the specified protocol. If the target is unreachable, a connection attempt will be made using the next target in the array. As soon as one reachable target is found, the device will be assumed to be connected during the current interval.

`sessions`

{
  "sessions": [
    {
      "captured_metrics": [
        "cpu/sum/percent/system",
        "cpu/sum/percent/user",
        "cpu/sum/percent/idle",
        "cpu/sum/percent/idle",
        "memory/memory/free",
        "memory/memory/used"
      ],
      "name": "camera-recording"
    }
  ]
}

Field	Description	Valid values
`name`	String name for this type of session	`[a-zA-Z][-a-zA-Z0-9_]`, with a max length of 64
`captured_metrics`	List of metric string keys to capture readings for reports of this session type	[Metric String Key]

The sessions config field is used to defined a list of sessions that can be captured by memfaultd on this device. "name" is the string name that must be passed to memfaultctl start-session or memfaultctl end-session to start and end the session respectively. Only session type names that are defined here can be started - names that are not present in this config will fail to start.

"captured_metrics" is the list of metric keys whose readings will be captured and aggregated for sessions of this type. In addition to the metrics listed here, all Memfault Core Metrics are captured in all session Metric Reports by default.

`linux_custom_trace`

"linux_custom_trace": {
  "log_compression": "gzip"
}

Field	Description	Valid values
`log_compression`	Compression format to use with Linux Custom Traces' attached log files (defaults to gzip)	`gzip`, `zlib`, or `none`

`persist_storage`

"persist_storage": {
    "min_headroom_kib": 1024,
    "max_usage_kib": 1024,
    "min_inodes": 1024,
    "reboots": true,
    "coredumps": true,
    "metrics": false,
    "logs": false
}

The persist_storage config field is used to configure the size of persistent storage, as well as which data types are stored there. Note that this setting is only relevant if tmp_dir[tmp-dir] is set. This field dictates which data types are stored in the [persist_dir][persist_dir], and how large it will be. The end result of this configuration is that certain data types will be stored persistently across reboots, while others will only be stored temporarily in tmp_dir.

Field	Description	Recommended value
`min_headroom_kib`	Minimum space to keep available on the persistent storage filesystem. `memfaultd` will stop writing and will delete buffered data when free space goes below this value.	10% of your filesystem or less
`max_usage_kib`	Maximum size of memfault data on the persistent storage filesystem. Memfault will start deleting older data and stop writing when this limit is reached.	Project-specific
`min_inodes`	Minimum number of inodes to keep available on the persistent storage filesystem. `memfaultd` will stop writing when free inode count goes below this value.	10% of filesystem inodes
`reboots`	Whether to store reboot MAR entries in persistent storage.	`true` or `false`
`coredumps`	Whether to store coredump MAR entries in persistent storage.	`true` or `false`
`metrics`	Whether to store metric MAR entries in persistent storage.	`true` or `false`
`logs`	Whether to store log MAR entries in persistent storage.	`true` or `false`

`metrics`

  "metrics": {
    "enable_daily_heartbeats": false,
    "statsd_server": {
      "bind_address": "127.0.0.1:8125",
      "legacy_key_names": false
    },
    "system_metric_collection": {
      "enable": false,
      "poll_interval_seconds": 10,
      "processes": null,
      "disk_space": null,
      "diskstats": null,
      "network_interfaces": null
    },
    "high_resolution_telemetry": {
      "enable": true,
      "max_samples_per_minute": 750
    }
  }

Field	Description	Valid values
`enable_daily_heartbeats`	Turn on Daily Heartbeat reports for this device	`true` or `false`
`statsd_server.bind_address`	Sets the address `memfaultd`'s StatsD server should bind to. Set the `statsd_server` field to `null` to disable the StatsD server entirely.	`<IP Address>:<Port>`
`statsd_server.legacy_key_names`	Enables legacy key names for the builtin statsd server, adding the `statsd/gauge\|counter/` prefix to metrics received. This is helpful for migrating from the `collectd-statsd` server as the metric names will be consistent.	`true` or `false`
`system_metric_collection.enable`	Turn on `memfaultd`'s built-in system metric collection.	`true` or `false`
`system_metric_collection.poll_interval_seconds`	Sets the frequency at which `memfaultd` will poll the system and generate metric readings based on its state.	Integer (10 is the recommended default)
`system_metric_collection.processes`	Configure which processes should be monitored by `memfaultd`. If `null`, only the `memfaultd` process will be monitored	`[<process names>]`
`system_metric_collection.network_interfaces`	Configure which network interfaces should be monitored by `memfaultd`. If `null`, all non-loopback interfaces will be monitored.	`[<network interface IDs>]`
`system_metric_collection.disk_space`	Configure which disks or partitions should have their capacity utilization monitored by `memfaultd`. If `null`, all disks whose ID begins with `/dev/` will be monitored. RAM and loop disks will be ignored.	`[<mount points>]`
`system_metric_collection.diskstats`	Configure which disks or partitions should have their usage monitored by `memfaultd`. If `null`, all disks whose ID begins with `/dev/` will be monitored. RAM and loop disks will be ignored	`[<mount points>]`
`high_resolution_telemetry.enable`	Turn on the capture of high-resolution telemetry data alongside Heartbeat Metric Reports	`true` or `false`
`high_resolution_telemetry.max_samples_per_minute`	Configure the maximum number of readings ingested by `memfaultd` per minute. This value is set to its maximum of 750 by default and can be lowered to limit the size of HRT data uploaded.	Integer in the range `[0,750]`

Basic Usage​

Start/Stop scripts​

/etc/memfaultd.conf​

Top-level /etc/memfaultd.conf configuration​

http_server​

swupdate​

reboot​

coredump​

Capture Strategy​

threads​

stacktrace​

kernel_selection​

Filters​

executable_name​

executable_path​

fluent-bit​

logs​

filtering​

mar​

battery_monitor​

connectivity_monitor​

sessions​

linux_custom_trace​

persist_storage​

metrics​