Skip to main content

Reference: memfaultd Configuration

Useful links:

Basic Usage​

Running memfaultd --help prints the following:

Usage: memfaultd [-c <config-file>] [-s] [-Z] [-v] [-V]

Memfault daemon.

Options:
  -c, --config-file use configuration file
  -s, --show-settings
                    show settings and exit immediately
  -Z, --daemonize   daemonize (fork to background)
  -v, --version     show version
  -V, --verbose     verbose output
  --help            display usage information

The --config-file path defaults to /etc/memfaultd.conf. The settings you add in /etc/memfaultd.conf extend the built-in configuration file.

Start/Stop scripts​

The meta-memfault layer provides a systemd unit file, as well as a SysVinit start/stop script. The proper file is selected automatically based on your Yocto configuration. If you are not using Yocto, you can refer to these files to add your own start/stop scripts.

Note that for SysVinit systems, the init script is installed with sequence number 15 which places memfaultd after the networking stack (01) but before collectd and swupdate in the boot order.

/etc/memfaultd.conf​

A full configuration example can be found in the source directory of memfaultd. Here's a copy:

{
  "persist_dir": "/media/memfault",
  "tmp_dir": null,
  "tmp_dir_min_headroom_kib": 10240,
  "tmp_dir_min_inodes": 100,
  "tmp_dir_max_usage_kib": 102400,
  "upload_interval_seconds": 3600,
  "heartbeat_interval_seconds": 3600,
  "enable_data_collection": false,
  "enable_dev_mode": false,
  "software_version": "<YOUR_SOFTWARE_VERSION>",
  "software_type": "<YOUR_SOFTWARE_TYPE>",
  "project_key": "<YOUR_PROJECT_KEY>",
  "base_url": "https://device.memfault.com",
  "swupdate": {
    "input_file": "/etc/swupdate.cfg",
    "output_file": "/tmp/swupdate.cfg"
  },
  "reboot": {
    "last_reboot_reason_file": "/media/last_reboot_reason",
    "capture_pstore": false
  },
  "coredump": {
    "coredump_max_size_kib": 96000,
    "compression": "gzip",
    "rate_limit_count": 5,
    "rate_limit_duration_seconds": 3600,
    "capture_strategy": {
      "type": "threads",
      "max_thread_size_kib": 32
    },
    "log_lines": 100
  },
  "http_server": {
    "bind_address": "127.0.0.1:8787"
  },
  "fluent-bit": {
    "extra_fluentd_attributes": [],
    "bind_address": "127.0.0.1:5170",
    "max_buffered_lines": 1000,
    "max_connections": 4
  },
  "logs": {
    "compression_level": 1,
    "max_lines_per_minute": 500,
    "rotate_size_kib": 10240,
    "rotate_after_seconds": 3600,
    "storage": "persist",
    "source": "fluent-bit"
  },
  "mar": {
    "mar_file_max_size_kib": 10240,
    "mar_entry_max_age_seconds": 604800
  },
  "battery_monitor": null,
  "connectivity_monitor": null,
  "metrics": {
    "enable_daily_heartbeats": false,
    "statsd_server": null,
    "system_metric_collection": {
      "poll_interval_seconds": 10,
      "processes": null,
      "disk_space": null,
      "network_interfaces": null,
      "enable": false
    }
  }
}

The settings you add in /etc/memfaultd.conf extend the built-in configuration file.

Top-level /etc/memfaultd.conf configuration​

FieldDescriptionRecommended value
upload_interval_seconds

The period (in seconds) to flush the queue and post it to Memfault. Note that a recovery system with exponential back-off is in place for network failures.

Setting this to 0 disables uploads. You can still manually trigger an upload with the memfaultctl sync command, the SIGUSR1 signal or export the data manually with memfaultctl export.

We recommend setting this to 3600 (1 hour) or more. Values lower than this will cause the uploads to get throttled by the Memfault cloud. See the

Rate Limiting section for more information.

3600
heartbeat_interval_seconds

The period (in seconds) of metrics aggregation into a Heartbeat event.

3600
enable_data_collection

Whether memfaultd should collect and post data to Memfault by default. Read more here.

false (ask for user consent first)

enable_dev_mode

Enable or disable developer mode. Read more here.

false (only use in development)

base_urlThe base URL to Memfault's device API.https://device.memfault.com
software_version

The current version of your software. Read more here. If unset, the default is the value associated with VERSION_ID in /etc/os-release on your device.

Project-specific
software_type

The current type of your software. Read more here. If unset, the default is the value associated with ID in /etc/os-release on your device.

Project-specific
project_key

A write-only key for your Project. Find yours in Project → Settings in the app.

Project-specific
persist_dir

A directory where memfaultd can store application data persistently (needs to survive firmware upgrades). Read more here.

Project-specific
tmp_dir

A directory where memfaultd can store temporary data. This can be a temporary filesystem. Read more here

"" (will use persist_dir)

tmp_dir_min_headroom_kib

Minimum space to keep available on the tmp_dir filesystem. memfaultd will stop writing and will delete buffered data when free space goes below this value.

10% of the filesystem space - or less if your application also uses this filesystem.

tmp_dir_min_inodes

Minimum number of inodes to keep available on the tmp_dir filesystem. memfaultd will stop writing when free inode count goes below this value.

10% of the filesystem inodes.
tmp_dir_max_usage_kib

Maximum size of memfault data on the tmp_dir filesystem. Memfault will start deleting older data and stop writing when this limit is reached.

Project-specific.
http_server

Configuration values for the built-in HTTP server. Read more.

See

http_server
swupdate

Configuration values for the swupdate feature if enabled in memfaultd (default). Read more.

See

swupdate
reboot

Configuration values for the reboot feature. Read more.

See

reboot
coredump

Configuration values for the coredump feature if enabled in memfaultd (default). Read more

See

coredump

http_server​

{
  "http_server": {
    "bind_address": "127.0.0.1:8787"
  }
}
FieldDescriptionRecommended value
bind_addressAddress (including port) that the server will bind to."127.0.0.1:8787"

The built-in http server is used to receive metrics from collectd and to manually export the data collected by Memfault.

swupdate​

{
  "swupdate": {
    "input_file": "/etc/swupdate.cfg",
    "output_file": "/tmp/swupdate.cfg"
  }
}
FieldDescriptionRecommended value
input_file

Will be used as the base SWUpdate configuration when generating $output_file. May specify a suricatta section (gets merged with generated parameters). If an identify section exists it will get replaced. See upstream SWUpdate docs.

/etc/swupdate.cfg
output_file

Generated by memfaultd using $input_file as a base. Needs to be passed to SWUpdate as its config file. See an example here.

/tmp/swupdate.cfg

reboot​

{
  "reboot": {
    "last_reboot_reason_file": "/media/last_reboot_reason",
    "capture_pstore": false
  }
}
FieldDescriptionRecommended value
last_reboot_reason_file

The path where memfaultd's reboot reason tracking feature will attempt to find the device-specific reboot reason . If the file does not exist, memfaultd will interpret this as "no device specific reason known".

Project-specific
capture_pstore

Whether or not to capture the contents of the pstore log buffer on reboot. This can be helpful for diagnosing the cause of kernel panics. Read more on kernel panics here. Logs written to the pstore buffer by Ramoops will be uploaded as a custom data recording.

false

coredump​

{
  "coredump": {
    "compression": "gzip",
    "coredump_max_size_kib": 96000,
    "rate_limit_count": 5,
    "rate_limit_duration_seconds": 3600,
    "capture_strategy": {
      "type": "threads"
    },
    "log_lines": 100
  }
}

To configure the location where coredumps are stored during processing, see tmp_dir.

FieldDescriptionRecommended value
capture_strategy

The strategy used to select which regions of memory to capture. See capture_strategy

kernel_selection
compression

Compression to use when storing on disk and uploading to Memfault ( none or gzip).

gzip
coredump_max_size_kibThe maximum size of a coredump that can be processed.96000
rate_limit_count*

The maximum amount of coredumps to process in a given period of rate_limit_duration_seconds.

5*

rate_limit_duration_seconds*

A window in which a maximum of rate_limit_count coredumps can be processed.

3600*

log_lines

The maximum number of log lines to capture in the coredump. Setting to 0 will disable the capture of log lines in the coredump.

100

* Please consult with the Memfault team if you need to change rate-limiting settings for your integration, as the Memfault Web App will further enforce rate limiting rules.

When a program crashes, the kernel will attempt to produce a coredump for the crashing process. When memfaultd receives the coredump, it will first apply the rate limiting policy, to limit the number of coredumps that can get generated within a period of time. If the rate limit is exceeded, the coredump is dropped.

Next, memfaultd will determine the maximum size that is allowed, based on the coredump_max_size_kib, storage_max_usage_kib and storage_min_headroom_kib configuration values and the amount of available storage space.

Unless there is no available storage, the coredump is written into a temporary holding area inside the tmp_dir.

Finally, the coredump is added to the upload queue. This queue is serviced periodically (see the top-level upload_interval_seconds).

Capture Strategy​

memfaultd can use different strategies to capture the memory of the crashed program. Refer to our coredump guide for more information.

threads​
{
  "coredump": {
    "capture_strategy": {
      "type": "threads",
      "max_thread_size_kib": 32
    }
  }
}
kernel_selection​
{
  "coredump": {
    "capture_strategy": {
      "type": "kernel_selection"
    }
  }
}

fluent-bit​

{
  "fluent-bit": {
    "extra_fluentd_attributes": [],
    "bind_address": "127.0.0.1:5170",
    "max_buffered_lines": 1000,
    "max_connections": 4
  }
}
FieldDescriptionRecommended value
extra_fluentd_attributes

List of fluentd attributes to save beyond the defaults.

[]
bind_address

Address and port to bind the fluent-bit listener to.

Replacing 127.0.0.1 by 0.0.0.0 will open the log collection service to the network (not recommended).

127.0.0.1:5170
max_buffered_lines

Maximum number of lines to buffer in memory before applying backpressure to fluent-bit.

10000
max_connectionsMaximum number of simultaneous connected sockets with fluent-bit.4

logs​

{
  "logs": {
    "compression_level": 1,
    "log_to_metrics": null,
    "max_lines_per_minute": 500,
    "rotate_size_kib": 10240,
    "rotate_after_seconds": 3600,
    "source": "fluent-bit",
    "storage": "persist"
  }
}
FieldDescriptionRecommended value
compression_levelCompression level (0 - none, 1 - fast to 9 - best).1 - Fast
log_to_metrics

Configuration for log to metrics conversion.

null (disabled)

max_lines_per_minuteMaximum number of lines to write per minute.500
rotate_after_secondsLog files will be rotated when they reach this number of seconds.3600
rotate_size_kib

Log files will be rotated when they reach this size (in kibibytes).

10240 (10 MiB)
source

Whether memfaultd should pull logs from fluent-bit or journald

"fluent-bit" or "journald"
storageWhether memfaultd should store processed logs on disk or not "persist" or "disabled"

log_to_metrics​

To configure the conversion of log into metrics, you define a key rules in the log_to_metrics section of the configuration file. This key must contain a list of valid rules.

Each rule is a JSON object with:

  • A type which most always be count_matching for now,
  • An optional filter which is a list of key-value pairs that must be present in the log line for the rule to apply. This is useful to quickly filter out logs that should be ignored without trying to apply the regular expression.
  • A pattern which is a regular expression that must match the log line for the rule to apply. Note that the regular expression must be a valid JSON string, so you need to escape the backslashes.
  • A metric_name which is a string which can contain $1, $2, etc. to reference the capture groups in the regular expression.
Example​

The following configuration (included in our meta-memfault-example image) will count how many time SystemD services were restarted and how often the Linux Out Of Memory Killer decided to kill a process:

{
  // ...
  "logs": {
    // ...
    "log_to_metrics": {
        "rules": [
            {
                "type": "count_matching",
                "filter": {
                  "_SYSTEMD_UNIT": "init.scope"
                },
                "pattern": "(.*): Scheduled restart job, restart counter is at",
                "metric_name": "systemd_restarts_$1"
            },
            {
              "type": "count_matching",
              "filter": {
              },
              "pattern": "Out of memory: Killed process \\d+ \\((.*)\\)",
              "metric_name": "oomkill_$1"
            }
        ]
    }
  }
}

mar​

{
  "mar": {
    "mar_file_max_size_kib": 10240,
    "mar_entry_max_age_seconds": 604800
  }
}
FieldDescriptionRecommended value
mar_file_max_size_kibMaximum size of one MAR ZIP file for upload.10240 (10 MiB)
mar_entry_max_age_seconds

Maximum age of one MAR entry on disk.

Setting this to 0 disables age-based MAR cleanup

604800 (1 week)

memfaultd will transform log files and coredumps into MAR entries on disk and keep them in the MAR staging area until they are uploaded.

To upload them, a ZIP file is generated "on the fly", grouping multiple MAR entries together. The mar_file_max_size_kib controls how large this file is allowed to grow. If the individual entries are larger than this setting, they will be uploaded one by one. The mar_entry_max_age_seconds config controls how long an individual MAR entry is allowed to live on disk before it is deleted.

We recommend lowering this value if your Internet connection is not fast or reliable.

battery_monitor​

The battery monitor is an optional feature to easily capture battery metrics from battery powered devices. It is disabled by default.

The battery monitor will run a command at fixed interval to get battery charging state and state of charge. You configure both the interval (in seconds) and the command to run.

{
  "battery_monitor": {
    "interval_seconds": 10,
    "battery_info_command": "/usr/bin/battery_monitor.sh"
  }
}

The output of the battery_info_command should follow the format <CHARGING STATE>:<STATE OF CHARGE>, just like the [battery readings of the memfaultctl add-battery-reading][memfaultctl-add-battery-reading] command.

connectivity_monitor​

The connectivity monitor is an optional feature to easily capture connectivity metrics from devices. It is disabled by default.

To enable the connectivity monitor, add the following to your /etc/memfaultd.conf:

{
  "connectivity_monitor": {
    "interval_seconds": 10,
    "targets": [{ "host": "8.8.8.8", "port": 443, "protocol": "tcp" }]
  }
}

The only supported protocol is "tcp".

memfaultd will attempt to make a connection to the specified targets in the order they appear in, using the specified protocol. If the target is unreachable, a connection attempt will be made using the next target in the array. As soon as one reachable target is found, the device will be assumed to be connected during the current interval.

sessions​

{
  "sessions": [
    {
      "captured_metrics": [
        "cpu/sum/percent/system",
        "cpu/sum/percent/user",
        "cpu/sum/percent/idle",
        "cpu/sum/percent/idle",
        "memory/memory/free",
        "memory/memory/used"
      ],
      "name": "camera-recording"
    }
  ]
}
FieldDescriptionValid values
nameString name for this type of session[a-zA-Z][-a-zA-Z0-9_], with a max length of 64
captured_metrics

List of metric string keys to capture readings for reports of this session type

[Metric String Key]

The sessions config field is used to defined a list of sessions that can be captured by memfaultd on this device. "name" is the string name that must be passed to memfaultctl start-session or memfaultctl end-session to start and end the session respectively. Only session type names that are defined here can be started - names that are not present in this config will fail to start.

"captured_metrics" is the list of metric keys whose readings will be captured and aggregated for sessions of this type. In addition to the metrics listed here, all Memfault Core Metrics are captured in all session Metric Reports by default.

metrics​

  "metrics": {
    "enable_daily_heartbeats": false,
    "statsd_server": {
      "bind_address": "127.0.0.1:8125"
    },
    "system_metric_collection": {
      "enable": false,
      "poll_interval_seconds": 10,
      "processes": null,
      "disk_space": null,
      "network_interfaces": null
    }
  }
FieldDescriptionValid values
enable_daily_heartbeatsTurn on Daily Heartbeat reports for this devicetrue or false
statsd_server.bind_address

Sets the address memfaultd's StatsD server should bind to. Set the statsd_server field to null to disable the StatsD server entirely.

<IP Address>:<Port>
system_metric_collection.enableTurn on memfaultd's built-in system metric collection.true or false
system_metric_collection.poll_interval_secondsSets the frequency at which memfaultd will poll the system and generate metric readings based on its state. Integer (10 is the recommended default)
system_metric_collection.processesConfigure which processes should be monitored by memfaultd. If null, only the memfaultd process will be monitored[<process names>]
system_metric_collection.network_interfacesConfigure which network interfaces should be monitored by memfaultd. If null, all non-loopback interfaces will be monitored.[<network interface IDs>]
system_metric_collection.disk_spaceConfigure which disks or partitions should have their usage monitored by memfaultd. If null, all disks whose ID begins with /dev/ will be monitored.[<mount points>]
Last updated on
This page is human-authored. If you see any errors, outdated information, or any other suggestions for improving the contents, please contact us!