Linux Logging

Introduction

memfaultd captures logs from your device automatically using journald. Logs are automatically attached to coredumps captured by memfault. Continuous log capture can also be enabled on a per device basis.

If your system does not make use of journald, logs can be forwarded from either fluent-bit or a syslog daemon to memfaultd as well.

Logs are parsed directly on the device and can be used to generate metrics even when they are not uploaded to Memfault.

Logs collected by memfaultd will be shown on the device timeline.

Prerequisites

The memfaultd daemon, built with logging

Follow the integration guide to learn how to install memfaultd on your device.

The logging feature is enabled by default in the meta-memfault layer as of v1.4.0 of Memfault SDK.

Read more on how to configure which features memfaultd builds with.

Log Sources

The choice of which source memfaultd should use for logs is configured via the source field in the logs configuration

journald

Note

The systemd feature also must be enabled to use the journald option for logs.source.

The journald log source config provides a way to capture logs from your device and send them to Memfault via memfaultd out-of-the-box.

When using the journald logs source option, memfaultd will read the logs sent to systemd's journal since the last boot (similar to the output of journalctl --boot). On shutdown or restart, memfaultd will save the location of its current cursor to disk and resume from there so as to avoid re-processing logs.

If you are running your application or set of applications via systemd, nothing else will need to be done to see their logs along with logs from the Linux system in Memfault.

syslog

The syslog log source can be used if your system uses a syslog daemon such as BusyBox's syslogd or rsyslogd. These programs must be configured to collect logs from various sources on the system and forward them to the UDP server that runs inside memfaultd when the syslog logs.source configuration is set.

Configuring memfaultd's syslog Server

To configure memfaultd's syslog server, we need to set the field logs.source.syslog.bind_address. This will tell memfaultd the address to use to host its server that will accept syslog messages over UDP. Below is an example of what this might look like, using localhost and the standard of port 514 for UDP syslog ingestion:

{
  // ...
  "logs": {
    // ...
    "source": {
      "syslog": {
        "bind_address": "127.0.0.1:514"
      }
    }
  }
}

Example syslog daemon Configurations

Most syslog daemons have the capability to forward messages to a server over UDP but may not be configured to do so out of the box. We've included two examples of how to configure the common syslog daemons of BusyBox syslogd and rsyslog to forward messages to 127.0.0.1:514 here for reference.

rsyslog

Similar to memfaultd, rsyslog can be configured via an rsyslog.conf file, commonly located at /etc/rsyslog.conf. We can make use of the omfwd plugin that comes bundled with rsyslog to forward messages from rsyslog to memfaultd over UDP:

# ...
# Rest of rsyslog.conf

# Forward messages to memfaultd over UDP
action(
    type="omfwd"
    target="127.0.0.1"
    port="514"
    protocol="udp"
)

BusyBox syslogd

BusyBox's syslogd is not configured via a config file but rather exclusively via command line flags passed when the program is started.

We can use the combination of the -R flag to tell syslogd to log to memfaultd over the network and -L to tell it to continue to log to the /var/log/messages as well. Below is an example of how to do this for a system that starts syslogd via systemd:

[Unit]
Description=System Logging Service
Wants=busybox-klogd.service

[Service]
EnvironmentFile=-/etc/default/busybox-syslog
ExecStart=/usr/sbin/syslogd -n -L -R 127.0.0.1:514 $OPTIONS
Sockets=syslog.socket

[Install]
WantedBy=multi-user.target
Also=busybox-klogd.service
Alias=syslog.service

fluent-bit

The fluent-bit log source configuration can be used on system without systemd, or when logs filtering is required before processing by memfaultd. If neither of these apply to you, our recommendation is to use the journald option.

The fluent-bit source code includes a Yocto recipe to compile and install fluent-bit.

We include the same recipe (with some minor changes for compatibility with recent versions of Yocto) and a sample configuration file in our meta-memfault-example layer.

See more information on using fluent-bit with Memfault in our dedicated fluent-bit guide.

Flow of logs

Regardless of the logs source, memfaultd buffers logs in memory for performance reasons. memfautld will maintain a single log file for all the log messages until the file reaches a specific size or age, at which point it is written and will be uploaded during the next synchronization.

You can suppress all writes to disks with the logs.storage option. When set to disable, memfaultd will not write any logs to disk but it will continue to process them to generate metrics and maintain a small buffer of recent messages in memory to attach to coredumps.

memfaultd compresses logs on disk using the ZLib Deflate algorithm. Multiple protections are in place to ensure that logs cannot fill the disk, including rate limiting (in lines per minute) and size limits.

Important Log Configs

logs.compression_level

Log files are compressed using the Deflate algorithm before writing to disk. This setting controls which compression level to apply.

• 0: No compression.
• 1: Fastest compression
• 9: Best compression

The default is 1. After in-house testing, Memfault believes this is the best compromise of CPU-cost to space saving for most use-cases.

logs.max_lines_per_minute

This setting controls how many lines per minute can be saved before discarding new logs.

When logging resumes, memfaultd will print a message indicating how many lines were skipped:

Memfaultd rate limited 42 messages."

logs.rotate_after_seconds

Regardless of size, log files are rotated when they reach a certain age.

logs.rotate_size_kib

Rotate the current log file when it reaches this size. After rotation, it will remain in the MAR staging area until the next upload (see general configuration).

logs.source

Whether memfaultd should use journald, fluent-bit, or syslog as its source of logs.

logs.filtering

A list of log filtering rules memfaultd should use to determine which logs should be sent to Memfault and which should be excluded. See details in the Log Filtering section below.

Enabling data collection

By default, enable_data_collection is false. This is to enable asking end users for consent before collecting or transmitting any data to Memfault services.

Once the end user has given their consent, you can enable data collection like so:

$ memfaultctl enable-data-collection

To disable it:

$ memfaultctl disable-data-collection

The memfaultd service will restart automatically whenever you run either of those commands if called with a value different from the current configuration.

Take a look at the /etc/memfaultd.conf reference for more information.

Log Filtering

By default, memfaultd will include all logs it receives in the log files sent to Memfault. This may not be the desired behavior, especially if there are a few services on the system emitting a high volume of verbose debugging logs.

memfaultd allows the user to define which log lines should be included and excluded from the logs sent to Memfault with a rich log filtering system.

Log rules are evaluated in the order they appear in the configuration file. If a log line matches the pattern of a rule that has the include or exclude action, rule evaluation is terminated and the log line will be included or excluded. Log lines that match a rule that has the pass action will continue evaluation. Log lines that continue evaluation to the end of the list of rules will take the configured default_action.

Example

Here is an example configuration:

{
  // ...
  "logs": {
    // ...
    "filtering": {
      "default_action": "include"
      "rules": [
        {
          "service": "spammy_application.service",
          "action": "exclude"
        },
        {
          "service": "sensors.service",
          "pattern": "Logging sensor value: .*",
          "level": "DEBUG",
          "action": "exclude"
        },
        {
          "counter_name": "ssh_sessions_$1_count",
          "pattern": "session opened for user (\\w*)\\(uid="",
          "action": "pass"
        }
      ]
    }
  }
}

This configuration snippet instructs memfaultd to do the following:

• Exclude all logs from spammy_application.service
• Exclude all logs from sensors.service that have the DEBUG log level and match the regex Logging sensor value: .*
• Increment a counter ssh_sessions_$1_count anytime a ssh session is started for a particular user.
• Include all log lines that do not match one of the two exclude rules ( because of the include default_action)

Converting logs into metrics

The log filtering system can also be used to convert logs into metrics, as shown above with the ssh_sessions_$1_count rule. Specific patterns in log messages will be captured directly on the device and efficiently transformed into metrics. Edge processing of logs is much more efficient than trying to upload all the logs and process them in the cloud.

This enables:

• Monitoring and alerting on Kernel and System logs,
• Monitoring and alerting on application logs when it's not convenient or easy to instrument the application with StatsD metrics directly,
• Monitoring for security events and reporting them.

caution

On the edge conversion of logs to metrics is a powerful feature but it comes at a cost that should be considered in the context of your project:

• Depending on how many rules you define and how much logs are generated by your system there will also be a CPU impact that should be carefully evaluated.

We are here to help understand and evaluate this feature.

Refer to our configuration reference for the full syntax.

Built-in Log Counter Metrics

memfaultd includes two built-in log counter metrics that are enabled by default. You do not need to include this in your configuration for them to be tracked by memfaultd, but we provide the configuration for these rules below as a reference:

  // ...
  "logs": {
    // ...
    "filtering": {
      // ...
      "rules": [
        {
          "counter_name": "oomkill_$1",
          "pattern": "Out of memory: Killed process \\d+ \\((.*)\\)",
          "action": "pass"
        },
        {
          "counter_name": "systemd_restarts_$1",
          "service": "init.scope",
          "pattern": "(.*): Scheduled restart job, restart counter is at",
          "action": "pass"
        }
      ]
    }
  }
}

These track the number of systemd service restarts and number of times the Linux Out-of-Memory (OOM) Killer has terminated a process due to low available memory on the system.

Testing your integration

During development, you can use memfaultctl sync to force memfaultd to rotate the current logfile and upload it.

The log will be visible in Memfault Dashboard as soon as it has been processed (usually a few seconds later).

Viewing Logs in the Web Application

To see detailed reports from a specific device, find the device in Fleet → Devices, and then open its Timeline tab.