Linux Logging
Introduction
memfaultd
captures logs from your device automatically using
journald.
Logs are automatically attached to coredumps captured by memfault. Continuous
log capture can also be enabled on a per device basis.
Logs are parsed directly on the device and can be used to generate metrics even when they are not uploaded to Memfault.
Logs collected by memfaultd
will be shown on the device
timeline.
Prerequisites
The memfaultd
daemon, built with logging
Follow the integration guide to learn how to install
memfaultd
on your device.
The logging feature is enabled by default in the meta-memfault
layer as of
v1.4.0
of Memfault SDK.
Read more on how to configure which features memfaultd
builds
with.
Log Sources
The choice of which source memfaultd
should use for logs is configured via the
source
field in the logs
configuration
journald
The systemd
feature also must be enabled to use the journald
option for
logs.source
.
The journald
log source config provides a way to capture logs from your device
and send them to Memfault via memfaultd
out-of-the-box.
When using the journald
logs source option, memfaultd
will read the logs
sent to systemd
's journal since the last boot (similar to the output of
journalctl --boot
). On shutdown or restart, memfaultd
will save the location
of its current cursor to disk and resume from there so as to avoid re-processing
logs.
If you are running your application or set of applications via systemd
,
nothing else will need to be done to see their logs along with logs from the
Linux system in Memfault.
fluent-bit
The fluent-bit
log source configuration can be used on system without
systemd
, or when logs filtering is required before processing by memfaultd
.
If neither of these apply to you, our recommendation is to use the journald
option.
The fluent-bit source code includes a Yocto recipe to
compile and install fluent-bit
.
We include the same recipe (with some minor changes for compatibility with
recent versions of Yocto) and a sample configuration file in our
meta-memfault-example
layer.
See more information on using fluent-bit
with Memfault in our dedicated
fluent-bit
guide.
Flow of logs
Regardless of the logs source, memfaultd
buffers logs in memory for
performance reasons. memfautld
will maintain a single log file for all the log
messages until the file reaches a specific size or
age, at which point it is written and will be
uploaded during the next synchronization.
You can suppress all writes to disks with the logs.storage
option. When set to
disable
, memfaultd
will not write any logs to disk but it will continue to
process them to generate metrics and maintain a small buffer of recent messages
in memory to attach to coredumps.
memfaultd
compresses logs on disk using the ZLib Deflate algorithm. Multiple
protections are in place to ensure that logs cannot fill the disk, including
rate limiting (in lines per minute) and size limits.
Important Log Configs
logs.compression_level
Log files are compressed using the Deflate algorithm before writing to disk. This setting controls which compression level to apply.
- 0: No compression.
- 1: Fastest compression
- 9: Best compression
The default is 1. After in-house testing, Memfault believes this is the best compromise of CPU-cost to space saving for most use-cases.
logs.max_lines_per_minute
This setting controls how many lines per minute can be saved before discarding new logs.
When logging resumes, memfaultd
will print a message indicating how many lines
were skipped:
Memfaultd rate limited 42 messages."
logs.rotate_after_seconds
Regardless of size, log files are rotated when they reach a certain age.
logs.rotate_size_kib
Rotate the current log file when it reaches this size. After rotation, it will remain in the MAR staging area until the next upload (see general configuration).
logs.source
Whether memfaultd
should use journald
or fluent-bit
as a source of logs.
logs.filtering
A list of log filtering rules memfaultd
should use to determine which logs
should be sent to Memfault and which should be excluded. See details in the Log
Filtering section below.
Enabling data collection
By default, enable_data_collection
is false
. This is to enable asking end
users for consent before collecting or transmitting any data to Memfault
services.
Once the end user has given their consent, you can enable data collection like so:
$ memfaultctl enable-data-collection
To disable it:
$ memfaultctl disable-data-collection
The memfaultd
service will restart automatically whenever you run either of
those commands if called with a value different from the current configuration.
Take a look at the /etc/memfaultd.conf
reference for
more information.
Log Filtering
By default, memfaultd
will include all logs it receives in the log files sent
to Memfault. This may not be the desired behavior, especially if there are a few
services on the system emitting a high volume of verbose debugging logs.
memfaultd
allows the user to define which log lines should be included and
excluded from the logs sent to Memfault with a rich log filtering system.
Log rules are evaluated in the order they appear in the configuration file. If a
log line matches the pattern
of a rule that has the include
or exclude
action
, rule evaluation is terminated and the log line will be included or
excluded. Log lines that match a rule that has the pass
action
will continue
evaluation. Log lines that continue evaluation to the end of the list of rules
will take the configured default_action
.
Example
Here is an example configuration:
{
// ...
"logs": {
// ...
"filtering": {
"default_action": "include"
"rules": [
{
"service": "spammy_application.service",
"action": "exclude"
},
{
"service": "sensors.service",
"pattern": "Logging sensor value: .*",
"level": "DEBUG",
"action": "exclude"
},
{
"counter_name": "ssh_sessions_$1_count",
"pattern": "session opened for user (\\w*)\\(uid="",
"action": "pass"
}
]
}
}
}
This configuration snippet instructs memfaultd
to do the following:
- Exclude all logs from
spammy_application.service
- Exclude all logs from
sensors.service
that have the DEBUG log level and match the regexLogging sensor value: .*
- Increment a counter
ssh_sessions_$1_count
anytime a ssh session is started for a particular user. - Include all log lines that do not match one of the two
exclude
rules ( because of theinclude
default_action
)
Converting logs into metrics
The log filtering system can also be used to convert logs into metrics, as shown
above with the ssh_sessions_$1_count
rule. Specific patterns in log messages
will be captured directly on the device and efficiently transformed into
metrics. Edge processing of logs is much more efficient than trying to
upload all the logs and process them in the cloud.
This enables:
- Monitoring and alerting on Kernel and System logs,
- Monitoring and alerting on application logs when it's not convenient or easy to instrument the application with StatsD metrics directly,
- Monitoring for security events and reporting them.
On the edge conversion of logs to metrics is a powerful feature but it comes at a cost that should be considered in the context of your project:
- Depending on how many rules you define and how much logs are generated by your system there will also be a CPU impact that should be carefully evaluated.
We are here to help understand and evaluate this feature.
Refer to our configuration reference for the full syntax.
Built-in Log Counter Metrics
memfaultd
includes two built-in log counter metrics that are enabled by
default. You do not need to include this in your configuration for them to be
tracked by memfaultd
, but we provide the configuration for these rules below
as a reference:
// ...
"logs": {
// ...
"filtering": {
// ...
"rules": [
{
"counter_name": "oomkill_$1",
"pattern": "Out of memory: Killed process \\d+ \\((.*)\\)",
"action": "pass"
},
{
"counter_name": "systemd_restarts_$1",
"service": "init.scope",
"pattern": "(.*): Scheduled restart job, restart counter is at",
"action": "pass"
}
]
}
}
}
These track the number of systemd
service restarts and number of times the
Linux Out-of-Memory (OOM) Killer has terminated a process due to low available
memory on the system.
Testing your integration
During development, you can use memfaultctl sync
to force memfaultd
to
rotate the current logfile and upload it.
The log will be visible in Memfault Dashboard as soon as it has been processed (usually a few seconds later).
Viewing Logs in the Web Application
To see detailed reports from a specific device, find the device in Fleet → Devices, and then open its Timeline tab.