Forwarding Logs to Memfault with Fluent-bit

On systems that are not using journald, Fluent-bit can be used to forward logs to memfaultd and ultimately Memfault.

Configuring fluent-bit

Fluent-bit provides a rich set of input and filter plugins to control precisely what gets collected. Our default configuration for fluent-bit sets up collection of kernel and systemd log messages.

memfaultd does not generate the fluent-bit configuration file. It just expects a connection from fluent-bit on the default fluent-bit TCP port 5170. Fluent-bit messages are expected to be delivered in the fluent-bit native msgpack format.

This is the required output configuration:

[OUTPUT]
  Name tcp
  Host 127.0.0.1
  Port 5170
  Format msgpack
  Match *
  net.keepalive on
  net.keepalive_idle_timeout 10
  # Default retry limit is 1. We recommend setting to a higher value to
  # decrease the chance of losing logs in the event that memfaultd is
  # (re)starting while fluent-bit is attempting to flush logs:
  Retry_Limit 5

Fluent-bit /etc/memfaultd.conf settings

You can adjust the behavior of memfaultd when it comes to logging using the following configuration options.

See a full configuration reference here.

fluent-bit.bind_address

Change the listening address and port of fluent-bit connector.

fluent-bit.max_buffered_lines

In most cases, memfaultd will immediately write to disk new log lines. Some buffering is required while rotating log files. This controls how many lines may be buffered before back pressure is applied to fluent-bit. The default will be safe for most use-cases.

fluent-bit.max_connections

This limits the number of open connections with fluent-bit. Fluent-bit will typically open one question for each input plugin. Connection keep-alive is optional but we recommend turning it on.

The default is set to 4. Increase this if you have more input plugins.

fluent-bit.extra_fluentd_attributes

To reduce the size of the log files, memfaultd will only save the keys "MESSAGE", "_PID", "_SYSTEMD_UNIT" and "PRIORITY" by default.

If your fluent-bit sources generate more keys that you need to save, add them to this list. They will be visible in the memfault dashboard and searchable.

Recommended Fluent-bit configuration

We recommend starting with our example configuration file. If some applications are too verbose, you can use one of the fluent-bit filter plugins to limit the amount of logs collected.

Filtering out specific messages

To filter out specific messages, you can use the fluent-bit grep plugin.

# Exclude all messages containing the string "Connection timeout. Will retry."
[FILTER]
    name    grep
    match   *
    exclude MESSAGE Connection timeout. Will retry.