Forwarding `collectd` Metrics to Memfault

memfaultd has the ability to ingest metrics from collectd to capture further data from the system. All metrics from collectd are sent to the local HTTP server memfaultd exposes.

note

collectd is not required to capture basic, widely applicable metrics like CPU utilization, memory usage, and per-process resource usage. The collectd integration described in this doc is largely meant to be used if there is a specific collectd plugin that is critical for monitoring your device - otherwise the metrics captured by memfaultd out-of-the-box are likely sufficient.

collectd

collectd is a lightweight daemon that collects system and application metrics. It is widely used in the Linux ecosystem and has a large number of plugins available to collect metrics from a variety of sources.

The collectd feature of memfaultd enables a collectd-compatible HTTP endpoint. This endpoint will receive metrics pushed by the collectd daemon at any frequency. memfaultd aggregates the metrics (in memory) into Heartbeats and sends them to the Memfault cloud.

In addition to standard metrics captured by collectd, you can capture custom metrics from your application using the StatsD protocol to collectd.

To use collectd with Memfault, we will configure collectd to send metrics to memfaultd (over HTTP on localhost). This is our recommended configuration and demonstrated in [meta-memfault-example][source-yocto-example].

collectd with Yocto

For Yocto, the meta-oe layer includes a recipe for collectd, so you may be able to just add collectd to your dependencies, e.g. by appending it to IMAGE_INSTALL. In our example project, we've opted for adding it to MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS in [layer.conf][source-layer-conf].

Configure the HTTP server

memfaultd exposes an HTTP server on port 8787 by default. The port must match the port that you will define in the collectd configuration. You can change the port by setting the http_port configuration option in /etc/memfaultd.conf.

Metrics are expected to be sent to the /v1/collectd path (this is not configurable).

Configuring collectd

We recommend familiarizing yourself with collectd and how it's configured in order to make the best use of it alongside memfaultd.

A minimal configuration file to configure collectd to post to Memfault looks like this:

<LoadPlugin write_http>
  FlushInterval 10
</LoadPlugin>
<Plugin write_http>
  <Node "memfaultd">
    URL "http://127.0.0.1:8787/v1/collectd"
    Format "JSON"
    Metrics true
    Notifications false
    StoreRates true
    BufferSize 65536
    Timeout 10000
  </Node>
</Plugin>

This file configures the write_http plugin to send data to the memfaultd daemon on localhost. By default the port 8787 is used. We also set a FlushInterval of 10 seconds.

Note

The FlushInterval is the interval at which collectd will send data to memfaultd. Because memfaultd will re-aggregate the data in memory, a faster frequency will not result in more points visible in the Memfault backend. We recommend anywhere between 10 and 60 seconds.

Setting this interval to a value that is too high will increase the time error on the measurement (heartbeats are time-stamped when they are generated but the data may be much older if the FlushInterval is too high). With a FlushInterval greater than the Heartbeat interval, you would have Heartbeats without data points.

Note that this minimal example does not capture any data. You will need to activate some collectd plugins to capture useful data. This will be covered in the next section.

Recommended configuration

See our recommended collectd configuration in meta-memfault-example. This configuration makes use of standard plugins that enjoy special support on the Memfault platform. Copying this configuration over to your project will guarantee a good first experience.