Skip to main content

Linux Metrics

Introduction​

The collectd plugin for memfaultd (see the source code) configures collectd to regularly upload collected metrics to Memfault. These metrics:

See Metrics and Attributes for an introduction to Memfault terminology related to metrics and to learn how the features you'll set up here can be accessed in the Memfault Web App.

Prerequisites​

The memfaultd daemon​

Follow the integration guide to learn how to set this up for your device. A key function of memfaultd is to correctly configure and control collectd to upload metrics to Memfault.

tip

Keep meta-memfault-example open as a reference implementation. Your integration should look similar to it once you're done following the steps in this tutorial.

Dependencies​

collectd​

If you're using Yocto, the meta-oe layer includes a recipe for collectd, so you may be able to just add collectd to your dependencies, e.g. by appending it to IMAGE_INSTALL. In our example project, we've opted for adding it to MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS in layer.conf.

Optional: an application metrics client​

There are different ways to track application metrics using collectd. We recommend using a StatsD client, but other mechanisms are expected to work equally well. StatsD clients can be configured to communicate with collectd using the StatsD plugin. This will allow you to send application metrics through to Memfault. You can find a list of StatsD clients for a diversity of languages in the StatsD repository.

In our example project, we've added StatsD clients as dependencies:

Configuring collectd​

We strongly recommend familiarizing yourself with collectd and how it's configured in order to make the best use of the Memfault platform.

The memfaultd daemon generates two configuration files, by default in /tmp/collectd-header-include.conf and /tmp/collectd-footer-include.conf, with information about the device provided by memfault-device-info (see the [getting-started guide][docs-linux-getting-started] for more information). Note that the path where this configuration file is generated can be configured in /etc/memfaultd.conf using the collectd_plugin.output_file configuration key.

A minimal configuration file to configure collectd to post to Memfault looks like this:

Include "/tmp/collectd-header-include.conf" # Or the value of collectd_plugin.header_include_output_file

# Your configuration here.

Include "/tmp/collectd-footer-include.conf" # Or the value of collectd_plugin.footer_include_output_file

It is important that the Include of the memfaultd-generated files are the first and last statements in your collectd configuration. This ensures each HTTP request that collectd's write_http plugin performs to the Memfault backend contains the full set of configured metrics and is sent out at the correct interval.

If you inspect the generated files, you'll find that they configure the write_http plugin to talk to Memfault, and also set an Interval and a FlushInterval.

note

You must not add a LoadPlugin write_http in your own config (either before or after Includeing the generated config file). One such line is included in the generated configuration. Subsequent LoadPlugin lines for the same plugin will have no effect.

Relevant /etc/memfaultd.conf settings​

The generated collectd configuration can be controlled using the collectd section of /etc/memfaultd.conf. See a full configuration reference here.

collectd.interval_seconds​

The collectd.interval_seconds setting in /etc/memfaultd.conf controls both Interval and FlushInterval. This means that you should not configure either setting in your own configuration file.

In collectd, the Interval setting regulates how often read plugins will attempt a reading by default (plugin instances may override this for their own metrics), and the FlushInterval setting for the write_http plugin controls how often collectd will flush its cache and attempt to post metrics to Memfault.

note

The FlushInterval parameter to write_http is shared among all instances of the plugin. That means that if you have your own, it will flush at the Memfault-configured FlushInterval, too. This is a limitation of write_http.

collectd.write_http_buffer_size_kib​

The default value of 64KiB gives collectd plenty of space to fit a payload generated by our recommended configuration. A typical payload grows up to around 20KiB.

You may want to fine-tune write_http_buffer_size_kib if you modify the recommended configuration. Bear in mind the interaction between this and collectd.interval_seconds: if the write buffer is full, collectd will be forced to flush before an interval ends.

collectd.non_memfaultd_chain​

See Configuring additional chains.

See our recommended collectd configuration in meta-memfault-example. This configuration makes use of standard plugins that enjoy special support on the Memfault platform. Copying this configuration over to your project will guarantee a good first experience.

Note that as long as the generated configuration file /tmp/collectd.conf is included at the very bottom of your collectd configuration, you're free to add your own directives before it, and in addition to the recommended settings.

Configuring additional chains​

See collectd chains for an introduction.

Jumping to your custom chain after the memfaultd-generated chain​

By default, the memfaultd-generated chain will bottom out in a Target stop directive. You can change this final directive to instruct collectd to jump to your own chain. To do this, set the value of collectd.non_memfaultd_chain in /etc/memfaultd.conf and add a chain with a matching name. See a full configuration reference here.

On setting PostCacheChain​

The value of the PostCacheChain is set by the memfaultd-generated configuration. That means it's the first node of the PostCacheChain. If you change its value to something else, you need to make sure that your chains bottom out in a call to the memfaultd-generated chain. Please inspect /tmp/collectd.conf to see its identifier.

Set enable_data_collection​

By default, enable_data_collection is false (see the default configuration). This is to enable asking end users for consent before collecting or transmitting any data to Memfault services.

Once the end user has given their consent, you can enable data collection like so:

$ memfaultctl enable-data-collection

To disable it:

$ memfaultctl disable-data-collection

The memfaultd service will restart automatically whenever you run either of those commands if called with a value different from the current configuration.

Take a look at the /etc/memfaultd.conf reference for more information.

Application metrics​

First, make sure you've followed the prerequisites section and installed a StatsD client for your application.

Enable the statsd plugin in your /etc/collectd.conf (note that this is already included in the recommended configuration):

LoadPlugin statsd

The StatsD plugin exposes a UDP port (by default 8125). You'll need to configure your StatsD client to talk to it. Read on to see some examples.

Example: using C​

See the full module in our example layer.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <statsd-client.h>
#include <unistd.h>

#define MAX_LINE_LEN 200
#define PKT_LEN 1400

int main(int argc, char *argv[])
{
  statsd_link *link;

  link = statsd_init_with_namespace("localhost", 8125, "mycapp");

  char pkt[PKT_LEN] = {'\0'};
  char tmp[MAX_LINE_LEN] = {'\0'};

  statsd_prepare(link, "mygauge", 42, "g", 1.0, tmp, MAX_LINE_LEN, 1);
  strncat(pkt, tmp, PKT_LEN - 1);
  statsd_send(link, pkt);

  statsd_finalize(link);
}

Example: using Python​

See the full module in our example layer.

from statsd import StatsClient

statsd = StatsClient(
    host="localhost",
    port=8125,
    prefix="mypythonapp",
)

statsd.gauge("mygauge", 42)

Custom Device Attributes​

The memfaultctl command provides an easy way to set a device-specific attribute with the write-attributes command.

# memfaultctl write-attributes APP_VERSION=1.4.2 ACTIVATED=true

Refer to memfaultctl write-attributes documentation.

Testing your integration​

During the development phase, we recommend setting a low value (e.g. 60 seconds) for the collectd.interval_seconds setting in /etc/memfaultd.conf. Take a look at the /etc/memfaultd.conf reference for more information.

For changes in /etc/memfaultd.conf to take effect, you'll need to restart the memfaultd daemon:

$ systemctl restart memfaultd

The following section should help you figure out where you may expect data to be accessible in the Memfault Web Application.

Viewing Metrics in the Web Application​

To see detailed reports from a specific device, find the device in Fleet -> Devices, and then open its Timeline tab.

Open Dashboards -> Metrics to create Metric Charts that monitor metrics at the fleet level by aggregating the data from each device.

To receive notifications when your metrics exceed a certain threshold or meet any complex set of criteria, you can set up Alerts. Navigate to Alerts using the main menu on the Memfault Web App.