Issue Management

What is an Issue?

When devices in your fleet upload diagnostics data to Memfault, the data is analyzed by Memfault. Any detected problems are converted into Traces. Similar Traces are logically grouped into an Issue, based on the signature of the Trace.

In other words, a Trace is a single occurrence of a problem and an Issue is a group of Traces manifesting the same problem.

Grouping Traces into Issues

A Trace's signature is a SHA256 hash of various pieces of information generated by the device at the time the Trace was captured. These pieces of information include, but are not limited to, the error code, active thread, the backtrace of the active thread, the software type running, and other attributes provided at the time of the Trace.

Memfault's Trace processing is constantly being improved and updated over time. After an update to our trace signature algorithms, new Issues might be created that seem similar to old ones but now appear as multiple Issues. In these instances, you can make the decision to merge the issues so that future Traces will be grouped into only a single parent Issue.

A similar occurrence can take place in a customer's code base. Let's take the example below, which shows two backtraces, A and B.

Backtrace A

2  write_something
1  do_something
0  prv_main

Backtrace B

3  write_something
2  do something
1  prv_update_context  <-- New stack frame added in new release
0  prv_main

If a backtrace for a Trace changed from A to B between two releases, two Issues would likely be created. This is because the backtraces changed, and Memfault cannot be 100% sure they are the same bug. The best practice is to merge the Issues in this scenario.

Missing Symbols

On MCU and Linux, Traces can be created from coredumps even with some or all symbol files missing, using a less granular trace signature algorithm. Issues containing these Traces will be marked with a "Symbols Missing" indicator in the Issue list, with further details about the missing symbols available in the Trace details page. After symbol files are uploaded, reprocessing will result in a new Trace and potentially a new Issue. Because limited stacktrace information is available when processing without symbol files, Traces grouped into a "Symbols Missing" Issue can represent different underlying problems. The best practice is to resolve these Issues after uploading symbol files. We strongly recommend against merging them with other Issues.

Ignore Traces (Skip grouping Traces into Issues)

In some cases, you may find it useful to ignore Traces from certain devices. To do this, you can enable the "Ignore Traces from Devices" setting for a given Cohort on the Cohort settings tab.

When enabled, new Traces from Devices in this Cohort will not be grouped into any Issues. However, Traces will still be recorded, and will be visible with limited details in the Timeline and Traces tabs on the Device detail page.

Similarly, if you notice Issues and Traces data do not match for a given device, this setting may be enabled for its Cohort.

Issue States

There are four states an Issue can be in: Unresolved, Resolved, Merged, and Muted.

All Issues are Unresolved by default when created.

A Resolved Issue is intended for Issues that are believed to be fixed. Memfault will automatically alert you of a regression if this Issue is seen on a later software version. See more below.

A Merged Issue has been merged into another Issue (the "parent"), and its Traces have been moved to the parent Issue. The Merged Issue's detail page provides very limited information and displays a link to the parent Issue's detail page, where more information is consolidated. An Issue's "child" Issues can be viewed in the Merged tab of the Issue detail page. Any new Traces that would have been grouped under a Merged Issue will instead be grouped under its parent Issue.

A Muted Issue is not intended to be fixed or represents known and accepted behavior. Once an Issue is Muted, it will not reopen to Unresolved unless the Issue State is manually changed.

Once an issue has been Resolved or Muted, it will no longer appear in the default Issue list filter named Unresolved. To see a list of unresolved and muted Issues, toggle the filter to All. If reopened either manually, or automatically in the case of a regression (see details below), the issue will again appear under Unresolved Issues in the Issue list.

Memfault will continue to collect Traces of Resolved and Muted Issues. These traces will continue to populate Issue Charts in the Metrics Dashboard. Traces will also be available on the Device page under the Traces tab and on the Device Timeline.

Manually changing Issue States

The state of an Issue can be viewed and changed manually from the Issue’s detail page. You can navigate to an Issue detail page by clicking on an Issue on the Issue page.

Below the title of the Issue, you will find a button to change the state to either Resolved or Muted. The Issue can be Reopened at any time by clicking Reopen. Note the Reopen button will have a blue outline if the Issue has been resolved, and a red outline if the issue has been muted.

Bulk Issue Merge

In addition to the Merge button on the Issue detail page, Issues can also be merged in bulk from the Issue list page. After adding a filter for software type, the Bulk Actions dropdown will be enabled, with a preview count of the number of selected Issues. From there, you can select the Bulk Merge action to merge the selected Issues together. The oldest selected Issue will become the parent of all the others.

Issues can be selected for Bulk Merge in multiple ways:

• Execute an Issue search using the filter panel on the Issue list page
• Select individual Issues using the checkboxes in the Issue list table

Note: a software type filter MUST be applied in order to use Bulk Merge. Bulk actions across different software types are currently not supported.

Automatic Issue reopening upon regression

When Memfault receives a Trace from a greater Software Version compared to the last Software Version when the Issue was Resolved, it will automatically be changed to Unresolved, and subscribed users will automatically be notified.

You can also "Mute" an issue if you never want it to reopen and keep it Resolved indefinitely or until you manually reopen it again.

Example

Issue A is seen on Software Versions 1.0, 1.1, and 1.2. You close Issue A when 1.2 was the greatest Software Version it was seen on. In the future, when a new Trace is received from Software Version 1.3 for the same Issue, Memfault automatically reopens it and changes the Issue to Unresolved.

Monitoring Issue Occurrence

Memfault offers different ways to monitor the occurrence of Issues over time.

To visualize Issue occurrence at the fleet level, you can create Issue Charts. Issue Charts group together related Issues and allow you to visualize their occurrence over time.

Additionally, you can visualize the occurrence of an Issue over different Software Versions, Hardware Versions, Cohorts, and current Device Attribute values using the distribution visualization charts present in the Issue details view:

Correlating Issue Occurrence with Device Attributes

The Current Device Attribute distribution feature enables you to analyze how different Device Attribute values correlate with an Issue across your fleet. This analysis is performed in real-time, reflecting the current state of Device Attributes when you load the Issue page.

There are two ways to use it:

• Open an Issue and click on Add attributes to quickly add a new Device Attribute to the distribution visualization
• To permanently add a Device Attribute to every Issue page, go to Settings → Metrics, select a Custom Metric, and enable "Display on Issue details page"

Since the data reflects the current state of Device Attributes, we recommend picking Device Attributes whose value does not change over the lifetime of a Device, and ideally Device Attributes with low cardinality, so that more meaning can be drawn from the distribution of their values.

Automated Hiding of Issues with Zero Traces

As traces are deleted after 60 days, Memfault will hide the related Issues automatically. This helps declutter your dashboards so that only relevant Issues are displayed.

To view Issues that have been hidden due to their traces being expired, you can click "All" in the filter panel of the Issues view.

Issue Notifications

Issues will trigger email notifications to users on the following Issue events:

• Creation
• Comment added
• Modified
• Re-open

These emails can be disabled from the User Settings panel: