Issue Management

What is an Issue?

When devices in your fleet upload diagnostics data to Memfault, the data is analyzed by Memfault. Any detected problems are converted into Traces. Similar Traces are logically grouped into an Issue, based on the Signature of the Trace.

In other words, a Trace is a single occurrence of a problem and an Issue is a group of Traces manifesting the same problem.

Grouping Traces into Issues

A Trace's Signature is a SHA256 hash of various pieces of information generated by the device at the time the Trace was captured. These pieces of information include, but are not limited to, the error code, active thread, the backtrace of the active thread, the Software Type running, and other attributes provided at the time of the Trace.

Memfault's Trace processing is constantly being improved and updated over time. After an update to our trace Signature algorithms, new Issues might be created that seem similar to old ones but now appear as multiple Issues. In these instances, you can make the decision to merge the issues so that future Traces will be grouped into only a single parent Issue.

A similar occurrence can take place in a customer's code base. Let's take the example below, which shows two backtraces, A and B.

Backtrace A

2  write_something
1  do_something
0  prv_main

Backtrace B

3  write_something
2  do something
1  prv_update_context  <-- New stack frame added in new release
0  prv_main

If a backtrace for a Trace changed from A to B between two releases, two Issues would likely be created. This is because the backtraces changed, and Memfault cannot be 100% sure they are the same bug. The best practice is to merge the Issues in this scenario.

Missing Symbols

On MCU and Linux, Traces can be created from coredumps even with some or all symbol files missing, using a less granular trace Signature algorithm. Issues containing these Traces will be marked with a "Symbols Missing" indicator in the Issue list, with further details about the missing symbols available in the Trace details page. After symbol files are uploaded, reprocessing will result in a new Trace and potentially a new Issue. Because limited stacktrace information is available when processing without symbol files, Traces grouped into a "Symbols Missing" Issue can represent different underlying problems. The best practice is to resolve these Issues after uploading symbol files. We strongly recommend against merging them with other Issues.

Ignore Traces (Skip grouping Traces into Issues)

In some cases, you may find it useful to ignore Traces from certain devices. To do this, you can enable the "Ignore Traces from Devices" setting for a given Cohort on the Cohort settings tab.

When enabled, new Traces from Devices in this Cohort will not be grouped into any Issues. However, Traces will still be recorded, and will be visible with limited details in the Timeline and Traces tabs on the Device detail page.

Similarly, if you notice Issues and Traces data do not match for a given device, this setting may be enabled for its Cohort.

Issue Search

The Issue list has powerful filters that you can use to quickly find the Issues you are looking for.

Each type of filter is explained below in more detail.

Combining Multiple Filters

When multiple filters are applied at the same time, they are combined with an AND operator. Some filters support entering multiple values, in which case an OR operator is used within that filter. This works the same way as combining filters in Device Search.

For filters that support excluding values (a "not" operator), only a single value can be excluded per row. To exclude more than one value, add another filter row of the same kind.

Searching Issues by Title

Matches Issues whose title contains the given text (case-insensitive substring match). Only a single value is supported per row; add another Title filter row to match on additional text.

Searching Issues by Issue ID

One or more Issues can be selected by searching for their title or numeric ID.

Searching Issues by Software Type

Select one or more Software Types to narrow the Issue list to. The dropdown lists every Software Type reporting to your project, along with a count of active devices for each, to help you distinguish similarly-named types.

A Software Type filter is required to use Bulk Issue Merge.

Searching Issues by Software Version

The Software Version filter matches Issues with at least one Trace captured on the given Software Versions.

Comparison operators: Type a comparison operator followed by a version string to match versions using natsort ordering (e.g. >1.0 matches 1.0.1, 2.0, etc.). Semver-aware comparison is also available when entering a comparison operator followed by a valid semver version (e.g. >1.0.1). This only matches valid semver versions.

Regex: Type a regular expression pattern to match versions using regex (e.g. ^1\.1\.[0-9]* to match all 1.1.x semver versions).

Multiple version expressions within the same filter row are OR'd together. This works the same way as Device Search's Current Software Version filter.

Searching Issues by Introduced In

Finds Issues that were first seen on the given Software Versions, i.e. the earliest Software Version a matching Trace was captured on. Supports the same comparison-operator, semver, and regex matching described under Searching Issues by Software Version.

Searching Issues by Latest Affected

Finds Issues that were most recently seen on the given Software Versions, i.e. the latest Software Version a matching Trace was captured on. Supports the same comparison-operator, semver, and regex matching described under Searching Issues by Software Version.

Searching Issues by Hardware Version

One or more Hardware Versions can be used in this field. Values are pulled live from your project.

Searching Issues by Cohort

One or more Cohorts can be used in this field, matching Issues with at least one Trace from a device currently in the selected Cohorts.

Searching Issues by Reason

This filter finds Issues by the crash/error reason recorded on their Traces (for example Hard Fault, Assert, or SIGSEGV, depending on platform). Values are pulled live from your project; on Android projects, options are grouped into "User-Visible Crashes" and "Other Errors".

Searching Issues by Tags

One or more Issue Tags can be used in this field. Values are pulled live from your project.

Searching Issues by Devices

Rather than a simple dropdown, this filter embeds the full Device Search filter builder, letting you filter Issues down to a specific, arbitrarily complex population of devices with at least one matching Trace.

Searching Issues by First/Last Seen

These filters can be used to find Issues that were first or last seen within a certain time frame.

Searching Issues by Trace Captured Date

Matches Issues having Traces captured between the specified date range.

note

Your account retains Trace Data for a limited number of days. If the selected date range exceeds this limit, only Traces within the retention window will be loaded and displayed. Contact us if you need to increase this limit.

This is also the filter required to enable creating an Issue Chart directly from the Issue list page.

Searching Issues by Stacktrace

In addition to filtering by Software Type, Software Version, and other metadata, you can filter Issues by the symbolicated function names in their backtrace using the Stacktrace filter, available from the "Add Filter" menu on the Issue search page.

Because this searches over the same signature data used to group Traces into Issues (rather than a full-text search of every thread and frame), there are a couple of caveats:

• Only symbolicated functions in the crashing thread are searched (on Android, the primary thread)
• Very long threads may be truncated

A missing or blank value in the columns indicates:

• The Issue's stacktrace could not be symbolicated (missing symbol files); open the Issue to view details
• The Issue's stacktrace could not be processed; open the Issue to view details

Jumping from a Trace to Issue search

From the thread/backtrace viewer on a Trace's detail page, hovering over a stack frame reveals a magnifying glass icon ("Search issues by this function"). Clicking it opens Issue search with a Stacktrace filter pre-applied for that function. This is disabled for address-only frames, since those can't be searched by function name.

Issue States

There are four states an Issue can be in: Unresolved, Resolved, Merged, and Muted.

All Issues are Unresolved by default when created.

A Resolved Issue is intended for Issues that are believed to be fixed. Memfault will automatically alert you of a regression if this Issue is seen on a later software version. See more below.

A Merged Issue has been merged into another Issue (the "parent"), and its Traces have been moved to the parent Issue. The Merged Issue's detail page provides very limited information and displays a link to the parent Issue's detail page, where more information is consolidated. An Issue's "child" Issues can be viewed in the Merged tab of the Issue detail page. Any new Traces that would have been grouped under a Merged Issue will instead be grouped under its parent Issue.

A Muted Issue is not intended to be fixed or represents known and accepted behavior. Once an Issue is Muted, it will not reopen to Unresolved unless the Issue State is manually changed.

Once an issue has been Resolved or Muted, it will no longer appear in the default Issue list filter named Unresolved. To see a list of unresolved and muted Issues, toggle the filter to All. If reopened either manually, or automatically in the case of a regression (see details below), the issue will again appear under Unresolved Issues in the Issue list.

Memfault will continue to collect Traces of Resolved and Muted Issues. These traces will continue to populate Issue Charts in the Metrics Dashboard. Traces will also be available on the Device page under the Traces tab and on the Device Timeline.

Manually changing Issue States

The state of an Issue can be viewed and changed manually from the Issue's detail page. You can navigate to an Issue detail page by clicking on an Issue on the Issue page.

Below the title of the Issue, you will find a button to change the state to either Resolved or Muted. The Issue can be Reopened at any time by clicking Reopen. Note the Reopen button will have a blue outline if the Issue has been resolved, and a red outline if the issue has been muted.

Bulk Issue Merge

In addition to the Merge button on the Issue detail page, Issues can also be merged in bulk from the Issue list page. After adding a filter for software type, the Bulk Actions dropdown will be enabled, with a preview count of the number of selected Issues. From there, you can select the Bulk Merge action to merge the selected Issues together. The oldest selected Issue will become the parent of all the others.

Issues can be selected for Bulk Merge in multiple ways:

• Execute an Issue search using the filter panel on the Issue list page
• Select individual Issues using the checkboxes in the Issue list table

Note: a Software Type filter MUST be applied in order to use Bulk Merge. Bulk actions across different Software Types are currently not supported.

Automatic Issue reopening upon regression

When Memfault receives a Trace from a greater Software Version compared to the last Software Version when the Issue was Resolved, it will automatically be changed to Unresolved, and subscribed users will automatically be notified.

You can also "Mute" an issue if you never want it to reopen and keep it Resolved indefinitely or until you manually reopen it again.

Example

Issue A is seen on Software Versions 1.0, 1.1, and 1.2. You close Issue A when 1.2 was the greatest Software Version it was seen on. In the future, when a new Trace is received from Software Version 1.3 for the same Issue, Memfault automatically reopens it and changes the Issue to Unresolved.

Monitoring Issue Occurrence

Memfault offers different ways to monitor the occurrence of Issues over time.

To visualize Issue occurrence at the fleet level, you can create Issue Charts. Issue Charts group together related Issues and allow you to visualize their occurrence over time.

Additionally, you can visualize the occurrence of an Issue over different Software Versions, Hardware Versions, Cohorts, and current Device Attribute values using the distribution visualization charts present in the Issue details view:

Correlating Issue Occurrence with Device Attributes

The Current Device Attribute distribution feature enables you to analyze how different Device Attribute values correlate with an Issue across your fleet. This analysis is performed in real-time, reflecting the current state of Device Attributes when you load the Issue page.

There are two ways to use it:

• Open an Issue and click on Add attributes to quickly add a new Device Attribute to the distribution visualization
• To permanently add a Device Attribute to every Issue page, go to Settings → Metrics, select a Metric, and enable "Display on Issue details page"

Since the data reflects the current state of Device Attributes, we recommend picking Device Attributes whose value does not change over the lifetime of a Device, and ideally Device Attributes with low cardinality, so that more meaning can be drawn from the distribution of their values.

Automated Hiding of Issues with Zero Traces

As traces are deleted after 60 days, Memfault will hide the related Issues automatically. This helps declutter your dashboards so that only relevant Issues are displayed.

To view Issues that have been hidden due to their traces being expired, you can click "All" in the filter panel of the Issues view.

Saved Issue Searches

After filtering the Issues list using the filter panel, you can save the current filters as a Saved Issue Search using the buttons in the filter panel:

• Save: creates a new Saved Issue Search
• Save As: saves the current filters as a separate, new Saved Issue Search (shown when you view a saved search)
• Delete: removes the Saved Issue Search you currently have open

Managing Saved Issue Searches

Saved searches can be renamed, deleted, or favorited from the All Saved Searches page, available in the expandable Issues section of the sidebar. Your favorited searches will also appear in this section.

Issue Notifications

Issues will trigger email notifications to users on the following Issue events:

• Creation
• Comment added
• Modified
• Re-open

These emails can be disabled from the User Settings panel: