Issue Management
What is an Issue?
When devices in your fleet upload diagnostics data to Memfault, the data is analyzed by Memfault. Any detected problems are converted into Traces. Similar Traces are logically grouped into an Issue, based on the signature of the Trace.
In other words, a Trace is a single occurrence of a problem and an Issue is a group of Traces manifesting the same problem.
De-Duplication of Traces into Issues
A Trace's signature is a SHA256 hash of various pieces of information generated by the device at the time the Trace was captured. These pieces of information include, but are not limited to, the error code, active thread, the backtrace of the active thread, the software type running, and other attributes provided at the time of the Trace.
The process in which Memfault groups Traces into Issues is constantly being improved over time. During such updates to our algorithms, new Issues might be created that seem similar to old ones but now appear as multiple Issues. In these instances, it's best to merge the issues so that all future Traces will be grouped into a single parent Issue.
A similar occurrence can take place in a customer's code base. Let's take the example below, which shows two backtraces, A and B.
Backtrace A
2 write_something
1 do_something
0 prv_main
Backtrace B
3 write_something
2 do something
1 prv_update_context <-- New stack frame added in new release
0 prv_main
If a backtrace for a Trace changed from A to B between two releases, two Issues would likely be created. This is because the backtraces changed, and Memfault can not be 100% sure they are the same bug. The best practice is to merge the Issues in this scenario.
Issue States
There are four states an Issue can be in: Unresolved, Resolved, Merged, and Muted.
All Issues are Unresolved by default.
A Resolved Issue is intended for Issues that are believed to be fixed. Memfault will automatically alert you of a regression if this Issue is seen on a later software version. See more below.
A Muted Issue is intended for Issues that are not intended to be fixed or are known and accepted behavior. Once an Issue is Muted, it will not reopen to Unresolved unless the Issue State is manually changed.
Once an issue has been Resolved or Muted, it will no longer appear in the default Issue list filter named Unresolved. To see a list of unresolved and muted Issues, toggle the filter to All. If reopened either manually, or automatically in the case of a regression (see details below), the issue will again appear under Unresolved Issues in the Issue list.
Memfault will continue to collect Traces of Resolved and Muted Issues. These traces will continue to populate Issue Charts in the Metrics Dashboard. Traces will also be available on the Device page under the Traces tab and on the Device Timeline.
Manually changing Issue States
The state of an Issue can be viewed and changed manually from the Issue’s detail page. You can navigate to an Issue detail page by clicking on an Issue on the Issue page.
Below the title of the Issue, you will find a button to change the state to either Resolved or Muted. The Issue can be Reopened at any time by clicking Reopen. Note the Reopen button will have a blue outline if the Issue has been resolved, and a red outline if the issue has been muted.
Automatic Issue reopening upon regression
When Memfault receives a Trace from a greater Software Version compared to the last Software Version when the Issue was Resolved, it will automatically be changed to Unresolved, and subscribed users will automatically be notified.
You can also "Mute" an issue if you never want it to reopen and keep it Resolved indefinitely or until you manually reopen it again.
Issue A is seen on Software Versions 1.0, 1.1, and 1.2. You close Issue A when 1.2 was the greatest Software Version it was seen on. In the future, when a new Trace is received from Software Version 1.3 for the same Issue, Memfault automatically reopens it and changes the Issue to Unresolved.
Monitoring Issue Occurrence
Memfault offers different ways to monitor the occurrence of Issues over time.
To visualize Issue occurrence at the fleet level, you can create Issue Charts. Issue Charts group together related Issues and allow you to visualize their occurrence over time.
Additionally, you can visualize the occurrence of an Issue over different Software Versions, Hardware Versions and Cohorts using the distribution visualization charts present in the Issue details view:
Automated Hiding of Issues with zero Traces
As traces are deleted after 60 days, Memfault will hide the related Issues automatically. This helps declutter your dashboards so that only relevant Issues are displayed.
To view Issues that have been hidden due to their traces being expired, you can click "All" in the filter panel of the Issues view.