Troubleshooting

Integration

Here are some steps to assist in diagnosing some possible integration failures.

System components missing during validation

If the validation is failing due to missing system components, verify if the components are being built by searching the build artifacts.

$ find $AOSP_ROOT/out/ -iname "MemfaultDumpstateRunner"
...
out/target/product/$YOUR_PRODUCT/system/bin/MemfaultDumpstateRunner
...

If the binary is not being built, then verify the Makefile configuration:

• Is the bort product.mk file included in your device.mk file?
  • Note that this include appends to the PRODUCT_PACKAGES variable: is this variable being redefined elsewhere via an assignment? (:=) after including the product.mk file?

If you have confirmed the binary is being built, the next step is to verify if the binary is inside the (system) image as expected. To do this, you can mount the system image and search inside it. You should see the following files included in the system image:

$ sudo find /path/to/volume/ -iname memfault\*
/path/to/volume/system/priv-app/MemfaultUsageReporter
/path/to/volume/system/priv-app/MemfaultUsageReporter/MemfaultUsageReporter.apk
...
/path/to/volume/system/priv-app/MemfaultBort
...
/path/to/volume/system/priv-app/MemfaultBort/MemfaultBort.apk
/path/to/volume/system/bin/MemfaultDumpstateRunner
/path/to/volume/system/bin/MemfaultDumpster
/path/to/volume/system/etc/init/memfault_dumpster.rc
/path/to/volume/system/etc/init/memfault_init.rc

If these files are not present, then there may be an issue with your build process, you may be inspecting the wrong system image, or the SDK may be included in a different image.

sepolicy not correctly configured

If Bort's SELinux policy is not fully applied, then you might see:

• Errors from the SDK validation tool (validate-sdk-integration)
• AOSP build errors e.g. ERROR: memfault_dumpster_service not found in sepolicy contexts or ERROR: bort_app_data_file not found in sepolicy contexts.

If SELinux policy is not setup correctly, this could be because:

• Is the bort BoardConfig.mk file being included in your BoardConfig.mk file?
• Verify that the BOARD_SEPOLICY_DIRS and BOARD_PLAT_PRIVATE_SEPOLICY_DIR variables are not being over-written. You can search for other BoardConfig.mk files in the build tree which might be over-writing Bort's configuration by running grep "_SEPOLICY_DIR" . -R -l -s :=. If any are found then:
  • Reference the Bort BoardConfig.mk from a file which is processesed later (after the other makefiles with := on the same variables).
  • Or, := (set) operations should be changed to += (append) on the variables which Bort sets in BoardConfig.mk.

Verify that the Bort sepolicy configuration is being correctly applied from BoardConfig.mk by running this in an AOSP build environment where build/envsetup.sh has been run. This can be done before running the AOSP build (to quickly iterate on fixing the issue). vendor/memfault/bort/sepolicy will be printed if sepolicy is configured correctly, and there will be no output if not configured correctly:

$ get_build_var BOARD_SEPOLICY_DIRS 2> /dev/null | grep vendor/memfault/bort/sepolicy

Bort OTA app is missing

If the Bort OTA app is not installed when expected, errors will be reported from validate-sdk-integration when passing the --bort-ota-app-id argument.

Ensure that the TARGET_USES_MFLT_OTA environment variable is set to 1 when running the AOSP build.

Missing data / Device not reporting

Here are some steps to help diagnose why a device may not be reporting data when expected.

Data does not appear in the Memfault application

If you encounter any issues in your data transfer implementation, Memfault has tools to help debug!

• To troubleshoot data not getting uploaded or processed correctly by the Memfault cloud, take a look at the Integration Hub → Processing Log view. This provides a filterable, chronological view of recent errors that have occurred while processing received data. See this documentation page for more information on the Integration Hub.
• Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.

Checklist

If data is not appearing in the processing log or on the device timeline, check these common causes:

• Is Bort enabled on the device? If RUNTIME_ENABLE_REQUIRED=true was configured when building Bort, then Bort will only run once it has been enabled. If not enabled, no data will be either collected or uploaded.
• Did the device check in to Memfault? If the "Last Seen" timestamp is updated on the device page, but no data was uploaded, then the Bort OTA app may have OTA check (this happens even when the SDK is disabled), or the Bort app may have checked for updated SDK settings (only happens when the SDK is enabled) - in which case check for any of the issues below.
• Is the "Allow Uploads on Metered Networks" setting enabled on the Data Sources page? If so, then no data will be uploaded when the device is connected to a metered network.
• Was the SDK validation tool run on this AOSP build, before it was released? This will rule out most potential integration problems, and should always be checked after every new build.
• Is the device reporting to the correct project? Check which project key was used when building Bort (or which key was configured at runtime, if using this function). Check any other projects in your organization, in case the device is reporting to a different one.

If the device is local and can be connected over ADB to debug further:

• Run the SDK validation tool on this device.
• Collect logcat logs using adb logcat or Android Studio, checking for any bort logs.
• Check whether Bort's scheduled jobs are running as expected, by running adb shell dumpsys jobscheduler and checking for Bort jobs in the output.
• Enable Dev Mode on the device, and trigger a manual metrics collection.

If nothing was abnormal in any of these steps, and the manual collection did not upload anything, then try clearing Bort's data, then restart Bort (using your own Bort application ID in place of your.app.id):

adb shell pm clear your.app.id
./bort_cli.py enable-bort --bort-app-id your.app.id

Then optionally Enable Dev Mode on the device, and trigger a manual metrics collection (else wait up to two hours for data to be uploaded).

After checking everything above, contact us including details such as the device identifier, all debugging steps attempted, and logcat/dumpsys outputs collected as described above.

Bug report is generated but fails to upload

If the SDK is enabled, a bug report is generated, but the logcat logs show that the upload has failed, then capture and upload a bug report manually over ADB so that we can assist with a detailed investigation:

$ adb bugreport upload-failure.zip

Then navigate to Issues in Memfault and upload the bug report via the Manual Upload button in the top right hand side.

Bug report contains fewer logs than expected

If you are viewing logs from a bug report and the timespan covered by the logcat logs or device timeline is less than expected, this may be because:

1. The device recently rebooted and its non-persistent data was wiped.
2. The system is logging a lot (being very "chatty") and is quickly filling up the buffer, causing older logs to be quickly evicted.

Logcat logs are collected by dumpstate with the command:

logcat -v threadtime -v printable -v uid -d *:v

This means that the bug report will collect whatever logs are available in the system buffer. Making the logs less chatty or increasing the size of the log buffer are two possible solutions.

Making the logs less chatty is likely to be the more effective solution. However, you can quickly evaluate effect of changing the size of the log buffer over adb using logcat -G (docs). Note that this setting is transient and will be lost when the system reboots. To change the log buffer size permanently, use the persist.logd.size system property; see also android_logger.h source.