The Intermittent Embedded Failure Toolkit

Intermittent embedded failures are some of the most expensive problems an engineering team can face.

The system resets once every three weeks. A deadline is missed only after hours of operation. Adding logging makes the problem disappear. An I²C peripheral hangs once every few days. The product works perfectly whenever a debugger is attached. A communications task crashes because memory was corrupted somewhere else hours earlier.

These failures are difficult not because engineers lack debugging tools, but because the evidence often disappears before anyone knows what to measure.

The Intermittent Embedded Failure Toolkit presents a practical, systematic approach to investigating rare embedded-system failures that conventional debugging often misses.

Rather than providing a collection of debugging tips, the toolkit teaches a repeatable engineering thought process:

DETECT → PRESERVE → CLASSIFY → CORRELATE → STRESS → PROVE

Using realistic embedded-system failure scenarios, you'll learn how to:

• Preserve useful evidence across watchdog resets and other failures.
• Build low-overhead breadcrumb histories and retained diagnostic records.
• Design watchdog monitoring around meaningful system health rather than simple code execution.
• Track worst-case timing and resource margins before they become failures.
• Investigate problems that disappear when logging or a debugger is introduced.
• Correlate firmware behavior with electrical, timing, power, and peripheral activity.
• Diagnose partial-state startup and interrupted-power failures.
• Design robust non-volatile data updates that survive power interruption.
• Catch memory corruption closer to the offending write rather than debugging the eventual victim.
• Use GPIO, trace, RTT, scopes, logic analyzers, retained RAM, guards, MPU protection, and other observability techniques effectively.
• Accelerate rare failures without accidentally changing the mechanism being investigated.
• Separate recovery from root-cause diagnosis.
• Design observability into the next product before the first difficult field failure occurs.

Included with the toolkit

The package includes the complete Intermittent Embedded Failure Toolkit plus practical companion resources:

• Intermittent Failure Investigation Workbook
• Design-for-Observability Review Checklist
• Startup & Partial-State Test Matrix
• Benchside Failure Triage Quick Reference
• Intermittent Failure Evidence & Metrics Workbook

Who this is for

Embedded firmware engineers, technical leads, embedded systems engineers, verification engineers, and engineering managers responsible for products where failures cross firmware, RTOS, peripheral, timing, hardware, or power boundaries.

This isn't an introduction to embedded debugging. It is intended for engineers who already know how to use a debugger and need a more systematic way to attack the failures that don't cooperate with the debugger.

About the author

Drew Karnes is Founder and Principal Embedded Systems Engineer at BitMetrics, LLC. The toolkit draws on nearly four decades of embedded-system development and debugging experience across real-time firmware, hardware/firmware integration, industrial systems, medical devices, communications, and other embedded products.