Dead Letter Queue Analyzer

---

name: dead-letter-queue-analyzer description: Analyze dead letter queue (DLQ) messages to identify failure patterns, root causes, and remediation strategies. Supports AWS SQS, RabbitMQ, Kafka, Azure Service Bus, and generic message queues.

---

Dead Letter Queue Analyzer

Stop ignoring your dead letter queue. Analyze DLQ messages to find failure patterns, identify root causes, determine which messages are replayable, and generate remediation plans — turning your DLQ from a black hole into an actionable error stream.

Use when: "analyze DLQ", "dead letter queue growing", "why are messages failing", "replay failed messages", "DLQ backlog", "message processing failures", or when unprocessed messages accumulate.

Commands

1. analyze — Categorize DLQ Messages

#### Step 1: Read DLQ Messages

AWS SQS:

aws sqs receive-message \
  --queue-url "$DLQ_URL" \
  --max-number-of-messages 10 \
  --attribute-names All \
  --message-attribute-names All | python3 -c "
import json, sys
msgs = json.load(sys.stdin).get('Messages', [])
for m in msgs:
    body = json.loads(m['Body']) if m['Body'].startswith('{') else m['Body']
    attrs = m.get('Attributes', {})
    print(f'ID: {m[\"MessageId\"]}')
    print(f'  Received count: {attrs.get(\"ApproximateReceiveCount\", \"?\")}')
    print(f'  First received: {attrs.get(\"ApproximateFirstReceiveTimestamp\", \"?\")}')
    print(f'  Body preview: {str(body)[:200]}')
    print()
"
Count total DLQ depth
aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages | python3 -c "
import json, sys
attrs = json.load(sys.stdin)['Attributes']
print(f'DLQ depth: {attrs[\"ApproximateNumberOfMessages\"]} messages')
"

RabbitMQ:

# List DLQ queues
rabbitmqctl list_queues name messages | grep -i "dead\|dlq\|error"
Peek at messages
rabbitmqadmin get queue="dead_letter_queue" count=10 2>/dev/null

Kafka:

# Read from DLT (dead letter topic)
kafka-console-consumer --bootstrap-server $KAFKA_BROKER \
  --topic "$DLT_TOPIC" --from-beginning --max-messages 20 \
  --property print.headers=true --property print.timestamp=true

#### Step 2: Classify Failure Causes

Group DLQ messages by failure reason:

| Category | Signal | Replayable? | Action | |----------|--------|------------|--------| | Schema error | Validation failure, missing field | After fix | Fix producer or consumer schema | | Timeout | Processing exceeded deadline | Yes | Increase timeout or optimize processing | | Dependency down | Connection refused, 503 | Yes | Wait for recovery, then replay | | Poison message | Crash/exception on processing | No | Fix handler, then replay | | Data integrity | FK violation, duplicate key | Maybe | Fix data, then replay | | Permission | Auth error, access denied | After fix | Fix credentials, then replay | | Deserialization | Invalid JSON/Protobuf/Avro | No | Discard or fix producer |

# Group messages by error pattern
from collections import Counter
errors = Counter()
for msg in dlq_messages:
    # Extract error reason from message attributes or headers
    error = msg.get('error_reason', msg.get('x-death-reason', 'unknown'))
    errors[error] += 1
for error, count in errors.most_common(10):
    print(f'{count:>5}x  {error}')

#### Step 3: Generate Report

# DLQ Analysis Report
Summary
Queue: orders-processing-dlq
Total messages: 1,247
Oldest message: 3 days ago
Growth rate: ~400/day (increasing)
Failure Categories
| Category | Count | % | Replayable | Root Cause |
|----------|-------|---|------------|------------|
| Timeout | 823 | 66% | ✅ | DB slow queries since Tuesday deploy |
| Schema error | 312 | 25% | ✅ (after fix) | New field currency not in consumer schema |
| Poison message | 67 | 5% | ❌ | NullPointer in price calculation |
| Permission | 45 | 4% | ✅ (after fix) | Expired service account token |
Root Cause
Primary: DB slow queries causing processing timeouts (66% of failures)
Started: Tuesday 14:30 UTC (correlates with deploy)
Impact: 823 orders stuck in DLQ
Remediation Plan
1. Fix DB performance — add missing index on orders.status (immediate)
2. Replay timeout messages (823) — safe, operations are idempotent
3. Update consumer schema to accept currency field (312 messages)
4. Rotate service account token (45 messages)
5. Fix NullPointer in OrderPriceCalculator.java:67 (67 messages — investigate first)
6. Set up DLQ depth alerting (threshold: 50 messages)

2. replay — Generate Replay Script

# SQS: move messages from DLQ back to main queue
aws sqs start-message-move-task \
  --source-arn "$DLQ_ARN" \
  --destination-arn "$MAIN_QUEUE_ARN" \
  --max-number-of-messages-per-second 10
Or selective replay (only timeout errors)
Read, filter, re-send

3. monitor — Set Up DLQ Alerting

Generate CloudWatch alarm / Prometheus alert for DLQ depth:

Alert when DLQ depth > 0 (any message is a signal)

Alert when growth rate > N/hour (active problem)

Alert when oldest message > 24h (messages going stale)

Dashboard showing DLQ depth over time + categorization

4. prevent — Improve Message Handling

Recommend changes to prevent future DLQ accumulation:

Add retry with backoff before sending to DLQ

Add idempotency keys for safe replay

Add dead letter reason headers for faster triage

Add message TTL to prevent infinite accumulation

Add schema validation before publishing (catch at source)