NewMeet PingPage — status pages for your users

When a webhook delivery fails, what happens next matters more than you'd think. Retry too aggressively and you'll overwhelm a server that's already struggling. Retry too conservatively and your users wait hours for a critical notification. The right strategy depends on your use case.

Why Retries Matter

Webhook delivery fails more often than most people realize:

• Temporary network issues: DNS timeouts, connection resets, packet loss
• Server overload: 503 responses during traffic spikes
• Deployment windows: Brief downtime during deploys
• Rate limiting: Provider-side throttling (429 responses)
• Application errors: Bugs in webhook handlers (500 responses)

Without retries, a single transient failure means a permanently lost event. With the wrong retry strategy, you can make a bad situation worse.

The Three Strategies

1. Linear Retry (Fixed Interval)

Retry at a constant interval:

Code

Attempt 1:  0s
Attempt 2:  60s
Attempt 3:  120s
Attempt 4:  180s
Attempt 5:  240s

Total window: 4 minutes

Javascript

async function linearRetry(webhook, maxAttempts, interval) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const result = await deliver(webhook);

if (result.success) {
      return { delivered: true, attempts: attempt };
    }

if (attempt < maxAttempts) {
      await sleep(interval);
    }
  }

return { delivered: false, attempts: maxAttempts };
}

// Retry every 60 seconds, up to 5 attempts
await linearRetry(webhook, 5, 60000);

When to use linear retry:

• Internal services with predictable recovery times
• Simple use cases where you want predictable behavior
• Low-volume endpoints where retry storms aren't a concern

Drawbacks:

• Can overwhelm recovering servers with constant retry pressure
• No adaptation to the severity of the failure
• Wastes resources retrying at the same rate regardless of the problem

2. Exponential Backoff

Double the wait time between each retry:

Code

Attempt 1:  0s
Attempt 2:  30s
Attempt 3:  60s    (30 × 2)
Attempt 4:  120s   (60 × 2)
Attempt 5:  240s   (120 × 2)
Attempt 6:  480s   (240 × 2)

Total window: ~15 minutes

Javascript

async function exponentialBackoff(webhook, maxAttempts, baseDelay) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const result = await deliver(webhook);

if (result.success) {
      return { delivered: true, attempts: attempt };
    }

if (attempt < maxAttempts) {
      const delay = baseDelay * Math.pow(2, attempt - 1);
      await sleep(delay);
    }
  }

return { delivered: false, attempts: maxAttempts };
}

// Start at 30s, double each time, up to 6 attempts
await exponentialBackoff(webhook, 6, 30000);

When to use exponential backoff:

• External services that may need time to recover
• High-volume endpoints where retry storms are a real risk
• When you don't know the nature of the failure

With jitter (recommended for high-volume):

Javascript

function delayWithJitter(baseDelay, attempt) {
  const exponentialDelay = baseDelay * Math.pow(2, attempt - 1);
  const jitter = Math.random() * exponentialDelay * 0.5;
  return exponentialDelay + jitter;
}

Jitter prevents the "thundering herd" problem where thousands of failed webhooks all retry at exactly the same moment.

3. Custom Intervals

Define specific wait times for each retry attempt:

Code

Attempt 1:  0s
Attempt 2:  10s    (quick retry for transient issues)
Attempt 3:  60s    (maybe a brief deploy)
Attempt 4:  300s   (give it 5 minutes)
Attempt 5:  3600s  (something's really wrong, try in an hour)

Total window: ~66 minutes

Javascript

async function customRetry(webhook, intervals) {
  for (let attempt = 0; attempt < intervals.length; attempt++) {
    if (attempt > 0) {
      await sleep(intervals[attempt]);
    }

const result = await deliver(webhook);

if (result.success) {
      return { delivered: true, attempts: attempt + 1 };
    }
  }

return { delivered: false, attempts: intervals.length };
}

// Custom intervals in milliseconds
await customRetry(webhook, [
  0,       // Immediate first attempt
  10000,   // 10 seconds
  60000,   // 1 minute
  300000,  // 5 minutes
  3600000  // 1 hour
]);

When to use custom intervals:

• When you understand your failure patterns well
• Payment webhooks where the first retry should be fast
• Mixed failure scenarios (quick retries for glitches, longer for outages)

Choosing the Right Strategy

Decision Framework

Code

Is the destination an external service you don't control?
  ├── Yes → Exponential backoff with jitter
  └── No → Is recovery time predictable?
        ├── Yes → Linear retry
        └── No → Custom intervals

Configuring Retries in HookWatch

HookWatch lets you configure retry behavior per endpoint:

Bash

# Set retry count and interval for an endpoint
hookwatch endpoints create \
  --name "Payment Webhooks" \
  --destination "https://api.myapp.com/webhooks/payments" \
  --retry-count 5 \
  --retry-interval 60

From the dashboard, you can adjust retry settings at any time:

• Retry count: How many times to retry (1-10)
• Retry interval: Base interval between retries in seconds
• Per-endpoint configuration: Different endpoints can have different retry strategies

Monitoring Retry Behavior

Understanding your retry patterns helps you optimize:

Bash

hookwatch events list --status retrying

Code

ID            ENDPOINT          ATTEMPT    NEXT RETRY
evt_001       payment-webhook   2/5        in 45s
evt_002       order-processor   3/3        in 120s
evt_003       payment-webhook   1/5        in 58s

Key metrics to watch:

• First-attempt success rate: If this drops, something changed at the destination
• Average attempts to deliver: Should ideally be close to 1
• Retry-to-failure ratio: High ratio means your retry strategy is working; low ratio means events are failing faster than retries can help

Anti-Patterns

Retrying Non-Retryable Errors

Not all failures should be retried:

Javascript

// Don't retry these status codes
const nonRetryable = [400, 401, 403, 404, 405, 422];

if (nonRetryable.includes(response.status)) {
  // Log and alert, but don't retry
  return { delivered: false, reason: 'non-retryable' };
}

Retrying Without Idempotency

If your webhook handler isn't idempotent, retries can cause duplicate processing:

Javascript

// Bad: Creates duplicate orders on retry
app.post('/webhooks', async (req, res) => {
  await createOrder(req.body);  // Runs again on retry!
  res.json({ ok: true });
});

// Good: Idempotent with event ID check
app.post('/webhooks', async (req, res) => {
  const eventId = req.headers['x-webhook-id'];
  const existing = await db.events.findOne({ eventId });

if (existing) {
    return res.json({ ok: true, duplicate: true });
  }

await createOrder(req.body);
  await db.events.create({ eventId, processedAt: new Date() });
  res.json({ ok: true });
});

Unlimited Retries

Always set a maximum retry count. Retrying forever wastes resources and can mask underlying issues that need human attention.

Getting Started

HookWatch handles retries automatically for every endpoint. Configure the retry count and interval that matches your use case, and let the worker service handle delivery.

Start with the defaults (3 retries, 60-second interval) and adjust based on what you observe in your delivery metrics.