Emergency Pause for DEP Inbox Consumers

This runbook covers how to pause a DEP inbox consumer in a way that survives continuous deployment (CD-safe).

The pause mechanism lives in the shared next_inbox gem, so it applies to every DEP inbox consumer — Mailgun (email) and Firebase (push) alike — not just one channel.

When to use this

Use this when you need to stop an inbox consumer from pulling messages — for example, during an incident where downstream processing is failing, is overwhelmed, or is producing bad side effects, and you want messages to safely back up rather than be processed or lost.

Why the obvious options don't work

• Scaling pods to 0 — reverted by Argo/Helm on the next deploy.
• Detaching/purging the Pub/Sub subscription — loses messages.
• NOTIFICATIONS_MAILGUN_DLQ_MODE (and equivalents) — not a pause; it acks failures to nowhere.

How the pause works

An opt-in env var makes Next::Inbox::Consumer#start skip listener.start (the streaming pull) while keeping the health-check thread alive.

• The pod boots healthy but never pulls.
• Messages back up safely in the subscription — no loss, up to the 7-day retention window.
• It is config-driven, so it survives continuous deployment.

The pod does not crashloop: the process is not killed. start returns early (after the health-check thread is already running), the consumer binary's sleep keeps it alive, and /tmp/healthy keeps getting touched — so the pod stays Running/healthy and Kubernetes leaves it alone. It is simply idle and not pulling.

The env var

Each consumer reads a per-subscription pause variable. The name is the subscription name upcased with dashes replaced by underscores, suffixed with _INBOX_PAUSED:

<SUBSCRIPTION_UPCASED_WITH_UNDERSCORES>_INBOX_PAUSED = true

Set on the specific deployment, this pauses only that one consumer.

Notifications inbox consumers

Channel	Deployment (k8s)	Subscription	Pause env var
Email	notifications-mailgun-requested-notifications-inbox	notifications-mailgun-requested-notifications	NOTIFICATIONS_MAILGUN_REQUESTED_NOTIFICATIONS_INBOX_PAUSED=true
Email	notifications-mailgun-lp-requested-notifications-inbox	notifications-mailgun-low-priority-requested-notifications	NOTIFICATIONS_MAILGUN_LOW_PRIORITY_REQUESTED_NOTIFICATIONS_INBOX_PAUSED=true
Email	notifications-mailgun-hai-requested-notifications-inbox	notifications-mailgun-hai-requested-notifications	NOTIFICATIONS_MAILGUN_HAI_REQUESTED_NOTIFICATIONS_INBOX_PAUSED=true
Push	notifications-firebase-requested-notifications-inbox	notifications-firebase-requested-notifications	NOTIFICATIONS_FIREBASE_REQUESTED_NOTIFICATIONS_INBOX_PAUSED=true
Push	notifications-firebase-lp-requested-notifications-inbox	notifications-firebase-low-priority-requested-notifications	NOTIFICATIONS_FIREBASE_LOW_PRIORITY_REQUESTED_NOTIFICATIONS_INBOX_PAUSED=true
Push	notifications-firebase-app-badge-requested-notifications-inbox	notifications-firebase-app-badge-update-requested-notifications	NOTIFICATIONS_FIREBASE_APP_BADGE_UPDATE_REQUESTED_NOTIFICATIONS_INBOX_PAUSED=true

Pause a consumer

1. In the ops repo, open the deployment template for the consumer and environment you want to pause:

   helm/handshake/templates/<env>/<deployment>.yaml

   Environments are demo, staging, production (US prod), and eu-production (EU prod). Prod runs in both US and EU, so to fully pause a consumer in production you must set the var in both production/ and eu-production/.

2. Add the pause var to that deployment's env: block:

   env:
     - name: NOTIFICATIONS_MAILGUN_REQUESTED_NOTIFICATIONS_INBOX_PAUSED
       value: "true"

3. Merge and let it deploy (or trigger the Argo sync). The pause takes effect on the next deploy.

When paused, the consumer logs a warning and increments a metric:

• Log: [next_inbox] PAUSED — not starting listener for <subscription>
• Metric: DEP.inbox.paused

Unpause a consumer

Remove the env var (or set it to "false") and redeploy. The backed-up messages will drain once the listener starts pulling again.

Caveats

• Not instant. The pause (and unpause) takes effect on the next deploy, not immediately.
• Resume within 7 days. Messages are retained in the subscription for up to 7 days. Resume within that window or you will lose messages.
• Don't set the global INBOX_PAUSED. Setting the unprefixed INBOX_PAUSED in a shared config map pauses every inbox consumer. Always use the consumer-specific prefixed variable unless you truly intend to pause all of them.
• Set it per region. Each environment (and US vs EU prod) has its own deployment template; a var set in only one won't pause the other.