Process Dead Letter Queue

Dashboards

• Dead Letter Queue Dashboard

Handshake Admin

You may quickly process the DLQ by going to the admin page here

Infrastructure

If there are thousands of messages in the DLQ, the quickest way to process them is to bring up a separate inbox/worker with the DLQ subscription.

Batch Requests Worker

1. Get a pod name from the GCP console
2. Log onto the pod using seira

seira production notifications pods connect --pod=<pod-name

example:
seira production notifications pods connect --pod=notification-batch-requests-worker-7b9f5dc5bb-hg72n

3. Start the application in dlq mode

Batch Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-batch-requests-worker dep dlq p -t notifications-notification-batch-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

Low Priority Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-lp-requests-worker dep dlq p -t notifications-notification-low-priority-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

HAI Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-hai-requests-worker dep dlq p -t notifications-notification-hai-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

Offline Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-offline-requests-worker dep dlq p -t notifications-notification-offline-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

DEP Monolith Inboxes

For things like the firebase and mailgun inboxes, you can follow log onto a pod:

$ seira production handshake pods connect --dedicated
$ gems/next_inbox/bin/next_inbox_consumer --subscription notifications-mailgun-low-priority-requested-notifications-dead-letter --handler Next::Notifications::Handlers --module Next::Notifications --client_max_messages "50" --no-ledger

you can "scale" this if needed by running on multiple tabs

Scaled DLQ Drain Deployment (DEP Monolith)

When a dead-letter queue has a large backlog, connecting to individual pods and running consumers by hand (the section above) doesn't scale. Instead, launch a short-lived Kubernetes Deployment with N replicas that all consume from the dead-letter subscription in parallel, then delete it when the queue is drained.

For the Mailgun requested handlers you can also run this in DLQ mode (NOTIFICATIONS_MAILGUN_DLQ_MODE=true, added in handshake#102584). In DLQ mode, any delivery error that would normally retry (and get re-driven back into the DLQ forever — e.g. Mailgun's "options limited to 16 kB" permanent failure) is instead acked: it notifies Bugsnag, records failed_at on the notification, and drops the message. Leave the flag unset to preserve normal retry behavior.

⚠️ DLQ mode drops permanently-failing messages (with a Bugsnag record). Only enable it when you intend to clear un-sendable messages, not reprocess them.

1. Get Teleport access (prod-workload-admin)

Creating/editing workloads in the production cluster with raw kubectl is a mutating API call, which requires the prod-workload-admin Teleport role. (Seira requests roles for you automatically, but raw kubectl does not, so you must request it yourself.) Access uses a +1 system — a colleague must approve; you cannot approve your own request. See Teleport Access Requests.

# Request the role (optionally @-mention a reviewer to ping them in Slack)
tsh request create --roles prod-workload-admin --reviewers <colleague>@joinhandshake.com

# After approval, find your approved request id
tsh request ls --my-requests

# Assume the role by re-logging in with the request id
tsh login --request-id=<request_id>

Make sure kubectl is pointed at the production cluster (e.g. via tsh kube ls + tsh kube login <prod-cluster>, or by running any seira production handshake ... command first to set the context).

2. Create the drain Deployment manifest

Save the following as dlq-drain.yaml. Adjust the marked values:

• replicas — how many parallel consumers (each requests 2 CPU / 2Gi; e.g. 20 replicas ≈ 40 CPU / 40Gi requested — watch for Pending pods).
• --subscription — the dead-letter subscription to drain.
• image — the tag currently running in prod (see command below the manifest).
• NOTIFICATIONS_MAILGUN_DLQ_MODE — keep "true" to drop un-sendable Mailgun messages; remove the env entry to just reprocess/retry.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mailgun-lp-dead-letter-drain
  namespace: handshake
  labels:
    team: network
    squad: notifications
    app: handshake
    tier: notifications-mailgun-lp-requested-notifications-inbox
    app.kubernetes.io/name: mailgun-lp-dead-letter-drain
    app.kubernetes.io/component: domain-events
    app.kubernetes.io/part-of: notifications
spec:
  replicas: 20 # <-- number of parallel consumers
  selector:
    matchLabels:
      app: handshake
      app.kubernetes.io/name: mailgun-lp-dead-letter-drain
  template:
    metadata:
      annotations:
        owner: <your-username>
      labels:
        # REQUIRED in prod: triggers the kube-secrets-init webhook to inject secrets
        kube-secrets-init.doit-intl.com/mutate: "true"
        team: network
        squad: notifications
        app: handshake
        tier: notifications-mailgun-lp-requested-notifications-inbox
        app.kubernetes.io/name: mailgun-lp-dead-letter-drain
        app.kubernetes.io/component: domain-events
        app.kubernetes.io/part-of: notifications
    spec:
      # REQUIRED: workload-identity SA with access to the pubsub subscription
      serviceAccountName: notifications
      securityContext:
        runAsNonRoot: true
      containers:
        - name: handshake
          # <-- current prod image tag (see command below)
          image: "us-central1-docker.pkg.dev/handshake-artifacts/handshake/handshake:REPLACE_WITH_CURRENT_TAG"
          command:
            - gems/next_inbox/bin/next_inbox_consumer
            - --subscription
            # <-- the dead-letter subscription to drain
            - notifications-mailgun-low-priority-requested-notifications-dead-letter
            - --handler
            - Next::Notifications::Handlers
            - --module
            - Next::Notifications
            - --client_max_messages
            - "50"
            - --no-ledger
          imagePullPolicy: IfNotPresent
          env:
            - name: EXIT_EARLY
              value: "true"
            # DLQ drain mode: ack + Bugsnag-notify permanently-failing messages
            # instead of retrying them back into the dead-letter queue.
            # Remove this entry to reprocess/retry instead of dropping.
            - name: NOTIFICATIONS_MAILGUN_DLQ_MODE
              value: "true"
            - name: DB_POOL
              value: "8"
            - name: NOTIFICATIONS_MAILGUN_LOW_PRIORITY_REQUESTED_NOTIFICATIONS_INBOX_CALLBACK_THREADS
              value: "4"
            - name: DOGSTATSD_HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: DD_TRACE_AGENT_URL
              value: "unix:///var/run/datadog/apm.socket"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          envFrom:
            - configMapRef:
                name: handshake-env-config-ksi
            - configMapRef:
                name: handshake-toggle-configs
            - configMapRef:
                name: handshake-domain-events-platform-config
          resources:
            requests:
              cpu: 2000m
              memory: 2Gi
            limits:
              cpu: 3000m
              memory: 4Gi
          securityContext:
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: apmsocketpath
              mountPath: /var/run/datadog
      volumes:
        - name: apmsocketpath
          hostPath:
            path: /var/run/datadog/

Get the current prod image tag to fill into image: (mirrors the live inbox deployment so the drain runs the same code that's deployed):

kubectl get deploy notifications-mailgun-lp-requested-notifications-inbox -n handshake \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

The manifest above is modeled on the notifications-mailgun-lp-requested-notifications-inbox deployment. To drain a different DLQ, base it on that consumer's deployment instead (copy its command, env, configmaps, and service account) and change the --subscription to the matching *-dead-letter subscription.

3. Launch and scale

# Create the deployment
kubectl create -f dlq-drain.yaml

# Scale up/down at any time without editing the file
kubectl scale deploy mailgun-lp-dead-letter-drain -n handshake --replicas=20

# Watch the pods come up (look out for Pending = not enough cluster capacity)
kubectl get pods -n handshake -l app.kubernetes.io/name=mailgun-lp-dead-letter-drain -w

# Tail logs across all replicas
kubectl logs -f -n handshake -l app.kubernetes.io/name=mailgun-lp-dead-letter-drain --max-log-requests=20

Monitor progress on the Dead Letter Queue Dashboard and via the subscription's num_undelivered_messages metric. In DLQ mode, drops increment the notifications.mailgun.requested.dlq_mode_ack metric and appear in Bugsnag.

4. Stop / clean up when done

Once the DLQ backlog reaches zero, delete the deployment (this also removes its ReplicaSet and all pods):

kubectl delete deploy mailgun-lp-dead-letter-drain -n handshake

# Verify everything is gone
kubectl get deploy,pods -n handshake -l app.kubernetes.io/name=mailgun-lp-dead-letter-drain

There is no KEDA/HPA on this deployment, so it stays at the replica count you set until you delete it — don't leave it running after the queue is drained.