Process Dead Letter Queue

Dashboards

• Dead Letter Queue Dashboard

Batch Requests Worker

1. Get a pod name from the GCP console
2. Log onto the pod using seira

seira production notifications pods connect --pod=<pod-name

example:
seira production notifications pods connect --pod=notification-batch-requests-worker-7b9f5dc5bb-hg72n

3. Start the application in dlq mode

Batch Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-batch-requests-worker dep dlq p -t notifications-notification-batch-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

Low Priority Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-lp-requests-worker dep dlq p -t notifications-notification-low-priority-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

HAI Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-hai-requests-worker dep dlq p -t notifications-notification-hai-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

Offline Requests Worker

/helper/bin/secrets-init --provider=google /app/notification-offline-requests-worker dep dlq p -t notifications-notification-offline-requested --disable-dlq=false --timer-duration 10 --subscription-num-go-routines 1 --subscription-max-messages 1

DEP Monolith Inboxes

For things like the firebase and mailgun inboxes, you can follow log onto a pod:

$ seira production handshake pods connect --dedicated
$ gems/next_inbox/bin/next_inbox_consumer --subscription notifications-mailgun-low-priority-requested-notifications-dead-letter --handler Next::Notifications::Handlers --module Next::Notifications --client_max_messages "50" --no-ledger

you can "scale" this if needed by running on multiple tabs

Scaled DLQ Drain Deployment (DEP Monolith)

When a dead-letter queue has a large backlog, connecting to individual pods and running consumers by hand (the section above) doesn't scale. Instead, launch a short-lived Kubernetes Deployment with N replicas that all consume from the dead-letter subscription in parallel, then delete it when the queue is drained.

For the Mailgun requested handlers you can also run this in DLQ mode (NOTIFICATIONS_MAILGUN_DLQ_MODE=true, added in handshake#102584). In DLQ mode, any delivery error that would normally retry (and get re-driven back into the DLQ forever — e.g. Mailgun's "options limited to 16 kB" permanent failure) is instead acked: it notifies Bugsnag, records failed_at on the notification, and drops the message. Leave the flag unset to preserve normal retry behavior.

⚠️ DLQ mode drops permanently-failing messages (with a Bugsnag record). Only enable it when you intend to clear un-sendable messages, not reprocess them.

1. Get Teleport access (prod-workload-admin)

Creating/editing workloads in the production cluster with raw kubectl is a mutating API call, which requires the prod-workload-admin Teleport role. (Seira requests roles for you automatically, but raw kubectl does not, so you must request it yourself.) Access uses a +1 system — a colleague must approve; you cannot approve your own request. See Teleport Access Requests.

# Request the role (optionally @-mention a reviewer to ping them in Slack)
tsh request create --roles prod-workload-admin --reviewers <colleague>@joinhandshake.com

# After approval, find your approved request id
tsh request ls --my-requests

# Assume the role by re-logging in with the request id
tsh login --request-id=<request_id>

Make sure kubectl is pointed at the production cluster (e.g. via tsh kube ls + tsh kube login <prod-cluster>, or by running any seira production handshake ... command first to set the context).

2. Create the drain Deployment manifest

Save the following as dlq-drain.yaml. Adjust the marked values:

• replicas — how many parallel consumers (each requests 2 CPU / 2Gi; e.g. 20 replicas ≈ 40 CPU / 40Gi requested — watch for Pending pods).
• --subscription — the dead-letter subscription to drain.
• image — the tag currently running in prod (see command below the manifest).
• NOTIFICATIONS_MAILGUN_DLQ_MODE — keep "true" to drop un-sendable Mailgun messages; remove the env entry to just reprocess/retry.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mailgun-lp-dead-letter-drain
  namespace: handshake
  labels:
    team: network
    squad: notifications
    app: handshake
    tier: notifications-mailgun-lp-requested-notifications-inbox
    app.kubernetes.io/name: mailgun-lp-dead-letter-drain
    app.kubernetes.io/component: domain-events
    app.kubernetes.io/part-of: notifications
spec:
  replicas: 20 # <-- number of parallel consumers
  selector:
    matchLabels:
      app: handshake
      app.kubernetes.io/name: mailgun-lp-dead-letter-drain
  template:
    metadata:
      annotations:
        owner: <your-username>
      labels:
        # REQUIRED in prod: triggers the kube-secrets-init webhook to inject secrets
        kube-secrets-init.doit-intl.com/mutate: "true"
        team: network
        squad: notifications
        app: handshake
        tier: notifications-mailgun-lp-requested-notifications-inbox
        app.kubernetes.io/name: mailgun-lp-dead-letter-drain
        app.kubernetes.io/component: domain-events
        app.kubernetes.io/part-of: notifications
    spec:
      # REQUIRED: workload-identity SA with access to the pubsub subscription
      serviceAccountName: notifications
      securityContext:
        runAsNonRoot: true
      containers:
        - name: handshake
          # <-- current prod image tag (see command below)
          image: "us-central1-docker.pkg.dev/handshake-artifacts/handshake/handshake:REPLACE_WITH_CURRENT_TAG"
          command:
            - gems/next_inbox/bin/next_inbox_consumer
            - --subscription
            # <-- the dead-letter subscription to drain
            - notifications-mailgun-low-priority-requested-notifications-dead-letter
            - --handler
            - Next::Notifications::Handlers
            - --module
            - Next::Notifications
            - --client_max_messages
            - "50"
            - --no-ledger
          imagePullPolicy: IfNotPresent
          env:
            - name: EXIT_EARLY
              value: "true"
            # DLQ drain mode: ack + Bugsnag-notify permanently-failing messages
            # instead of retrying them back into the dead-letter queue.
            # Remove this entry to reprocess/retry instead of dropping.
            - name: NOTIFICATIONS_MAILGUN_DLQ_MODE
              value: "true"
            - name: DB_POOL
              value: "8"
            - name: NOTIFICATIONS_MAILGUN_LOW_PRIORITY_REQUESTED_NOTIFICATIONS_INBOX_CALLBACK_THREADS
              value: "4"
            - name: DOGSTATSD_HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: DD_TRACE_AGENT_URL
              value: "unix:///var/run/datadog/apm.socket"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          envFrom:
            - configMapRef:
                name: handshake-env-config-ksi
            - configMapRef:
                name: handshake-toggle-configs
            - configMapRef:
                name: handshake-domain-events-platform-config
          resources:
            requests:
              cpu: 2000m
              memory: 2Gi
            limits:
              cpu: 3000m
              memory: 4Gi
          securityContext:
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: apmsocketpath
              mountPath: /var/run/datadog
      volumes:
        - name: apmsocketpath
          hostPath:
            path: /var/run/datadog/

Get the current prod image tag to fill into image: (mirrors the live inbox deployment so the drain runs the same code that's deployed):

kubectl get deploy notifications-mailgun-lp-requested-notifications-inbox -n handshake \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

The manifest above is modeled on the notifications-mailgun-lp-requested-notifications-inbox deployment. To drain a different DLQ, base it on that consumer's deployment instead (copy its command, env, configmaps, and service account) and change the --subscription to the matching *-dead-letter subscription.

3. Launch and scale

# Create the deployment
kubectl create -f dlq-drain.yaml

# Scale up/down at any time without editing the file
kubectl scale deploy mailgun-lp-dead-letter-drain -n handshake --replicas=20

# Watch the pods come up (look out for Pending = not enough cluster capacity)
kubectl get pods -n handshake -l app.kubernetes.io/name=mailgun-lp-dead-letter-drain -w

# Tail logs across all replicas
kubectl logs -f -n handshake -l app.kubernetes.io/name=mailgun-lp-dead-letter-drain --max-log-requests=20

Monitor progress on the Dead Letter Queue Dashboard and via the subscription's num_undelivered_messages metric. In DLQ mode, drops increment the notifications.mailgun.requested.dlq_mode_ack metric and appear in Bugsnag.

4. Stop / clean up when done

Once the DLQ backlog reaches zero, delete the deployment (this also removes its ReplicaSet and all pods):

kubectl delete deploy mailgun-lp-dead-letter-drain -n handshake

# Verify everything is gone
kubectl get deploy,pods -n handshake -l app.kubernetes.io/name=mailgun-lp-dead-letter-drain

There is no KEDA/HPA on this deployment, so it stays at the replica count you set until you delete it — don't leave it running after the queue is drained.