My Kafka producer said it delivered 500 messages. It delivered zero.

I was wiring a Kubernetes smoke test for a fraud-detection stream processor when I found the kind of bug that never shows up in tutorials and always shows up in production: a component that fails while reporting success.

The setup

The system is simple by streaming standards: a producer replays card transactions onto a Redpanda topic, a Python consumer validates each message against a strict schema, scores it with a LightGBM model, and routes it — scores to one topic, above-threshold alerts to another, malformed payloads to a dead-letter queue. Prometheus counters track every outcome.

The CI test I was adding is the kind I now think every streaming repo needs: spin up a kind cluster (Kubernetes-in-Docker, free on GitHub runners), build and load the image, kubectl apply the whole stack, replay exactly 500 transactions — 5 of them deliberately malformed — and assert, straight from the pipeline's metrics endpoint:

sf_transactions_total{outcome="scored"}  == 500
sf_transactions_total{outcome="invalid"} == 5

Everything deployed green. The producer Job showed Completed. And the assertion found... nothing. Not a wrong number — the counters didn't exist.

The hunt

The debug step dumped pod states and logs. Three clues, in the order I noticed them:

1. Every pod was 36 seconds old. kubectl apply -f k8s/ starts everything concurrently — there is no depends_on in Kubernetes.
2. The consumer's logs showed Connection refused against the broker service for the first few seconds — then a clean connection, a clean subscription, and silence.
3. The broker's logs showed it finishing its raft bootstrap after the producer Job had already completed.

So the timeline was: producer started, broker wasn't up yet, producer "sent" 500 messages, producer exited successfully, broker came up, consumer connected to a topic containing nothing.

How does a producer send 500 messages to a broker that isn't running — successfully?

The bug

confluent-kafka's producer is asynchronous: produce() queues messages client-side and a background thread delivers them. flush(timeout) waits for delivery. And here's the line I wrote:

def flush(self) -> None:
    self._producer.flush(10)

flush() doesn't raise when messages can't be delivered. It returns the number of messages still undelivered — and I threw that return value away. Broker unreachable → 500 messages sit in the queue → flush times out after 10 seconds → returns 500 → my wrapper returns None → process exits 0 → Kubernetes marks the Job Completed.

Every layer of the stack reported success. The only honest component was the empty topic.

The fix

Four lines, plus a philosophy adjustment:

def flush(self) -> None:
    remaining = self._producer.flush(30)
    if remaining:
        raise RuntimeError(
            f"{remaining} messages not delivered before flush timeout"
        )

Now the producer racing the broker crashes — which is exactly what I want, because the Job's backoffLimit turns that crash into retry-with-backoff, and the retry lands after the broker is ready. Kubernetes doesn't give you startup ordering; loud failure plus retries is the ordering primitive.

The CI assertion changed too: polling the metrics endpoint until the expected counts appear (with a timeout), instead of sleep 10 and hope. Fixed sleeps in integration tests are just race conditions with good PR.

What I took away

1. Silent success is worse than loud failure. An exception gets retried, logged, alerted. An exit-0 with no side effects gets discovered weeks later as missing data, by someone else, with no stack trace.
2. Check the return value of every fire-and-forget API. Async client libraries often report delivery problems through returns, callbacks, or counters rather than exceptions — the ergonomic default is the dangerous one.
3. Infrastructure tests earn their keep on the first run. Unit tests passed throughout; the in-memory bus can't lose messages. Only deploying the real topology in CI — pods racing, brokers bootstrapping — could surface this. The kind cluster costs about two minutes per CI run and caught in an evening what production would have charged much more for.

The failing run, the diagnosis, and the fix are all still in the PR history — I left them there on purpose: github.com/minhazda/streaming-fraud-detection