Skip to content

Attempts for no-response delivery failures carry no diagnostic information #1028

Description

@alexluong

Problem

When a webhook delivery fails before receiving an HTTP response — DNS failure, connection refused, TLS error, dial timeout — the attempt records a code and nothing else. response_data is NULL.

failure code response_data
destination returns 4xx/5xx "503" status, body
payload format failure "ERR" the error message
publisher resolution failure "ERR" the error message
connection-level failure connection_refused, dns_error, network_error, … NULL

A destination that returns a 500 with an empty body leaves more diagnostic trace than one that could not be reached at all. The network_error catch-all is the worst case: an attempt with that code carries no information whatsoever, so the end user cannot distinguish "my DNS points at a stale IP" from "my service is down" from "my TLS cert expired" without contacting support.

Constraints on a fix

Message content must be safe by construction. Raw Go transport error strings cannot be stored verbatim: proxy failures embed operator-side proxy diagnostics, net.OpError can include a source (pod) address, and errno text reveals sender-host state. Any recorded cause should be assembled from allowlisted parts — classification code, the customer-configured destination host/port, optionally the resolved IP — never from err.Error() passthrough.

Field design needs a decision. Two positions:

  1. response_data is the attempt's explanation field and the name is historical. Two existing no-response paths (format failure, publisher resolution) already store {"error": ...} there, and NewFormatError's doc comment calls it "the customer-facing string persisted on the attempt."
  2. response_data should strictly mean an HTTP response, and failure causes belong in a dedicated field (failure_reason / error) — cleaner, but touches models.Attempt, both log store schemas, the API type, the SDKs, and the {"error": ...} values already in customers' attempt histories.

The inconsistency is customer-visible either way, so whichever path a fix takes should settle this first.

Prior art

PR #1018 implemented position 1 with raw err.Error() on the non-proxy path (proxy errors already stored classification only). Parked: no user demand yet, and the raw passthrough could not be proven leak-free. See discussion there.

Related, severable piece from that PR: classifying source-side ephemeral port exhaustion (EADDRNOTAVAIL) as address_unavailable instead of the network_error catch-all, so sender-side failures are not charged to the destination.

Metadata

Metadata

Assignees

No one assigned

    Labels

    destinationRelates to destinations support

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions