Two issues surfaced from running mode = "odoh" against the live Hetzner
relay as system DNS:
1. **Bootstrap deadlock.** The reqwest HTTPS client resolves the relay
and target hostnames via system DNS. When numa is itself the system
resolver, the ODoH client loops trying to resolve through itself.
Adds optional `relay_ip` and `target_ip` to `[upstream]`, plumbed
into reqwest's `resolve()` so the HTTPS client bypasses system DNS
for those two hostnames. TLS still validates against the URL
hostname, so a stale IP fails loudly rather than silently MITM'ing.
2. **2x relay load.** Default `hedge_ms = 10` triggers a duplicate
in-flight query for every request. Useful for UDP/DoH/DoT (rescues
tail latency cheaply); wasteful for ODoH (doubles HPKE seal/unseal,
doubles sealed-byte footprint a passive observer can correlate, no
latency win — relay hop dominates either way). Force-zero in
oblivious mode regardless of configured hedge_ms.
Validated end-to-end against odoh-relay.numa.rs → Cloudflare:
3 digs produced 3 forwarded_ok on the relay (was 6 before the hedge
fix), upstream_transport.odoh ticks correctly.
Client (mode = "odoh"): URL-query target routing per RFC 9230 §5,
/.well-known/odohconfigs TTL cache with 60s backoff on failure, HPKE
seal/open via odoh-rs, strict-mode default that SERVFAILs on relay
failure instead of silently downgrading. Host-equality config
validation rejects same-operator relay/target pairs.
Relay (`numa relay [PORT]`): axum server with /relay + /health.
SSRF-hardened hostname validator (RFC 1035 ASCII + dot + dash),
4 KiB body cap at the axum layer, 5s full-transaction timeout, and
static 502 on target failure (reqwest internals logged, not leaked).
Aggregate counters only — no per-request logs.
Observability: new `UpstreamTransport { Udp, Doh, Dot, Odoh }`
orthogonal to `QueryPath`, so /stats can tally wire protocols
symmetrically. Recursive mode records `Some(Udp)` for honest
"bytes egressing in cleartext" accounting.
Tests: Suite 8 exercises the client end-to-end via Frank Denis's
public relay + Cloudflare target; Suite 9 exercises `numa relay`
forwarding + guards against Cloudflare as the real far end. Full
probe script at tests/probe-odoh-ecosystem.sh verifies the entire
public ODoH ecosystem (4 targets + 1 relay per DNSCrypt's curated
list — confirms deploying Numa's relay doubles global supply).
The SRTT ordering + failure penalty path was UDP-only, so a DoT primary
in a forwarding-rule pool was never deprioritized on failure and all
DoT entries tied at INITIAL_SRTT_MS in the sort key. With [[forwarding]]
now accepting arrays of upstreams, DoT pools are a first-class case and
need the same healthiest-first behavior the default pool gets for UDP.
- Add Upstream::tracked_ip() → Some(ip) for Udp/Dot, None for Doh
(DoH has no stable IP — reqwest pools connections by hostname).
- Rewire the three SRTT call sites in forward_with_failover_raw.
- Hoist srtt.read() out of the candidate-scoring loop — one lock per
query instead of N (matters now that pools commonly have N>1).
- Drop unused #[derive(Debug)] on UpstreamPool and ForwardingRule.
- Regression tests: udp_failure_records_in_srtt + dot_failure_records_in_srtt.
Mirrors `[upstream] address` — `upstream` accepts string or array
of strings, builds an `UpstreamPool` and routes queries through
`forward_with_failover_raw` so SRTT ordering and failover apply to
matched `[[forwarding]]` rules the same way they do for the default
pool.
Single-string rules keep their current behavior (one-element pool,
equivalent single-upstream path). Empty array errors at config load.
Addresses item 1 of issue #102. Plan: docs/102_item1.md.
Config-level forwarding rules were parsed with the UDP-only
`parse_upstream_addr` helper, silently rejecting the DoT/DoH schemes
that the rest of the forwarding pipeline already supports.
Widen `ForwardingRule.upstream` from `SocketAddr` to `Upstream` so
config rules reuse the same parser as `[upstream].address` and
`fallback`. Demote `parse_upstream_addr` to `pub(crate)` to prevent
the same mistake recurring.
Closes#100.
Hedging was DoH-only (hyper dispatch spike mitigation). Now applies to
UDP (rescues packet loss) and DoT (rescues TLS handshake stalls) too.
Same-upstream hedging: fires a second independent request after hedge_ms
delay. First response wins. Disable with hedge_ms = 0.
- Remove forward_with_failover (parsed): warm_domain now uses _raw + insert_wire
- forward_udp delegates to forward_udp_raw (single UDP socket implementation)
- forward_query uses unified _raw path for all protocols
- Fix send_query_hedged warm branch: bare select! dropped secondary on primary
error instead of waiting for it — now drains both futures like the cold branch
- Remove pointless raw_len = len rename
Wire-level forwarding path skips DnsPacket parse/serialize on the hot
path. Cache stores raw wire bytes with pre-scanned TTL offsets — patches
ID + TTLs in-place on lookup instead of cloning parsed packets.
Request hedging (Dean & Barroso "Tail at Scale") fires a second
parallel request after a configurable delay (default 10ms) when
the primary upstream stalls. DoH keepalive loop prevents idle
HTTP/2 + TLS connection teardown.
Recursive resolver now hedges across multiple NS addresses and
caches NS delegation records to skip TLD re-queries.
Integration test harness polls /blocking/stats instead of fixed
sleep, eliminating the blocklist-download race condition.
Adds tls:// upstream support for forwarding queries over DNS-over-TLS
(RFC 7858). Parses tls://IP:PORT#hostname format, with default port 853.
- New Upstream::Dot variant with TLS connector
- forward_dot: length-prefixed DNS over TLS stream
- build_dot_connector: system root CAs via webpki-roots
- parse_upstream handles tls:// prefix
Example config:
address = ["tls://9.9.9.9#dns.quad9.net"]
* feat: multi-forwarder with SRTT-based failover
address accepts string or array, with optional per-server port override.
New fallback pool tried only when all primaries fail. Sequential failover
with SRTT ranking ensures fastest upstream is tried first.
Closes#34 (items 1, 2, 3)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: simplify failover candidate list and deduplicate recursive pool
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract maybe_update_primary for testable upstream re-detection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: rustfmt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add DnsPacket::query(id, domain, qtype) constructor; replace mock_query,
make_query, and 4 inline constructions across ctx/forward/recursive/api
- Add record_to_addr() in recursive.rs; replace 4 identical A/AAAA match
blocks with filter_map one-liners
- Add sinkhole_record() in ctx.rs; consolidate localhost and blocklist
A/AAAA branching into single calls
- Remove now-unused DnsQuestion imports
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: DNS-over-HTTPS upstream forwarding
Encrypt upstream queries via DoH — ISPs see HTTPS traffic on port 443,
not plaintext DNS on port 53. URL scheme determines transport:
https:// = DoH, bare IP = plain UDP. Falls back to Quad9 DoH when
system resolver cannot be detected.
- Upstream enum (Udp/Doh) with Display and PartialEq
- BytePacketBuffer::from_bytes constructor
- reqwest http2 feature for DoH server compatibility
- network_watch_loop guards against DoH→UDP silent downgrade
- 5 new tests (mock DoH server, HTTP errors, timeout)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: cargo fmt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add DoH to README — Why Numa, comparison table, roadmap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
recv_from can never return more bytes than the buffer size — the kernel
truncates silently. == is the correct heuristic for detecting truncation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Set TC (truncation) bit when response exceeds 4096-byte buffer
instead of dropping the response silently. Clients can retry via TCP.
- Log when upstream response is truncated in forward.rs.
- Dockerfile: bump to Rust 1.88, include site/service files, use
alpine runtime instead of scratch, add cmake/perl for aws-lc-sys.
- Makefile deploy: platform-aware — codesign on macOS, systemctl on Linux.
- README: trim roadmap to near-term items only.
- Verified: Docker build + smoke test passes on Linux (Alpine musl).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
