Files
coredns/test/shed_test.go

48 lines
1.3 KiB
Go
Raw Normal View History

plugin/shed: add UDP overload protection plugin (#8312) * plugin/shed: add UDP overload protection plugin UDP responses written back through one listener socket serialize on the Go runtime's internal fdMutex, which allows at most 2^20-1 concurrent operations per file descriptor and panics the process when exceeded. CoreDNS serves UDP with one goroutine per query, all writing through the shared packet connection, so a sustained overload parks every excess in-flight query in that wait queue until the process dies with "too many concurrent operations on a single file or socket". Observed in production: ~2.8M goroutines and 60GiB RSS before the panic. The shed plugin makes the panic structurally unreachable. It installs, via Config.UDPDecorateWriterFunc, a per-socket bounded evict-oldest stack drained newest-first by a single writer goroutine, so the fd never sees more than one writer and residual capacity under overload always goes to the freshest response. While a socket's stack is full, arriving queries are dropped before any plugin runs. Drops are silent (the client's resolver retries elsewhere) and counted in coredns_shed_dropped_total{server, reason}. plugin/shed/fdmutex_test.go demonstrates the failure and the fix with one shared flood harness. Two subprocess tests reproduce the exact runtime panic without the plugin's write discipline - one deterministic (a held write plus >2^20 queued writers), one with nothing held or mocked; both exercise the Go runtime rather than the plugin, so they are gated behind SHED_FLOOD_TEST=1. The counterfactual - the same load through the plugin's stack, completing with every response accounted for as written or dropped - runs in every test invocation, including -race, at 50k responders, and at the full 1.5M with SHED_FLOOD_TEST=1: SHED_FLOOD_TEST=1 go test ./plugin/shed/ Signed-off-by: Ryan Brewster <rpb@anthropic.com> * test: add shed e2e test Query a shed-enabled server over UDP (the plugin's deferred single-writer path) and TCP (which shed passes through), and check that coredns_shed_dropped_total is exported with its reason label. No-Verification-Needed: test-only change Signed-off-by: Ryan Brewster <rpb@anthropic.com> --------- Signed-off-by: Ryan Brewster <rpb@anthropic.com>
2026-07-27 05:13:25 -04:00
package test
import (
"testing"
"github.com/coredns/coredns/plugin/metrics"
"github.com/coredns/coredns/plugin/test"
"github.com/miekg/dns"
)
// TestShed checks that with the shed plugin installed a query is answered
// over both UDP (through the plugin's deferred single-writer path) and
// TCP (which shed passes through), and that its drop counter is exported.
func TestShed(t *testing.T) {
corefile := `.:0 {
shed
prometheus localhost:0
whoami
}`
i, udp, tcp, err := CoreDNSServerAndPorts(corefile)
if err != nil {
t.Fatalf("Could not get CoreDNS serving instance: %s", err)
}
defer i.Stop()
m := new(dns.Msg)
m.SetQuestion("whoami.example.org.", dns.TypeA)
if r, err := dns.Exchange(m, udp); err != nil || r.Rcode != dns.RcodeSuccess {
t.Fatalf("Expected UDP reply, got %v: %v", r, err)
}
c := &dns.Client{Net: "tcp"}
if r, _, err := c.Exchange(m, tcp); err != nil || r.Rcode != dns.RcodeSuccess {
t.Fatalf("Expected TCP reply, got %v: %v", r, err)
}
data := test.Scrape("http://" + metrics.ListenAddr + "/metrics")
got, labels := test.MetricValue("coredns_shed_dropped_total", data)
if got != "0" {
t.Errorf("Expected coredns_shed_dropped_total 0, but got %s", got)
}
if labels["reason"] == "" {
t.Errorf("Expected coredns_shed_dropped_total to carry a reason label, got %v", labels)
}
}