plugin/forward: Increase minimum read timeout to 200ms (#1889)

After several experiments at SoundCloud we found that the current
minimum read timeout of 10ms is too low. A single request against a
slow/unavailable authoritative server can cause all TCP connections to
get closed. We record a 50th percentile forward/proxy latency of <5ms,
and a 99th percentile latency of 60ms. Using a minimum timeout of 200ms
seems to be a fair trade-off between avoiding unnecessary high
connection churn and reacting to upstream failures in a timely manner.

This change also renames hcDuration to hcInterval to reflect its usage,
and removes the duplicated timeout constant to make code comprehension
easier.

plugin/forward/proxy.go

@@ -34,7 +34,7 @@ func NewProxy(addr string, tlsConfig *tls.Config) *Proxy {
 		fails:     0,
 		probe:     up.New(),
 		transport: newTransport(addr, tlsConfig),
-		avgRtt:    int64(timeout / 2),
+		avgRtt:    int64(maxTimeout / 2),
 	}
 	p.client = dnsClient(tlsConfig)
 	runtime.SetFinalizer(p, (*Proxy).finalizer)
@@ -106,8 +106,7 @@ func (p *Proxy) start(duration time.Duration) {
 }
 
 const (
-	timeout    = 2 * time.Second
 	maxTimeout = 2 * time.Second
-	minTimeout = 10 * time.Millisecond
-	hcDuration = 500 * time.Millisecond
+	minTimeout = 200 * time.Millisecond
+	hcInterval = 500 * time.Millisecond
 )