plugin/cache: Fix prefetching issues (#1363)

* Improve plugin/cache metrics

* Add coredns_cache_prefetch_total metric to track number of prefetches.
* Remove unnecessary Cache.get() call which would incorrectly increment
  cache counters.
* Initialize all counters and gauges at zero.

* Allow prefetching of a single request per ttl

The original implementation didn't allow prefetching queries which are
only requested once during the duration of a TTL. The minimum amount of
queries which had to be seen was therefore capped at 2.

This change also implements a real prefetch test. The existing test was
a noop and always passed regardless of any prefetch implementation.

* Fix prefetching for items with a short TTL

The default prefetch threshold (percentage) is 10% of the lifetime of a
cache item. With the previous implementation, this disabled prefetching
for all items with a TTL < 10s (the resulting percentage would be 0, at
which point a cached item is already discarded).

This change uses a time based threshold calculation and ensures that
a prefetch is triggered at a TTL of 1 at the latest.

* Fix wrong duration reporting of cached responses

The logging and metrics plugins (among others) included the duration of
a cache prefetch in the request latency of client request. This change
fixes this wrong reporting and executes the prefetch request in a
goroutine in the background.

plugin/cache/item.go

@@ -23,7 +23,7 @@ type item struct {
 	*freq.Freq
 }
 
-func newItem(m *dns.Msg, d time.Duration) *item {
+func newItem(m *dns.Msg, now time.Time, d time.Duration) *item {
 	i := new(item)
 	i.Rcode = m.Rcode
 	i.Authoritative = m.Authoritative
@@ -44,7 +44,7 @@ func newItem(m *dns.Msg, d time.Duration) *item {
 	i.Extra = i.Extra[:j]
 
 	i.origTTL = uint32(d.Seconds())
-	i.stored = time.Now().UTC()
+	i.stored = now.UTC()
 
 	i.Freq = new(freq.Freq)
 
@@ -53,7 +53,7 @@ func newItem(m *dns.Msg, d time.Duration) *item {
 
 // toMsg turns i into a message, it tailors the reply to m.
 // The Authoritative bit is always set to 0, because the answer is from the cache.
-func (i *item) toMsg(m *dns.Msg) *dns.Msg {
+func (i *item) toMsg(m *dns.Msg, now time.Time) *dns.Msg {
 	m1 := new(dns.Msg)
 	m1.SetReply(m)
 
@@ -67,7 +67,7 @@ func (i *item) toMsg(m *dns.Msg) *dns.Msg {
 	m1.Ns = make([]dns.RR, len(i.Ns))
 	m1.Extra = make([]dns.RR, len(i.Extra))
 
-	ttl := uint32(i.ttl(time.Now()))
+	ttl := uint32(i.ttl(now))
 	for j, r := range i.Answer {
 		m1.Answer[j] = dns.Copy(r)
 		m1.Answer[j].Header().Ttl = ttl