New cache implementation and prefetch handing in mw/cache (#731)

* cache: add sharded cache implementation

Add Cache impl and a few tests. This cache is 256-way sharded, mainly
so each shard has it's own lock. The main cache structure is a readonly
jump plane into the right shard.

This should remove the single lock contention on the main lock and
provide more concurrent throughput - Obviously this hasn't been tested
or measured.

The key into the cache was made a uint32 (hash.fnv) and the hashing op
is not using strings.ToLower anymore remove any GC in that code path.

* here too

* Minimum shard size

* typos

* blurp

* small cleanups no defer

* typo

* Add freq based on Johns idea

* cherry-pick conflict resolv

* typo

* update from early code review from john

* add prefetch to the cache

* mw/cache: add prefetch

* remove println

* remove comment

* Fix tests

* Test prefetch in setup

* Add start of cache

* try add diff cache options

* Add hacky testcase

* not needed

* allow the use of a percentage for prefetch

If the TTL falls below xx% do a prefetch, if the record was popular.
Some other fixes and correctly prefetch only popular records.

middleware/pkg/cache/shard_test.go

@@ -0,0 +1,60 @@
+package cache
+
+import "testing"
+
+func TestShardAddAndGet(t *testing.T) {
+	s := newShard(4)
+	s.Add(1, 1)
+
+	if _, found := s.Get(1); !found {
+		t.Fatal("Failed to find inserted record")
+	}
+}
+
+func TestShardLen(t *testing.T) {
+	s := newShard(4)
+
+	s.Add(1, 1)
+	if l := s.Len(); l != 1 {
+		t.Fatalf("Shard size should %d, got %d", 1, l)
+	}
+
+	s.Add(1, 1)
+	if l := s.Len(); l != 1 {
+		t.Fatalf("Shard size should %d, got %d", 1, l)
+	}
+
+	s.Add(2, 2)
+	if l := s.Len(); l != 2 {
+		t.Fatalf("Shard size should %d, got %d", 2, l)
+	}
+}
+
+func TestShardEvict(t *testing.T) {
+	s := newShard(1)
+	s.Add(1, 1)
+	s.Add(2, 2)
+	// 1 should be gone
+
+	if _, found := s.Get(1); found {
+		t.Fatal("Found item that should have been evicted")
+	}
+}
+
+func TestShardLenEvict(t *testing.T) {
+	s := newShard(4)
+	s.Add(1, 1)
+	s.Add(2, 1)
+	s.Add(3, 1)
+	s.Add(4, 1)
+
+	if l := s.Len(); l != 4 {
+		t.Fatalf("Shard size should %d, got %d", 4, l)
+	}
+
+	// This should evict one element
+	s.Add(5, 1)
+	if l := s.Len(); l != 4 {
+		t.Fatalf("Shard size should %d, got %d", 4, l)
+	}
+}