* create pkg/reuseport
Move the core server listening functions to a new package so plugins can
use them.
Also make *all* servers use the functions here; as only the udp/tcp
listeners where using SO_REUSEPORT (if available). This is the only
actual change in this PR; in it's core it's just a move of 2 files.
This can also be used to cleanup the dance we're doing now for
re-acquiring the sockets in e.g. the metrics plugins and the ready
plugin.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Also push a small doc update
Signed-off-by: Miek Gieben <miek@miek.nl>
When reloading we need to disable debug output when the debug plugin is
removed from the config file. Add a `Clear` function to pkg/log and use
it in the server server.
Add test case in pkg/log, for actuall check I manually checked the
output by sprinkling some debug statements in the startup and checking
with sending SIGUSR1.
Also clear up the comments in pkg/log to remove the text about time
stamping.
Fixes: #3035
Signed-off-by: Miek Gieben <miek@miek.nl>
* Make request.Request smaller
This makes the request struct smaller and removes the pointer to the do
boolean (tri-bool) as size == 0 will indicate if we have cached it.
Family can be a int8 because it only carries 3 values, Size itself is
just a uint16 under the covers.
This is a more comprehensive fix than #3292Closes#3292
Signed-off-by: Miek Gieben <miek@miek.nl>
* cache: fix test
this now needs a valid response writter
Signed-off-by: Miek Gieben <miek@miek.nl>
journald timestamps, kubernetes timestamps, syslog timestamps. It seems
silly to add our own timestamps to the logging output as these external
ones *also* do it. Only when just running coredns this might be weird.
Remove the timestamping from pkg/log.
Remove test that tested for this.
Fixes: #3211
Signed-off-by: Miek Gieben <miek@miek.nl>
This can't be used in its current form; revert the entire PR.
Revert "fuzzing: allow setup function to be called (#3175)"
This reverts commit 62451fd3eb.
* pkg/cache: fix race in Add() and Evict()
This fixes a race in Add() when the shard is at max capacity and the key
being added is already stored. Previously, the shard would evict a
random value - when all it needed to do was replace an existing value.
There was a race in how Evict() picked which key to remove, which would
cause concurrent calls to Evict() to remove the same key.
Additionally, this commit removes a lot of the lock contention and a
race around Add() and Evict() by changing them to immediately hold the
write lock. Previously, they would check conditions with the read lock
held and not re-check those conditions once the write lock was acquired
(this is a race).
* pkg/cache: code review comments
* pkg/cache: simplify Add() logic
* replacer: evaluate format once and improve perf by ~3x
This improves the performance of logging by almost 3x and reduces memory
usage by ~8x.
Benchmark results:
benchmark old ns/op new ns/op delta
BenchmarkReplacer-12 644 324 -49.69%
BenchmarkReplacer_CommonLogFormat-12 4228 1471 -65.21%
benchmark old allocs new allocs delta
BenchmarkReplacer-12 8 2 -75.00%
BenchmarkReplacer_CommonLogFormat-12 51 17 -66.67%
benchmark old bytes new bytes delta
BenchmarkReplacer-12 240 48 -80.00%
BenchmarkReplacer_CommonLogFormat-12 3723 446 -88.02%
* replacer: code review comments
* bufPool: document why we use a pointer to a slice
* parseFormat: fix confusing comment
* TestParseFormat_Nodes: rename to TestParseFormatNodes
* replacer: use a value for bufPool instead of a pointer
* replacer: remove comment
* replacer: replace labels with knownLabels
The previous slice of label names is no longer needed.
* Update Caddy to 1.0.1, and update import path
This fix updates caddy to 1.0.1 and also
updates the import path to github.com/caddyserver/caddy
This fix fixes 2959
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Also update plugin.cfg
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Update and bump zplugin.go
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Add OnRestartFailed to the ready plugin and some various cleanups.
Document slightly better how things are supposed to work with multiple
`ready`'s in the multiple Server Blocks.
All manually tested with this Corefile:
~~~
. {
log
ready
}
example.org {
log
chaos
ready
}
~~~
And then `kill -SIGUSR1` and curling the ready endpoint. This works
well, the FailedReload is triggered by adding a syntax error in the
Corefile.
See #2659
Signed-off-by: Miek Gieben <miek@miek.nl>
* pkg/log: fix data race on d
Wrap d in a mutex to prevent data race. This makes is slower, but this
is a debugging aid anyway. It's not used normally.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix tests compilation
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix test compile
Signed-off-by: Miek Gieben <miek@miek.nl>
This fixes a data race on the listener(s) that get started in the
metrics plugins.
It also restore pkg/uniq to its former glory and removes and state being
carried in there; this means for metrics that registry.go was to
replicate that behavior *with* locking (as pkg/uniq doesn't do, or need
that).
Also renamed uniqAddr to just u, to make it slightly shorter.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Remove context.Context from request.Request
This removes the context from request.Request and makes all the changes
in the code to make it compile again. It's all mechanical. It did
unearth some weirdness in that the context was kept in handler structs
which may cause havoc with concurrently handling of requests.
Fixes#2721
Signed-off-by: Miek Gieben <miek@miek.nl>
* Make test compile
Signed-off-by: Miek Gieben <miek@miek.nl>
Add another test case for impossible DNS messages which should not be
cached. This adds a check for a message that denies its own existence.
Fixes#2724.
Signed-off-by: Miek Gieben <miek@miek.nl>
* pkg/replace: make it more efficient.
Remove the map that is allocated on every write and make it more static,
but just defining a function that gets called for a label and returns
its value.
Remove the interface definition and just implement what is needed in our
case. Add benchmark test for replace as well.
Extend metadata test to test multiple values (pretty sure this didn't
work, but there wasn't a test for it, so can't be sure).
Update all callers to use it - concurrent use should be fine as we pass
everything by value.
Benchmarks in replacer:
new: BenchmarkReplacer-4 300000 4717 ns/op 240 B/op 8 allocs/op
old: BenchmarkReplacer-4 300000 4368 ns/op 384 B/op 11 allocs/op
Added benchmark function to the old code to test it.
~~~
func BenchmarkReplacer(b *testing.B) {
w := dnstest.NewRecorder(&test.ResponseWriter{})
r := new(dns.Msg)
r.SetQuestion("example.org.", dns.TypeHINFO)
r.MsgHdr.AuthenticatedData = true
b.ResetTimer()
b.ReportAllocs()
repl := New(context.TODO(), r, w, "")
for i := 0; i < b.N; i++ {
repl.Replace("{type} {name} {size}")
}
}
~~~
New code contains (of course a different one). The amount of ops is
more, which might be good to look at some more. For all the allocations
is seems it was quite performant.
This looks to be 50% faster, and there is less allocations in log
plugin:
old: BenchmarkLogged-4 20000 70526 ns/op
new: BenchmarkLogged-4 30000 57558 ns/op
Signed-off-by: Miek Gieben <miek@miek.nl>
* Stickler bot
Signed-off-by: Miek Gieben <miek@miek.nl>
* Improve test coverage
Signed-off-by: Miek Gieben <miek@miek.nl>
* typo
Signed-off-by: Miek Gieben <miek@miek.nl>
* Add test for malformed log lines
Signed-off-by: Miek Gieben <miek@miek.nl>
This was added, but didn't see any use. For a large, complex chunk of
code we should have some users of it.
Remove all watch functionally from plugins, servers and packages.
Fixes: #2548
Signed-off-by: Miek Gieben <miek@miek.nl>
* Default to upstream to self
This is a backwards incompatible change.
This is a massive (cleanup) PR where we default to resolving external
names by the coredns process itself, instead of directly forwarding them
to some upstream.
This ignores any arguments `upstream` may have had and makes it depend
on proxy/forward configuration in the Corefile. This allows resolved
upstream names to be cached and we have better healthchecking of the
upstreams. It also means there is only one way to resolve names, by
either using the proxy or forward plugin.
The proxy/forward lookup.go functions have been removed. This also
lessen the dependency on proxy, meaning deprecating proxy will become
easier. Some tests have been removed as well, or moved to the top-level
test directory as they now require a full coredns process instead of
just the plugin.
For the etcd plugin, the entire StubZone resolving is *dropped*! This
was a hacky (but working) solution to say the least. If someone cares
deeply it can be brought back (maybe)?
The pkg/upstream is now very small and almost does nothing. Also the
New() function was changed to return a pointer to upstream.Upstream. It
also returns only one parameter, so any stragglers using it will
encounter a compile error.
All documentation has been adapted. This affected the following plugins:
* etcd
* file
* auto
* secondary
* federation
* template
* route53
A followup PR will make any upstream directives with arguments an error,
right now they are ignored.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix etcd build - probably still fails unit test
Signed-off-by: Miek Gieben <miek@miek.nl>
* Slightly smarter lookup check in upstream
Signed-off-by: Miek Gieben <miek@miek.nl>
* Compilez
Signed-off-by: Miek Gieben <miek@miek.nl>
The miekg/dns API is a bit scruffy (documentation PR in flight soon),
but these both need to be set for the correct RCODE to be reflected in
the returned packet.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix EDNS0 compliance
Do SizeAndDo in the server (ScrubWriter) and remove all uses of this
from the plugins. Also *always* do it. This is to get into compliance
for https://dnsflagday.net/.
The pkg/edns0 now exports the EDNS0 options we understand; this is
exported to allow plugins add things there. The *rewrite* plugin used
this to add custom EDNS0 option codes that the server needs to
understand.
This also needs a new release of miekg/dns because it triggered a
race-condition that was basicly there forever.
See:
* https://github.com/miekg/dns/issues/857
* https://github.com/miekg/dns/pull/859
Running a test instance and pointing the https://ednscomp.isc.org/ednscomp
to it shows the tests are now fixed:
~~~
EDNS Compliance Tester
Checking: 'miek.nl' as at 2018-12-01T17:53:15Z
miek.nl. @147.75.204.203 (drone.coredns.io.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok edns512tcp=ok optlist=ok
miek.nl. @2604:1380:2002:a000::1 (drone.coredns.io.): dns=ok edns=ok edns1=ok edns@512=ok ednsopt=ok edns1opt=ok do=ok ednsflags=ok docookie=ok edns512tcp=ok optlist=ok
All Ok
Codes
ok - test passed.
~~~
Signed-off-by: Miek Gieben <miek@miek.nl>
Signed-off-by: Miek Gieben <miek@miek.nl>
* typos in comments
Signed-off-by: Miek Gieben <miek@miek.nl>
* pkg/up: implement backoff
Every 2nd failure we double the interval until we hit 4 * interval. This
to have some sort of backoff, esp when a large cluster of coredns shares
an upstream (original intent of up package) they will hammer the
upstream. This put some back pressure on that.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Update plugin/pkg/up/up.go
Co-Authored-By: miekg <miek@miek.nl>
* Redo the plugin log PR
Remove the code duplication and call of the "official" functions. This
is the second(?) time we forgot to update the other half, so remove that
problem entirely.
Also add a test if the correct (within limits) time in front of the log
line.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Remove pFormat
Signed-off-by: Miek Gieben <miek@miek.nl>
999 chops of suffix zero, 000 allows for the milliseconds to be always
printed with 3 chars. This makes the log the printed with the same
columns.
(partial logs below)
2018-11-13T21:13:28.249Z [INFO] [::1]
2018-11-13T21:13:48.414Z [INFO] [::1]
2018-11-13T21:13:49.1Z [INFO] [::1]
vs:
2018-11-13T21:20:22.262Z [INFO] [::1]
2018-11-13T21:20:22.436Z [INFO] [::1]
2018-11-13T21:20:22.608Z [INFO] [::1]
Signed-off-by: Miek Gieben <miek@miek.nl>
* - UT on metrics verifying that all plugins of all blocs have their metrics collectors declared
* - fix error msg
* - redirect Registry of metric to the one that handle the listener
- allow duplicate of metrics collector on the same Registry (case of same plugin in 2 blocs listening metrics on the same address)
* - fix change of signature
* - ensure cleaning metrics before starting the test (metrics collectors are global vars .. and re-used by several tests)
* - I think I fixed this test. Ensure correct mn of hits and clean metrics before test.
* - fix typo in error msg - proposed at review
* - fix typo in comment
* - remove ResetMetrics functions
- change a way to test the numeric metrics : get the diff between begin and end of test
* - oops. removing debug logs
Default to using pkg/log for all logging and use a fixed time prefix
which is RFC3339Millli (doesn't exist in time, so we just extended
RFC3339), i.e. Nano might be pushing it.
Logs go from:
2018/10/30 19:14:55 [INFO] CoreDNS-1.2.5
2018/10/30 19:14:55 [INFO] linux/amd64, go1.11,
to:
2018-10-30T19:10:07.547Z [INFO] CoreDNS-1.2.5
2018-10-30T19:10:07.547Z [INFO] linux/amd64, go1.11,
Which includes the timezone - which oddly the std log package doesn't
natively do.
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/kubernetes: remove bunch a string ops
This removes a bunch of appends to where not needed, makes dnsutil.Join
take variadic args which removes the need to wrap in a new string slice.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix calls to dnsutil.Join
Signed-off-by: Miek Gieben <miek@miek.nl>
* Revert these
Signed-off-by: Miek Gieben <miek@miek.nl>
* Move functions from pkg/transport to pkg/parse
Although "parse" is a fairly generic name I believe this is somewhat
better named. pkg/transport keeps a few constants that are uses
throughout for the rest is is renaming a bunch (and the fallout from
there to make things compile again).
Signed-off-by: Miek Gieben <miek@miek.nl>
* Fix tests
Signed-off-by: Miek Gieben <miek@miek.nl>
Create plugin/pkg/transport that holds the transport related functions.
This needed to be a new pkg to prevent cyclic import errors.
This cleans up a bunch of duplicated code in core/dnsserver that also
tried to parse a transport (now all done in transport.Parse).
Signed-off-by: Miek Gieben <miek@miek.nl>
* Create test to verify correct listener behavior
* Create Unset function to remove todo items
* Reset address for prometheus listener before restarting
* Add inline documentation for Unset function
* Make shutdownTimeout a constant and change to five seconds
* Revert ForEach behavior in uniq package
* New plugin: loop
Add a plugin that detects loops. It does this by sending an unique query
to our selves. If we see the query more than twice we stop the process.
If there isn't a loop, the plugin disables it self and becomes a noop
plugin.
Signed-off-by: Miek Gieben <miek@miek.nl>
* bliep
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/log: add log.Fatal[f]
Add log.Fatal(f) to mimic more of the log package. The first and only
use is in the (new) loop plugin.
Signed-off-by: Miek Gieben <miek@miek.nl>
