mirror of
https://github.com/coredns/coredns.git
synced 2025-10-31 10:13:14 -04:00
BK8s datasource middleware -- PoC for A records (#153)
* Laying down kubernetes middleware foundation * Duplicated a bunch of code form etcd middleware * Duplicated code hacked to compile and load as a separate middleware * Adding verbose build option to Makefile * Removing stubzone and tls support tls and stubzone support was carried over from base etcd middleware code. Removing to simplify the kube middleware implementation. (For now.) * Adding conf directory for sample conf files * Removing stubzone support from query handler * Remove upstream and proxy from k8s corefile. Not sure that upstream or proxy makes sense for a k8s backed zone. * Comment out use of singleflight serialization * Removing parsing support for "upstream" directive from k8s * Removing upstream directive parsing code * Removing CNAME and TXT lookup implementation * Create README.md Brain-dump of DNS record name assembly and open work items. * Adding notes about wildcard handling * Adding basic k8s API client * Fleshing out methods on k8s connector * Remove PathPrefix from middleware init * Removing incorrect plural * Adding brute-force k8s service lookup functions * Initializing k8s API connector during startup * Hacking around to call k8s connector * Parsing incoming domain name into serviceName and namespace * Improving and simplifying k8s zone matching and label segmentation * Removing unused functions carried over from etcd middleware * Adding basic return of k8s data to DNS client * updated debugging println statements to flag with "[debug]" * removed code in kubernetes.go::Records that was a hold-over from etcd middleware. * Removed some random exploratory hacking. * Minior README.md updates * Updating with demo instructions * Updating README.md with CoreFile and removing completed TODO items * Updating conf file and README to reflect DNS response cache works * Disabling DNS response caching * Adding debug statement on entry to Records() * Changing port number in exampes to port 53. * Misc style and clarity changes * Removing empty function definitions * Adding comment to track future cleanup * Refactoring README to follow style of other middleware * Exposing dataobject field (typo)
This commit is contained in:
Michael Richmond
committed by
Miek Gieben
Miek Gieben
parent
446eaa957d
commit
d04abdf422
284
middleware/kubernetes/README.md
Normal file
284
middleware/kubernetes/README.md
Normal file
@@ -0,0 +1,284 @@
|
||||
# kubernetes
|
||||
|
||||
`kubernetes` enables reading zone data from a kubernetes cluster. Record names
|
||||
are constructed as "myservice.mynamespace.coredns.local" where:
|
||||
|
||||
* "myservice" is the name of the k8s service (this may include multiple DNS labels, such as "c1.myservice"),
|
||||
* "mynamespace" is the k8s namespace for the service, and
|
||||
* "coredns.local" is the zone configured for `kubernetes`.
|
||||
|
||||
|
||||
## Syntax
|
||||
|
||||
~~~
|
||||
kubernetes [zones...]
|
||||
~~~
|
||||
|
||||
* `zones` zones kubernetes should be authorative for.
|
||||
|
||||
|
||||
~~~
|
||||
kubernetes [zones] {
|
||||
endpoint http://localhost:8080
|
||||
}
|
||||
~~~
|
||||
|
||||
* `endpoint` the kubernetes API endpoint, default to http://localhost:8080
|
||||
|
||||
## Examples
|
||||
|
||||
This is the default kubernetes setup, with everything specified in full:
|
||||
|
||||
~~~
|
||||
# Serve on port 53
|
||||
.:53 {
|
||||
# use kubernetes middleware for domain "coredns.local"
|
||||
kubernetes coredns.local {
|
||||
# Use url for k8s API endpoint
|
||||
endpoint http://localhost:8080
|
||||
}
|
||||
# cache 160 coredns.local
|
||||
}
|
||||
~~~
|
||||
|
||||
### Basic Setup
|
||||
|
||||
#### Launch Kubernetes
|
||||
|
||||
Kubernetes is launched using the commands in the following `run_k8s.sh` script:
|
||||
|
||||
~~~
|
||||
#!/bin/bash
|
||||
|
||||
# Based on instructions at: http://kubernetes.io/docs/getting-started-guides/docker/
|
||||
|
||||
#K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/latest.txt)
|
||||
K8S_VERSION="v1.2.4"
|
||||
|
||||
ARCH="amd64"
|
||||
|
||||
export K8S_VERSION
|
||||
export ARCH
|
||||
|
||||
#DNS_ARGUMENTS="--cluster-dns=10.0.0.10 --cluster-domain=cluster.local"
|
||||
DNS_ARGUMENTS=""
|
||||
|
||||
docker run -d \
|
||||
--volume=/:/rootfs:ro \
|
||||
--volume=/sys:/sys:ro \
|
||||
--volume=/var/lib/docker/:/var/lib/docker:rw \
|
||||
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
|
||||
--volume=/var/run:/var/run:rw \
|
||||
--net=host \
|
||||
--pid=host \
|
||||
--privileged \
|
||||
gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \
|
||||
/hyperkube kubelet \
|
||||
--containerized \
|
||||
--hostname-override=127.0.0.1 \
|
||||
--api-servers=http://localhost:8080 \
|
||||
--config=/etc/kubernetes/manifests \
|
||||
${DNS_ARGUMENTS} \
|
||||
--allow-privileged --v=2
|
||||
~~~
|
||||
|
||||
#### Configure kubectl and test
|
||||
|
||||
The kubernetes control client can be downloaded from the generic URL:
|
||||
`http://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/${GOOS}/${GOARCH}/${K8S_BINARY}`
|
||||
|
||||
For example, the kubectl client for Linux can be downloaded using the command:
|
||||
`curl -sSL "http://storage.googleapis.com/kubernetes-release/release/v1.2.4/bin/linux/amd64/kubectl"
|
||||
|
||||
The following `setup_kubectl.sh` script can be stored in the same directory as
|
||||
kubectl to setup
|
||||
kubectl to communicate with kubernetes running on the localhost:
|
||||
|
||||
~~~
|
||||
#!/bin/bash
|
||||
|
||||
BASEDIR=`realpath $(dirname ${0})`
|
||||
|
||||
${BASEDIR}/kubectl config set-cluster test-doc --server=http://localhost:8080
|
||||
${BASEDIR}/kubectl config set-context test-doc --cluster=test-doc
|
||||
${BASEDIR}/kubectl config use-context test-doc
|
||||
|
||||
alias kubctl="${BASEDIR}/kubectl"
|
||||
~~~
|
||||
|
||||
|
||||
Verify that kubectl is working by querying for the kubernetes namespaces:
|
||||
|
||||
~~~
|
||||
$ ./kubectl get namespaces
|
||||
NAME STATUS AGE
|
||||
default Active 8d
|
||||
test Active 7d
|
||||
~~~
|
||||
|
||||
|
||||
#### Launch a kubernetes service and expose the service
|
||||
|
||||
The following commands will create a kubernetes namespace "demo",
|
||||
launch an nginx service in the namespace, and expose the service on port 80:
|
||||
|
||||
~~~
|
||||
$ ./kubectl create namespace demo
|
||||
$ ./kubectl get namespace
|
||||
|
||||
$ ./kubectl run mynginx --namespace=demo --image=nginx
|
||||
$ /kubectl get deployment --namespace=demo
|
||||
|
||||
$ ./kubectl expose deployment mynginx --namespace=demo --port=80
|
||||
$ ./kubectl get service --namespace=demo
|
||||
~~~
|
||||
|
||||
|
||||
#### Launch CoreDNS
|
||||
|
||||
Build CoreDNS and launch using the configuration file in `conf/k8sCorefile`.
|
||||
This configuration file sets up CoreDNS to use the zone `coredns.local` for
|
||||
the kubernetes services.
|
||||
|
||||
The command to launch CoreDNS is:
|
||||
|
||||
~~~
|
||||
$ ./coredns -conf conf/k8sCoreFile
|
||||
~~~
|
||||
|
||||
In a separate terminal a dns query can be issued using dig:
|
||||
|
||||
~~~
|
||||
$ dig @localhost mynginx.demo.coredns.local
|
||||
|
||||
; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> @localhost mynginx.demo.coredns.local
|
||||
; (2 servers found)
|
||||
;; global options: +cmd
|
||||
;; Got answer:
|
||||
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47614
|
||||
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
|
||||
|
||||
;; OPT PSEUDOSECTION:
|
||||
; EDNS: version: 0, flags:; udp: 4096
|
||||
;; QUESTION SECTION:
|
||||
;mynginx.demo.coredns.local. IN A
|
||||
|
||||
;; ANSWER SECTION:
|
||||
mynginx.demo.coredns.local. 0 IN A 10.0.0.10
|
||||
|
||||
;; Query time: 2 msec
|
||||
;; SERVER: ::1#53(::1)
|
||||
;; WHEN: Thu Jun 02 11:07:18 PDT 2016
|
||||
;; MSG SIZE rcvd: 71
|
||||
~~~
|
||||
|
||||
|
||||
|
||||
## Implementation Notes/Ideas
|
||||
|
||||
### Basic Zone Mapping (implemented)
|
||||
The middleware is configured with a "zone" string. For
|
||||
example: "zone = coredns.local".
|
||||
|
||||
The Kubernetes service "myservice" running in "mynamespace" would map
|
||||
to: "myservice.mynamespace.coredns.local".
|
||||
|
||||
The middleware should publish an A record for that service and a service record.
|
||||
|
||||
Initial implementation just performs the above simple mapping. Subsequent
|
||||
revisions should allow different namespaces to be published under different zones.
|
||||
|
||||
For example:
|
||||
|
||||
# Serve on port 53
|
||||
.:53 {
|
||||
# use kubernetes middleware for domain "coredns.local"
|
||||
kubernetes coredns.local {
|
||||
# Use url for k8s API endpoint
|
||||
endpoint http://localhost:8080
|
||||
}
|
||||
# Perform DNS response caching for the coredns.local zone
|
||||
# Cache timeout is provided by the integer argument in seconds
|
||||
# This works for the kubernetes middleware.)
|
||||
#cache 20 coredns.local
|
||||
#cache 160 coredns.local
|
||||
}
|
||||
|
||||
|
||||
### Internal IP or External IP?
|
||||
* Should the Corefile configuration allow control over whether the internal IP or external IP is exposed?
|
||||
* If the Corefile configuration allows control over internal IP or external IP, then the config should allow users to control the precidence.
|
||||
|
||||
For example a service "myservice" running in namespace "mynamespace" with internal IP "10.0.0.100" and external IP "1.2.3.4".
|
||||
|
||||
This example could be published as:
|
||||
|
||||
| Corefile directive | Result |
|
||||
|------------------------------|---------------------|
|
||||
| iporder = internal | 10.0.0.100 |
|
||||
| iporder = external | 1.2.3.4 |
|
||||
| iporder = external, internal | 10.0.0.100, 1.2.3.4 |
|
||||
| iporder = internal, external | 1.2.3.4, 10.0.0.100 |
|
||||
| _no directive_ | 10.0.0.100, 1.2.3.4 |
|
||||
|
||||
|
||||
### Wildcards
|
||||
|
||||
Publishing DNS records for singleton services isn't very interesting. Service
|
||||
names are unique within a k8s namespace therefore multiple services will be
|
||||
commonly run with a structured naming scheme.
|
||||
|
||||
For example, running multiple nginx services under the names:
|
||||
|
||||
| Service name |
|
||||
|--------------|
|
||||
| c1.nginx |
|
||||
| c2.nginx |
|
||||
|
||||
or:
|
||||
|
||||
| Service name |
|
||||
|--------------|
|
||||
| nginx.c3 |
|
||||
| nginx.c4 |
|
||||
|
||||
A DNS query with wildcard support for "nginx" in these examples should
|
||||
return the IP addresses for all services with "nginx" in the service name.
|
||||
|
||||
TBD:
|
||||
* How does this relate the the k8s load-balancer configuration?
|
||||
* Do wildcards search across namespaces?
|
||||
* Initial implementation assumes that a namespace maps to the first DNS label below the zone managed by the kubernetes middleware. This assumption may need to be revised.
|
||||
|
||||
|
||||
## TODO
|
||||
* Implement namespace filtering to different zones.
|
||||
* Implement IP selection and ordering (internal/external).
|
||||
* Implement SRV-record queries using naive lookup.
|
||||
* Flatten service and namespace names to valid DNS characters. (service names
|
||||
and namespace names in k8s may use uppercase and non-DNS characters. Implement
|
||||
flattening to lower case and mapping of non-DNS characters to DNS characters
|
||||
in a standard way.)
|
||||
* Do we need to generate synthetic zone records for namespaces?
|
||||
* Implement wildcard-based lookup.
|
||||
* Improve lookup to reduce size of query result obtained from k8s API.
|
||||
(namespace-based?, other ideas?)
|
||||
* How to support label specification in Corefile to allow use of labels to
|
||||
indicate zone? (Is this even useful?) For example, the following configuration
|
||||
exposes all services labeled for the "staging" environment and tenant "customerB"
|
||||
in the zone "customerB.stage.local":
|
||||
|
||||
~~~
|
||||
kubernetes customerB.stage.local {
|
||||
# Use url for k8s API endpoint
|
||||
endpoint http://localhost:8080
|
||||
label "environment" : "staging", "tenant" : "customerB"
|
||||
}
|
||||
~~~
|
||||
|
||||
* Test with CoreDNS caching. CoreDNS caching for DNS response is working using
|
||||
the `cache` directive. Tested working using 20s cache timeout and A-record queries.
|
||||
* DNS response caching is good, but we should also cache at the http query
|
||||
level as well. (Take a look at https://github.com/patrickmn/go-cache as
|
||||
a potential expiring cache implementation for the http API queries.)
|
||||
|
||||
Reference in New Issue
Block a user