Open in app

Sign in

Write

Sign in

Modeling Environments with Linkerd Ingress (Part 2/3)

Another Use Case

Jeff Gensler
5 min readApr 13, 2017

This blog post is meant as an exploratory post. I am not sure if there are better patterns for the following.

Use Case: I want to expose my Kubernetes + Linkerd cluster to the world! How do I expose only some of the registered services? (internal, external)

What we don’t want to happen is a client outside of the cluster to hit a gateway (an entry point into our cluster from the internet) and gain knowledge about what services might exist beyond the gateway. Translating this into Linkerd’s terms, we want to allow requests into our cluster where Host: QQQ gets you to service QQQ if their service is external and deny requests with Host: YYY .

A namer binds a concrete name to a physical address.
- link

Ideally, external services will allow such a binding and internal ones won’t. Could we allow everything to be bound (resolution), but use another policy control mechanism to enforce authorization? Sure. Looping back to our original task, let’s try and not expose anything we don’t have to.

Back to our requirements:

  • Teams deploy per namespace (because of Secrets)
  • Teams have some authority over what gets placed in a Kubernetes Ingress Object (with some “standards/guidelines” from our end)

Square 1: Basic Ingress Dtab

/svc => /#/io.l5d.k8s

Make sure you understand this picture (how Identifiers work, why use them), this documentation (our specific Identifier), and this documentation (our specific Namer).

Square 2: Separate Clusters

Let’s make this simple. Say we have two clusters. One cluster will have the internal services and another will have the external services. The external services cluster has the following dtab:

/external => /#/io.l5d.k8s ;
/svc => /external ;

and internal cluster will have the the follow dtab:

/internal => /#/internal/io.l5d.k8s ;
/external => /#/external/io.l5d.k8s ;
/svc => /internal | /external ;

with multiple namers:

# We have two proxies in our linkerd containers
namers:
- kind: io.l5d.k8s
  host: localhost
  port: 8001
  prefix: /internal/io.l5d.k8s
- kind: io.l5d.k8s
  host: localhost
  port: 8002
  prefix: /external/io.l5d.k8s

After applying the “echoheaders” folder only to the external cluster:

Path: “/svc/applications/first-thing/mydeployment”

Remember, these clusters have to live under one Flannel cluster. If you weren’t using an overlay, you need to make sure Pod IP ranges don’t overlap. Note: Resolvable != Routable!

Square 3: Turning two clusters into one

So the internal cluster’s dtab configuration is looking pretty useful. The problem is that when we turn the two cluster into one, something has to give. This config:

/internal => /#/io.l5d.k8s ;
/external => /#/io.l5d.k8s ;
/svc => /internal | /external ;

has just one namer and won’t distinguish between internal and external.

Labels to the rescue!

We will use two namers, one selecting over service that opt-in to being exposed externally and one that selects over everything!

namers:
    # Back to one proxy!
    - kind: io.l5d.k8s
      port: 8001
      prefix: /internal/io.l5d.k8s
    - kind: io.l5d.k8s
      port: 8001
      prefix: /external/io.l5d.k8s
      labelSelector: linkerd-expose-external

Next, we need to create our routers. We will have one listener for externally exposed services (consumed by something like NGINX) while all of the other apps can use the standard listener. We don’t really need the internal/external distinction in the first example because all external services are registered internally.

- protocol: http
      ...
      servers:
        - port: 80
          ip: 0.0.0.0
          clearContext: true
        - port: 1080
          ip: 0.0.0.0
          clearContext: true
      dtab: |
        /internal => /#/internal/io.l5d.k8s ;
        /external => /#/external/io.l5d.k8s ;
        /svc => /internal | /external ;
    - protocol: http
      ...
      servers:
        - port: 81
          ip: 0.0.0.0
          clearContext: true
        - port: 1081
          ip: 0.0.0.0
          clearContext: true
      dtab: |
        /external => /#/external/io.l5d.k8s ;
        /fix/applications/first-thing/mydeployment => /external/applications/first-thing/mydeployment/on ;
        /svc => /fix | /external;

Because of these two routers, we end up having two Dtab configurations, but far fewer pods and Kubernetes components. At the time of this article, the Kubernetes Ingress Identifier doesn’t add any sort of labels. This means we have have to manually add the value of the labelSelector (which can be anything you want).

Remember: the NGINX container could also run its own Linkerd container as a sidecar to isolate the configuration that really only exists for it to use.

Using the examples from the previous blog article, we can see they aren’t exposed on the “external” servers.

$ curl --proxy 172.17.4.4:80 addition-operator/operate?args=1,2
{"value":3}$ curl --proxy 172.17.4.4:81 addition-operator/operate?args=1,2
No hosts are available for /svc/team-addition-operator/my-http/addition-operator, Dtab.base=[/external=>/#/external/io.l5d.k8s;/fix/applications/first-thing/mydeployment=>/external/applications/first-thing/mydeployment/on;/svc=>/fix | /external], Dtab.local=[]. Remote Info: Not Available

As for echoheaders:

$ curl --proxy 172.17.4.4:80 echoheaders/hello/world
GET /hello/world HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
Proxy-Connection: Keep-Alive
Host: echoheaders
l5d-dst-logical: /svc/applications/first-thing/mydeployment
Via: 1.1 linkerd
l5d-dst-concrete: /#/internal/io.l5d.k8s/applications/first-thing/mydeployment
l5d-ctx-trace: LZwBnv+jPiEtnAGe/6M+IS2cAZ7/oz4hAAAAAAAAAAA=
l5d-reqid: 2d9c019effa33e21$ curl --proxy 172.17.4.4:81 echoheaders/hello/world
GET /hello/world HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
Proxy-Connection: Keep-Alive
Host: echoheaders
l5d-dst-logical: /svc/applications/first-thing/mydeployment
Via: 1.1 linkerd
l5d-dst-concrete: /#/external/io.l5d.k8s/applications/first-thing/mydeployment
l5d-ctx-trace: AxP54eDHcT4DE/nh4MdxPgMT+eHgx3E+AAAAAAAAAAA=
l5d-reqid: 0313f9e1e0c7713e

What about fiddling with Host from NGINX?

Assume the following Ingress Object:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mydeployment
  namespace: applications
  labels:
    app: mydeployment
    linkerd-expose-external: true
  annotations:
    kubernetes.io/ingress.class: "linkerd"
spec:
  rules:
  - host: "echoheaders"
    http:
      paths:
      - backend:
          serviceName: mydeployment
          servicePort: first-thing
# External
  - host: "echoheaders.external"
    http:
      paths:
      - backend:
          serviceName: mydeployment
          servicePort: first-thing

and have a Dtab like this:

/internal => /#/internal/io.l5d.k8s ;
/external => /#/external/io.l5d.k8s ;
/svc => /internal | /external

We can have NGINX append .external to the host and use Dtab to search from there on. With Host: X.external values coming from NGINX, our Identifier will only every identify anything ending in .external . If a development team wanted to expose developer-api.shoes.com to the world, they would write developer-api.shoes.com.external in their NGINX config.

This might be reminiscent of DNS with AWS Private Hosted Zones. Hostname modification might work if your Ingress Obejcts are being created from a template as part of a build/deployment pipeline and/or if your company already integrates with different hosts for internal and external (meaning, hosts not on Kubernetes that will continue using a specific routing host).

$ curl -H "Host: echoheaders" http://172.17.4.4:31111/echo/health
GET /echo/health HTTP/1.0
Host: echoheaders.external
X-Forwarded-For: 10.2.48.1
User-Agent: curl/7.43.0
Accept: */*
l5d-dst-logical: /svc/applications/first-thing/mydeployment
Via: 1.0 linkerd
l5d-dst-concrete: /#/external/io.l5d.k8s/applications/first-thing/mydeployment
l5d-ctx-trace: MkP717D2ahQyQ/vXsPZqFDJD+9ew9moUAAAAAAAAAAA=
l5d-reqid: 3243fbd7b0f66a14

Note the change in Host: QQQ.external header and the X-Forwarded-For header.

$ curl -H "Host: addition-operator" "http://172.17.4.4:31111/operate?args=1,2"
Unknown destination: Request("GET /operate?args=1,2", from /10.2.48.10:36794) / no ingress rule matches$ curl -H "Host: addition-operator" "http://172.17.4.4:81/operate?args=1,2"
No hosts are available for /svc/team-addition-operator/my-http/addition-operator, Dtab.base=[/external=>/#/external/io.l5d.k8s;/fix/applications/first-thing/mydeployment=>/external/applications/first-thing/mydeployment/on;/svc=>/fix | /external], Dtab.local=[]. Remote Info: Not Available$ curl -H "Host: addition-operator" "http://172.17.4.4:80/operate?args=1,2"
{"value":3}
Kubernetes
Docker
Linkerd
Ingress

--

--

Jeff Gensler

Written by Jeff Gensler

288 Followers

https://linktr.ee/jeffzzq

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams