Home ¦ Posts ¦ Books ¦ Articles ¦ Talks ¦ Notes

Kubernetes

Similar approach: https://medium.com/bench-engineering/deploying-kubernetes-clusters-with-kops-and-terraform-832b89250e8e

Problem exposing a service via internal AWS ELB

I see this issue about 10% of the time I have tried to create a service an expose it via an internal ELB.

Issue

Setup:

  • 1 master, 1 node
  • Private networking mode in AWS
  • AWS VPC CNI Plugin
  • kube-proxy in iptables mode
  • Kubernetes 1.10.7
NAME                                               STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION   CONTAINER-RUNTIME
ip-172-16-48-234.ap-southeast-2.compute.internal   Ready     master    1d        v1.10.7   <none>        Debian GNU/Linux 8 (jessie)   4.4.121-k8s      docker://17.3.2
ip-172-16-90-217.ap-southeast-2.compute.internal   Ready     node      19h       v1.10.7   <none>        Debian GNU/Linux 8 (jessie)   4.4.121-k8s      docker://17.3.2

Example service config:

    apiVersion: v1
    kind: Service
    metadata:
      name: kube-state-metrics
      namespace: kube-system
      labels:
        k8s-app: kube-state-metrics
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    spec:
      ports:
      - name: http-metrics
        port: 8080
        targetPort: http-metrics
        protocol: TCP
      - name: telemetry
        port: 8081
        targetPort: telemetry
        protocol: TCP
      selector:
        k8s-app: kube-state-metrics
      type: LoadBalancer

Once the load balancer was created, the AWS console showed that it could not reach the service on the instance. I verified that the service was running and reachable via the Cluster IP from the master.

Having faced this issue before, I created a new node. I expected the new node to be reachable via the load balancer. That is what exactly happened. The load balancer now shows that it can reach one of two instances. That is, it can now reach the service via the new Node IP. I still had two pods running on my old node. Now, I increased the number of replicas of my service by 1, and hence I had pods running on the new node too. Now, my load balancer shows that it can reach both instances. My theory is that, the old node is forwarding traffic to the pods on the new nodes since the iptables rules were modified by kube-proxy to probabilistically forward traffic to one of the two containers running on different nodes.

What could be going on? I took a look at the IPtables rules and they look fine to me.

tcpdump on the node showed SYN flag being received, but no SYN+ACK going out.

Steps to reproduce

Example service to deploy kube-state-metrics.

Not reproducible with any definitive steps. It just happens at times and at other times it doesn't. For eaxmple, on a fresh single node, tried creating a service with a LoadBalacner ~15 times and I could reproduce it once.

Working service state

$ kubectl describe svc kube-state-metrics -n kube-system
Name:                     kube-state-metrics
Namespace:                kube-system
Labels:                   k8s-app=kube-state-metrics
Annotations:              prometheus.io/scrape=true
                          service.beta.kubernetes.io/aws-load-balancer-internal=0.0.0.0/0
Selector:                 k8s-app=kube-state-metrics
Type:                     LoadBalancer
IP:                       172.16.14.143
LoadBalancer Ingress:     internal-aff6cbb43b70511e8ac7b02d7a666f10-310572322.ap-southeast-2.elb.amazonaws.com
Port:                     http-metrics  8080/TCP
TargetPort:               http-metrics/TCP
NodePort:                 http-metrics  32191/TCP
Endpoints:                172.16.86.169:8080
Port:                     telemetry  8081/TCP
TargetPort:               telemetry/TCP
NodePort:                 telemetry  30408/TCP
Endpoints:                172.16.86.169:8081
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  27s   service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   25s   service-controller  Ensured load balancer
 ```

### Iptables when ELB was able to reach

Generated by iptables-save v1.4.21 on Thu Sep 13 02:44:47 2018

*nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [25:1500] :POSTROUTING ACCEPT [9:540] :DOCKER - [0:0] :KUBE-MARK-DROP - [0:0] :KUBE-MARK-MASQ - [0:0] :KUBE-NODEPORTS - [0:0] :KUBE-POSTROUTING - [0:0] :KUBE-SEP-52ZZZJAODWAA6K3Q - [0:0] :KUBE-SEP-AWEDAGM4NTKY7ZYU - [0:0] :KUBE-SEP-TFUG4HAKX25QORXT - [0:0] :KUBE-SEP-TXUYR3XZGLQTXVWJ - [0:0] :KUBE-SEP-XB3E6ZIF2F5R5V7R - [0:0] :KUBE-SEP-Y3X6YUAVQQTO6AGY - [0:0] :KUBE-SEP-ZM3JDWU6GRJZPEVT - [0:0] :KUBE-SERVICES - [0:0] :KUBE-SVC-6CEJ7SGDDYPX3QFE - [0:0] :KUBE-SVC-DLFQ6QP4ICS3WVQP - [0:0] :KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0] :KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0] :KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0] -A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER -A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE -A POSTROUTING ! -d 172.16.0.0/16 -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 172.16.90.217 -A DOCKER -i docker0 -j RETURN -A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000 -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000 -A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics" -m tcp --dport 32603 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics" -m tcp --dport 32603 -j KUBE-SVC-6CEJ7SGDDYPX3QFE -A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry" -m tcp --dport 31125 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry" -m tcp --dport 31125 -j KUBE-SVC-DLFQ6QP4ICS3WVQP -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE -A KUBE-SEP-52ZZZJAODWAA6K3Q -s 172.16.86.169/32 -m comment --comment "kube-system/kube-state-metrics:http-metrics" -j KUBE-MARK-MASQ -A KUBE-SEP-52ZZZJAODWAA6K3Q -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics" -m tcp -j DNAT --to-destination 172.16.86.169:8080 -A KUBE-SEP-AWEDAGM4NTKY7ZYU -s 172.16.86.169/32 -m comment --comment "kube-system/kube-state-metrics:telemetry" -j KUBE-MARK-MASQ -A KUBE-SEP-AWEDAGM4NTKY7ZYU -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry" -m tcp -j DNAT --to-destination 172.16.86.169:8081 -A KUBE-SEP-TFUG4HAKX25QORXT -s 172.16.93.227/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ -A KUBE-SEP-TFUG4HAKX25QORXT -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 172.16.93.227:53 -A KUBE-SEP-TXUYR3XZGLQTXVWJ -s 172.16.48.234/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ -A KUBE-SEP-TXUYR3XZGLQTXVWJ -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-TXUYR3XZGLQTXVWJ --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.48.234:443 -A KUBE-SEP-XB3E6ZIF2F5R5V7R -s 172.16.89.198/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ -A KUBE-SEP-XB3E6ZIF2F5R5V7R -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 172.16.89.198:53 -A KUBE-SEP-Y3X6YUAVQQTO6AGY -s 172.16.89.198/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ -A KUBE-SEP-Y3X6YUAVQQTO6AGY -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 172.16.89.198:53 -A KUBE-SEP-ZM3JDWU6GRJZPEVT -s 172.16.93.227/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ -A KUBE-SEP-ZM3JDWU6GRJZPEVT -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 172.16.93.227:53 -A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y -A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 172.16.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU -A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 172.16.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4 -A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.1.212/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 172.16.1.212/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics cluster IP" -m tcp --dport 8080 -j KUBE-SVC-6CEJ7SGDDYPX3QFE -A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.1.212/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry cluster IP" -m tcp --dport 8081 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 172.16.1.212/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry cluster IP" -m tcp --dport 8081 -j KUBE-SVC-DLFQ6QP4ICS3WVQP -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS -A KUBE-SVC-6CEJ7SGDDYPX3QFE -m comment --comment "kube-system/kube-state-metrics:http-metrics" -j KUBE-SEP-52ZZZJAODWAA6K3Q -A KUBE-SVC-DLFQ6QP4ICS3WVQP -m comment --comment "kube-system/kube-state-metrics:telemetry" -j KUBE-SEP-AWEDAGM4NTKY7ZYU -A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-Y3X6YUAVQQTO6AGY -A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-ZM3JDWU6GRJZPEVT -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-TXUYR3XZGLQTXVWJ --mask 255.255.255.255 --rsource -j KUBE-SEP-TXUYR3XZGLQTXVWJ -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-TXUYR3XZGLQTXVWJ -A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-XB3E6ZIF2F5R5V7R -A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-TFUG4HAKX25QORXT COMMIT

Completed on Thu Sep 13 02:44:47 2018

Generated by iptables-save v1.4.21 on Thu Sep 13 02:44:47 2018

*filter :INPUT ACCEPT [323:57114] :FORWARD DROP [0:0] :OUTPUT ACCEPT [248:30992] :DOCKER - [0:0] :DOCKER-ISOLATION - [0:0] :KUBE-EXTERNAL-SERVICES - [0:0] :KUBE-FIREWALL - [0:0] :KUBE-FORWARD - [0:0] :KUBE-SERVICES - [0:0] -A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES -A INPUT -j KUBE-FIREWALL -A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD -A FORWARD -j DOCKER-ISOLATION -A FORWARD -o docker0 -j DOCKER -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i docker0 ! -o docker0 -j ACCEPT -A FORWARD -i docker0 -o docker0 -j ACCEPT -A FORWARD -p tcp -j ACCEPT -A FORWARD -p udp -j ACCEPT -A FORWARD -p icmp -j ACCEPT -A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES -A OUTPUT -j KUBE-FIREWALL -A DOCKER-ISOLATION -j RETURN -A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP -A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT -A KUBE-FORWARD -s 172.16.128.0/17 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A KUBE-FORWARD -d 172.16.128.0/17 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT COMMIT

Completed on Thu Sep 13 02:44:47 2018

## Not working service state

Name: kube-state-metrics Namespace: kube-system Labels: k8s-app=kube-state-metrics Annotations: prometheus.io/scrape=true service.beta.kubernetes.io/aws-load-balancer-internal=0.0.0.0/0 Selector: k8s-app=kube-state-metrics Type: LoadBalancer IP: 172.16.8.6 LoadBalancer Ingress: internal-a2bddd51eb70811e8ac7b02d7a666f10-1440534267.ap-southeast-2.elb.amazonaws.com Port: http-metrics 8080/TCP TargetPort: http-metrics/TCP NodePort: http-metrics 31764/TCP Endpoints: 172.16.80.46:8080 Port: telemetry 8081/TCP TargetPort: telemetry/TCP NodePort: telemetry 32082/TCP Endpoints: 172.16.80.46:8081 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal EnsuringLoadBalancer 5m service-controller Ensuring load balancer Normal EnsuredLoadBalancer 5m service-controller Ensured load balancer ```

On the node itself:

Primary node IP address: 172.16.90.217

$ curl 172.16.8.6:8080 # POD IP
<OK>
$ curl 127.0.0.1:31764 # Host IP - localhost
<OK>
$ curl 172.16.90.217:31764 #Node IP - primary interface
<OK>

From master:

admin@ip-172-16-48-234:~$ curl 172.16.90.217:31764
<HANGS>

Any other port gets an instant connection refused:

$ curl 172.16.90.217:31112
curl: (7) Failed to connect to 172.16.90.217 port 31112: Connection refused

Not working iptables

# Generated by iptables-save v1.4.21 on Thu Sep 13 04:07:04 2018
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [22:1320]
:POSTROUTING ACCEPT [14:840]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-SNUX2HG2DTMM57QJ - [0:0]
:KUBE-SEP-TFUG4HAKX25QORXT - [0:0]
:KUBE-SEP-TXUYR3XZGLQTXVWJ - [0:0]
:KUBE-SEP-VPXUKJACYB2NZE6B - [0:0]
:KUBE-SEP-XB3E6ZIF2F5R5V7R - [0:0]
:KUBE-SEP-Y3X6YUAVQQTO6AGY - [0:0]
:KUBE-SEP-ZM3JDWU6GRJZPEVT - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-6CEJ7SGDDYPX3QFE - [0:0]
:KUBE-SVC-DLFQ6QP4ICS3WVQP - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING ! -d 172.16.0.0/16 -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 172.16.90.217
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics" -m tcp --dport 31764 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics" -m tcp --dport 31764 -j KUBE-SVC-6CEJ7SGDDYPX3QFE
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry" -m tcp --dport 32082 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry" -m tcp --dport 32082 -j KUBE-SVC-DLFQ6QP4ICS3WVQP
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-SNUX2HG2DTMM57QJ -s 172.16.80.46/32 -m comment --comment "kube-system/kube-state-metrics:http-metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-SNUX2HG2DTMM57QJ -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics" -m tcp -j DNAT --to-destination 172.16.80.46:8080
-A KUBE-SEP-TFUG4HAKX25QORXT -s 172.16.93.227/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-TFUG4HAKX25QORXT -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 172.16.93.227:53
-A KUBE-SEP-TXUYR3XZGLQTXVWJ -s 172.16.48.234/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-TXUYR3XZGLQTXVWJ -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-TXUYR3XZGLQTXVWJ --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.48.234:443
-A KUBE-SEP-VPXUKJACYB2NZE6B -s 172.16.80.46/32 -m comment --comment "kube-system/kube-state-metrics:telemetry" -j KUBE-MARK-MASQ
-A KUBE-SEP-VPXUKJACYB2NZE6B -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry" -m tcp -j DNAT --to-destination 172.16.80.46:8081
-A KUBE-SEP-XB3E6ZIF2F5R5V7R -s 172.16.89.198/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-XB3E6ZIF2F5R5V7R -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 172.16.89.198:53
-A KUBE-SEP-Y3X6YUAVQQTO6AGY -s 172.16.89.198/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-Y3X6YUAVQQTO6AGY -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 172.16.89.198:53
-A KUBE-SEP-ZM3JDWU6GRJZPEVT -s 172.16.93.227/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZM3JDWU6GRJZPEVT -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 172.16.93.227:53
-A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 172.16.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 172.16.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.8.6/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 172.16.8.6/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:http-metrics cluster IP" -m tcp --dport 8080 -j KUBE-SVC-6CEJ7SGDDYPX3QFE
-A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.8.6/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry cluster IP" -m tcp --dport 8081 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 172.16.8.6/32 -p tcp -m comment --comment "kube-system/kube-state-metrics:telemetry cluster IP" -m tcp --dport 8081 -j KUBE-SVC-DLFQ6QP4ICS3WVQP
-A KUBE-SERVICES ! -s 172.16.128.0/17 -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-6CEJ7SGDDYPX3QFE -m comment --comment "kube-system/kube-state-metrics:http-metrics" -j KUBE-SEP-SNUX2HG2DTMM57QJ
-A KUBE-SVC-DLFQ6QP4ICS3WVQP -m comment --comment "kube-system/kube-state-metrics:telemetry" -j KUBE-SEP-VPXUKJACYB2NZE6B
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-Y3X6YUAVQQTO6AGY
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-ZM3JDWU6GRJZPEVT
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-TXUYR3XZGLQTXVWJ --mask 255.255.255.255 --rsource -j KUBE-SEP-TXUYR3XZGLQTXVWJ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-TXUYR3XZGLQTXVWJ
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-XB3E6ZIF2F5R5V7R
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-TFUG4HAKX25QORXT
COMMIT
# Completed on Thu Sep 13 04:07:04 2018
# Generated by iptables-save v1.4.21 on Thu Sep 13 04:07:04 2018
*filter
:INPUT ACCEPT [453:106224]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [392:36882]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -p tcp -j ACCEPT
-A FORWARD -p udp -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 172.16.128.0/17 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 172.16.128.0/17 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Thu Sep 13 04:07:04 2018

Trace when working/not working

I enabled iptables tracing to collect traces.

IP addr on node

dmin@ip-172-16-90-217:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:b0:e8:73:2d:c4 brd ff:ff:ff:ff:ff:ff
    inet 172.16.90.217/20 brd 172.16.95.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::8b0:e8ff:fe73:2dc4/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:17:65:a4:a0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:a0:b0:7b:5d:ca brd ff:ff:ff:ff:ff:ff
    inet 172.16.91.239/20 brd 172.16.95.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::8a0:b0ff:fe7b:5dca/64 scope link
       valid_lft forever preferred_lft forever
5: eni88d33fadcc8@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether a2:a0:ee:96:c7:78 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0a0:eeff:fe96:c778/64 scope link
       valid_lft forever preferred_lft forever
6: eni31811c0de29@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether fe:21:51:a3:a9:ef brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc21:51ff:fea3:a9ef/64 scope link
       valid_lft forever preferred_lft forever
7: eni0f88c6c62d8@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether e6:5b:93:c0:a5:1e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e45b:93ff:fec0:a51e/64 scope link
       valid_lft forever preferred_lft forever
9: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:67:09:f3:73:60 brd ff:ff:ff:ff:ff:ff
    inet 172.16.88.9/20 brd 172.16.95.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::867:9ff:fef3:7360/64 scope link
       valid_lft forever preferred_lft forever
26: eni341e3e59da4@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 1a:60:86:b7:62:d1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1860:86ff:feb7:62d1/64 scope link
       valid_lft forever preferred_lft forever

Iptables filter table

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
2    KUBE-FIREWALL  all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
num  target     prot opt source               destination
1    KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
2    DOCKER-ISOLATION  all  --  anywhere             anywhere
3    DOCKER     all  --  anywhere             anywhere
4    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
5    ACCEPT     all  --  anywhere             anywhere
6    ACCEPT     all  --  anywhere             anywhere
7    ACCEPT     tcp  --  anywhere             anywhere
8    ACCEPT     udp  --  anywhere             anywhere
9    ACCEPT     icmp --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
2    KUBE-FIREWALL  all  --  anywhere             anywhere

Chain DOCKER (1 references)
num  target     prot opt source               destination

Chain DOCKER-ISOLATION (1 references)
num  target     prot opt source               destination
1    RETURN     all  --  anywhere             anywhere

Chain KUBE-EXTERNAL-SERVICES (1 references)
num  target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
num  target     prot opt source               destination
1    DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
num  target     prot opt source               destination
1    ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
2    ACCEPT     all  --  ip-172-16-128-0.ap-southeast-2.compute.internal/17  anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
3    ACCEPT     all  --  anywhere             ip-172-16-128-0.ap-southeast-2.compute.internal/17  /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (1 references)
num  target     prot opt source               destination
admin@ip-172-16-90-217:~$

IPtables NAT table

admin@ip-172-16-90-217:~$ sudo iptables -L --line-numbers -t nat
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
2    DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
2    DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
2    MASQUERADE  all  --  ip-172-17-0-0.ap-southeast-2.compute.internal/16  anywhere
3    SNAT       all  --  anywhere            !ip-172-16-0-0.ap-southeast-2.compute.internal/16  /* AWS, SNAT */ ADDRTYPE match dst-type !LOCAL to:172.16.90.217

Chain DOCKER (2 references)
num  target     prot opt source               destination
1    RETURN     all  --  anywhere             anywhere

Chain KUBE-MARK-DROP (0 references)
num  target     prot opt source               destination
1    MARK       all  --  anywhere             anywhere             MARK or 0x8000

Chain KUBE-MARK-MASQ (14 references)
num  target     prot opt source               destination
1    MARK       all  --  anywhere             anywhere             MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */ tcp dpt:31764
2    KUBE-SVC-6CEJ7SGDDYPX3QFE  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */ tcp dpt:31764
3    KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */ tcp dpt:32082
4    KUBE-SVC-DLFQ6QP4ICS3WVQP  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */ tcp dpt:32082

Chain KUBE-POSTROUTING (1 references)
num  target     prot opt source               destination
1    MASQUERADE  all  --  anywhere             anywhere             /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-SNUX2HG2DTMM57QJ (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-80-46.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-state-metrics:http-metrics */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */ tcp to:172.16.80.46:8080

Chain KUBE-SEP-TFUG4HAKX25QORXT (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-93-227.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns */
2    DNAT       udp  --  anywhere             anywhere             /* kube-system/kube-dns:dns */ udp to:172.16.93.227:53

Chain KUBE-SEP-TXUYR3XZGLQTXVWJ (2 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  api.internal.rsaus-non-production.k8s.local  anywhere             /* default/kubernetes:https */
2    DNAT       tcp  --  anywhere             anywhere             /* default/kubernetes:https */ recent: SET name: KUBE-SEP-TXUYR3XZGLQTXVWJ side: source mask: 255.255.255.255 tcp to:172.16.48.234:443

Chain KUBE-SEP-VPXUKJACYB2NZE6B (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-80-46.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-state-metrics:telemetry */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */ tcp to:172.16.80.46:8081

Chain KUBE-SEP-XB3E6ZIF2F5R5V7R (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-89-198.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns */
2    DNAT       udp  --  anywhere             anywhere             /* kube-system/kube-dns:dns */ udp to:172.16.89.198:53

Chain KUBE-SEP-Y3X6YUAVQQTO6AGY (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-89-198.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns-tcp */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */ tcp to:172.16.89.198:53

Chain KUBE-SEP-ZM3JDWU6GRJZPEVT (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-93-227.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns-tcp */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */ tcp to:172.16.93.227:53

Chain KUBE-SERVICES (2 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-1.ap-southeast-2.compute.internal  /* default/kubernetes:https cluster IP */ tcp dpt:https
2    KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  anywhere             ip-172-16-0-1.ap-southeast-2.compute.internal  /* default/kubernetes:https cluster IP */ tcp dpt:https
3    KUBE-MARK-MASQ  udp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
4    KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  anywhere             ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
5    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
6    KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  anywhere             ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
7    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-8-6.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:http-metrics cluster IP */ tcp dpt:http-alt
8    KUBE-SVC-6CEJ7SGDDYPX3QFE  tcp  --  anywhere             ip-172-16-8-6.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:http-metrics cluster IP */ tcp dpt:http-alt
9    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-8-6.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:telemetry cluster IP */ tcp dpt:tproxy
10   KUBE-SVC-DLFQ6QP4ICS3WVQP  tcp  --  anywhere             ip-172-16-8-6.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:telemetry cluster IP */ tcp dpt:tproxy
11   KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-6CEJ7SGDDYPX3QFE (2 references)
num  target     prot opt source               destination
1    KUBE-SEP-SNUX2HG2DTMM57QJ  all  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */

Chain KUBE-SVC-DLFQ6QP4ICS3WVQP (2 references)
num  target     prot opt source               destination
1    KUBE-SEP-VPXUKJACYB2NZE6B  all  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num  target     prot opt source               destination
1    KUBE-SEP-Y3X6YUAVQQTO6AGY  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.50000000000
2    KUBE-SEP-ZM3JDWU6GRJZPEVT  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num  target     prot opt source               destination
1    KUBE-SEP-TXUYR3XZGLQTXVWJ  all  --  anywhere             anywhere             /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-TXUYR3XZGLQTXVWJ side: source mask: 255.255.255.255
2    KUBE-SEP-TXUYR3XZGLQTXVWJ  all  --  anywhere             anywhere             /* default/kubernetes:https */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num  target     prot opt source               destination
1    KUBE-SEP-XB3E6ZIF2F5R5V7R  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns */ statistic mode random probability 0.50000000000
2    KUBE-SEP-TFUG4HAKX25QORXT  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns */

Random notes

sudo iptables -t raw -A OUTPUT -p tcp --dport 31764 -j TRACE
sudo iptables -t raw -A PREROUTING -p tcp --dport 31764 -j TRACE
sudo modprobe nf_log_ipv4
sudo sysctl net.netfilter.nf_log.2=nf_log_ipv4

https://backreference.org/2010/06/11/iptables-debugging/

When I try to curl from master:

logs on node

admin@ip-172-16-90-217:~$ sudo cat /var/log/kern.log | grep "172.16.48.234"

Non-working trace:

Sep 13 05:02:39 ip-172-16-90-217 kernel: [74324.894886] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309)
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74324.909934] TRACE: nat:PREROUTING:rule:1 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP
SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309)
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74324.925605] TRACE: nat:KUBE-SERVICES:rule:11 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309)
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74324.940970] TRACE: nat:KUBE-NODEPORTS:rule:1 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309)
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74324.956481] TRACE: nat:KUBE-MARK-MASQ:rule:1 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309)
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74324.972106] TRACE: nat:KUBE-MARK-MASQ:return:2 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74324.988336] TRACE: nat:KUBE-NODEPORTS:rule:2 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74325.003898] TRACE: nat:KUBE-SVC-6CEJ7SGDDYPX3QFE:rule:1 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74325.020403] TRACE: nat:KUBE-SEP-SNUX2HG2DTMM57QJ:rule:2 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57671 DF PROTO=TCP SPT=47054 DPT=31764 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74325.036970] TRACE: filter:FORWARD:rule:1 IN=eth0 OUT=eni341e3e59da4 MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.80.46 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=57671
DF PROTO=TCP SPT=47054 DPT=8080 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74325.053711] TRACE: filter:KUBE-FORWARD:rule:1 IN=eth0 OUT=eni341e3e59da4 MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.80.46 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=57671 DF PROTO=TCP SPT=47054 DPT=8080 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74325.070918] TRACE: nat:POSTROUTING:rule:1 IN= OUT=eni341e3e59da4 SRC=172.16.48.234 DST=172.16.80.46 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=57671 DF PROTO=TCP SPT=47054 DPT=8080 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000
Sep 13 05:02:39 ip-172-16-90-217 kernel: [74325.085027] TRACE: nat:KUBE-POSTROUTING:rule:1 IN= OUT=eni341e3e59da4 SRC=172.16.48.234 DST=172.16.80.46 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=57671 DF PROTO=TCP SPT=47054 DPT=8080 SEQ=3925179987 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A02719F140000000001030309) MARK=0x4000

I recreate the service and then the curl from master works:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:b0:e8:73:2d:c4 brd ff:ff:ff:ff:ff:ff
    inet 172.16.90.217/20 brd 172.16.95.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::8b0:e8ff:fe73:2dc4/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:17:65:a4:a0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:a0:b0:7b:5d:ca brd ff:ff:ff:ff:ff:ff
    inet 172.16.91.239/20 brd 172.16.95.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::8a0:b0ff:fe7b:5dca/64 scope link
       valid_lft forever preferred_lft forever
5: eni88d33fadcc8@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether a2:a0:ee:96:c7:78 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a0a0:eeff:fe96:c778/64 scope link
       valid_lft forever preferred_lft forever
6: eni31811c0de29@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether fe:21:51:a3:a9:ef brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc21:51ff:fea3:a9ef/64 scope link
       valid_lft forever preferred_lft forever
7: eni0f88c6c62d8@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether e6:5b:93:c0:a5:1e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e45b:93ff:fec0:a51e/64 scope link
       valid_lft forever preferred_lft forever
9: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:67:09:f3:73:60 brd ff:ff:ff:ff:ff:ff
    inet 172.16.88.9/20 brd 172.16.95.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::867:9ff:fef3:7360/64 scope link
       valid_lft forever preferred_lft forever
28: eni2c639369d29@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 5a:c5:f6:6c:b4:5b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::58c5:f6ff:fe6c:b45b/64 scope link
       valid_lft forever preferred_lft forever


# filter table

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
2    KUBE-FIREWALL  all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
num  target     prot opt source               destination
1    KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
2    DOCKER-ISOLATION  all  --  anywhere             anywhere
3    DOCKER     all  --  anywhere             anywhere
4    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
5    ACCEPT     all  --  anywhere             anywhere
6    ACCEPT     all  --  anywhere             anywhere
7    ACCEPT     tcp  --  anywhere             anywhere
8    ACCEPT     udp  --  anywhere             anywhere
9    ACCEPT     icmp --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
2    KUBE-FIREWALL  all  --  anywhere             anywhere

Chain DOCKER (1 references)
num  target     prot opt source               destination

Chain DOCKER-ISOLATION (1 references)
num  target     prot opt source               destination
1    RETURN     all  --  anywhere             anywhere

Chain KUBE-EXTERNAL-SERVICES (1 references)
num  target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
num  target     prot opt source               destination
1    DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
num  target     prot opt source               destination
1    ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
2    ACCEPT     all  --  ip-172-16-128-0.ap-southeast-2.compute.internal/17  anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
3    ACCEPT     all  --  anywhere             ip-172-16-128-0.ap-southeast-2.compute.internal/17  /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (1 references)
num  target     prot opt source               destination

# nat table
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
2    DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
2    DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
2    MASQUERADE  all  --  ip-172-17-0-0.ap-southeast-2.compute.internal/16  anywhere
3    SNAT       all  --  anywhere            !ip-172-16-0-0.ap-southeast-2.compute.internal/16  /* AWS, SNAT */ ADDRTYPE match dst-type !LOCAL to:172.16.90.217

Chain DOCKER (2 references)
num  target     prot opt source               destination
1    RETURN     all  --  anywhere             anywhere

Chain KUBE-MARK-DROP (0 references)
num  target     prot opt source               destination
1    MARK       all  --  anywhere             anywhere             MARK or 0x8000

Chain KUBE-MARK-MASQ (14 references)
num  target     prot opt source               destination
1    MARK       all  --  anywhere             anywhere             MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */ tcp dpt:32682
2    KUBE-SVC-6CEJ7SGDDYPX3QFE  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */ tcp dpt:32682
3    KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */ tcp dpt:31441
4    KUBE-SVC-DLFQ6QP4ICS3WVQP  tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */ tcp dpt:31441

Chain KUBE-POSTROUTING (1 references)
num  target     prot opt source               destination
1    MASQUERADE  all  --  anywhere             anywhere             /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-52ZZZJAODWAA6K3Q (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-86-169.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-state-metrics:http-metrics */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */ tcp to:172.16.86.169:8080

Chain KUBE-SEP-AWEDAGM4NTKY7ZYU (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-86-169.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-state-metrics:telemetry */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */ tcp to:172.16.86.169:8081

Chain KUBE-SEP-TFUG4HAKX25QORXT (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-93-227.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns */
2    DNAT       udp  --  anywhere             anywhere             /* kube-system/kube-dns:dns */ udp to:172.16.93.227:53

Chain KUBE-SEP-TXUYR3XZGLQTXVWJ (2 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  api.internal.rsaus-non-production.k8s.local  anywhere             /* default/kubernetes:https */
2    DNAT       tcp  --  anywhere             anywhere             /* default/kubernetes:https */ recent: SET name: KUBE-SEP-TXUYR3XZGLQTXVWJ side: source mask: 255.255.255.255 tcp to:172.16.48.234:443

Chain KUBE-SEP-XB3E6ZIF2F5R5V7R (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-89-198.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns */
2    DNAT       udp  --  anywhere             anywhere             /* kube-system/kube-dns:dns */ udp to:172.16.89.198:53

Chain KUBE-SEP-Y3X6YUAVQQTO6AGY (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-89-198.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns-tcp */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */ tcp to:172.16.89.198:53

Chain KUBE-SEP-ZM3JDWU6GRJZPEVT (1 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  all  --  ip-172-16-93-227.ap-southeast-2.compute.internal  anywhere             /* kube-system/kube-dns:dns-tcp */
2    DNAT       tcp  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */ tcp to:172.16.93.227:53

Chain KUBE-SERVICES (2 references)
num  target     prot opt source               destination
1    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
2    KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  anywhere             ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
3    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-12.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:http-metrics cluster IP */ tcp dpt:http-alt
4    KUBE-SVC-6CEJ7SGDDYPX3QFE  tcp  --  anywhere             ip-172-16-0-12.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:http-metrics cluster IP */ tcp dpt:http-alt
5    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-12.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:telemetry cluster IP */ tcp dpt:tproxy
6    KUBE-SVC-DLFQ6QP4ICS3WVQP  tcp  --  anywhere             ip-172-16-0-12.ap-southeast-2.compute.internal  /* kube-system/kube-state-metrics:telemetry cluster IP */ tcp dpt:tproxy
7    KUBE-MARK-MASQ  tcp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-1.ap-southeast-2.compute.internal  /* default/kubernetes:https cluster IP */ tcp dpt:https
8    KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  anywhere             ip-172-16-0-1.ap-southeast-2.compute.internal  /* default/kubernetes:https cluster IP */ tcp dpt:https
9    KUBE-MARK-MASQ  udp  -- !ip-172-16-128-0.ap-southeast-2.compute.internal/17  ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
10   KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  anywhere             ip-172-16-0-10.ap-southeast-2.compute.internal  /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
11   KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-6CEJ7SGDDYPX3QFE (2 references)
num  target     prot opt source               destination
1    KUBE-SEP-52ZZZJAODWAA6K3Q  all  --  anywhere             anywhere             /* kube-system/kube-state-metrics:http-metrics */

Chain KUBE-SVC-DLFQ6QP4ICS3WVQP (2 references)
num  target     prot opt source               destination
1    KUBE-SEP-AWEDAGM4NTKY7ZYU  all  --  anywhere             anywhere             /* kube-system/kube-state-metrics:telemetry */

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num  target     prot opt source               destination
1    KUBE-SEP-Y3X6YUAVQQTO6AGY  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.50000000000
2    KUBE-SEP-ZM3JDWU6GRJZPEVT  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num  target     prot opt source               destination
1    KUBE-SEP-TXUYR3XZGLQTXVWJ  all  --  anywhere             anywhere             /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-TXUYR3XZGLQTXVWJ side: source mask: 255.255.255.255
2    KUBE-SEP-TXUYR3XZGLQTXVWJ  all  --  anywhere             anywhere             /* default/kubernetes:https */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num  target     prot opt source               destination
1    KUBE-SEP-XB3E6ZIF2F5R5V7R  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns */ statistic mode random probability 0.50000000000
2    KUBE-SEP-TFUG4HAKX25QORXT  all  --  anywhere             anywhere             /* kube-system/kube-dns:dns */
admin@ip-172-16-90-217:~$




Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.813764] TRACE: raw:PREROUTING:policy:3 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309)
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.833819] TRACE: nat:PREROUTING:rule:1 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP
SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309)
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.837752] TRACE: nat:KUBE-SERVICES:rule:11 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309)
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.853361] TRACE: nat:KUBE-NODEPORTS:rule:3 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309)
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.880551] TRACE: nat:KUBE-MARK-MASQ:rule:1 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309)
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.895612] TRACE: nat:KUBE-MARK-MASQ:return:2 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.929049] TRACE: nat:KUBE-NODEPORTS:rule:4 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.958605] TRACE: nat:KUBE-SVC-DLFQ6QP4ICS3WVQP:rule:1 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.958828] TRACE: nat:KUBE-SEP-AWEDAGM4NTKY7ZYU:rule:2 IN=eth0 OUT= MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 sDST=172.16.90.217 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26854 DF PROTO=TCP SPT=32440 DPT=31441 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76414.980823] TRACE: filter:FORWARD:rule:1 IN=eth0 OUT=eni2c639369d29 MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.86.169 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=26854 DF PROTO=TCP SPT=32440 DPT=8081 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76415.000832] TRACE: filter:KUBE-FORWARD:rule:1 IN=eth0 OUT=eni2c639369d29 MAC=0a:b0:e8:73:2d:c4:0a:a9:4f:33:0a:74:08:00 SRC=172.16.48.234 DST=172.16.86.169 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=26854 DF PROTO=TCP SPT=32440 DPT=8081 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76415.029038] TRACE: nat:POSTROUTING:rule:1 IN= OUT=eni2c639369d29 SRC=172.16.48.234 DST=172.16.86.169 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=26854 DF PROTO=TCP SPT=32440 DPT=8081 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000
Sep 13 05:37:29 ip-172-16-90-217 kernel: [76415.044192] TRACE: nat:KUBE-POSTROUTING:rule:1 IN= OUT=eni2c639369d29 SRC=172.16.48.234 DST=172.16.86.169 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=26854 DF PROTO=TCP SPT=32440 DPT=8081 SEQ=131264737 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A027998070000000001030309) MARK=0x4000

Generic Issues faced/gotchas

  • https://github.com/kubernetes/kops/issues/4049#issuecomment-352152838: Adopted solution to create systemd dropin file
  • Use existing AWS keypair - modified kops
  • For AWS Internal ELB backed master/API, make sure the DNS is api.<cluster name>. Needed for certificate validation to reach cluster from outside the cluster

# AWS ELB issue reachability

  • See gist

# Links

Non-Kubernetes:

  • AWS VPC subnets guide: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html
  • Networking & subnetting: https://0x00sec.org/t/a-dive-into-subnetting/468
  • WSL + ssh agent: https://gist.github.com/copperlight/e9eaa40d6f3bb9e1e6e6b1dd8ba0a8d5
  • iptables: https://www.booleanworld.com/depth-guide-iptables-linux-firewall/, man page: https://linux.die.net/man/8/iptables

Generic:

  • Kubernetes components: https://kubernetes.io/docs/concepts/overview/components/
  • AWS specific components: https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws
  • AWS specific use cases: https://blog.giantswarm.io/load-balancer-service-use-cases-on-aws
  • Kubernetes notes: https://github.com/darshanime/notes/blob/master/kubernetes.org
  • https://blog.codersociety.com/creating-a-highly-available-secured-kubernetes-cluster-on-aws-with-kops-89598ce0c5aa
  • Magic of Kuberenetes networking: https://www.youtube.com/watch?v=7OFw3lgSb1Q

kops:

  • kops create cluster command: https://github.com/kubernetes/kops/blob/master/docs/cli/kops_create_cluster.md
  • kops + terraform: https://github.com/kubernetes/kops/blob/master/docs/terraform.md
  • kops network topology: https://github.com/kubernetes/kops/blob/master/docs/topology.md
  • kops + networking: https://github.com/kubernetes/kops/blob/master/docs/networking.md#amazon-vpc-backend
  • bastion ssh: https://github.com/kubernetes/kops/blob/master/docs/bastion.md
  • kops + HA: https://github.com/kubernetes/kops/blob/master/docs/high_availability.md
  • kops rolling update + terraform: https://github.com/kubernetes/kops/blob/master/docs/upgrade.md#terraform-users

Networking:

  • https://medium.com/@anne_e_currie/kubernetes-aws-networking-for-dummies-like-me-b6dedeeb95f3

Securing kubernetes:

  • https://kubernetes.io/docs/reference/access-authn-authz/rbac/
  • https://medium.com/devopslinks/security-problems-of-kops-default-deployments-2819c157bc90
  • https://github.com/kubernetes/kops/blob/master/docs/security.md
  • https://docs.bitnami.com/kubernetes/how-to/configure-rbac-in-your-kubernetes-cluster/
  • https://github.com/kubernetes-sigs/aws-iam-authenticator
  • https://rancher.com/understanding-kubernetes-rbac/

Kubernetes Federation

  • https://kubernetes.io/docs/tasks/federation/

Tooling

  • https://brigade.sh/
  • https://draft.sh/
  • https://github.com/GoogleContainerTools/skaffold
  • https://ksonnet.io/

Serverless

  • https://github.com/kubeless/kubeless

Developer/user access

  • https://github.com/coreos/dex
  • https://thenewstack.io/single-sign-on-for-kubernetes-dashboard-experience/
  • https://medium.com/@evnsio/setting-up-google-auth-rbac-for-kubernetes-2d91d68af356
  • https://docs.bitnami.com/kubernetes/how-to/configure-rbac-in-your-kubernetes-cluster/

Ingress

  • Ingress controller: https://blog.giantswarm.io/load-balancer-service-use-cases-on-aws/
  • Ingress controller: https://kubernetes.io/docs/concepts/services-networking/ingress/
  • https://danielfm.me/posts/painless-nginx-ingress.html
  • http://blog.wercker.com/troubleshooting-ingress-kubernetes

DR/Backup

  • kubectl Cordon/drain
  • https://heptio.github.io/ark/v0.9.0/about (https://www.youtube.com/watch?v=qRPNuT080Hk)

AWS/IAM

Notes

kops

Get clusters:

$ kops get cluster --state s3://ratesetter-k8s-state-non-production
NAME                            CLOUD   ZONES
rsaus-non-production.k8s.local  aws     ap-southeast-2a,ap-southeast-2b,ap-southeast-2c

Get cluster config as yaml:

$ kops get cluster rsaus-non-production.k8s.local --state s3://ratesetter-k8s-state-non-production -o yaml

Rolling update:

$kops rolling-update  cluster rsaus-non-production.k8s.local --state s3://ratesetter-k8s-state-non-productionUnable to reach the kubernetes API.
Use --cloudonly to do a rolling-update without confirming progress with the k8s API

error listing nodes in cluster: Get https://internal-api-rsaus-non-production--j92ql2-282872028.ap-southeast-2.elb.amazonaws.com/api/v1/nodes: dial tcp: lookup internal-api-rsaus-non-production--j92ql2-282872028.ap-southeast-2.elb.amazonaws.com on 10.0.64.1:53: no such host

Rolling update preview:

$ kops rolling-update  cluster rsaus-non-production.k8s.local --state s3://ratesetter-k8s-state-non-production --cloudonly
NAME                    STATUS          NEEDUPDATE      READY   MIN     MAX
master-ap-southeast-2a  NeedsUpdate     1               0       1       1
nodes                   NeedsUpdate     1               0       1       1

Must specify --yes to rolling-update.

Rolling update:

$ kops rolling-update  cluster rsaus-non-production.k8s.local --state s3://ratesetter-k8s-state-non-production --cloudonly --yes
NAME                    STATUS          NEEDUPDATE      READY   MIN     MAX
master-ap-southeast-2a  NeedsUpdate     1               0       1       1
nodes                   NeedsUpdate     1               0       1       1
W0911 14:03:31.087352    1477 instancegroups.go:152] Not draining cluster nodes as 'cloudonly' flag is set.
I0911 14:03:31.087570    1477 instancegroups.go:280] Stopping instance "i-0623da4b8b3a5bbf4", in group "master-ap-southeast-2a.masters.rsaus-non-production.k8s.local" (this may take a while).
W0911 14:08:31.235302    1477 instancegroups.go:184] Not validating cluster as cloudonly flag is set.
W0911 14:08:31.235598    1477 instancegroups.go:152] Not draining cluster nodes as 'cloudonly' flag is set.
I0911 14:08:31.235857    1477 instancegroups.go:280] Stopping instance "i-011caeb317ee46cc6", in group "nodes.rsaus-non-production.k8s.local" (this may take a while).
W0911 14:12:27.359113    1477 instancegroups.go:184] Not validating cluster as cloudonly flag is set.
I0911 14:12:27.360176    1477 rollingupdate.go:184] Rolling update completed for cluster "rsaus-non-production.k8s.local"!

Rolling update notes:

  1. kops edit cluster $NAME
  2. set the KubernetesVersion to the target version (e.g. v1.3.5)
  3. kops update cluster rsaus-non-production.k8s.local --state s3://ratesetter-k8s-state-non-production --target=terraform --out=kubernetes/
  4. terraform plan
  5. terraform apply
  6. kops rolling-update cluster $NAME to preview, then kops rolling-update cluster $NAME --yes --force

DR/Backup

Cluster states to preserve which is not in the cluster itself:

  • Certificates

Cluster Sanity check

On the master:

$ kubectl get nodes -o wide
NAME                                               STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION   CONTAINER-RUNTIME
ip-172-16-49-31.ap-southeast-2.compute.internal    Ready     master    1h        v1.10.3   <none>        Debian GNU/Linux 8 (jessie)   4.4.121-k8s      docker://17.3.2
ip-172-16-52-202.ap-southeast-2.compute.internal   Ready     node      1h        v1.10.3   <none>        Debian GNU/Linux 8 (jessie)   4.4.121-k8s      docker://17.3.2
admin@ip-172-16-49-31:~$

On the master:

admin@ip-172-16-60-207:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Cluster Check if RBAC is enabled

On the master

ps -aef | grep -i apiserver
root      2408  2390  2 03:51 ?        00:01:51 /usr/local/bin/kube-apiserver --allow-privileged=true --anonymous-auth=false --apiserver-count=1 --authorization-mode=RBAC --basic-auth-file=/srv/kubernetes/basic_auth.csv --bind-address=0.0.0.0 --client-ca-file=/srv/kubernetes/ca.crt --cloud-provider=aws --enable-admission-plugins=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,ResourceQuota --etcd-quorum-read=false --etcd-servers-overrides=/events#http://127.0.0.1:4002 --etcd-servers=http://127.0.0.1:4001 --insecure-bind-address=127.0.0.1 --insecure-port=8080 --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP --proxy-client-cert-file=/srv/kubernetes/apiserver-aggregator.cert --proxy-client-key-file=/srv/kubernetes/apiserver-aggregator.key --requestheader-allowed-names=aggregator --requestheader-client-ca-file=/srv/kubernetes/apiserver-aggregator-ca.cert --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=443 --service-cluster-ip-range=172.16.0.0/19 --storage-backend=etcd2 --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key --token-auth-file=/srv/kubernetes/known_tokens.csv --v=2
root      2422  2408  0 03:51 ?        00:00:01 tee -a /var/log/kube-apiserver.log
admin    15426  7891  0 05:16 pts/0    00:00:00 grep -i apiserver

kubectl RAW API requests

$ kubectl get --raw https://api.rsaus-non-production.k8s.local/apis/metrics.k8s.io/v1beta1/pods/

Monitoring

Generic links:

  • kops component ports: https://github.com/kubernetes/kops/blob/master/docs/development/ports.md
  • https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-b190cc97f0f6

Monitoring Cluster Components

Node level metrics:

  • Promteheus Node exporter

Individual components export prometheus metrics:

API server

  • kubectl get --raw https://api.rsaus-non-production.k8s.local/metrics

etcd

  • curl -k http://localhost:4002/metrics

kubelet

  • curl -k https://localhost:10250/metrics
  • curl -k https://localhost:10250/metrics/cadvisor (cadvisor) metrics

amazon-vpc-cni-k8s:

  • curl http://localhost:61678/metrics

Monitoring Cluster Objects

© Amit Saha. Built using Pelican. Customised theme based on the one by Giulio Fidente on github.