Creating custom kubeconfig with limited access to K8s cluster

In this blog, we will go step by step on how to create a custom kubeconfig file with limited access to kubernetes cluster using service account, secret token and RBAC

These are the mulitple ways to authenticate with Kubernetes:

Note: For this demo I am using AWS EKS cluster, but the same applies to Kubernetes running anywhere. Moreover, this lab assumes that you have access to create, manage and view ClusterRole, Role, ClusterRoleBinding, RoleBinding, ServiceAccount and Secret API resources.

Let's start with creating custom RBAC role for our users.

There are two categories of it:

ClusterRole API resource as the name says is used to provide access at Cluster level where as with Role you can restrict access at namespace level.

We will create both ClusterRole and Role in this demo for our users Adam and Bill.

Adam is our Senior DevOps Engineer and is working on mulitple projects. Hence, will create ClusterRole for him which will not restricting his permission at namespace level but at resource level.

Bill is a Junior DevOps Engineer whi has recently joined the team and is working on Brown Fox project hence, will create a Role for him to restrict his permission both at namespace and resource level.

ClusterRole for Adam:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: adam
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "namespaces", "nodes"]
    verbs: ["create", "get", "update", "list", "watch", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployment"]
    verbs: ["create", "get", "update", "list", "delete", "watch", "patch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterroles", "clusterrolebindings"]
    verbs: ["create", "get", "list", "watch"]

Role for Bill:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bill
  namespace: brown-fox
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "namespaces", "nodes"]
    verbs: ["create", "get", "update", "list", "watch", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployment"]
    verbs: ["create", "get", "update", "list", "delete", "watch", "patch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterroles", "clusterrolebindings"]
    verbs: ["create", "get", "list", "watch"]

Let us now bind ClusterRole to a Adam using ClusterRoleBinding API resource.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: adam
subjects:
  - kind: ServiceAccount
    name: adam-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: adam
  apiGroup: rbac.authorization.k8s.io

For Bill, we will be using RoleBinding API resource.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bill
  namespace: brown-fox
subjects:
  - kind: ServiceAccount
    name: bill-sa
    namespace: brown-fox
roleRef:
  kind: Role
  name: bill
  apiGroup: rbac.authorization.k8s.io

We are now required to create Service Account for both Adam and Bill which are attached to their roles we create above.

Service Account for Adam:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: adam-sa
  namespace: kube-system
secrets:
  - name: adam-secret

Service Account for Bill:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: bill-sa
  namespace: brown-fox
secrets:
  - name: bill-secret

Now we need to create tokens using Secret API resource which will be used in our kubeconfig file.

Secret for Adam:

apiVersion: v1
kind: Secret
metadata:
  name: adam-secret
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: adam-sa
type: kubernetes.io/service-account-token

Secret for Bill:

apiVersion: v1
kind: Secret
metadata:
  name: bill-secret
  namespace: brown-fox
  annotations:
    kubernetes.io/service-account.name: bill-sa
type: kubernetes.io/service-account-token

Finally, let us generate kubeconfig file for Adam and Bill.

Sample File:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: {cluster-ca}
    server: {server-dns}
  name: {cluster-name}
contexts:
- context:
    cluster: {cluster-name}
    user: {user-name}
  name: {context-name}
current-context: {context-name}
kind: Config
users:
- name: {user-name}
  user:
    token: {secret-token}

We need to replace all the values within curly braces with their actual values, so let's keep going.

Type the below command to fetch Certificate and Server information:

kubectl config view --flatten --minify

Copy certificate-authority-data, server and name field from the output and replace them with their respective fields in the sample file.

It's now time to grab the secret token. Use the following command:

# Fetching Adam's secret
kubectl get secrets -n kube-system

# Copy the token name starting adam-secret
kubectl describe secrets {token-name} -n kube-system

# Fetching Bill's secret
kubectl get secrets -n brown-fox

# Copy the token name starting bill-secret
kubectl describe secrets {token-name} -n brown-fox

Copy the token value and paste it in the sample file. The kubeconfig file is now complete and should look like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0...............Sk1TTG5Ndk1yNmYzdFNl
    server: https://01F4..............BEA3.sk1.ap-south-1.eks.amazonaws.com
  name: arn:aws:eks:ap-south-1:006378141167:cluster/brown-fox-cluster
contexts:
- context:
    cluster: arn:aws:eks:ap-south-1:006378141167:cluster/brown-fox-cluster
    user: adam
  name: arn:aws:eks:ap-south-1:006378141167:cluster/brown-fox-cluster
current-context: arn:aws:eks:ap-south-1:006378141167:cluster/brown-fox-cluster
kind: Config
users:
- name: adam
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJ...............SNFZmRdIfU

To test custom kubeconfig file type the following command:

kubectl get pods --kubeconfig {kubeconfig_file}

You must receive output with Adam's kubeconfig file but not with Bill's kubeconfig file.

Bingo! You have created kubeconfig file for Adam and Bill with limited access to Kubernetes cluster.