Skip to main content

Mutual authentication between microservices in kubernetes cluster

Earlier in this post, we have talked about TLS Working model and configuring it for application in kubernetes. Basically any HTTPS server open for client connections, will present a server certificate to client to verify against its trusted certificate authorities and if success, it does basic TLS handshake. Its more of a validating whether sever domain is authentic, using server certificate.

What if there is a security requirement where client needs a valid certificate before it access server content, this is where mutual authentication fits in. Its basically establishing secure encrypted communication between two parties and authenticity of each party will be verified at other party end with presented certificate against certificate authority. The following diagram demonstrates steps involved in mutual authentication



1. Client requests sever for its content
2. Server replies back by presenting its server certificate
3. Server's identity will be tested by client using its trusted certificate authorities (digital ca certs are normally pre loaded for browser type of clients)
4. Now client provides its certificate to server
5. Server verifies client certificate is valid or not by using  its trusted certificate authorities
6. Once its success, it establishes encrypted channel between server and client

Prerequisites:

* Assuming kubernetes cluster is already set up with ingress controller and openssl command installed
 
Steps:

1. Generate self signed certificate authority certificate and key
openssl req -new -x509 -days 365 -keyout ca-key.pem -out ca-crt.pem

2. Generate server key
openssl genrsa -out server-key.pem 4096

3. Now generate certificate signing request for server using its private key, provide all the configurations, subject alternative names and details in san.cnf. Ensure server hosted address is matching with either of common name or san fields in config file


openssl req -new -sha256 -key server-key.pem -out server-csr.pem -config san.cnf

Sample san.cnf file listed below:

4.  Now lets sign server certificate, using ca certs and server csr file

openssl x509 -req -days 365 -in server-csr.pem -CA ca-crt.pem -CAkey ca-key.pem -CAcreateserial -out server-crt.pem -extfile san.cnf -extensions req_ext

5. Perform steps 2,3 and 4 again for client certificates, if client can be accessed with external address, then use san.cnf


* openssl genrsa -out clientA-key.pem 4096
* openssl req -new -sha256 -key clientA-key.pem -out clientA-csr.pem (if SAN need to be included, add -config san.cnf)
* openssl x509 -req -days 365 -in clientA-csr.pem -CA ca-crt.pem -CAkey ca-key.pem -CAcreateserial -out clientA-crt.pem (if SAN need to be included, add -extfile san.cnf -extensions req_ext, here req_ext should be getting from req_extensions field in san.cnf)


6. Lets verify whether certificates are properly generated and also check mutual authentication
Following command runs server with its certificates on port number 8765

openssl s_server -cert ./server-crt.pem -key ./server-key.pem -CAfile ./ca-crt.pem -accept 8765 -Verify on -verify_return_error

In another tab, run following command to run client with certs and connect to server

openssl s_client -connect localhost:8765 -cert ./clientA-crt.pem -key ./clientA-key.pem -CAfile ./ca-crt.pem -verify on -verify_return_error If status is connected and its hung means everything working fine. Connection will be dropped in case of problem with certificates.

Mutual authentication between two Microservices running in kubernetes cluster - Demo:

For simplicity purpose, lets run nginx as server and mutual authentication, ssl termination will be done at kubernetes Ingress level. We will run small Go client code to access server with valid certs.

* Lets create a kubernetes certificate for ingress to verify mutual auth and to terminate ssl

kubectl create secret generic nginx-tls-certs --from-file=tls.crt=server-crt.pem --from-file=tls.key=server-key.pem --from-file=ca.crt=ca-crt.pem

* Now deploy nginx and expose it via service

kubectl run nginx --image=nginx --port=80; kubectl expose deploy/nginx

Create kubernetes ingress for nginx application with ssl termination and mutual auth verfication. Copy the following content to a yaml and apply via kubectl apply -f ingress.yaml
* Now lets access the URl without any cert(browser or from any program). Make an /etc/hosts entry for nginx.testdomain.com, matching to kubernetes node ip, where ingress controller running.

 curl -k -v https://nginx.testdomain.com

 496 SSL Certificate Required
A client certificate must be provided.

* Lets access this url from go client program with proper certs. copy following code to a go file and run go run test.go, we will be able to access server content (nginx home page)

Comments

Post a Comment

Popular posts from this blog

Automation scripts for Admins to run on Jenkins Script Console

Jenkins script console is one of the key feature for administrators to automate tasks, troubleshoot on server side like cleaning up older job builds, deleting/creating bunch of jobs, setting up log rotation on all jobs etc. It's hard and monotonous for them to do this tasks from UI repeatedly. The following scripts certainly helps to maintain and operate Jenkins efficiently.

Release and Tagging strategy for Microservices running on Kubernetes

Releasing product is one of the critical phase of software development life cycle, so much brainstorming will go in to design versioning and release strategy for a product. In olden days, it was all different, applications were monolithic, build and release management teams used to build this, entirely as a package (.WAR or .EAR files) by creating one version and putting it in artifact repository, triggering a process to deploy package in application servers. Versioning is pretty straight forward because it is monolithic and treated as one application. Now things are changed, considering complexity and inflexibility of monolithic application, people has opted to build application as set of smaller and interconnected services called microservices. In effect of this, versioning and releasing of individual microservices and for overall application became complex. There are no straight forward principles which solves problem, it entirely depends on various factors like branching strateg

Insights about Certified Kubernetes Administrator(CKA)

Attending certification exams is one of the way to test our skills and to check whether we are moving in right direction towards understanding technology properly. I have taken CKA exam in December 2018 and passed it with 92%. Overall it was very good experience, it's not that tough if you are already working on kubernetes clusters. Major advantage of taking up CKA is we will be able to cover almost all kubernetes concepts, core components and features etc during preparation. Normally in day to day work, we do not get opportunity to cover everything. Experiences: As a first step, I have read CKA experiences in Internet, it certainly gave me an idea how to prepare and how exam pattern looks like. Sharing some of the links. https://linuxacademy.com/community/posts/show/topic/25094-cka-exam-experience-and-some-useful-tips https://suraj.pro/post/journey-to-cka/ https://suraj.io/post/road-to-cka/ https://medium.com/@walidshaari/kubernetes-certified-administrator-cka-43a25ca4c61