Skip to main content

Mutual authentication between microservices in kubernetes cluster

Earlier in this post, we have talked about TLS Working model and configuring it for application in kubernetes. Basically any HTTPS server open for client connections, will present a server certificate to client to verify against its trusted certificate authorities and if success, it does basic TLS handshake. Its more of a validating whether sever domain is authentic, using server certificate.

What if there is a security requirement where client needs a valid certificate before it access server content, this is where mutual authentication fits in. Its basically establishing secure encrypted communication between two parties and authenticity of each party will be verified at other party end with presented certificate against certificate authority. The following diagram demonstrates steps involved in mutual authentication



1. Client requests sever for its content
2. Server replies back by presenting its server certificate
3. Server's identity will be tested by client using its trusted certificate authorities (digital ca certs are normally pre loaded for browser type of clients)
4. Now client provides its certificate to server
5. Server verifies client certificate is valid or not by using  its trusted certificate authorities
6. Once its success, it establishes encrypted channel between server and client

Prerequisites:

* Assuming kubernetes cluster is already set up with ingress controller and openssl command installed
 
Steps:

1. Generate self signed certificate authority certificate and key
openssl req -new -x509 -days 365 -keyout ca-key.pem -out ca-crt.pem

2. Generate server key
openssl genrsa -out server-key.pem 4096

3. Now generate certificate signing request for server using its private key, provide all the configurations, subject alternative names and details in san.cnf. Ensure server hosted address is matching with either of common name or san fields in config file


openssl req -new -sha256 -key server-key.pem -out server-csr.pem -config san.cnf

Sample san.cnf file listed below:

4.  Now lets sign server certificate, using ca certs and server csr file

openssl x509 -req -days 365 -in server-csr.pem -CA ca-crt.pem -CAkey ca-key.pem -CAcreateserial -out server-crt.pem -extfile san.cnf -extensions req_ext

5. Perform steps 2,3 and 4 again for client certificates, if client can be accessed with external address, then use san.cnf


* openssl genrsa -out clientA-key.pem 4096
* openssl req -new -sha256 -key clientA-key.pem -out clientA-csr.pem (if SAN need to be included, add -config san.cnf)
* openssl x509 -req -days 365 -in clientA-csr.pem -CA ca-crt.pem -CAkey ca-key.pem -CAcreateserial -out clientA-crt.pem (if SAN need to be included, add -extfile san.cnf -extensions req_ext, here req_ext should be getting from req_extensions field in san.cnf)


6. Lets verify whether certificates are properly generated and also check mutual authentication
Following command runs server with its certificates on port number 8765

openssl s_server -cert ./server-crt.pem -key ./server-key.pem -CAfile ./ca-crt.pem -accept 8765 -Verify on -verify_return_error

In another tab, run following command to run client with certs and connect to server

openssl s_client -connect localhost:8765 -cert ./clientA-crt.pem -key ./clientA-key.pem -CAfile ./ca-crt.pem -verify on -verify_return_error If status is connected and its hung means everything working fine. Connection will be dropped in case of problem with certificates.

Mutual authentication between two Microservices running in kubernetes cluster - Demo:

For simplicity purpose, lets run nginx as server and mutual authentication, ssl termination will be done at kubernetes Ingress level. We will run small Go client code to access server with valid certs.

* Lets create a kubernetes certificate for ingress to verify mutual auth and to terminate ssl

kubectl create secret generic nginx-tls-certs --from-file=tls.crt=server-crt.pem --from-file=tls.key=server-key.pem --from-file=ca.crt=ca-crt.pem

* Now deploy nginx and expose it via service

kubectl run nginx --image=nginx --port=80; kubectl expose deploy/nginx

Create kubernetes ingress for nginx application with ssl termination and mutual auth verfication. Copy the following content to a yaml and apply via kubectl apply -f ingress.yaml
* Now lets access the URl without any cert(browser or from any program). Make an /etc/hosts entry for nginx.testdomain.com, matching to kubernetes node ip, where ingress controller running.

 curl -k -v https://nginx.testdomain.com

 496 SSL Certificate Required
A client certificate must be provided.

* Lets access this url from go client program with proper certs. copy following code to a go file and run go run test.go, we will be able to access server content (nginx home page)

Comments

Post a Comment

Popular posts from this blog

Insights about Certified Kubernetes Administrator(CKA)

Attending certification exams is one of the way to test our skills and to check whether we are moving in right direction towards understanding technology properly. I have taken CKA exam in December 2018 and passed it with 92%. Overall it was very good experience, it's not that tough if you are already working on kubernetes clusters. Major advantage of taking up CKA is we will be able to cover almost all kubernetes concepts, core components and features etc during preparation. Normally in day to day work, we do not get opportunity to cover everything. Experiences: As a first step, I have read CKA experiences in Internet, it certainly gave me an idea how to prepare and how exam pattern looks like. Sharing some of the links. https://linuxacademy.com/community/posts/show/topic/25094-cka-exam-experience-and-some-useful-tips https://suraj.pro/post/journey-to-cka/ https://suraj.io/post/road-to-cka/ https://medium.com/@walidshaari/kubernetes-certified-administrator-cka-43a25ca4c61...

Generate and configure TLS certs for an application in Kubernetes at Ingress level

TLS basic working model:  Secure communication between the two entities is paramount in today's digital world. TLS provides the secure end to end communication between two machines over a network. Classic example of TLS model is accessing www.google.com(server) via browser(client). Following basic steps are involved * Client makes a call, when URL was entered in browser * Server sends its certificate * Browser will check any Root CA present for server certificate, if its present it will validate via digital sign. If not it asks user to trust the certificate, happens when we use self signed certificates. *  Client sends its certificates, will be verified at server, its optional * Once verification is done, it starts exchanging encrypting messages, which will be only decrypted using certs and keys at their end. Generating TLS certs: There are many tools in market through which we can generate certs. Openssl is one of the widely used opensource tool, we will ...

Automation scripts for Admins to run on Jenkins Script Console

Jenkins script console is one of the key feature for administrators to automate tasks, troubleshoot on server side like cleaning up older job builds, deleting/creating bunch of jobs, setting up log rotation on all jobs etc. It's hard and monotonous for them to do this tasks from UI repeatedly. The following scripts certainly helps to maintain and operate Jenkins efficiently.