We were trying to integrate Spring Cloud Gateway running on K8S with HashiCorp Vault. Wanted to share info how we resolved these issues. Issue 1: [ERROR] auth.kubernetes.auth_kubernetes: login unauthorized due to: Post “https://10.0.0.:6443/apis/authentication.k8s.io/v1/tokenreviews”: x509: certificate signed by unknown authority Solution: We used following kubernetes auth config to authenticate client to Continue Reading
Kubernetes
Troubleshooting HashiCorp Vault Kubernetes Auth Error
We were trying to integrate Spring Cloud Gateway (SCG) on Kubernetes with HashiCorp. We followed the steps mentioned in vault documentation. We were able to bring up vault and vault injector but SCG pods were stuck in init state. Found following error in SCG application pod vault-agent-init container logs: NAME Continue Reading
Wavefront Alert Queries for Kubernetes Monitoring
Recently, we have setup monitoring of Kubernetes using wavefront. Here are some of the useful alerts and their queries: POD Memory Utilization:ts(“kubernetes.pod.memory.working_set”, namespace_name=”xyz”)/ts(“kubernetes.pod.memory.limit”, namespace_name=”xyz”) * 100 Kong POD CPU Utilization:ts(“kubernetes.pod.cpu.usage_rate”, namespace_name=”xyz”)/ts(“kubernetes.pod.cpu.limit”, namespace_name=”xyz”) * 100 Kong Replica Count Mismatch:ts(“kubernetes.deployment.desired_replicas”, namespace_name=”xyz”) – ts(“kubernetes.deployment.available_replicas” and namespace_name=”xyz”) New Pod Created/Pod Deleted:highpass(0, ts(“kubernetes.pod.uptime”, namespace_name=”xyz”) < Continue Reading
Issues Faced during rebuilding a Kubernetes Cluster
We had a Kubernetes POC cluster with version 1.18. This cluster got corrupted during experimentation by our team. Instead of starting with clean slate, which is comparatively easy, we tried to rebuild cluster with existing kubelet, etcd, kubeadm. This post is basically my own reference/notes of the issues faced and Continue Reading
Useful bashrc files
We as sysadmin spend lot of time at command line and most of the time we use few regular commands like going to logs directory, searching history. I have prepared bash aliases that can be used to save few keystrokes for various applications like weblogic, kubernetes, Boomi etc. And these Continue Reading
It is DNS again !!! Kong API was stopped in its tracks
We have a two node Kubernetes cluster with one master and one worker node. Kong api is installed as containers on this cluster. Log files in our Kong API server were located in root directory of the host machine. As the number of logs were getting increased, there was a Continue Reading
Kong API Installation Failure due to Cassandra Config
We have Kong API running on Kubernetes. It uses Cassandra to store meta data. The setup used in production is clustered one but due to cost consideration we were asked to build a POC environment with single node. We were given two VMs – one to run Kubernetes control plane Continue Reading