Script to detect "orphaned" TLS secrets when Cert manager (cainjector) complains about "unable to fetch certificate that owns the secret", because deleting a Certificate will not (default) delete the…

This article provides a Bash script that detects and cleans up "orphaned" TLS secrets in Kubernetes—secrets that remain after their corresponding cert-manager Certificate resources have been deleted. The script addresses the cert-manager error "unable to fetch certificate that owns the secret" by identifying secrets without a matching certificate and optionally deleting them, with support for dry-run mode, namespace filtering, and verbose output.

cleanup-cert-manager-secrets.sh This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters /bin/bash inspired by https://gist.github.com/lisawolderiksen/8c6026ef55f04e4f5d6a288b5e53214b Apache 2.0 License usage { cat << EOF This script detects TLS secrets which refer to certificates that don't exist anymore . This is the case when error "unable to fetch certificate that owns the secret" occurs in cert-manager cainjector logs. The reason is that a certificate has been removed without the secret being deleted. The solution is to clean up by deleting any secret which belonged to a certificate that no longer exists. Usage: $0 -n namespace -v $0 -n | --namespace <namespace optional, default="all" -d | --dry-run <true/false optional, default=true -v | --verbose optional -h | --help displays help and exits Examples: Check the TLS secrets and matching certificates in all namespaces, print only the end result $0 Check the TLS secrets and matching certificates in the "mynamespace" namespace, print only the end result $0 -n mynamespace Check the TLS secrets and matching certificates in all namespaces, print info about all TLS secrets and certificates $0 -v Check the TLS secrets and matching certificates in all namespaces, delete all "safe" secrets $0 -d false EOF } namespace="all" dry run=true printfLog='silentPrintf' Read commandline arguments while "$1" = "" do case $1 in -n | --namespace shift namespace=$1 ;; -d | --dry-run shift if "$1" == "true" ; then dry run=true elif "$1" == "false" ; then dry run=false else echo "Invalid value for -d|--dry-run. Use 'true' or 'false'." exit 1 fi ;; -v | --verbose shift printfLog='printf' ;; -h | --help usage exit ;; usage exit 1 esac shift done Don't print a string. This method is used when -v/--verbose is NOT specified. silentPrintf { : } Function to check if a TLS certificate is still valid check tls cert expired { local secret name=$1 local namespace=$2 Extract the certificate from the secret cert data=$ kubectl get secret "$secret name" -n "$namespace" -o jsonpath='{.data.tls\.crt}' | base64 --decode if -z "$cert data" ; then echo "Secret $secret name in namespace $namespace does not contain a valid TLS certificate." return 1 fi Check if the certificate is valid using openssl if echo "$cert data" | openssl x509 -noout -text /dev/null 2 &1; then echo "Secret $secret name in namespace $namespace contains an invalid TLS certificate." return 1 fi Check if the certificate is expired expiration date=$ echo "$cert data" | openssl x509 -noout -enddate | cut -d= -f2 Convert expiration date to epoch time if "$OSTYPE" == "darwin" ; then macOS expiration epoch=$ date -j -f "%b %d %T %Y %Z" "$expiration date" "+%s" else Linux expiration epoch=$ date -d "$expiration date" "+%s" fi current epoch=$ date "+%s" if -z "$expiration epoch" ; then echo "Failed to parse expiration date for secret $secret name." return 1 fi "$current epoch" -gt "$expiration epoch" return $? } Variables to store the last queried namespace and its resources last namespace="" last namespace resources="" check tls cert referenced { local secret name=$1 local namespace=$2 Check if the resources for the current namespace are already cached if "$namespace" = "$last namespace" ; then printf "Fetching resources for namespace %s ...\n" "$namespace" Fetch all resources in the namespace and cache them last namespace resources=$ kubectl get all -n "$namespace" -o json last namespace="$namespace" fi printf "Searching for secret %s in namespace %s ...\n" "$secret name" "$namespace" resources="$last namespace resources" Check if the secret is referenced anywhere in the JSON output if echo "$resources" | grep -q "$secret name"; then printf "Secret %s is used in the namespace %s\n" "$secret name" "$namespace" return 0 else printf "Secret %s is not used in the namespace %s\n" "$secret name" "$namespace" return 1 fi } Function to check TLS secrets and certificates in a given namespace check tls secrets and certs { local namespace=$1 printf "\nProcessing namespace: %s\n" "$namespace" secrets with certificate name=$ kubectl get secret -n "$namespace" -o jsonpath="{range .items ? .type=='kubernetes.io/tls' }{.metadata.name},{.metadata.annotations.cert-manager\.io/certificate-name} {end}" $printfLog "Listing secrets of type kubernetes.io/tls with their annotated certificate:\n" for i in ${secrets with certificate name @ }; do $printfLog "$i \n" done $printfLog "\nListing certificates:\n" $printfLog "$ kubectl get certificate -n $namespace|awk '{print $1}'|grep -v "NAME" \n" secrets without certs= $printfLog "\nLooking for certificates matching the secrets:\n" for secret with cert name in ${secrets with certificate name @ }; do IFS="," read -r -a details <<< "$secret with cert name" secret name=${details 0 } cert=${details 1 } $printfLog "Secret $secret name is made using cert $cert. Looking for that cert...\n" certmatch=$ kubectl get certificate -n "$namespace" -o jsonpath="{range .items ? .metadata.name=='$cert' }{.metadata.name} {end}" if $certmatch == "" ; then secrets without certs+= "$secret name" $printfLog "\n WARNING: Secret $secret name has no matching certificate $cert . \n\n" else $printfLog "Certificate found.\n" fi done Just because there is no associated Certificate doesn't mean it's not in use Other services could create secrets without associated Certificates that are not managed by cert-manager, sources could be from helm installs/nginx ingress expired secrets without certs= secrets seemingly unused= secrets in use= $printfLog "\nChecking secrets in namespace $namespace that have no matching certificates:\n" for secret in ${secrets without certs @ }; do A simple safety check to see if the tls cert is expired if check tls cert expired "$secret" "$namespace"; then expired secrets without certs+= "$secret" We can also check if the secret is used/reference anywhere like the k9s functionality elif check tls cert referenced "$secret" "$namespace"; then secrets seemingly unused+= "$secret" else secrets in use+= "$secret" fi done if ${ secrets in use @ } -ne 0 ; then printf "\nSecrets still in use with no matching certificates should not be deleted :\n" for secret in ${secrets in use @ }; do printf "%s \n" "$secret" done fi if ${ secrets seemingly unused @ } -ne 0 ; then printf "\nSecrets probably not in use with no matching certificates possibly could be deleted :\n" if "$dry run" = true ; then for secret in ${secrets seemingly unused @ }; do printf "%s \n" "$secret" done else for secret in ${secrets seemingly unused @ }; do kubectl delete secret -n "$namespace" "$secret" done fi fi if ${ expired secrets without certs @ } -ne 0 ; then printf "\nExpired Secrets with no matching certificates probably can be deleted :\n" if "$dry run" = true ; then for secret in ${expired secrets without certs @ }; do printf "%s \n" "$secret" done else for secret in ${expired secrets without certs @ }; do kubectl delete secret -n "$namespace" "$secret" done fi fi } if "$namespace" == "all" ; then namespaces=$ kubectl get namespaces -o jsonpath='{.items .metadata.name}' for ns in $namespaces; do check tls secrets and certs "$ns" done else check tls secrets and certs "$namespace" fi