Prometheus Operator install

Prometheus operator allows setting up and configuring Prometheus servers running in your cluster. As part of the helm installation we also set up by default ServiceMonitor and PodMonitor objects which define how to scrape our services.

If your cluster doesn’t have Prometheus operator installed already, you can install one from our repository or the official helm, depending on your setup. Note that, unlike the official helm, Tabnine’s version doesn’t install Prometheus server by default (.prometheus.enabled is set to false). If you opt-in to install the prometheus server as part of the kube-prometheus-stack - either from the official helm or by setting .prometheus.enabled=true in our chat’s values, you will need to fine-tune the server configuration in the helm chart and not in the Prometheus server object shown later on, as it will be created by the helm chart.

helm upgrade --install --create-namespace -n monitoring monitoring oci://registry.tabnine.com/self-hosted/kube-prometheus-stack

Check if there is an installed operator in your cluster

If you are unsure if the operator is in your cluster, you can run the following commands

# Make sure you have the relevant CRDs installed
$ kubectl get crd | grep monitoring.coreos.com
alertmanagerconfigs.monitoring.coreos.com        2022-12-08T13:15:57Z
alertmanagers.monitoring.coreos.com              2022-12-08T13:15:57Z
podmonitors.monitoring.coreos.com                2022-12-08T13:15:58Z
probes.monitoring.coreos.com                     2022-12-08T13:15:58Z
prometheuses.monitoring.coreos.com               2022-12-08T13:15:59Z
prometheusrules.monitoring.coreos.com            2022-12-08T13:15:59Z
servicemonitors.monitoring.coreos.com            2022-12-08T13:16:00Z
thanosrulers.monitoring.coreos.com               2022-12-08T13:16:00Z

# Make sure there is a prometheus operator running. Note that depening on the helm installation the name might be slightly different.
$ kubectl get pods -A | grep operator
monitoring                     kube-prometheus-stack-operator-XX

Check if you already have Prometheus server in your cluster

$ kubectl get prometheus -A
NAMESPACE    NAME
monitoring   kube-prometheus-stack-prometheus

Note that if you have enabled telemetry as part of Tabnine installation, you will see a Prometheus server created by Tabnine. That server is used for remote-writing metrics to Tabnine and doesn’t persist data locally. If that is the only server you see in the list, or there are none, you can create a server based on the example below.

If you have a Prometheus server, It might be configured the collect data only from Pod/ServiceMontors with specific labels and/or namespaces. Run the following command (based on the example output above, this might be different in your environment) and check the output of the following fields

• Pod Monitor Namespace Selector

• Pod Monitor Selector

• Service Monitor Namespace Selector

• Service Monitor Selector

In its default setup kube-prometheus-stack requires some labels to be present on the monitor objects for them to be propagated to the Prometheus server. If that is the case, write down the required labels - we will use them below