Prometheus

Installing Prometheus and Grafana

This is based on https://eksworkshop.com/monitoring/deploy-prometheus/

Define SSL certificate for Grafana:

cat <<'END_OF_GRAFANA_TF' > grafana.tf
module "grafana_cert" {
  source         = "./modules/validated_acm_cert/"
  zone_name      = "example.com"
  subdomain_name = "grafana"
  tags = {
    Name = "Grafana"
  }
}

# SSL Certificate for Grafana
output "grafana_helm_values" {
  value = <<EOF
persistence:
  enabled: true
  storageClassName: "gp2"

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-server.prometheus.svc.cluster.local
        access: proxy
        isDefault: true

service:
  type: LoadBalancer
  port: 443
  annotations:
    # https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ${module.grafana_cert.cert.arn}
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: service
EOF
}
END_OF_GRAFANA_TF

Make sure that your workspace is correct:

WS=
export KUBECONFIG=~/.kube/foo_${WS}.yml
terraform workspace select ${WS}
terraform apply

kubectl create namespace prometheus
helm install stable/prometheus \
    --name prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"
kubectl get all -n prometheus
kubectl port-forward -n prometheus deploy/prometheus-server 9090:9090
# check http://localhost:9090/targets

Install Grafana

terraform output grafana_helm_values > /tmp/grafana_helm_values.yml
view /tmp/grafana_helm_values.yml  # review
helm upgrade --install grafana stable/grafana \
    --namespace grafana \
    --values /tmp/grafana_helm_values.yml
watch -dn1 kubectl -n grafana get all

Get Grafana admin password

kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Point Grafana ELB to grafana.example.com:

# wait until EXTERNAL-IP is set:
watch kubectl -n grafana get service/grafana
elb=$(kubectl -n grafana get service/grafana -ojsonpath='{ .status.loadBalancer.ingress[0].hostname }')
echo $elb | clipboard

Go to https://console.aws.amazon.com/route53/home?#hosted-zones: and add manually:

• name: grafana

• type: A

• Alias: yes

• ttl: 300

• value: <paste>

After a little while you can log in to https://grafana.example.com

Monitoring Pods in k8s using Prom Helm Chart

Simply annotate your pods. port and path are optional.

prometheus.io/scrape: 'true'
prometheus.io/port: '8000'
prometheus.io/path: '/metrics'

Example Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo
  labels:
    app: foo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: foo
  template:
    metadata:
      labels:
        app: foo
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8000'
        prometheus.io/path: '/metrics'
    spec:
      restartPolicy: Always
      containers:
        - name: foo
          image: fooimage:d34db33f
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
              name: "http"
          readinessProbe:
            exec:
              command:
              - ./is_ready.sh
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: "/healthz"
              port: "http"
            periodSeconds: 10
            initialDelaySeconds: 30

Source: helm/charts