# Prometheus

## Installing Prometheus and Grafana
This is based on https://eksworkshop.com/monitoring/deploy-prometheus/

Define SSL certificate for Grafana:
```bash
cat <<'END_OF_GRAFANA_TF' > grafana.tf
module "grafana_cert" {
  source         = "./modules/validated_acm_cert/"
  zone_name      = "example.com"
  subdomain_name = "grafana"
  tags = {
    Name = "Grafana"
  }
}

# SSL Certificate for Grafana
output "grafana_helm_values" {
  value = <<EOF
persistence:
  enabled: true
  storageClassName: "gp2"

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-server.prometheus.svc.cluster.local
        access: proxy
        isDefault: true

service:
  type: LoadBalancer
  port: 443
  annotations:
    # https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ${module.grafana_cert.cert.arn}
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: service
EOF
}
END_OF_GRAFANA_TF
```

Make sure that your workspace is correct:
```
WS=
export KUBECONFIG=~/.kube/foo_${WS}.yml
terraform workspace select ${WS}
terraform apply
```

```
kubectl create namespace prometheus
helm install stable/prometheus \
    --name prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"
kubectl get all -n prometheus
kubectl port-forward -n prometheus deploy/prometheus-server 9090:9090
# check http://localhost:9090/targets
```

Install Grafana
```
terraform output grafana_helm_values > /tmp/grafana_helm_values.yml
view /tmp/grafana_helm_values.yml  # review
helm upgrade --install grafana stable/grafana \
    --namespace grafana \
    --values /tmp/grafana_helm_values.yml
watch -dn1 kubectl -n grafana get all
```

Get Grafana admin password
```
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

Point Grafana ELB to `grafana.example.com`:
```
# wait until EXTERNAL-IP is set:
watch kubectl -n grafana get service/grafana
elb=$(kubectl -n grafana get service/grafana -ojsonpath='{ .status.loadBalancer.ingress[0].hostname }')
echo $elb | clipboard
```

Go to https://console.aws.amazon.com/route53/home?#hosted-zones:
and add manually:

- name: grafana
- type: A
- Alias: yes
- ttl: 300
- value: `<paste>`

After a little while you can log in to https://grafana.example.com 


## Monitoring Pods in k8s using Prom Helm Chart
Simply annotate your pods. `port` and `path` are optional.
```
prometheus.io/scrape: 'true'
prometheus.io/port: '8000'
prometheus.io/path: '/metrics'
```

Example Deployment:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo
  labels:
    app: foo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: foo
  template:
    metadata:
      labels:
        app: foo
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8000'
        prometheus.io/path: '/metrics'
    spec:
      restartPolicy: Always
      containers:
        - name: foo
          image: fooimage:d34db33f
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
              name: "http"
          readinessProbe:
            exec:
              command:
              - ./is_ready.sh
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: "/healthz"
              port: "http"
            periodSeconds: 10
            initialDelaySeconds: 30
```

Source: https://github.com/helm/charts/blob/af1e617/stable/prometheus/values.yaml#L1178