TIL: Loki LogQL Pattern Matching for uWSGI Logs

Suppose you have log lines like this as uWSGI generates them:

[pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar  7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)

First, let’s show the logs in a “Logs” view and filter them:

{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"

Extract fields by using the pattern parser [1]:

{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"
# [pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar  7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)
| pattern `[<_>] <_> () {<_>} [<_>] <method> <path> => <_> (HTTP/1.1 <status>)`

You can now expand the line inside the explore view. There you will see the extracted log labels method, path and status.

Finally, format the lines using the extracted fields:

{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"
# [pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar  7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)
| pattern `[<_>] <_> () {<_>} [<_>] <method> <path> => <_> (HTTP/1.1 <status>)`
| line_format `{{ .method }} {{ .path }} {{ .status }}`

This results in nicely readable output:

GET /foo 200

Bonus

To filter by HTTP 4xx and 5xx error codes, you can use the following hack:

{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"
# [pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar  7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)
| pattern `[<_>] <_> () {<_>} [<_>] <method> <path> => <_> (HTTP/1.1 <status>)`
| line_format `{{ .method }} {{ .path }} {{ .status }}`
# poor man's "status >= 400" follows, because there are no conditionals in golang's text/template 
# levels with color code: https://grafana.com/docs/grafana/latest/explore/logs-integration/#log-level
# if / else with hasPrefix: https://stackoverflow.com/a/69976898/241240
| label_format level=`{{ if hasPrefix "4" .status }}error{{else if hasPrefix "5" .status}}error{{else}}info{{end}}`
| level = "error"

---

[1]

https://grafana.com/blog/2021/08/09/new-in-loki-2.3-logql-pattern-parser-makes-it-easier-to-extract-data-from-unstructured-logs/

TIL: Prometheus file_sd_configs to Organize Static Scrape Configs   Python Ellipsis