TIL: Loki LogQL Pattern Matching for uWSGI Logs
Suppose you have log lines like this as uWSGI generates them:
[pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar 7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)
First, let’s show the logs in a „Logs“ view and filter them:
{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"
Extract fields by using the pattern parser 1:
{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"
# [pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar 7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)
| pattern `[<_>] <_> () {<_>} [<_>] <method> <path> => <_> (HTTP/1.1 <status>)`
You can now expand the line inside the explore view.
There you will see the extracted log labels method, path and status.
Finally, format the lines using the extracted fields:
{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"
# [pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar 7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)
| pattern `[<_>] <_> () {<_>} [<_>] <method> <path> => <_> (HTTP/1.1 <status>)`
| line_format `{{ .method }} {{ .path }} {{ .status }}`
This results in nicely readable output:
GET /foo 200
Bonus
To filter by HTTP 4xx and 5xx error codes, you can use the following hack:
{instance="prod",compose_service="core"}
|= "generated"
!= "/metrics"
# [pid: 13|app: 0|req: 8331/12326] 172.20.0.11 () {34 vars in 999 bytes} [Tue Mar 7 16:08:57 2023] GET /foo => generated 4289 bytes in 45 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1)
| pattern `[<_>] <_> () {<_>} [<_>] <method> <path> => <_> (HTTP/1.1 <status>)`
| line_format `{{ .method }} {{ .path }} {{ .status }}`
# poor man's "status >= 400" follows, because there are no conditionals in golang's text/template
# levels with color code: https://grafana.com/docs/grafana/latest/explore/logs-integration/#log-level
# if / else with hasPrefix: https://stackoverflow.com/a/69976898/241240
| label_format level=`{{ if hasPrefix "4" .status }}error{{else if hasPrefix "5" .status}}error{{else}}info{{end}}`
| level = "error"