Skip to content

What is LogQL

LogQl is a query language developed by Grafana labs. It allows you to filter and search across your log content and can also generate metric data based on your log data.

Logs can be both read historically and streamed as they are ingested.

You can read more on LogQL, its syntax and functions in the LogQL online documentation.

LogQL examples

Search for Logs with a specific job

{job="job1"}

Search for specific filename

{filename=~"request"}

Search for logs with a job using wildcard

{job=~"job.*"}

Search for any stdout or stderr logs for any jobs matching the wildcard

{job=~"job.*",filename=~"std.*"}

Query for the volume of jobs per job and instance

sum(count_over_time({job=~".+"}[5m])) by (job,instance)

Query for the top IP addresses triggering requests in FusionReactor

topk(10,sum by (clientAddress)(rate({filename="request"} | logfmt | line_format "{{.clientAddress}}" | __error__="" [1m])))

Query nginx for the top IP address hitting the load balancer

topk(10,sum by (remote_address)(rate({filename="/opt/access.log"} | logfmt | line_format "{{.remote_address}}" | __error__="" [1m])))

Search all logs for the text Exception { job=~".+"} |= "Exception"

Process the request log for top hit page URLs

sum by (url)(rate({filename="request"} | logfmt | line_format "{{.url}}" | __error__="" [10s]))

Graph the number of Error or Exception error lines

sum by (job) (count_over_time({job=~"store-.*"} |= "error" != "exception" [5m]))

Process the avg CPU time per page URL

topk(10, sum by (url)(avg_over_time({ filename="request"} | logfmt | __error__="" | unwrap cpuTime[5m])))

Graph previous crashes caused by OutOfMemory Errors

sum by (job) (count_over_time({job=~".+"} |= "java.lang.OutOfMemory" [1m]))

Labels and Fields

When shipping logs to FusionReactor Cloud, you can apply labels to them. These labels can be used to filter your log content using LogQL.

Default labels applied by FusionReactor include job, instance, app_name and filename but are not limited to these specific entries. If you ship logs through FusionReactor a label for the instance will be applied, and if you ship your logs through an alternate log client you can specify any labels you require.

If you are shipping logs from outside of the FusionReactor agent, it is important to ensure the labels job and instance are applied o the logs. Job is typically the name of the service you are running and instance will be the host the service is running on. These labels allow you to isolate your LogQL queries to a selection of your online logging clients, without these you may have difficulty differentiating where logs are coming from.

In order to maintain performance when querying your data you should avoid high cardinality labels (such as having labels with similar names but incorporating a unique ID). You read more on why this is in the loki documentation.

Fields are derived from the log content using key value pairs (“key=value”); these fields will give you the ability to collate log entries into a graph.

Warning

LogQL has a lot of power and can run and aggregate complex queries on your data, similar to SQL though bad queries can result in an expensive and inefficient runtime.

FusionReactor Cloud has protections in place to protect against as queries taking too long or using excessive resources, this may result in some queries failing.

If this occurs, consider reducing the timeframe for your query, or potentially optimizing your query so it can be more efficient.

What is Logfmt

Logfmt is a log parser that allows FusionReactor Cloud to collate metric data from log fields that are represented as key=value data pairs.

When sending your logs you will gain greater value by using logfmt. FusionReactor logs themselves will be converted to logfmt when ingested.

You can read more on logfmt in documentation.