PromQL (Prometheus Query Language) is a built in query language for Prometheus. It is a powerful functional expression language, which lets you filter with Prometheus’ multi-dimensional time-series labels. The result of each expression can be shown either as a graph, viewed as tabular data or consumed via the HTTP API.
Prometheus Data Types
- Instant Vector : An ‘Instant Vector’ is a set of time series containing a single sample for each time series and all sharing the same timestamp. In other words, an instant vector is a set of zero, or more time series, where each time series will have one sample and a sample contains both a value and a timestamp.
- Range Vector : A range vector is a set of time series containing a range of data points over time for each time series. Unlike an instant vector which returns one sample per time series, a range vector can return many samples for each time series.
- Scalar : Scalars are single numbers with no dimensionality. They are just a simple numeric floating-point value that do not have any labels, they are just numbers. e.g. 15.21
- String : String is simply a string value without any labels. e.g. Hello
Selectors and Matchers
In Prometheus server you may have hundreds of different time series with different label values. Working with all these time series can be confusing if the metrics are coming from different targets.
The query ‘process_cpu_seconds_total’ will return us total user and system CPU time spent in seconds.
If you are working on node_exporter target, and you don’t want to see the metrics coming from any target jobs other than node exporter, then you can write the query with a filter. For the query process_cpu_seconds_total, we can apply filters by appending a set of curly braces and passing the label name and value.
Matcher Types
- Equality matcher (=) : Select labels that are exactly equal to the provided string.
e.g. process_cpu_seconds_total{job=’node_exporter’} - Negative Equality matcher (!=) : Select labels that are not equal to the provided string.
e.g. process_cpu_seconds_total{job!=’node_exporter’} - Regular Expression matcher (=~) : Select labels that regex-match with the provided string.
e.g. prometheus_http_requests_total{handler=~”/api.*”}
In the example the dot and * specifies that after /api there can be any letters. It can be /api1, /api2, /apixyz, etc. On executing this expression, we will get results for metrics having metric name prometheus_http_requests_total and whose label handler starts with slash api. - Negative Regular Expression matcher (!~) : Select labels that do not regex-match with the provided string.
e.g. prometheus_http_requests_total{handler!~”/api.*”}
PromQL Operators
PromQL supports two type of operators – binary operators and aggregation operators that allow us to apply more than simple arithmetic operations on instant vectors.
Binary Operators
Binary operators are the operators that take two operands and performs the specified calculations on them. It can be again classified into three main categories :-
- Arithmetic binary operators
- Comparison binary operators
- Logical/set binary operators
Arithmetic Binary Operators
The standard arithmetic operators are:
- + (Addition)
- – (Subtraction)
- * (Multiplication)
- / (Division)
- % (Modulo)
- ^ (Power)
Binary arithmetic operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs.
Eg: node_memory_Active_bytes/8
This is a valid example of applying an arithmetic operator between a vector and scalar where node_memory_active_bytes is a vector type and 8 is a scalar.
Comparison Binary Operators
A comparison operator is a binary symbol which is used for comparison. In Prometheus we have the following comparison operators:-
- == (equal)
- != (not equal)
- >(greater than)
- < (less than)
- >= (greater or equal)
- <= (less or equal)
Comparison operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs.
e.g. process_open_fds > 12
Logical/Set Binary Operators
The logical operators are used to combine simple relational expressions into more complex expressions. In PromQL, we have three types of logical operators :-
- and (intersection)
- or (union)
- unless (complement)
Unlike other operators, logical operators are defined between instant vectors only.
Aggregation Operators
Aggregation operators are special mathematical functions that are used to combine information. Prometheus supports the following built-in aggregation operators :
- sum
- min
- max
- avg
- group
- stddev
- stdvar
- count
- count_values
- bottomk
- topk
PromQL Functions
The most commonly used functions in Prometheus are :
1. rate()
Rate function is mainly used with counter type metrics. It calculates the per-second average rate increment of the time series in the range vector. In simpler words, this means that the rate function returns how fast a counter is increasing per second for each time series in the range vector passed to it.
rate should only be used with counters.It is best suited for alerting, and for graphing of slow-moving counters.
The following expression returns the per-second rate of HTTP requests as measured over the last 5 minutes, per time series in the range vector:
rate(http_requests_total{job="api-server"}[5m])
2. irate()
Unlike rate function that calculates the per-second ‘average’ rate of increment over time, the irate function calculates the ‘instant’ rate of increment of the time series in the range vector. Basically, it calculates the rate based on the last two data points gathered.
irate should only be used when graphing volatile, fast-moving counters.
The following expression returns the per-second rate of HTTP requests looking up to 5 minutes back for the two most recent data points, per time series in the range vector:
irate(http_requests_total{job="api-server"}[5m])3. changes()
With ‘changes’ function you can count how many times a gauge has changed its value over the time.
The following expression tell you how many times the process node_exporter has restarted in the past 1 hour.
changes(process_start_time_seconds{job="node_exporter"}[1h])4. deriv()
The deriv function estimates the slope of each time series in a range vector and calculates the per-second derivative of the time series in a range vector.
deriv should only be used with gauges.
The following expression calculate how far resident memory is changing per second based on samples from the past hour for the job prometheus.
deriv(process_resident_memory_bytes{job="prometheus"}[1h])5. predict_linear()
It predicts the future value of a Gauge by checking the previous patterns of a metric in the time range you provide.
The following expression will give us how much free memory would be left in next 2-hours based on the past 1 hour of samples.
predict_linear(node_memory_MemFree_bytes{job="node_exporter"}[1h], 2*60*60)/1024/1024Conclusion
PromQL is easy yet powerful query language for time series databases. It allows us to write typical TSDB queries in concise yet clear way. Prometheus collects a lot of metrics that you don’t need all of the time. You can filter those extra metrics out using PromQL.