Metrics

In addition to the types above, a datum may also be a histogram. Histograms let us compactly represent a large amount of data. Rather than individual values, we store a set of bins and counts within those bins. At each bin, we keep track of the total number of samples whose value falls between the upper and lower edges of the bin. For example, one might track the I/O latency on a disk using a histogram, because there may be a huge number of total I/Os in any time interval. For oximeter histograms, the bins may be any integer or floating-point type, and the counts are always 64-bit unsigned integers.

Let’s look at an example histogram, one tracking those I/O latencies on a disk.

The column "Bins" shows the right edges of the first few bins. Each is a time in milliseconds. As I/Os are completed, their latency is measured, and we look up the smallest bin in this array that is larger than the latency, and increment the count in that bin by 1.

Histograms greatly compress the underlying data. There may be thousands or millions of I/Os every second on modern SSDs, however our histogram may only have a few tens or hundreds of buckets. We lose information, since we don’t know the exactly latency for any I/O, only that it was within the appropriate bin. In exchange, we may accurately capture information about the distribution of a large number of I/Os, while using a fixed amount of memory per datum. The memory usage is a function of the number of bins, not the number of samples.

Sometimes we fail to collect a sample for a timeseries. For example, we might fail to access the hardware containing a temperature sensor, or be unable to reach a peer over the internal network to determine its state. In these situations, we may generate a missing datum.

Missing data points are represented by the JSON value null in the output of a query. Missing points still contain lots of the information that a valid sample does, including a timestamp, datum type, and start time for cumulative counters or histograms. It is often important to consider missing data. By providing an explicit null as the value along with the timing and type information present for other points, one can accurately distinguish the absence of valid data from the absence of any measured activity at all.

A gauge represents an instantaneous measurement. It captures the value of an underlying metric at one point in time. Each data point is completely independent of one another, different measurements of some underlying quantity taken at different timestamps. For example, the number of items resident in a queue at any time is a gauge, shown below.

At each timestamp, a new, independent sample is recorded. The values may go up or down arbitrarily — in this case, as the queue receives and processes items at different rates.

In contrast to gauges, which are instantaneous measurements, a cumulative counter tracks a total value over a range of time. It has start time in addition to its timestamp, which is the beginning of its time range. Its value represents the accumulated total encountered from the start time up to the recorded timestamp.

Cumulative counters are widely used in the Oxide product. It’s always possible that an attempt to collect a sample fails — software may be unable to access a hardware sensor or reach a peer on the internal network. Cumulative counters still provide information about the period of time we missed a sample, because the next acquired sample will count events that occurred during that time. Cumulative counters are also useful for identifying program crashes, denoted when the start time for the counter resets. See the section on missing data points for more information.

Here is an example cumulative counter.

Continuing with our example of a queue, suppose this cumulative counter tracks the total number of items sent on the queue. (The gauge above tracked the number of items in the queue at any one time.) The counter begins with a start time of 17:44:22, and a sample is taken immediately. The value is 0, meaning no items have been pushed onto the queue.

Another sample is taken 10 seconds later. The start time is the same, and the value has increased to 1. The queue received 1 new item in this 10 second window, and has now seen 1 item total.

After another 10 seconds, an additional sample is taken with the value 8. The queue has received 8 items in total now, and so 7 new items in this time window.

10 seconds later we see a new sample, but the start time has also changed! This could mean that the program managing the queue has restarted, for example. The value is now 1, which means between 17:44:44 and 17:44:52, there was 1 item sent on the queue.

The last metric type in OxQL is a delta. In a timeseries with a delta metric type, the values at each point in time represent the change over a time range. Similar to cumulative counters, deltas have a start time in addition to a timestamp. In contrast, the start times are always non-overlapping, with the start time of one bin exactly equal to the timestamp of the one before it.

Here is an example delta timeseries. We can see that each timestamp is equal to the start time of the following sample (if it exists). The value counts the total number of items just within that window.

Deltas are common in OxQL queries. That is mainly because, when selecting data from a cumulative table, it is automatically converted into a delta! Why is that?

As mentioned above, much of the data collected in the Oxide rack is cumulative, whether a counter or a histogram. Cumulative metrics have a number of nice properties that make the collected data more robust and useful in a complex distributed system. (See the section on cumulatives for details.) Despite their wide use, cumulative data is actually quite tricky to work with. The samples in a cumulative timeseries are not independent, since later values contain overlapping information with earlier values. This makes it difficult to combine data from multiple sources, either within the same timeseries (for example, temporally averaging) or across timeseries (say, averaging during a grouping operation). If one were to temporally average a cumulative counter into large time windows, it would weight earlier samples more heavily, since an incremement in an early time range is counted again for each later time range in which it also appears.

To address this, both cumulative counters and histograms are automatically and implicitly converted to deltas when they are selected from the database. After retrieving the data, OxQL computes the adjacent differences between each sample to convert a cumulative into a delta metric. This lets us aggregate, average, join, group, and align timeseries, since each data point is independent of its neighbors.

The cumulative data shown above is similar to the previous cumulative counter example. The first three rows all share a start time, and the timestamps increment regularly. The value is also monotonically increasing. In the fourth row, we see a restart of the counter, with the start time and value both resetting.

The delta data on the right is derived from the cumulative, depending on whether the counter reset. If it did not reset, the start time of each row in the delta is set to the timestamp of the previous sample (if one exists), and the value is difference between the current value and the previous (if it exists). If the cumulative counter did reset, then the start time, timestamp, and value are all equal to the corresponding value from the cumulative counter.

An identifier is a name, such as for a table operation or the field of a table.

For example, in the table operation filter link_name == "foo", both the operation name filter and the field name link_name are identifiers.

Identifiers are formatted as lowercase, ASCII characters or digits, possibly delimited by underscores, and have the following structure:

Identifiers may not start with a digit, though the underscore-delimited words after the first may start that way. The special identifiers are implicitly added to a table by the OxQL query engine, and are used to refer to the common elements that all tables and timeseries share, such as their timestamp or the datum value itself. See Terminology for details on the anatomy of a timeseries.

A table name is used to identify a table in the database or the output of an OxQL query.

The initial data input to OxQL queries is a table stored in the telemetry database. These are identified by a name with a particular structure: a target resource and a metric name, separated by a :. For example, the sled_data_link:bytes_sent table lists the number of bytes sent on a single sled’s NIC.

Table names are also used to name the output of a query. In many cases, this is the same as the input table name, but may change during processing. For example, the name of the output of a join operation is a concatenation of the input table names.

OxQL supports the common types of literals, usually used in filtering expressions or as arguments to table operations. This includes things like booleans, timestamps, numeric values, and strings. Where possible and helpful, the regular expression matching each literal has been included below.

Comparison operators are used to compare two values, and return a boolean. Most familiar operators are supported, including equality, less-than, etc. Here is the full list:

The last operator, ~= implements regular expression matching, and is only supported for strings. Expressions such as "[a-f]+0$ can be used, for example, to match any string ending with one or more character a-f and a single digit 0. The full supported regular expression syntax is described here.

Logical operators can also be used to combine one or two boolean values together. The supported operators are:

Note that the XOR operator is logical, not bitwise. The precendece of the operators follows the common programming order: in order of increasing precedence, they are OR, AND, XOR, and NOT.

The fundamental unit of an OxQL query is a table operation. Each table is zero or more timeseries, all with the same field names, field types, and data types. In this sense, a table is analogous to a table in a traditional relational database: it contains entries all with the same schema. In this case, an entry is a timeseries, which has a unique set of field values along with its timestamped data points. (See Terminology for more details.) Below is a summary of each supported table operation, and more details on each are in the following sections.

Each table operation may take zero or more tables, and produce zero or more new tables after processing their input. The number of supported tables, and any restrictions on data types, are listed in the section describing each operation.

All queries or subqueries must begin by extracting some data from the timeseries database, on which the rest of the query operates. This is done using the get table operation, described in detail here.

Complex queries are constructed by stringing together table operations with the familiar pipe operator, |. The output of each table operation is fed into the next. For example, the query

will first get the raw data from the sled_data_link:bytes_sent table, and then restrict it to the last minute from the time the query is run. This will result in one table, since there is one table named in the query. There may be zero, one, or many timeseries in the table, depending on the data available in the database when the query is run.

The above query’s output might look like this:

As mentioned above, queries can be either flat or nested. A query is nested if it contains subqueries, two or more queries inside braces ({}), separated by a semicolon (;). Each subquery itself can be flat or nested, so that quite complex processing pipelines can be built up using these basic elements. For example the query:

will get the number of bytes sent and received on all sleds' physical data links, over the last minute. Further filtering can be done to limit the data, described in detail below. Here is an example output:

Note that there are two tables here, printed one after the other. The first shows the table sled_data_link:bytes_sent, with its timeseries. This is first because the subquery listed it first. Next is the sled_data_link:bytes_received table, with all its timeseries.

The primordial table operation is get. This retrieves a single table by its name name from the database. It takes zero tables as input, and produces exactly one table as output. That output table may have any number of timeseries in it.

Example:

A filter table operation passes through data which matches the provided filtering expression. Filters take an identifier, a comparison operator, and a literal. (In the future, more complex expressions beyond literals may be supported.)

See Comparison and logical operators and Literals for details on those components.

The provided identifier may name any one of the fields in the table, or refer to a special identifier, such as timestamp. For cumulative or delta metric types, the start_time identifier is also supported. For scalar (non-histogram) timeseries, the datum identifier is used to refer to the value of the data point. For histograms, the bins and counts identifiers refer to the values of the data point instead.

Filters may include compound expressions, joined with the logical operators defined above. The precedence rules are also defined in that section, but note that comparison operators always bind more tightly than logical operators. In other words, foo == 0 && bar == 1 is equivalent to (foo == 0) && (bar == 1). Parentheses may be used to group expressions.

Below are several examples of filter table operations.

Note that the literals used to compare against a field must be comparable to the actual data type of that field. It is an error, for example, to write filter foo == 0 if the field "foo" of the input table doesn’t have an integer type.

The align table operation is used to generate output samples that are evenly spaced in time. This is critical for combining data from more than one timeseries or table. Timeseries data is usually sampled on an approximately regular interval, but almost never exactly so. The align operation can be used to generate exactly-spaced samples. This allows points from one timeseries to be confidently matched up with those of another, letting us perform grouping, aggregation, or other multi-timeseries operations.

The align table operation takes one argument, which is the alignment method. Currently the only supported method is mean_within, which computes an average of the input timeseries over the provided duration. For example:

will compute the number of bytes sent on a data link over the last one hour, averaged in five minute windows. The output samples will be spaced exactly five minutes apart.

The alignment method also specifies what it means for an input sample to "overlap" with the output period. For the mean_within method, any input timeseries that has any part of its sample period within the output period is considered to overlap, with the amount of overlap determining the contribution of that value to the output. This effectively prorates each point by the amount its time range overlaps with the output window.

For example, suppose there is an input sample over the range [0.1s, 1.1s], and the output period is exactly [0.0s, 1.0s]. Then the input sample will contribute its value weighted by 0.8, since 80% of its interval overlaps with the output window. That is, the input is linearly weighted in the output, where the weight is the amount of its interval which overlaps with the output period.

If there are zero data points in an output period, then a missing value is produced in that period, represented by the JSON value null in the output data array.

The group_by table operation can be used to group more than one timeseries in a table together. One specifies a list of field names on which to group, and optional a method for reducing all the points within a group.

Note that the input timeseries must be aligned first, so that samples from the input timeseries can be correlated across time. An error is generated if the inputs are not aligned. Example:

The above query will compute the average number of bytes received on each sled’s NICs, within one minute periods, and then group by the sled’s UUID. This will remove all the other fields, leaving only the sled_id field. Within each group, all data points at the same timestamp will be summed to generate the output result. In other words, this computes the total number of bytes received for all links on a compute sled.

In contrast, the query:

Will compute the average number of bytes received in each group.

Grouping is one way to merge data from multiple timeseries. However, one may also wish to correlate data from more than one timeseries over time. For example, one might want to know how many bytes are sent or received for a given sled.

One way to do this is the join table operation, which combines two or more tables together. For example:

This query will first retrieve data for both the bytes sent and bytes received tables. They will separately be aligned to one minute periods by averaging, since the align operation takes any number of tables as input, and produces the same number as its output.

Finally the join table operation (which takes no arguments), performs a natural inner join one two or more tables. This merges data from timeseries in each table with the exact same values for their fields. It combines data points by concatenating those values which occur at the same timestamp.

For example, the query:

Will produce data at 20s intervals, each with two values for the point: the first will give the average number of bytes sent in that interval, and the second the average bytes received. For example, entries might look like this:

The first and last table operations are used to limit the number of data points from each query. They both take an unsigned integer argument k, limiting the output to at most k samples, independently for each timeseries in each table.

For example:

will select the timeseries from the sled_data_link:bytes_sent table corresponding to the physical data links, and take the very last sample from each. The output might look like this:

Select the last sample from all timeseries in the sled_data_link:bytes_sent table. See this section for more details.

Select the last 5 minutes of data from all timeseries in the sled_data_link:bytes_sent table. See the section on the filter table operation and the section on timestamps.

Average all timeseries in the sled_data_link:bytes_sent table into non-overlapping 5 minute time windows. See the section on the align table operation.

Average all timeseries in the sled_data_link:bytes_sent table into non-overlapping 5 minute time windows, and also average all timeseries with the same value for the sled_id field. See the section on the group_by table operation.

Average all timeseries independently from the sled_data_link:bytes_sent and sled_data_link:bytes_received tables into 5 minute windows, and then concatenate the samples from timeseries with the same field values, at the same timestamps. See the section on the join table operation.