> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trace0hq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace0QL

> Search your transactions by their requests, calls, infrastructure, tags and logs with Trace0QL.

## Intro

Trace0 provides a query language called `Trace0QL` (Trace0 Query Language) that lets you search transactions in Trace0.

It is similar to Grafana's TraceQL, but simpler to read and write, with native support for querying transaction logs and [tags](https://docs.trace0hq.com/platform/transactions/tags.md).

A `Trace0QL` query is a list of [conditions](#conditions), combined with `AND` and `OR`.

![Trace0QL Light](https://mintcdn.com/trace0/L9un5Y7f7K7XbqHw/images/trace0ql-light.png?fit=max\&auto=format\&n=L9un5Y7f7K7XbqHw\&q=85\&s=8785cc96c4860b543513d6ba5c8ea504)

You can write `Trace0QL` directly in the Trace0QL query box, or describe what you're looking for in plain English with [Natural Language Search](https://docs.trace0hq.com/platform/transactions/search/natural-language-search.md), which is converted to `Trace0QL` for you. Either way, the query that was run is shown in the Trace0QL query box, where you can edit it and run it again.

With one query you can search across everything Trace0 knows about a transaction, such as:

* the request(s) each service handled, e.g. its endpoint, duration and status code
* the calls made, e.g. database queries, cloud service calls and requests to third-party APIs
* the infrastructure it ran on, e.g. the ECS task, Kubernetes pod or region
* the [tags](https://docs.trace0hq.com/platform/transactions/tags.md) your services added, e.g. a user ID
* the logs it wrote

The time range and service selected on the page apply to every query, so they aren't part of the query itself.

## Basics

### Conditions

A query is made of conditions, each in the form `attribute operator value`:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
status_code >= 500
```

Join conditions with `AND` and `OR`. `AND` takes precedence over `OR`, as in most programming languages, and brackets can be used to change the order:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
error = true AND (endpoint contains /checkout OR duration > 1000000000)
```

`AND` and `OR` can be written in upper or lower case. Attribute names are case-sensitive.

### Values

Values that contain spaces, brackets, quotes or any of the operator characters `= ! < >` must be double-quoted. Inside a quoted value, escape a double quote or backslash with a backslash:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
endpoint = "POST /orders"
result contains "Connection refused"
db.query.text contains "WHERE id = 5"
```

Values like `api.github.com`, `aws.dynamodb` and ARNs don't need quotes.

Durations (`duration` and `span.duration`) are always in **nanoseconds**:

| Duration | In nanoseconds |
| - | - |
| 250µs | `250000` |
| 29ms | `29000000` |
| 1.5s | `1500000000` |

Boolean attributes take `true` or `false`, e.g. `error = true`. Numbers must not contain commas.

### Attributes

There are four attribute scopes, which differ in what they match.

#### Trace Record

Trace Record attributes describe the request(s) each service in the transaction handled:

| Attribute | Type | Description |
| - | - | - |
| `endpoint` | Text | The endpoint that handled the request, as the HTTP method and route, e.g. `POST /orders`. |
| `duration` | Duration (ns) | How long the request took. |
| `status_code` | Number | The HTTP status code of the response. |
| `error` | Boolean | `true` if the request failed. |
| `result` | Text | The exception type, if an exception was thrown. |
| `summary` | Text | The transaction's summary. |
| `service.name` | Text | The name of the service. |

#### Span

Span attributes describe the calls made while handling the request, and match if any span in the transaction matches. They use the [OpenTelemetry semantic convention](https://opentelemetry.io/docs/specs/semconv/registry/attributes/) names, e.g. `db.system.name`, `rpc.method`, `server.address`, etc.

`span.duration` is a derived attribute calculated by Trace0 at ingestion time. It is calculated as follows: `span.endTimeUnixNano - span.startTimeUnixNano`. The value is always in nanoseconds.

#### Resource

Resource attributes describe the infrastructure the services ran on, and match if any service in the transaction ran on a matching resource. They use the [OpenTelemetry semantic convention](https://opentelemetry.io/docs/specs/semconv/registry/attributes/) names, e.g. `cloud.region`, `cloud.platform`, `aws.ecs.task.arn`, `k8s.pod.name`, etc.

#### Tag

[Tags](https://docs.trace0hq.com/platform/transactions/tags.md) are the custom key value pairs your services add, queried as `tag.<key>`, e.g. `tag.user_id = 123`. A tag condition matches if any service in the transaction has the tag. Tag values are always text.

The query box autocompletes every attribute you can use, including your environment's tag keys.

When a service is selected, span conditions only match that service's own spans. Transaction attributes are for that service's requests, while resource attributes and tags still match any service in the transaction.

***

You can hover over any attribute in a Trace0QL query to see its type (e.g. Text), scope (e.g. Span) and a More Details link. For example, the link for the `db.system.name` opens its [OpenTelemetry semantic convention](https://opentelemetry.io/docs/specs/semconv/registry/attributes/db/#db-system-name) page.

![Trace0QL Attribute Explainer Light](https://mintcdn.com/trace0/qrR-dq0OGMZ-T4of/images/trace0ql-attribute-explainer-light.png?fit=max\&auto=format\&n=qrR-dq0OGMZ-T4of\&q=85\&s=976af2f340c01e0950b1400696ecf53a)

### Logs

Search the logs a transaction wrote with `log contains "<term>"`:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
log contains "payment declined" AND status_code >= 400
```

Log conditions match transactions with at least one log message containing every term, and must be joined to the rest of the query with `AND`, not used inside brackets, an `OR` or `spans(...)`.

### Matching the same call with spans(...)

A span condition on its own matches if any span in the transaction matches it, so two span conditions joined with `AND` can be matched by two different spans. This query matches a transaction that made a fast KMS call and a separate slow database call:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
rpc.service = KMS AND span.duration > 29000000
```

To require that the same span matches every condition, put them inside `spans(...)` and compare the number of matching spans with a count:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
spans(rpc.service = KMS AND span.duration > 29000000) >= 1
```

`spans(...)` also lets you count calls. Each `spans(...)` is counted separately, so this finds transactions with two or more KMS calls, at least one of which took longer than 29ms:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
spans(rpc.service = KMS) >= 2 AND spans(rpc.service = KMS AND span.duration > 29000000) >= 1
```

Only span attributes can be used inside `spans(...)`, and it can't be nested inside another `spans(...)`. The count can be compared with `=`, `!=`, `>`, `>=`, `<` or `<=`, and `= 0` finds transactions with no matching calls.

## Operators

| Operator | Meaning | Use with |
| - | - | - |
| `=` | equals | all attributes |
| `!=` | doesn't equal | all attributes |
| `>` | greater than | number and duration attributes |
| `>=` | greater than or equal to | number and duration attributes |
| `<` | less than | number and duration attributes |
| `<=` | less than or equal to | number and duration attributes |
| `contains` | contains the value, ignoring case | text attributes and `log` |
| `!contains` | doesn't contain the value, ignoring case | text attributes |

| Keyword | Meaning |
| - | - |
| `AND` | both conditions must match |
| `OR` | either condition must match |
| `( )` | groups conditions, e.g. `a AND (b OR c)` |
| `spans( ) <comparison> <count>` | the number of spans that match every condition inside the brackets |

`contains` is the best choice when you only know part of a value, e.g. an endpoint without its HTTP method: `endpoint contains /assessment.cancel`.

## Examples

Failed requests that took longer than 50ms:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
error = true AND duration > 50000000
```

Client errors only:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
status_code >= 400 AND status_code < 500
```

Requests to one endpoint, with or without its HTTP method:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
endpoint = "POST /orders"
endpoint contains /orders
```

Timeouts or server errors:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
status_code >= 500 OR exception.type contains timeout
```

Slow requests that read from a DynamoDB table:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
duration > 1000000000 AND spans(db.system.name = aws.dynamodb AND db.collection.name = users) >= 1
```

Requests that made more than 10 Postgres queries (a possible N+1):

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
spans(db.system.name = postgresql) > 10
```

A slow call to a third-party API:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
spans(server.address contains api.github.com AND span.duration > 300000000) >= 1
```

Failed requests that didn't call S3:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
error = true AND spans(rpc.service = "Amazon S3" OR rpc.service = S3) = 0
```

Lambda cold starts in one region:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
faas.coldstart = true AND cloud.region = eu-west-1
```

Server errors from one Kubernetes namespace:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
status_code >= 500 AND k8s.namespace.name = payments
```

Requests for one user, by tag:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
tag.user_id = 123
tag.user_name contains Dave
```

Requests that logged a message and returned an error:

```trace0ql theme={"languages":{"custom":["/languages/trace0ql.tmLanguage.json"]}}
log contains "payment declined" AND status_code >= 400
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.