Basic Syntax
Each query is composed of one or more clauses, which can be combined using boolean operators. The basic structure of a query is as follows:field
: The log field to query (e.g.,time
,resource.labels.container_group_name
,json_log.custom_field
).operator
: The comparison operator to use (e.g.,=
,!=
,>
,<
,contains
).value
: The value to compare against (e.g., a string, number, or date).
Log Schema
Log events are JSON objects with the following structure:- Note that a log event will have EITHER a
text_log
OR ajson_log
, but not both. If your application emits a log that is valid JSON, it will be parsed and made available viajson_log
. Otherwise, the raw text will available intext_log
. - The
time
field indicates the system time on the node where the log was emitted. - the
receive_time
field indicates the time when Axiom (what we use under the hood for log storage) received the log entry. - Any of these fields can be used in the query language.
- Time and Date values take the form
YYYY-MM-DDTHH:MM:SSZ
, and are expressed in UTC.
Resource Types And Labels
You can query logs from three different resource types, indicated in the fieldresource.type
:
container
: Logs from a running container instance. This includes all logs emitted to stdout and stderr within the container.instance_controller
: System events that occur on the instance level, such as container start and stop events, probes passing and failing, etc.deployment_controller
: System events that occur on the container group level, such as adjusting replica count, and starting and stopping container groups.
project_name
: The name of your project in SaladCloud.container_group_name
: The name of your container group in SaladCloud.
container
and instance_controller
types have the following additional labels:
container_group_version
: The version of your container group when the log was emitted.instance_id
: A unique ID for a specific instantiation of a container instance. This is independent frommachine_id
, because a single machine ID can have multiple instance IDs associated with it, if a container has recreated or restarted a number of times on the same machine. In such a situation, each “run” has its owninstance_id
.machine_id
: The unique ID of the node running the container that created the log message.
Operators
The following operators are supported in log queries:=
: Equality (Case Sensitive)=~
: Equality (Case Insensitive)!=
: Inequality (Case Sensitive)!=~
: Inequality (Case Insensitive)>
: Greater than.<
: Less than.>=
: Greater than or equal to.<=
: Less than or equal to.contains
: String contains (Case Sensitive)contains~
: String contains (Case Insensitive)!contains
: String does not contain (Case Sensitive)!contains~
: String does not contain (Case Insensitive)startswith
: String starts-with (Case Sensitive)startswith~
: String starts-with (Case Insensitive)!startswith
: String does not start with (Case Sensitive)!startswith~
: String does not start with (Case Insensitive)endswith
: String ends-with (Case Sensitive)endswith~
: String ends-with (Case Insensitive)!endswith
: String does not end with (Case Sensitive)!endswith~
: String does not end with (Case Insensitive)
Combining Statements
Query statements can be combined with the following boolean operators:and
: Logical ANDor
: Logical ORnot
: Logical NOT
Field Promotion in Structured Logs
If you emit structured logs (meaning logs that are valid JSON) with any of the following fields, it will be removed from thejson_log
field and promoted to a top-level field in the log entry.
severity
severity_number
trace_id
span_id
parent_span_id
time
or receive_time
field in your structured log WILL NOT override the respective time
fields of the log entry.
Troubleshooting Common Issues
- Ensure that quotes are properly escaped in your
.query
field. - Ensure that times are specified in UTC and follow the ISO 8601 format, i.e.
YYYY-MM-DDTHH:MM:SSZ
- Ensure that times are within the retention period of your logs.
- Ensure that times are in the past, i.e. no future times.