Using Vector agent
This document describes how to configure a PacketAI Transformer and Sink for vector, in order to demonstrate PacketAI’s capacity to group log lines by patterns (aka: log patterns).

Setting credentials, cluster name and access control

In order to complete the setup, there are 2 things to configure :
  1. 1.
    You need to replace YOUR_PAI_TOKEN and YOUR_PAI_IID with your own credentials in the Sink block (see below). Those credentials can be obtained by simply contacting the PacketAI team. Please note that those credentials are non-transferable.
  2. 2.
    You need to replace yourclustername with your cluster name in the Transformer block (see below).
🔺 By default, the Sink block is configured to send logs to PacketAI’s staging environment on https://vector-ingester-gcpstage.packetai.co/vector/log and the actual log patterns are visible on https://app-gcpstage.packetai.co. Some customers, though, will be using PacketAI’s dev environment and should replace gcpstage by gcpdev in the two URLs above (so they need to update the URI field in the Sink block).

Setting Transformer

Packetai provides a kubernetes transformer that can be used to transform the data before it is ingested. In the kubernetes transformer we need our client to add packetai cluster name.
sources:
# assumes that you have a source called "kubernetes_logs"
kubernetes_logs:
type: kubernetes_logs
transforms:
# assumes that you already have a transformer your_filter
your_filter:
type: filter
inputs:
- kubernetes_logs
condition: null
# IMPORTANT:
# IN ORDER TO SEND LOG TO PACKETAI, CLUSTER_NAME MUST BE ADDED TO THE TRANSFORMER
# THIS IS DONE BY ADDING THE FOLLOWING LINE TO YOUR CONFIG
# CLUSTER NAME MUST MATCHES REGEX "^[a-z]+quot;
packetai_log:
type: remap
inputs:
- your_filter
source: |-
.packetai.cluster_name = "yourclustername"

Setting Sink

Packetai provides a http sink that can be used to ingest your data (Note: some customers will need to replace gcpstage by gcpdev below, and accordingly, log pattern site https://app-gcpstage.packetai.co would become https://app-gcpdev.packetai.co).
sinks:
http_packetai:
type: http
inputs: [packetai_log]
encoding:
codec: json
compression: gzip
uri: https://vector-ingester-gcpstage.packetai.co/vector/log
headers:
X-PAI-TOKEN: YOUR_PAI_TOKEN
X-PAI-IID: YOUR_PAI_IID