Fluent Bit

Fluent Bit integration with PacketAI

Input

The following section defines the fluent-bit input section. for more details see fluent-bit input here
[INPUT]
Name tail
# setting the path for the file to tail
Path /var/log/**/*.log
Tag mac_logs
[INPUT]
Name cpu
# setting the path for the file to tail
Tag my_cpu_metrics
Interval_Sec 10

Filters

We use filters to add some PacketAI metadata to easily process. It's important to have packetai.cluster_name and packetai.app_name. These 2 fields help us to quickly add logpatterns and more easy to manage opensearch indexes.
// Some code
[FILTER]
Name record_modifier
Match *_logs //here it's matches tag: mac_logs, it's impoortant that you tag your fluent-bit inputs
Record hostname ${HOSTNAME}
Record packetai.cluster_name macos
Record packetai.app_name varlogs # For metrics this is optional.

Output

The output section is used to send the logs/metrics to PacketAI.

Logs

For logs the below output configuration is being used. We need to configure the Host, Header, and Match sections.
The Host varies depending on your packetai URL. Please check with PacketAI team if you are not sure which URL to use.
Headers X-PAI-TOKEN and X-PAI-IID needs to get it from packetai URL. After login, you can navigate to Deploy PacketAI/Agent/logstash click on credentials It will show X-PAI-TOKEN and X-PAI-IID, copy these values and replace YOUR_PAI_TOKEN and YOUR_PAI_IID.
[OUTPUT]
Name http
Port 443
Tls On
Host vector-ingester-logpatterns.packetai.co # Check your URL, it might be different from this
URI /fluent/log
Header X-PAI-TOKEN YOUR_PAI_TOKEN # This token needs to get it from packetai UI section
Header X-PAI-IID YOUR_PAI_IID
Format json
json_date_format iso8601
json_date_key timestamp
Match *_logs

Metrics

Similarly for Metrics we need configure the Host, Header, and Match sections.
The Host varies depending on your packetai URL. Please check with PacketAI team if you are not sure which URL to use.
Headers X-PAI-TOKEN and X-PAI-IID needs to get it from packetai URL. After login, you can navigate to Deploy PacketAI/Agent/logstash click on credentials It will show X-PAI-TOKEN and X-PAI-IID, copy these values and replace YOUR_PAI_TOKEN and YOUR_PAI_IID.
// Some code
[OUTPUT]
Name http
Port 443
Tls On
Host vector-ingester-logpatterns.packetai.co # Check your URL, it might be different from this
URI /fluent/metric
Header X-PAI-TOKEN YOUR_PAI_TOKEN # This token needs to get it from packetai UI section
Header X-PAI-IID YOUR_PAI_IID # This token needs to get it from packetai UI section
Format json
json_date_format iso8601
json_date_key timestamp
Match *_metrics

Example configuration

[INPUT]
name cpu
tag my_cpu_metrics
Interval_Sec 10
[FILTER]
Name record_modifier
Match *
Record hostname ${HOSTNAME}
Record packetai.cluster_name macos
Record packetai.app_name macos
[OUTPUT]
Name http
Port 443
Tls On
Host vector-ingester-logpatterns.packetai.co
URI /fluent/metric
Header X-PAI-TOKEN YOUR_PAI_TOKEN
Header X-PAI-IID YOUR_PAI_ID
Format json
json_date_format iso8601
json_date_key timestamp
Match *_metrics
Last modified 14d ago