Using Logstash agent

This document describes how to send logs to PacketAI using the logstash http output plugin
Setting keys, cluster name and access control
In order to complete the setup, there are 2 main things to configure : 
1.Obtain your keys YOUR_PAI_TOKEN and  YOUR_PAI_IID by going to https://logpatterns.packetai.co/deploy/agent. Pick any of the integrations to find your keys.
2.Replace yourclustername with your cluster name in the mutate block (see below). ⚠
Note that even if you are not using a proper cluster, you must provide a value for this variable. 
Input conf
Logstash supports several input plugins, the list of input plugins it supports can be found here. The following file input configuration is provided as an example.
input { 
    file {
        path => [
            "/var/log/*.log",
            "/var/log/*/*.log"
        ]
    }
}
Filter conf
Logstash supports several plugins for filtering logs, the more details about several filters (plugins) available here.  PacketAI needs some additional fields needs to be attached to each message, see the below mutate section. We need to map the each application message to the following field, [kubernetes][controller][name] . for example if we have field application_name in the message, we could map [kubernetes][controller][name] => [application_name]. The following fields can be static
mutate {
    add_field => {
        "[kubernetes][namespace]" => "user" #don't change this value
        "[kubernetes][controller][type]" => "Deployment" #don't change this value
        "[kubernetes][controller][name]" => "test" #this needs to be mapped dynamically, it could be your application name
        "[packetai][cluster_name]" => "yourclustername" #needs to be update by user
    }
}
Output conf
Logstash several output plugins, more details can be found here. PacketAI use http output plugin of logstash to send the messages to packetai. The http output plugin code snippet can be found below. we need to replace the YOUR_PAI_TOKEN_HERE and YOUR_PAI_IID_HERE with the correct values.
output {
  http {
    id => "packetai_logstash_ingester"
    url => "https://vector-ingester-logpatterns.packetai.co/vector/log"
    http_method => "post"
    headers => ["X-PAI-TOKEN","YOUR_PAI_TOKEN_HERE","X-PAI-IID","YOUR_PAI_IID_HERE"]
    format => "json_batch"
    http_compression => true
  }
}
​
Sample logstash configuration
Please replace the YOUR_PAI_TOKEN_HERE and YOUR_PAI_IID_HERE with the correct values from the PacketAI account.
input { 
    file {
        path => [
            "/var/log/*.log",
            "/var/log/*/*.log"
        ]
    }
}
filter {
    ruby {
        #Here We are setting logfilename as app_name
        code => 'event.set("app_name",event.get("[log][file][path]").split("/")[-1][0..-5].gsub(/[^0-9a-zA-Z]/i, ""))'
    }
    mutate {
        add_field => {
            "[kubernetes][namespace]" => "user"
            "[kubernetes][controller][type]" => "Deployment"
            "[kubernetes][controller][name]" => "%{app_name}" # here we are setting the app_name as kubernetes.controller.name
            "[packetai][cluster_name]" => "prod"
        }
    }
}
output {
  http {
    id => "packetai_logstash_ingester"
    url => "https://vector-ingester-logpatterns.packetai.co/vector/log"
    http_method => "post"
    headers => ["X-PAI-TOKEN","YOUR_PAI_TOKEN_HERE","X-PAI-IID","YOUR_PAI_IID_HERE"]
    format => "json_batch"
    http_compression => true
  }
}
​
Validate your configuration
It is advised to test and validate your logstash configuration with below command
logstash --config.test_and_exit -f logstash-simple.conf
Last modified 3mo ago