Starting out with grok in Logstash

Published: 24 Sep 2012

grok seems to be the default way to filter events in Logstash. I got in contact with it last week and found some great documentation that I thought I'd save for a rainy day.

First thing to read is the excellent documentation about grok on the Logstash website. Then, jpmens has written some awesome and very informative posts, especially this one about grok.

In my example we're matching the following string:

1999-02-19 20:59:59 Hi, Peter. What's happening? We need to talk about your TPS reports.

A simple example of the grok filter can be seen below.

filter {
  grok {
    type    => 'innotech'
    pattern => "%{DATE} %{TIME} Hi, %{USERNAME:name}. What's happening\? We need to talk about your %{DATA:report_type} reports."
    add_tag => "to_%{name}"
  }
}

This matches the our string and we can collect the name in the %{name} variable, and TPS will be in the variable %{report_type}. The important thing to notice here is that the filter will only act on the input with the type set to innotech. If the input is not set to innotech it will be ignored by this filter.

Now, why do I use %{USERNAME} and %{DATA}? What do they match? In Logstash there are predefined patterns which are defined here. The easiest way to test grok out is to use the excellent grok debugger.

That's the quick introduction of how to get started with grok filters in Logstash. Below is a complete example of a shipper:

input {
  file {
    type   => 'innotech'
    path   => [ '/home/pgibbons/memoirs' ]
    format => 'plain'
  }
}

filter {
  grok {
    type    => 'innotech'
    pattern => "%{DATE} %{TIME} Hi, %{USERNAME:name}. What's happening\? We need to talk about your %{DATA:report_type} reports."
    add_tag => "to_%{name}"
  }
}

output {
  stdout { }
}

Log on %{name}!