Logstash doesn’t have to be that complicated. Last week’s example with log files from IIS looked so scary because the fields can vary from one IIS to the other. But when you want to use logstash to parse a well-known file format then all can be much simpler. Today I will show you the configuration to parse log files from the Apache web server.
This post is part of the Improve Your Log Messages series. You can find the other parts here:
- Part 1: The Missed Opportunities of Log Files
- Part 2: Structured Logging with Serilog
- Part 3: RavenDB as a Sink for Serilog
- Part 4: Seq as a Sink for Serilog
- Part 5: How to Influence the Output of Serilog
- Part 6: Monitor your Application with Seq
- Part 7: Debugging Serilog
- Part 8: Elasticsearch as a Sink for Serilog
- Part 9: Monitor your Applications with Kibana
- Part 10: Closing the Feedback Loop from Log Messages to Knowledge
- Part 11: How To Analyse IIS Log Files
- Part 12: Using Logstash to Analyse IIS Log Files with Kibana
- Part 13: Analysing Apache Log Files with Logstash and Kibana
- Part 14: How to Analyse SharePoint Log Files
Configuration for Apache
A typical entry in the log files of Apache may look like this one:
1 |
192.168.0.1 - - [1/Dec/2014:01:23:13 +0200] "GET /wp-content/uploads/2014/09/Nuget_Seq-t.png HTTP/1.1" 200 83350 "https://improveandrepeat.com/2014/09/seq-as-a-sink-for-serilog/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36" |
Since nearly no one tries to modify how those messages are created you can go with one of the default patterns that are shipped with logstash. All you now need to parse an Apache log file is this configuration:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
input { file { type => "apache" path => "C:/logs/*.log" start_position => "beginning" } } filter { grok { # Here you find more on the default patterns logstash ships with: # https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns pattern => "%{COMBINEDAPACHELOG}" } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] locale => "en" } } output { stdout { codec => rubydebug } elasticsearch { host => "127.0.0.1" port => "9200" protocol => "http" } } |
All you have to do is to modify the path to your log files. When you run logstash
with the agent flag and your configuration then all your log files are pushed into Elasticsearch:
Next
If all log files would be that simple we would not need to know much about the grok syntax. Next week we will look how SharePoint writes its log files and what tools you can use to read them.
Update 23 Nov 2015
The default patterns are now in the logstash-plugins repository at GitHub
Hi,
the link about default patterns doesn’t work (https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns). Can you give us the right one?
Thanks.
Hi Francesco,
the correct link is now https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
Regards
Johnny