In /this tutorial we will be covering the basics of setting up Graphite and Logster for Apache logs. First step is we setup an instance of Graphite. Read the following tutorial to install Graphite on your system and set it up. Later, we can use the same for Logster to implementing logging.
Apache is one of the most popular webserver, deployed by numerous organizations to serve documents to users. Apache logs every HTTP request in its Access logs which can be found at /var/log/httpd/access_log or /var/log/apache2/access.log in a usual installation on RHEL or Ubuntu. Access logs contain client IP Address, User Agent Information, Time of Request, HTTP Status Code, HTTP endpoint, etc.
In this tutorial we will focus on identifying all the significant classes of HTTP status codes found in Apache access logs parsing them via Logster and feeding them into Graphite as metrics to be stored.
- Step 2
Logster can be found at https://github.com/etsy/logster with installation instructions.
To install Logster, clone the Logster repository from GitHub.
git clone https://github.com/etsy/logster.git- Step 3
Install the pygtail module.
pip install pygtail- Step 4
Install the Logster module.
sudo python setup.py install- Step 5
After logster is installed, We collect some sample Apache acces logs from https://github.com/elastic/examples/tree/master/Common%20Data%20Formats/apache_logs. These logs will be parsed by logster.
wget https://raw.githubusercontent.com/elastic/examples/master/Common%20Data%20Formats/apache_logs/apache_logsDownload the sample logs.
- Step 6
sudo /usr/bin/logster --dry-run --output=graphite --graphite-host=localhost:2003 SampleLogster apache_logsExecuting this command will do a dry run and write the parsed output to stdout.
- Step 7
Executing the command without --dry-run will send the parsed metrics to Graphite running on localhost.
sudo /usr/bin/logster --output=graphite --graphite-host=localhost:2003 SampleLogster apache_logs
The newly added http_xxx metrics can be seen in Graphite-WebApp and visualized using Graphite Composer.
The metrics once published to Graphite will be stored by Whisper to a storage backend and are available via the Metrics API to access raw metrics or the Render API to generate graphs. Other 3rd Party tools like Grafana use these APIs for visualizing metrics stored in the Graphite time-series database on their own platforms.