Sending Rails Application Logs to Logstash over UDP
Tracking down issues by searching though log files can be difficult, especially if your application is deployed across multiple instances. Thankfully this can be made easier by offloading all of your Rails logs to an external ELK stack which consists of Elastic Search, LogStash and Kibanna.
To get our ELK stack up and running we will use Docker and Docker Compose pulling the
hugely popular sebp/elk
docker image. As it is though, sebp/elk
isn’t pre-configured
with support for UDP so we need to add a few modifications for that.
Updated For Rails 6 - 10th of October 2019
Following some requests I have now updated this guide to work with Rails 6 and provided an example application, put together using this guide, that is available over on GitHub.
Dockerfile configuration
Firstly create a new Dockerfile pulling from the sebp/elk
image and set our working
directory to ${LOGSTASH_HOME}
. Then for LogStash to support receiving events over UDP
we need to add a couple of plugins, these are logstash-input-udp
for enabling the UDP
protocol, and logstash-codec-json_lines
as we are going to send or logs
across in this format.
# ELK Stack Dockerfile with UDP
from sebp/elk
WORKDIR ${LOGSTASH_HOME}
# Plugins to install
RUN gosu logstash bin/logstash-plugin install logstash-input-udp
RUN gosu logstash bin/logstash-plugin install logstash-codec-json_lines
Next we need to configure a LogStash listener so it knows hoe to receive the logs from
our Rails Application. Create a new file called rails-udp-input.conf
alongside your
Dockerfile and place the following content in it. This simply tells LogStash what protocol
and port it should listen on along with the codec that it should use to deserialize any
logs received.
input {
udp {
port => 5228
codec => json_lines
}
}
Now we need to tell LogStash to use this config file by adding the following to our Dockerfile.
# Add LogStash Listeners
ADD ./rails-udp-input.conf /etc/logstash/conf.d/rails-udp-input.conf
Docker Compose configuration
To run your ELK stack you can put the following into a docker-compose.yml
file alongside
your Dockerfile, you can then run your stack with the docker-compose up -d
command. The
important thing to note here, is that for any LogStash listeners you have added you will
need to add a port mapping for them in the compose configuration. In this instance I
have added - "5228:5228/udp"
, the internal port mapping also needs to be prepended
with /udp
as our listener is using the UDP protocol.
version: "3"
services:
elk:
build: .
ports:
- "5228:5228/udp"
- "5601:5601"
- "9200:9200"
- "5044:5044"
volumes:
- elk-data:/var/lib/elasticsearch
volumes:
elk-data:
Rails configuration
Now that our ELK stack is up and running we need to tell our Rails Application to start
sending its logs to it. For this we will use the logstash-logger
gem which you will
need to add to your Gemfile
as below.
# Add LogstashLogger for sending logs to Elk
gem 'logstash-logger'
Then we need to create a logstash
initilaizer under config/initializers/logstash.rb
with the contents below. You will also need to make LOGSTASH_HOST
and LOGSTASH_PORT
available as environment variables, note that the value of LOGSTASH_HOST
should not
include any protocol such as https://
or http://
as we are using UDP.
# config/initializers/logstash.rb
# Configure LogStashLogger to send logs to our remote Elk stack in production
if ENV["LOGSTASH_ENABLED"] == "true"
Rails.logger = LogStashLogger.new(
type: :multi_delegator,
outputs: [
{ type: :file, path: "log/#{Rails.env}.log" },
{ type: :udp, host: ENV["LOGSTASH_HOST"], port: ENV["LOGSTASH_PORT"] }
]
)
# Add metadata for Logstash
LogStashLogger.configure do |config|
config.customize_event do |event|
event["application"] = Rails.application.config.session_options[:key].sub(/^_/,"").sub(/_session/,"")
event["environment"] = Rails.env
event["@metadata"] = {
beat: event["application"]
}
end
end
end
Thats it! You should now be able to run your Rails application and see your logs appearing in the Losgstash instance.
For further reference you can find a working example of a Rails application using Logstash Logging as detailed in this guide over on GitHub.