logstash_pdxdevops

Created: 2014-05-19 07:18
Updated: 2015-04-15 08:08

README.md

#Logstash PDX DevOps demo


This repo contains a few Vagrant environments demonstrating different Logstash setups.

I used VirtualBox for my hypervisor.

###Getting up and running

####Vagrant setup

The Vagrantfile in this repo gives the VMs host-only IP addresses on their own little private network. In order for your host machine to communicate with the VMs via their host-only addresses, you'll need to set up VirtualBox on your host machine first.

Go into VirtualBox's preferences (for the VirtualBox app itself, not the preferences for a specific VM), then Network -> Host-only networks. Click the + sign to add a virtual interface. Give your host machine the IP address 10.0.1.1 and use 255.255.255.0 for the subnet mask. You can leave the IPv6 settings blank.

The Vagrantfile also uses the vagrant-hostmanager plugin to distribute each VM's hostname and IP address to the /etc/hosts file of each other VM.

The host-only network and hostmanager plugin let the VMs communicate via actual hostnames and IP addresses like actual servers would and doesn't require any set up of Vagrant's port forwarding.

To install the vagrant-hostmanager plugin, run:

vagrant plugin install vagrant-hostmanager

####Puppet modules

After cloning the repo, run the following commands to grab the Puppet modules that are used:

mkdir logstash/modules/
cd logstash/modules/
git clone git@github.com:elasticsearch/puppet-logstash.git logstash
git clone git@github.com:elasticsearch/puppet-elasticsearch.git elasticsearch
git clone git@github.com:saz/puppet-rsyslog.git rsyslog
git clone git@github.com:puppetlabs/puppetlabs-apache.git apache
git clone git@github.com:puppetlabs/puppetlabs-stdlib.git stdlib
git clone git@github.com:puppetlabs/puppetlabs-concat.git concat
git clone git@github.com:puppetlabs/puppetlabs-apt.git apt
git clone git@github.com:electrical/puppet-lib-file_concat.git file_concat

####Booting the VMs

Go into any of the directories with a Vagrantfile in them.

Boot the VMs themselves:

vagrant up --no-provision

...sync the hostnames...

vagrant hostmanager

...and provision them:

vagrant provision

####Puppet file server

In order to serve the Logstash config file via Puppet, you have to set up a custom Puppet file server mount point.

On the Puppet master VM, edit the /etc/puppet/fileserver.conf config file.

To create a mountpoint, give it a name, like [logstash], a path and an allow rule:

[logstash]
  path /etc/puppet/files/logstash
  allow *

logstash is the mountpoint's name (inside of the []) and /etc/puppet/files/logstash is the base directory. /etc/puppet/files/ is mounted on the Puppet master VM via Vagrant's NFS-based folder sharing. The source is the files/ directory in each Vagrant project folder.

Edit /etc/puppet/auth.conf and add the following:

path ~ ^/file_(metadata|content)/logstash
auth yes 
allow *

The above needs to come before the following block:

path /file
allow *

Restart the Puppet master daemon:

sudo service puppetmaster restart

####Puppet agents

On the other VMs, log in with vagrant ssh and generate a CSR for Puppet:

sudo puppet agent -t

On the Puppet master, sign all of the CSRs that have been submitted:

sudo puppet cert sign --all

Run sudo puppet agent -t on the other VMs to configure them. Any VMs running Logstash or Elasticsearch will take a few minutes to be configured.

You'll probably want to configure the ElasticSearch VMs first, then the Logstash VM afterwards.

####Setting up Kibana

To install Kibana, log into the kibanathree VM. Set up the Puppet agent first, then do a Puppet run to set up Apache.

Once the VM is added to the Puppet master and you've done a Puppet run, download and install Kibana 3:

cd /tmp
wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
tar zxvf kibana-3.1.0.tar.gz
mv /tmp/kibana-3.1.0/* /sites/apps/kibana3

Edit /sites/apps/kibana3/config.js and change the value for elasticsearch: to the URL for one of your Elasticsearch nodes:

elasticsearch: "http://elasticsearch1.local:9200",

####Web pages

Once everything is set up, you should have the following web pages available:

###Specific pieces, what they are and what they do

####Base boxes

The base box the VMs are booted from is made from this Packer template:

https://github.com/nickchappell/packer-templates/tree/master/old_templates/ubuntu-13.10-server-amd64

If you already have an Ubuntu base box you'd prefer to use, just replace all instances of ubuntu-13.10-server-amd64 in the Vagrantfiles with the name of the base box you'd like to use.

Any plain Ubuntu 13.10 base box that does not have Puppet or Java already installed should work.

####Provisioning scripts

This repo has 2 provisioning scripts, one for a Puppet master and the other for a Puppet agent. They just install the latest stable versions of the Puppet master and agent packages from the Puppet Labs apt repo.

####Modules

This repo makes use of a few Puppet modules:

Cookies help us deliver our services. By using our services, you agree to our use of cookies Learn more