#Logstash PDX DevOps demo
This repo contains a few Vagrant environments demonstrating different Logstash setups.
I used VirtualBox for my hypervisor.
###Getting up and running
Vagrantfile in this repo gives the VMs host-only IP addresses on their own little private network. In order for your host machine to communicate with the VMs via their host-only addresses, you'll need to set up VirtualBox on your host machine first.
Go into VirtualBox's preferences (for the VirtualBox app itself, not the preferences for a specific VM), then Network -> Host-only networks. Click the + sign to add a virtual interface. Give your host machine the IP address 10.0.1.1 and use 255.255.255.0 for the subnet mask. You can leave the IPv6 settings blank.
Vagrantfile also uses the vagrant-hostmanager plugin to distribute each VM's hostname and IP address to the
/etc/hosts file of each other VM.
The host-only network and hostmanager plugin let the VMs communicate via actual hostnames and IP addresses like actual servers would and doesn't require any set up of Vagrant's port forwarding.
To install the vagrant-hostmanager plugin, run:
vagrant plugin install vagrant-hostmanager
After cloning the repo, run the following commands to grab the Puppet modules that are used:
mkdir logstash/modules/ cd logstash/modules/ git clone firstname.lastname@example.org:elasticsearch/puppet-logstash.git logstash git clone email@example.com:elasticsearch/puppet-elasticsearch.git elasticsearch git clone firstname.lastname@example.org:saz/puppet-rsyslog.git rsyslog git clone email@example.com:puppetlabs/puppetlabs-apache.git apache git clone firstname.lastname@example.org:puppetlabs/puppetlabs-stdlib.git stdlib git clone email@example.com:puppetlabs/puppetlabs-concat.git concat git clone firstname.lastname@example.org:puppetlabs/puppetlabs-apt.git apt git clone email@example.com:electrical/puppet-lib-file_concat.git file_concat
####Booting the VMs
Go into any of the directories with a
Vagrantfile in them.
Boot the VMs themselves:
vagrant up --no-provision
...sync the hostnames...
...and provision them:
####Puppet file server
In order to serve the Logstash config file via Puppet, you have to set up a custom Puppet file server mount point.
On the Puppet master VM, edit the
/etc/puppet/fileserver.conf config file.
To create a mountpoint, give it a name, like
[logstash], a path and an
[logstash] path /etc/puppet/files/logstash allow *
logstash is the mountpoint's name (inside of the
/etc/puppet/files/logstash is the base directory.
/etc/puppet/files/ is mounted on the Puppet master VM via Vagrant's NFS-based folder sharing. The source is the
files/ directory in each Vagrant project folder.
/etc/puppet/auth.conf and add the following:
path ~ ^/file_(metadata|content)/logstash auth yes allow *
The above needs to come before the following block:
path /file allow *
Restart the Puppet master daemon:
sudo service puppetmaster restart
On the other VMs, log in with
vagrant ssh and generate a CSR for Puppet:
sudo puppet agent -t
On the Puppet master, sign all of the CSRs that have been submitted:
sudo puppet cert sign --all
sudo puppet agent -t on the other VMs to configure them. Any VMs running Logstash or Elasticsearch will take a few minutes to be configured.
You'll probably want to configure the ElasticSearch VMs first, then the Logstash VM afterwards.
####Setting up Kibana
To install Kibana, log into the kibanathree VM. Set up the Puppet agent first, then do a Puppet run to set up Apache.
Once the VM is added to the Puppet master and you've done a Puppet run, download and install Kibana 3:
cd /tmp wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz tar zxvf kibana-3.1.0.tar.gz mv /tmp/kibana-3.1.0/* /sites/apps/kibana3
/sites/apps/kibana3/config.js and change the value for
elasticsearch: to the URL for one of your Elasticsearch nodes:
Once everything is set up, you should have the following web pages available:
- http://elasticsearch1.local:9200/_nodes: the status of the Elasticsearch cluster in JSON form
http://elasticsearch1.local:9200/_plugin/kopf/: the page for the ElasticSearch kopf plugin
- Lets you see cluster status, distribution of shards and the number of JSON documents (log events) the cluster has stored
- http://dashboard.kibanathree.local/index.html: Kibana dashboard
###Specific pieces, what they are and what they do
The base box the VMs are booted from is made from this Packer template:
If you already have an Ubuntu base box you'd prefer to use, just replace all instances of
ubuntu-13.10-server-amd64 in the
Vagrantfiles with the name of the base box you'd like to use.
Any plain Ubuntu 13.10 base box that does not have Puppet or Java already installed should work.
This repo has 2 provisioning scripts, one for a Puppet master and the other for a Puppet agent. They just install the latest stable versions of the Puppet master and agent packages from the Puppet Labs apt repo.
This repo makes use of a few Puppet modules: