Hey all,
I´ve been rather jealous over those very fancy reporting graphs[1] included in FreeNAS got me reading this[2] article that explains how to set that up manually. That information is priceless in times when your system just feels wrong, but you´re not exactly sure why. Those graphs can show that and more.
[1]: http://doc.freenas.org/index.php/Reporting
[2]: http://www.flagword.net/2014/01/installing-and-configuring-graphite-with-collectd-on-freebsd/
The article was centred around NGINX, but as I´m more of an Apache kind of guy, this is what I´ll be showing you here. So let´s get to it!
Install and configure graphite, then start carbon (graphite backend)
Be mindful of carbon´s storage retentions here! Read up on examples:
http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-schemas-conf
Install, configure and start uwsgi
The "-p 1" uwsgi flag you echo into rc.conf decides how many threads it should work on, in this case one thread. Most servers nowadays have many cores, so if you have four cores, put in "-p 4" for better performance.
Install, configure and start apache
Remember to set a HOST_NAME value here! Just like this:
Lastly install, configure and start collectd
You need the HOST_NAME variable set here as well. In case you logged out/in, or doing this from another session:
In collectd.conf, you have to specify the network interface you using. I´ve already added the usual ones, but If your interface is missing, just add another line in there and restart collectd for it show up. Same goes for hard drives or RAID devices.
You´ll end up with the dashboard accessible from https://myhostname.foo.bar/dashboard, no non-standard port that will always be a source of misunderstanding otherwise, but that´s just a matter of taste really. For Active Directory login you need a user that is used for LDAP-searches, so make it very unprivileged, only being able to read what it needs to read, since the password is written here in plain text. For smaller environments you can use "AuthBasicProvider file" instead, creating local apache accounts with htpasswd, instructions can found at http://httpd.apache.org/docs/current/howto/auth.html. The standard root URL http://myhostname.foo.bar/dashboard is automatically redirected to it´s https equivalent. For systems that are "out there" on the web, that is a must, in my opinion.
/Sebulon
I´ve been rather jealous over those very fancy reporting graphs[1] included in FreeNAS got me reading this[2] article that explains how to set that up manually. That information is priceless in times when your system just feels wrong, but you´re not exactly sure why. Those graphs can show that and more.
[1]: http://doc.freenas.org/index.php/Reporting
[2]: http://www.flagword.net/2014/01/installing-and-configuring-graphite-with-collectd-on-freebsd/
The article was centred around NGINX, but as I´m more of an Apache kind of guy, this is what I´ll be showing you here. So let´s get to it!
Install and configure graphite, then start carbon (graphite backend)
Be mindful of carbon´s storage retentions here! Read up on examples:
http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-schemas-conf
Code:
if [ "$(grep -c carbon /etc/rc.conf 2>&1)" -eq "0" ]; then
if [ "$(pkg_info | egrep -c '(graphite-web|carbon)' 2>&1)" -eq "0" ]; then
cd /usr/ports/www/py-graphite-web
make install distclean
if [ $? -gt "0" ]; then
echo '+-- Failed to install py-graphite-web. Aborting.'
exit 1
fi
cat > /usr/local/etc/carbon/storage-schemas.conf << EOF
[carbon]
pattern = ^carbon\.
retentions = 60:90d
[default]
pattern = .*
retentions = 15s:7d,1m:28d,60m:1y
EOF
cat > /usr/local/etc/carbon/carbon.conf << EOF
[cache]
GRAPHITE_ROOT = /usr/local/graphite
GRAPHITE_CONF_DIR = /usr/local/etc/carbon
GRAPHITE_STORAGE_DIR = /usr/local/graphite/storage/
STORAGE_DIR = /usr/local/graphite/storage/
LOCAL_DATA_DIR = /usr/local/graphite/storage/whisper/
CONF_DIR = /usr/local/etc/carbon
LOG_DIR = /usr/local/graphite/storage/log/
PID_DIR = /var/run
ENABLE_LOGROTATION = True
USER =
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500
MAX_CREATES_PER_MINUTE = 50
LINE_RECEIVER_INTERFACE = 127.0.0.1
LINE_RECEIVER_PORT = 2003
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 127.0.0.1
UDP_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2004
LOG_LISTENER_CONNECTIONS = True
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 127.0.0.1
CACHE_QUERY_PORT = 7002
USE_FLOW_CONTROL = True
LOG_UPDATES = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = True
CACHE_WRITE_STRATEGY = sorted
WHISPER_AUTOFLUSH = False
WHISPER_FALLOCATE_CREATE = True
[relay]
LINE_RECEIVER_INTERFACE = 127.0.0.1
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2014
LOG_LISTENER_CONNECTIONS = True
RELAY_METHOD = rules
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
[aggregator]
LINE_RECEIVER_INTERFACE = 127.0.0.1
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2024
LOG_LISTENER_CONNECTIONS = True
FORWARD_ALL = True
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5
EOF
echo 'carbon_enable="YES"' >> /etc/rc.conf
service carbon start
if [ $? -gt "0" ]; then
echo '+-- Failed to start carbon service. Aborting.'
exit 1
fi
while [ ! -d /usr/local/graphite ]; do
sleep 1
done
mkdir -p /usr/local/graphite/storage/log/webapp /usr/local/graphite/storage/rrd
for i in img js css html; do
cp -r /usr/local/share/graphite-web/content/$i /usr/local/graphite/webapp/
done
cat > /usr/local/lib/python2.7/site-packages/graphite/local_settings.py << EOF
#TIME_ZONE = 'Europe/Stockholm'
GRAPHITE_ROOT = '/usr/local/graphite'
CONF_DIR = '/usr/local/etc/graphite'
STORAGE_DIR = '/usr/local/graphite/storage'
CONTENT_DIR = '/usr/local/graphite/webapp'
DASHBOARD_CONF = '/usr/local/etc/graphite/dashboard.conf'
GRAPHTEMPLATES_CONF = '/usr/local/etc/graphite/graphTemplates.conf'
WHISPER_DIR = '/usr/local/graphite/storage/whisper'
RRD_DIR = '/usr/local/graphite/storage/rrd'
LOG_DIR = '/usr/local/graphite/storage/log/webapp'
DATABASES = {
'default': {
'NAME': '/usr/local/graphite/storage/graphite.db',
'ENGINE': 'django.db.backends.sqlite3',
'USER': '',
'PASSWORD': '',
'HOST': '',
'PORT': ''
}
}
EOF
cd /usr/local/lib/python2.7/site-packages/graphite
echo 'no' | python manage.py syncdb
if [ $? -gt "0" ]; then
echo '+-- Failed to run "manage.py syncdb". Aborting.'
exit 1
fi
cat > /usr/local/etc/graphite/graphite.wsgi << EOF
import os, sys
sys.path.append('/usr/local/graphite/webapp')
os.environ['DJANGO_SETTINGS_MODULE'] = 'graphite.settings'
import django
import django.core.handlers.wsgi
django.setup()
application = django.core.handlers.wsgi.WSGIHandler()
from graphite.logger import log
log.info("graphite.wsgi - pid %d - reloading search index" % os.getpid())
import graphite.metrics.search
EOF
cat > /usr/local/etc/graphite/dashboard.conf << EOF
[ui]
default_graph_width = 400
default_graph_height = 250
automatic_variants = true
refresh_interval = 60
autocomplete_delay = 375
merge_hover_delay = 750
theme = default
[keyboard-shortcuts]
toggle_toolbar = ctrl-z
toggle_metrics_panel = ctrl-space
erase_all_graphs = alt-x
save_dashboard = alt-s
completer_add_metrics = alt-enter
completer_del_metrics = alt-backspace
give_completer_focus = shift-space
EOF
cat > /usr/local/etc/graphite/graphTemplates.conf << EOF
[default]
background = black
foreground = white
majorLine = white
minorLine = grey
lineColors = blue,green,red,purple,brown,yellow,aqua,grey,magenta,pink,gold,rose
fontName = Sans
fontSize = 10
fontBold = False
fontItalic = False
[noc]
background = black
foreground = white
majorLine = white
minorLine = grey
lineColors = blue,green,red,yellow,purple,brown,aqua,grey,magenta,pink,gold,rose
fontName = Sans
fontSize = 10
fontBold = False
fontItalic = False
[plain]
background = white
foreground = black
minorLine = grey
majorLine = rose
[summary]
background = black
[alphas]
background = white
foreground = black
majorLine = grey
minorLine = rose
lineColors = 00ff00aa,ff000077,00337799
EOF
chown -R www:www /usr/local/graphite
else
echo 'Package "graphite-web" or "carbon" already installed. Aborting.'
exit 1
fi
else
echo 'Service "carbon" is already in "rc.conf". Aborting.'
exit 1
fiInstall, configure and start uwsgi
The "-p 1" uwsgi flag you echo into rc.conf decides how many threads it should work on, in this case one thread. Most servers nowadays have many cores, so if you have four cores, put in "-p 4" for better performance.
Code:
if [ "$(grep -c uwsgi /etc/rc.conf 2>&1)" -eq "0" ]; then
if [ "$(pkg_info | grep -c uwsgi 2>&1)" -eq "0" ]; then
cd /usr/ports/www/uwsgi
make install distclean
if [ $? -gt "0" ]; then
echo '+-- Failed to install uwsgi. Aborting.'
exit 1
fi
echo 'uwsgi_enable="YES"' >> /etc/rc.conf
echo 'uwsgi_flags="-L -M -p 1 --socket /tmp/uwsgi.sock --gid 80 --uid 80 --python-path /usr/local/lib/python2.7/site-packages/ --chdir /usr/local/etc/graphite/ -w graphite"' >> /etc/rc.conf
service uwsgi start
if [ $? -gt "0" ]; then
echo '+-- Failed to start uwsgi service. Aborting.'
exit 1
fi
else
echo 'Package "uwsgi" already installed. Aborting.'
exit 1
fi
else
echo 'Service "uwsgi" is already in "rc.conf". Aborting.'
exit 1
fiInstall, configure and start apache
Remember to set a HOST_NAME value here! Just like this:
# HOST_NAME="myhostname.foo.bar"
Code:
if [ -z ${HOST_NAME} ]; then
echo "You have to assign a HOST_NAME variable!"
exit 1
fi
if [ "$(grep -c apache22 /etc/rc.conf 2>&1)" -eq "0" ]; then
if [ "$(pkg_info | grep -c apache22 2>&1)" -eq "0" ]; then
cd /usr/ports/www/apache22
make install distclean
if [ $? -gt "0" ]; then
echo '+-- Failed to install apache22. Aborting.'
exit 1
fi
cd /usr/ports/www/mod_wsgi3
make install distclean
if [ $? -gt "0" ]; then
echo '+-- Failed to install mod_wsgi3. Aborting.'
exit 1
fi
if [ ! -d /etc/pki/graphite ]; then
mkdir -p /etc/pki/graphite
fi
openssl req -new -newkey rsa:4096 -days 3650 -nodes -x509 -subj "/CN=${HOST_NAME}" -keyout /etc/pki/graphite/${HOST_NAME}.key -out /etc/pki/graphite/${HOST_NAME}.cert
cat > /usr/local/etc/apache22/Includes/graphite.conf << EOF
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/\$1 [R,L]
WSGISocketPrefix /tmp/wsgi
Listen 443
<VirtualHost *:443>
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /etc/pki/graphite/${HOST_NAME}.cert
SSLCertificateKeyFile /etc/pki/graphite/${HOST_NAME}.key
ServerName ${HOST_NAME}
DocumentRoot "/usr/local/graphite/webapp/"
ErrorLog /usr/local/graphite/storage/log/webapp/error.log
CustomLog /usr/local/graphite/storage/log/webapp/access.log common
WSGIDaemonProcess graphite processes=5 threads=5 inactivity-timeout=120 display-name=graphite
WSGIProcessGroup graphite
WSGIApplicationGroup %{GLOBAL}
WSGIImportScript /usr/local/etc/graphite/graphite.wsgi process-group=graphite application-group=%{GLOBAL}
WSGIScriptAlias / /usr/local/etc/graphite/graphite.wsgi
Alias /content/ /usr/local/graphite/webapp/
<Location "/content/">
SetHandler None
</Location>
Alias /media/ /usr/local/lib/python2.7/site-packages/django
<Location "/media/">
SetHandler None
</Location>
Alias /static/ "/usr/local/lib/python2.7/site-packages/django/contrib/admin/static/"
<Location "/static/">
SetHandler None
</Location>
<Directory "/usr/local/lib/python2.7/site-packages/django/contrib/admin/static/">
Order deny,allow
Allow from all
</Directory>
<Directory /usr/local/etc/graphite/>
Order deny,allow
Allow from all
</Directory>
<Directory /usr/local/graphite/webapp/>
Order deny,allow
Allow from all
</Directory>
<Location "/">
AuthType basic
AuthName "Graphite"
AuthBasicProvider ldap
AuthLDAPBindDN "CN=someuser,OU=Users,DC=ad,DC=foo,DC=bar"
AuthLDAPBindPassword VerySecretPassword
AuthLDAPURL ldap://ad.foo.bar:3268/DC=ad,DC=foo,DC=bar?sAMAccountName?sub?(objectClass=*)
AuthLDAPGroupAttributeIsDN off
Require valid-user
</Location>
</VirtualHost>
EOF
echo 'apache22_enable="YES"' >> /etc/rc.conf
service apache22 start
if [ $? -gt "0" ]; then
echo '+-- Failed to start apache service. Aborting.'
exit 1
fi
else
echo 'Package "apache22" already installed. Aborting.'
exit 1
fi
else
echo 'Service "apache22" is already in "rc.conf". Aborting.'
exit 1
fiLastly install, configure and start collectd
You need the HOST_NAME variable set here as well. In case you logged out/in, or doing this from another session:
# HOST_NAME="myhostname.foo.bar"In collectd.conf, you have to specify the network interface you using. I´ve already added the usual ones, but If your interface is missing, just add another line in there and restart collectd for it show up. Same goes for hard drives or RAID devices.
Code:
if [ -z ${HOST_NAME} ]; then
echo "You have to assign a HOST_NAME variable!"
exit 1
fi
if [ "$(grep -c collectd /etc/rc.conf 2>&1)" -eq "0" ]; then
if [ "$(pkg_info | grep -c collectd5 2>&1)" -eq "0" ]; then
cd /usr/ports/net-mgmt/collectd5
make install distclean
if [ $? -gt "0" ]; then
echo '+-- Failed to install collectd5. Aborting.'
exit 1
fi
### I had this issue, not saying you will but best to leave this in, the rrdtool plugin wouldn´t work without it ###
if [ "$(ldd /usr/local/lib/collectd/rrdtool.so | grep libpixman-1 | grep -c 'not found' 2>&1)" -gt "0" ]; then
ln -s $(find /usr/local/lib/ | grep libpixman-1.so.) /usr/local/lib/$(ldd /usr/local/lib/collectd/rrdtool.so | grep libpixman-1 | grep 'not found' | awk '{print $1}')
fi
cat > /usr/local/etc/collectd.conf << EOF
LoadPlugin syslog
LoadPlugin cpu
LoadPlugin df
LoadPlugin disk
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin rrdtool
LoadPlugin write_graphite
LoadPlugin zfs_arc
<Plugin df>
FSType "zfs"
</Plugin>
<Plugin disk>
Disk "/^vtbd[0-9]+$/"
Disk "/^[hs]d[a-f][0-9]?$/"
Disk "/^d[a-f][0-9]+$/"
IgnoreSelected false
</Plugin>
<Plugin interface>
Interface "/^vtnet[0-9]+$/"
Interface "/^vlan[0-9]+$/"
Interface "/^lagg[0-9]+$/"
Interface "/^bxe[0-9]+$/"
Interface "/^de[0-9]+$/"
Interface "/^em[0-9]+$/"
Interface "/^igb[0-9]+$/"
Interface "/^ixgbe[0-9]+$/"
Interface "/^le[0-9]+$/"
Interface "/^ti[0-9]+$/"
Interface "/^txp[0-9]+$/"
Interface "/^vx[0-9]+$/"
Interface "/^miibus[0-9]+$/"
Interface "/^ae[0-9]+$/"
Interface "/^age[0-9]+$/"
Interface "/^alc[0-9]+$/"
Interface "/^ale[0-9]+$/"
Interface "/^bce[0-9]+$/"
Interface "/^bfe[0-9]+$/"
Interface "/^bge[0-9]+$/"
Interface "/^dc[0-9]+$/"
Interface "/^et[0-9]+$/"
Interface "/^fxp[0-9]+$/"
Interface "/^jme[0-9]+$/"
Interface "/^lge[0-9]+$/"
Interface "/^msk[0-9]+$/"
Interface "/^nfe[0-9]+$/"
Interface "/^nge[0-9]+$/"
Interface "/^nve[0-9]+$/"
Interface "/^pcn[0-9]+$/"
Interface "/^re[0-9]+$/"
Interface "/^rl[0-9]+$/"
Interface "/^sf[0-9]+$/"
Interface "/^sge[0-9]+$/"
Interface "/^sis[0-9]+$/"
Interface "/^sk[0-9]+$/"
Interface "/^ste[0-9]+$/"
Interface "/^stge[0-9]+$/"
Interface "/^tl[0-9]+$/"
Interface "/^tx[0-9]+$/"
Interface "/^vge[0-9]+$/"
Interface "/^vr[0-9]+$/"
Interface "/^wb[0-9]+$/"
Interface "/^xl[0-9]+$/"
Interface "/^cs[0-9]+$/"
Interface "/^ed[0-9]+$/"
Interface "/^ex[0-9]+$/"
Interface "/^ep[0-9]+$/"
Interface "/^fe[0-9]+$/"
Interface "/^sn[0-9]+$/"
Interface "/^xe[0-9]+$/"
IgnoreSelected false
</Plugin>
<Plugin rrdtool>
DataDir "/usr/local/graphite/storage/rrd"
CreateFilesAsync false
CacheTimeout 120
CacheFlush 900
WritesPerSecond 50
</Plugin>
<Plugin write_graphite>
<Node "${HOST_NAME}">
Host "127.0.0.1"
Port "2003"
Protocol "tcp"
LogSendErrors true
Prefix "collectd."
</Node>
</Plugin>
EOF
echo 'collectd_enable="YES"' >> /etc/rc.conf
service collectd start
if [ $? -gt "0" ]; then
echo '+-- Failed to start collectd service. Aborting.'
exit 1
fi
else
echo 'Package "collectd5" already installed. Aborting.'
exit 1
fi
else
echo 'Service "collectd" is already in "rc.conf". Aborting.'
exit 1
fi
echo 'Graphite installation complete.'You´ll end up with the dashboard accessible from https://myhostname.foo.bar/dashboard, no non-standard port that will always be a source of misunderstanding otherwise, but that´s just a matter of taste really. For Active Directory login you need a user that is used for LDAP-searches, so make it very unprivileged, only being able to read what it needs to read, since the password is written here in plain text. For smaller environments you can use "AuthBasicProvider file" instead, creating local apache accounts with htpasswd, instructions can found at http://httpd.apache.org/docs/current/howto/auth.html. The standard root URL http://myhostname.foo.bar/dashboard is automatically redirected to it´s https equivalent. For systems that are "out there" on the web, that is a must, in my opinion.
/Sebulon
Copy/paste the code into:
