Using Puppet Trusted Facts

How to improve the security around holding private information in Puppet.

What are Trusted Facts:

By default when Puppet runs on a node, Facter will discover system information and other custom values and report them back to the Puppet Master as Facts, however you have no guarantee the node is telling the truth, the facts are self-reporting.

This may not be such an issue with certain facts, i.e. kernelversion or swapfree, as if these are reported incorrectly it will probably just result in the Puppet run failng and not be of any real security concern.

However if you’re using the Roles/Profiles Pattern and you store sensitive/private information in your Puppet code or Hiera files (such as private keys) then if the role or tier facts were to be changed this could easily lead to data leakage and one server receiving the private information of another.

Trusted facts however are extracted from the node’s certificate, which can prove that the CA checked and approved them and prevents them been overridden.

Why Should I use them:

Suppose you have the following (very simple) setup, with web accessible servers sharing a Puppet Master. Each server sits in it’s own subnet and is firewalled off from the other subnets. The two web servers can not talk via internal networks directly to each other.

If the the Private key for the x509 cert used for the HTTPS connection on the so called “secure server” is stored within hiera and installed by Puppet, then if someone with malicious intent was able to compromise the “Corporate Site” server and gain root access they could easily change the role Fact over to the same as the “Secure Server” and subsequently gain access to the private key.

Simple server layout topology, each server in own subnet with FW preventing the two frontend servers communicating with each other.

If you were using Trusted Facts however this would not be possible as the role would be baked into the nodes certificate and as a result would require the Puppet Master to sign a new cert before giving up any private information.

Now you may argue that it requires root access (or at least access to the Puppet user account) to make this work. And if root access has been gained then it’s already game over. Well not entirely, because you only have root access to the one server, and this server doesn’t hold anything confidential (still bad obviously but could be a lot worse), there is also no easy way to pivot off this machine to target others. But using Puppet you could easily pull down all the private info about any other machines in other networks (sharing the same Puppet Master) without even gaining any access to them let alone privileged access (and you don’t need find vulnerabilities in Puppet).

How to use Trusted Facts:

On the Puppet Master if using Open Source < v4.0 you will need to enable trusted_node_data within your puppet.conf file. PE has this enabled by default.

trusted_node_data = true

Then when bringing up new nodes for the first time, before launching the first Puppet run add a new section in the csr_attributes.yaml setting your facts, for example:

# /etc/puppet/csr_attributes.yaml
  "": "secure-site"
  "": "prod"

These facts will then be added into the certificate signed by the Puppet CA (as long as the Puppet CA approves them).
The “” bit is an OID, you cannot use a string here unless it is a registered OID because as part of x509 spec this will be mapped to an OID if it’s not one already. Puppet 3.4 – 3.8 registered a few basic ones within the ppRegCertExt OID range: Puppet 3.8 ppRegCertExt OIDs. However Puppet 4 has now introduced a much more compressive list of OIDs: Puppet 4+ ppRegCertExt OIDs.

Note: I have picked the OIDs ""
and "" arbitrarily, simply using the ppRegCertExt OID range and bumping up the last number to way beyond what Puppet are currently using.
So the example above for Puppet 4 could be simplified to:

  pp_role: "secure-site"
  pp_environment: "prod"

Within your Puppet code the trusted facts are available through the $trusted hash, but to make them more friendly, and usable by your hiera structure you can set global variables to equal those of your trusted ones:

If you add the following to your initial point of entry .pp file (e.g. default.pp or entry.pp):

$role = $trusted['extensions']['']
$tier = $trusted['extensions']['']

You can then use the $role and $tier variables in your hiera hierarchy just as you would with normal facts.

  - "%{::environment}/hiera/role_%{::role}/tier_%{::tier}"
  - "%{::environment}/hiera/role_%{::role}"
  - "%{::environment}/hiera/osfamily/%{::os_family}"
  - "%{::environment}/hiera/virtual/%{::virtual}"
  - "%{::environment}/hiera/common"

Approving Puppet CSR extensions

Unfortunately Puppets build in cert list command does not have the ability to show CSR extension_requests. So you’ll need check these manually, this could easily be done by using openssl:

openssl req -noout -text -in .pem

Seeing it all in action

To help show this in action I have created Docker images, Puppet Master, Corporate site and Secure Site ones. There is a Vagrant template to enable launching these quickly. You will need to have the following installed:

Then simply clone the the pug-puppet repo onto your machine, and pull in the puppet modules:

git clone
librarian-puppet install

Then the Vagrant repo:

git clone

Copy the config.yaml.dist file to config.yaml and change the puppet_repo_dir line to point to the pug-puppet dir you created above.
Now you can launch the containers:

vagrant up pug-puppet-master
vagrant up pug-web-http
vagrant up pug-web-https

Launch a shell in the containers using the docker exec command:

docker exec -it pug-web-http /bin/bash

From within either of the web servers try launching a puppet run:

puppet agent -t

and see what happens. Then try overriding one of the facts like the role by setting it as an External Fact:

echo "role=secure-site" > /etc/facter/facts.d/role.txt

The pug-puppet repo contains 3 branches:

  • master (Trusted facts enabled and enforced)
  • migration (allows nodes with no trusted data in their cert to still connect, but certs with trusted data must always use those facts)
  • not_trusted (standard no-trusted-facts way of doing things)

The migration branch mentioned above shows a simple way to allow you to migrate your servers from not using Trusted Facts over to using them a few at a time, without breaking all the non-migrated ones.

On a side note:

You should use eyaml(or similar) for storing your private information securely at rest in Puppet. As well as making sure only personnel who actually need to day-to-day access to your Puppet/hiera repo that holds your secrets have access.

Nginx and PHP-FPM, bash script for deleting old vhost’s

If you’re using my bash script to create new nginx vhosts (with php-fpm support) you may also require an easy way to remove old vhosts you no longer need (along with all the configs associated with the vhost). I’ve put together this very simple bash script to automate the process:

# @author: Seb Dangerfield
# Created:   02/12/2012
# Modify the following to match your system
# --------------END 
SED=`which sed`
CURRENT_DIR=`dirname $0`
if [ -z $1 ]; then
	echo "No domain name given"
	exit 1
# check the domain is valid!
if [[ "$DOMAIN" =~ $PATTERN ]]; then
	DOMAIN=`echo $DOMAIN | tr '[A-Z]' '[a-z]'`
	echo "Removing vhost for:" $DOMAIN
	echo "invalid domain name"
	exit 1 
echo "What is the username for this site?"
# Remove the user and their home directory
userdel -rf $USERNAME
# Delete the users group from the system
groupdel $USERNAME
# Delete the virtual host config
# Delete the php-fpm config
rm -f $FPMCONF
$NGINX_INIT reload
$PHP_FPM_INIT restart
echo -e "\nSite removed for $DOMAIN"

How to use it:

Please note that because it deletes the users home directory all users files will be deleted, so if you want to keep a copy of the files as a backup do this before running the script.

Simply copy the script from above into a new file called something like If your not using Debian/Ubuntu or have modified the default directories and users for PHP and Nginx then you will need to change the paths and Nginx user at the top of the script to match your system. You will need to change the permissions on the file to make it executable (if it isn’t already):

chmod u+x

Then run the script passing to it the domain name as the only parameter.


It will then prompt you for the linux username used for this site and once that has been provided it will delete all the files related to that site and restart Nginx and PHP-FPM.

NSS error -8023 using AWS SDK for PHP

Please note this fix should also work on Fedora, CentOS and Redhat linux distros if you are seeing the NSS error -8023 when using CURL and PHP.

At work we came up against this odd error message from Amazon Web Services(AWS) SDK for PHP when using the SDK in a forked process on the AWS built AMI:

PHP Fatal error:  Uncaught exception 'cURL_Exception' with message 'cURL resource: Resource id #50; cURL error: SSL connect error (cURL error code 35). See for an explanation of error codes.' in /usr/share/pear/AWSSDKforPHP/lib/requestcore/requestcore.class.php:829

Stack trace:
#0 /usr/share/pear/AWSSDKforPHP/sdk.class.php(1035): RequestCore->send_request()
#1 /usr/share/pear/AWSSDKforPHP/services/swf.class.php(1305): CFRuntime->authenticate('TerminateWorkfl...', Array)
#2 ....php(189): AmazonSWF->terminate_workflow_execution(Array)
#3 ....php(83): daemon->checkSWFExecutions()
#4 ....php(350): daemon->run()
#5 {main}
  thrown in /usr/share/pear/AWSSDKforPHP/lib/requestcore/requestcore.class.php on line 829

Now, cURL error 35 means “A problem occurred somewhere in the SSL/TLS handshake. You really want the error buffer and read the message there as it pinpoints the problem slightly more. Could be certificates (file formats, paths, permissions), passwords, and others.” which is a bit vague and didn’t really help. After setting the CURLOPT_VERBOSE flag in the AWS SDK for PHP we were able to see the real error message:

NSS error -8023

Continue reading

Nginx and PHP-FPM, bash script for creating new vhost’s under separate fpm pools

Using worker pools in PHP-FPM can allow you to easily separate out and isolate virtual hosts that make use of PHP. PHP-FPM allows you to run multiple pools of processes all spawned from the master one and each pool can run as a different user and/or group. Each pool can be further isolated by running in a chroot environment and by overriding the default php.ini values on a per pool basis.

Running PHP for each vhost under a different user/group can help to stop a vulnerability in one site potentially exposing another vhost, it can also stop one malicious owner of a vhost from been able to use PHP to access the files of another site owned by someone else on the same server (in a shared hosting environment).

The process of setting up the web server config and a new PHP-FPM pool for each new vhost on a server can become a rather time consuming and boring process. However as this process follows a fairly standard set of steps it can be easily scripted.

Continue reading

Simple bandwidth monitoring on Linux

If you want a simple lightweight tool to monitor the network traffic in and out of your server vnstat might be just what you need. It keeps hourly, daily and monthly records and provides simple estimates of your expected use, it is also easy to link up to a web based frontend for fancy charts and reporting.

Installing and configuring vnstat is very simple, firstly install using your standard package manager, for example:


apt-get install vnstat


yum install vnstat

Then tell vnstat to create a database for the network interfaces you want to listen to (e.g. eth0):

vnstat -u -i eth0

That’s it, wait a few minutes then run vnstat to view a simple console display of the amount of traffic that has traveled though all the interfaces you’re monitoring:

   eth0 since 01/22/12
          rx:  177.59 MiB      tx:  7.78 MiB      total:  185.37 MiB
                     rx      |     tx      |    total    |   avg. rate
       Jan '12    177.59 MiB |    7.78 MiB |  185.37 MiB |    0.59 kbit/s
     estimated       183 MiB |       7 MiB |     190 MiB |
                     rx      |     tx      |    total    |   avg. rate
     yesterday     12.53 MiB |    1.36 MiB |   13.89 MiB |    1.32 kbit/s
         today      8.28 MiB |     127 KiB |    8.40 MiB |    0.88 kbit/s
     estimated        --     |      --     |      --     |

You can also get vnstat to dump its output in a programming friendly format (semicolon delimited):

vnstat --dumpdb

If you do want a nicer looking interface or one that doesn’t require shell access have a look at: vnstat PHP frontend

If you need a bandwidth monitoring solution that records the utilization of individual protocols instead of just received and transmitted traffic then have a look at bandwidthd

Setting up a centralised syslog server in the cloud

This post should help you get a basic syslog server and client(s) up and running in a virtual environment, It will take you through the implementation of a reasonably secure (using rsyslog’s TLS authentication) yet flexible setup useful to most virtual based server architectures I will assume if you’re reading this that you know what syslog is and what it’s used for. (if not have a quick Google then come back)

 Why Setup a Centralized Syslog Server

  • For convenience – If you for instance have a large number of web servers and you need to diagnose a problem on one of them (maybe not sure which one) you only have to check in one place, if you wanted to compile some statistics from all of them or check if they had all successfully completed a software upgrade.
  • For added security – If someone hacks into one of your servers they will probably try and cover their tracks by erasing any log records created by there presence, however if your logs are also sent to another (hardened) server then the logs will still be available to sysadmins.
  • Another very useful reason which only really applies to virtual servers is to help retain the log files from a terminated server (e.g. shut-down due to decreased demand on your application).

Continue reading

Bash Script to Create new virtual hosts on Nginx each under a different user

This script is a modified version of the original vhost creator script for nginx posted here (Actually it was a modified version of the original version of this script).

The script below will automatically create a new user on the system and adds the nginx user to the new users group. This allows FTP access to be given to the newly created user and the user will only have access to the their site and not to all vhosts running on your server, however as long as the users group has permission to access the files then nginx will still be able to serve them.

NOTE: This setup only helps lock down access to the vhost directories on a web server hosting only static sites, if CGI of any kind (i.e. PHP) is available then this will also need to be locked down so each user has access to a CGI process or set processes running as that user.

What does the script do:

  • Creates a new system user for the site
  • Creates a new vhost config file for nginx using a basic template
  • Creates a new directory for the site, within the new users home directory
  • Adds a simple index.html file to the new directory to show the site is working.
  • Makes sure the new nginx config syntax is correct before trying to reload nginx
  • Reloads Nginx to allow the new vhost to be detected

Debugging fail2ban not starting

If when you’re trying to start Fail2ban you just get the following response:

Starting Fail2ban:                                         [FAILED]

And you check in the fail2ban log file (or system log) and find no errors, it is probably caused by your Fail2ban init script writing the output of the fail2ban-client to /dev/null, effectively just discarding the output. The easy way to debug this is to try directly calling the fail2ban-client which will print out any syntax errors found in its config files. Use it like so:

$ fail2ban-client -x start
WARNING 'action' not defined in 'php-url-fopen'. Using default value
WARNING 'action' not defined in 'lighttpd-fastcgi'. Using default value
ERROR  Error in action definition #iptables[name=SSH, port=ssh, protocol=tcp]
ERROR  Errors in jail 'ssh-iptables'. Skipping...

This should then highlight where the errors are in your config file(s) and allow you to resolve them. You could obviously change the behavior of your init script to stop it discarding the output.