Using Puppet Trusted Facts

How to improve the security around holding private information in Puppet.

What are Trusted Facts:

By default when Puppet runs on a node, Facter will discover system information and other custom values and report them back to the Puppet Master as Facts, however you have no guarantee the node is telling the truth, the facts are self-reporting.

This may not be such an issue with certain facts, i.e. kernelversion or swapfree, as if these are reported incorrectly it will probably just result in the Puppet run failng and not be of any real security concern.

However if you’re using the Roles/Profiles Pattern and you store sensitive/private information in your Puppet code or Hiera files (such as private keys) then if the role or tier facts were to be changed this could easily lead to data leakage and one server receiving the private information of another.

Trusted facts however are extracted from the node’s certificate, which can prove that the CA checked and approved them and prevents them been overridden.

Why Should I use them:

Suppose you have the following (very simple) setup, with web accessible servers sharing a Puppet Master. Each server sits in it’s own subnet and is firewalled off from the other subnets. The two web servers can not talk via internal networks directly to each other.

If the the Private key for the x509 cert used for the HTTPS connection on the so called “secure server” is stored within hiera and installed by Puppet, then if someone with malicious intent was able to compromise the “Corporate Site” server and gain root access they could easily change the role Fact over to the same as the “Secure Server” and subsequently gain access to the private key.

Simple server layout topology, each server in own subnet with FW preventing the two frontend servers communicating with each other.

If you were using Trusted Facts however this would not be possible as the role would be baked into the nodes certificate and as a result would require the Puppet Master to sign a new cert before giving up any private information.

Now you may argue that it requires root access (or at least access to the Puppet user account) to make this work. And if root access has been gained then it’s already game over. Well not entirely, because you only have root access to the one server, and this server doesn’t hold anything confidential (still bad obviously but could be a lot worse), there is also no easy way to pivot off this machine to target others. But using Puppet you could easily pull down all the private info about any other machines in other networks (sharing the same Puppet Master) without even gaining any access to them let alone privileged access (and you don’t need find vulnerabilities in Puppet).

How to use Trusted Facts:

On the Puppet Master if using Open Source < v4.0 you will need to enable trusted_node_data within your puppet.conf file. PE has this enabled by default.

[master]
...
trusted_node_data = true
...

Then when bringing up new nodes for the first time, before launching the first Puppet run add a new section in the csr_attributes.yaml setting your facts, for example:

# /etc/puppet/csr_attributes.yaml
extension_requests:
  "1.3.6.1.4.1.34380.1.1.100": "secure-site"
  "1.3.6.1.4.1.34380.1.1.101": "prod"

These facts will then be added into the certificate signed by the Puppet CA (as long as the Puppet CA approves them).
The “1.3.6.1.4.1.34380.1.1.100” bit is an OID, you cannot use a string here unless it is a registered OID because as part of x509 spec this will be mapped to an OID if it’s not one already. Puppet 3.4 – 3.8 registered a few basic ones within the ppRegCertExt OID range: Puppet 3.8 ppRegCertExt OIDs. However Puppet 4 has now introduced a much more compressive list of OIDs: Puppet 4+ ppRegCertExt OIDs.

Note: I have picked the OIDs "1.3.6.1.4.1.34380.1.1.100"
and "1.3.6.1.4.1.34380.1.1.100" arbitrarily, simply using the ppRegCertExt OID range and bumping up the last number to way beyond what Puppet are currently using.
So the example above for Puppet 4 could be simplified to:

extension_requests:
  pp_role: "secure-site"
  pp_environment: "prod"

Within your Puppet code the trusted facts are available through the $trusted hash, but to make them more friendly, and usable by your hiera structure you can set global variables to equal those of your trusted ones:

If you add the following to your initial point of entry .pp file (e.g. default.pp or entry.pp):

$role = $trusted['extensions']['1.3.6.1.4.1.34380.1.1.100']
$tier = $trusted['extensions']['1.3.6.1.4.1.34380.1.1.101']

You can then use the $role and $tier variables in your hiera hierarchy just as you would with normal facts.

:hierarchy:
  - "%{::environment}/hiera/role_%{::role}/tier_%{::tier}"
  - "%{::environment}/hiera/role_%{::role}"
  - "%{::environment}/hiera/osfamily/%{::os_family}"
  - "%{::environment}/hiera/virtual/%{::virtual}"
  - "%{::environment}/hiera/common"

Approving Puppet CSR extensions

Unfortunately Puppets build in cert list command does not have the ability to show CSR extension_requests. So you’ll need check these manually, this could easily be done by using openssl:

openssl req -noout -text -in .pem

Seeing it all in action

To help show this in action I have created Docker images, Puppet Master, Corporate site and Secure Site ones. There is a Vagrant template to enable launching these quickly. You will need to have the following installed:

Then simply clone the the pug-puppet repo onto your machine, and pull in the puppet modules:

git clone https://github.com/sedan07/pug-puppet.git
librarian-puppet install

Then the Vagrant repo:

git clone https://github.com/sedan07/pug-vagrant.git

Copy the config.yaml.dist file to config.yaml and change the puppet_repo_dir line to point to the pug-puppet dir you created above.
Now you can launch the containers:

vagrant up pug-puppet-master
vagrant up pug-web-http
vagrant up pug-web-https

Launch a shell in the containers using the docker exec command:

docker exec -it pug-web-http /bin/bash

From within either of the web servers try launching a puppet run:

puppet agent -t

and see what happens. Then try overriding one of the facts like the role by setting it as an External Fact:

echo "role=secure-site" > /etc/facter/facts.d/role.txt

The pug-puppet repo contains 3 branches:

  • master (Trusted facts enabled and enforced)
  • migration (allows nodes with no trusted data in their cert to still connect, but certs with trusted data must always use those facts)
  • not_trusted (standard no-trusted-facts way of doing things)

The migration branch mentioned above shows a simple way to allow you to migrate your servers from not using Trusted Facts over to using them a few at a time, without breaking all the non-migrated ones.

On a side note:

You should use eyaml(or similar) for storing your private information securely at rest in Puppet. As well as making sure only personnel who actually need to day-to-day access to your Puppet/hiera repo that holds your secrets have access.

Leave a Reply

Your email address will not be published. Required fields are marked *