Securing a Private Docker Registry

So when I researched this a few weeks back most of the guidance I found suggested using Basic Auth. Now nothing wrong with this method as such, it works after all. However if you’re running a registry for more than one user you obviously don’t want to have just one username/password to access it. This then means having a way to add new users easily to it + “bot” users for your servers and so on.

However there is actually a much better way, using x.509 certificates (the same method used to verify and authenticate access to the docker daemon) with your own self-signed CA. Once a user has their client key and certificate setup the authentication is transparent to them. It also provides stronger security than a simple username/password combo.

The Docker docs go through the basic process for configuring this: https://docs.docker.com/articles/certificates/. However they don’t go through how to create your own CA. There are a few important things to keep in mind first: The security of the entire thing rests on keeping your root CA private key secret. Failure to do so means a miscreant with your private key could easily sign any number of new client certificates gaining full access to your registry. You will also need to ensure you have a process for signing new users certificate requests (CSRs), as you certainly don’t want to be generating the private key for them and emailing it over.

You may already have a a PKI implementation within your organisation that you can reuse here (such as used for providing access to build systems, internal wikis and so fourth), if not you have a few choices: You can either generate the Root CA using OpenSSL (or another encryption toolkit) and manage it all your self or use a PKI system, such as Cloudflare’s CFSSL lib which allows to specify all the config and CSRs in json files.

If you wanted to use OpenSSL here is a quick and dirty guide on how you would go about this:

Setup:
Create a new empty directory to create the root CA in then create the basic dir structure and required files:

mkdir certs crl newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial

Openssl Configuration:
Copy your systems default openssl.cnf file to the cwd and name it something like: “openssl-ca.cnf”. Then edit it to setup the CA and client sections (un-commenting and editing lines where appropriate):

[ usr_cert ]
basicConstraints=CA:FALSE
nsCertType = client
nsComment = "Docker Registry CA"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true
keyUsage = cRLSign, keyCertSign

And change the default directory to the current one:

[ CA_default ]
dir = . # Where everything is kept

The usr_cert section is used when creating the client certificates, the v3_ca is used for the CA creation, it simply configs the type of cert we want it what we want it it be usable for.

Generating the Private Key:

openssl genrsa -aes256 -out private/docker_ca.key 4096

Self-signing the certificate to be usable as a CA:

openssl req -new -x509 -days 3650 -key private/docker_ca.key -sha256 -extensions v3_ca -out certs/docker_ca.crt -config openssl-ca.cnf

Now you can go about signing CSRs for the client certificates:

Sign a signing request using your CA key:

openssl ca -keyfile private/docker_ca.key -cert certs/docker_ca.crt -extensions usr_cert -notext -md sha256 -in certs/sebdangerfield.csr -out certs/sebdangerfield.crt -config openssl-ca.cnf -days 366

Troubleshooting:

If when you try and authenticate with your Docker Registry Docker thinks the private key and certificate don’t match make sure your client certificate is the first one in your client.cert file (i.e. the certs making up the chain come after).

AngularJS Protractor “App Already Bootstrapped” error

When attempting to run an End-to-End test in Protractor for a AngularJS web app with a non-angular login page I received the following error message:

UnknownError: Error Message => '[ng:btstrpd] App Already Bootstrapped with this Element '<html lang="en" data-ng-app=

The test took the same structure as below (NOTE: this is not the real test):

describe('Test a page', function() {
    var ptor;
 
    beforeEach(function() {
        ptor = protractor.getInstance();
 
        browser.driver.get(ptor.baseUrl + '/login');
 
        browser.driver.findElement(by.id('user_email')).sendKeys('test@example.com');
        browser.driver.findElement(by.id('user_password')).sendKeys('passpass');
 
        browser.driver.findElement(by.id('login')).click();
 
        browser.driver.wait(function() {
            return browser.driver.getCurrentUrl().then(function(url) {
                return (/\/#\/$/.test(url));
            });
        });
 
        browser.get('#/');
    });
 
    it('Should list some item on this page', function() {
        browser.get('#/page2');
 
        browser.findElements(by.css('.my-item')).then(function(results) {
            expect(results.length).toBeGreaterThan(0);
        });
 
    });
});

The Angular docs suggests the error is caused when bootstrapping the application in both the HTML and in the JS: http://docs.angularjs.org/error/ng:btstrpd. However I was not doing this.

I had added the line browser.get('#/'); to resolve an issue where Protractor was reporting:

Angular could not be found on the page

This appeared to be caused by Protractor not detecting Angular on a page it had been redirected too (i.e. from the non-angular login page to the Angular app page) so I added in the get() call to explicitly load an Angular page in, this however caused the double bootstrap issue.

It turned out this was due to having two browser.get() in one test run (in the beforeEach() and the it()). Replacing the one in the beforeEach() with the following solved the problem:

browser.driver.get('about:blank');

This clears the currently loaded page, so on the next get() call it starts again and doesn’t think the Angular app has already been bootstrapped. I couldn’t see a better way to do this in the docs, however let me know if you do know of a nicer way to solve this problem.

Code Club – Space Invaders

I’ve been running a Code Club at a school only a few miles away from where I work for 5 weeks now and I’ve been thoroughly enjoying it. Although it does a good job of challenging for ‘busiest hour’ of my week award each time.

As the children had already done some scratch (namely “Whack-a-Witch”) in year 5 we decided to jump straight into the Level 2 projects provided by Code Club. Starting with the Fish Chomp one which the children completed in the first two weeks without any major hiccups. We then moved onto the fruit machine, this provided a fair bit of amusement with the images changing too fast to be able to tell what they were and similar. The main parts the children found challenging were:

  • The event broadcasting
  • The large nested conditional statement at the end of the project (to determine if the player had won).
  • Scoping of variables (using “For this sprite only” for some variables and “For all sprites” for others.)

About half the group advanced onto the “What’s that” project and faced some of the same complications.

Continue reading

Nginx and PHP-FPM, bash script for deleting old vhost’s

If you’re using my bash script to create new nginx vhosts (with php-fpm support) you may also require an easy way to remove old vhosts you no longer need (along with all the configs associated with the vhost). I’ve put together this very simple bash script to automate the process:

#!/bin/bash
# @author: Seb Dangerfield
# http://www.sebdangerfield.me.uk/ 
# Created:   02/12/2012
 
# Modify the following to match your system
NGINX_CONFIG='/etc/nginx/sites-available'
NGINX_SITES_ENABLED='/etc/nginx/sites-enabled'
PHP_INI_DIR='/etc/php5/fpm/pool.d'
NGINX_INIT='/etc/init.d/nginx'
PHP_FPM_INIT='/etc/init.d/php5-fpm'
# --------------END 
SED=`which sed`
CURRENT_DIR=`dirname $0`
 
if [ -z $1 ]; then
	echo "No domain name given"
	exit 1
fi
DOMAIN=$1
 
# check the domain is valid!
PATTERN="^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$";
if [[ "$DOMAIN" =~ $PATTERN ]]; then
	DOMAIN=`echo $DOMAIN | tr '[A-Z]' '[a-z]'`
	echo "Removing vhost for:" $DOMAIN
else
	echo "invalid domain name"
	exit 1 
fi
 
echo "What is the username for this site?"
read USERNAME
HOME_DIR=$USERNAME
 
# Remove the user and their home directory
userdel -rf $USERNAME
# Delete the users group from the system
groupdel $USERNAME
 
# Delete the virtual host config
rm -f $NGINX_CONFIG/$DOMAIN.conf
rm -f $NGINX_SITES_ENABLED/$DOMAIN.conf
 
# Delete the php-fpm config
FPMCONF="$PHP_INI_DIR/$DOMAIN.pool.conf"
rm -f $FPMCONF
 
$NGINX_INIT reload
$PHP_FPM_INIT restart
 
echo -e "\nSite removed for $DOMAIN"

How to use it:

Please note that because it deletes the users home directory all users files will be deleted, so if you want to keep a copy of the files as a backup do this before running the script.

Simply copy the script from above into a new file called something like remove_php_site.sh. If your not using Debian/Ubuntu or have modified the default directories and users for PHP and Nginx then you will need to change the paths and Nginx user at the top of the script to match your system. You will need to change the permissions on the remove_php_site.sh file to make it executable (if it isn’t already):

chmod u+x remove_php_site.sh

Then run the remove_php_site.sh script passing to it the domain name as the only parameter.

./remove_php_site.sh example.com

It will then prompt you for the linux username used for this site and once that has been provided it will delete all the files related to that site and restart Nginx and PHP-FPM.

NSS error -8023 using AWS SDK for PHP

Please note this fix should also work on Fedora, CentOS and Redhat linux distros if you are seeing the NSS error -8023 when using CURL and PHP.

At work we came up against this odd error message from Amazon Web Services(AWS) SDK for PHP when using the SDK in a forked process on the AWS built AMI:

PHP Fatal error:  Uncaught exception 'cURL_Exception' with message 'cURL resource: Resource id #50; cURL error: SSL connect error (cURL error code 35). See http://curl.haxx.se/libcurl/c/libcurl-errors.html for an explanation of error codes.' in /usr/share/pear/AWSSDKforPHP/lib/requestcore/requestcore.class.php:829

Stack trace:
#0 /usr/share/pear/AWSSDKforPHP/sdk.class.php(1035): RequestCore->send_request()
#1 /usr/share/pear/AWSSDKforPHP/services/swf.class.php(1305): CFRuntime->authenticate('TerminateWorkfl...', Array)
#2 ....php(189): AmazonSWF->terminate_workflow_execution(Array)
#3 ....php(83): daemon->checkSWFExecutions()
#4 ....php(350): daemon->run()
#5 {main}
  thrown in /usr/share/pear/AWSSDKforPHP/lib/requestcore/requestcore.class.php on line 829

Now, cURL error 35 means “A problem occurred somewhere in the SSL/TLS handshake. You really want the error buffer and read the message there as it pinpoints the problem slightly more. Could be certificates (file formats, paths, permissions), passwords, and others.” which is a bit vague and didn’t really help. After setting the CURLOPT_VERBOSE flag in the AWS SDK for PHP we were able to see the real error message:

NSS error -8023

Continue reading

Nginx and PHP-FPM, bash script for creating new vhost’s under separate fpm pools

Using worker pools in PHP-FPM can allow you to easily separate out and isolate virtual hosts that make use of PHP. PHP-FPM allows you to run multiple pools of processes all spawned from the master one and each pool can run as a different user and/or group. Each pool can be further isolated by running in a chroot environment and by overriding the default php.ini values on a per pool basis.

Running PHP for each vhost under a different user/group can help to stop a vulnerability in one site potentially exposing another vhost, it can also stop one malicious owner of a vhost from been able to use PHP to access the files of another site owned by someone else on the same server (in a shared hosting environment).

The process of setting up the web server config and a new PHP-FPM pool for each new vhost on a server can become a rather time consuming and boring process. However as this process follows a fairly standard set of steps it can be easily scripted.

Continue reading

Simple bandwidth monitoring on Linux

If you want a simple lightweight tool to monitor the network traffic in and out of your server vnstat might be just what you need. It keeps hourly, daily and monthly records and provides simple estimates of your expected use, it is also easy to link up to a web based frontend for fancy charts and reporting.

Installing and configuring vnstat is very simple, firstly install using your standard package manager, for example:

Debian

apt-get install vnstat

CentOS:

yum install vnstat

Then tell vnstat to create a database for the network interfaces you want to listen to (e.g. eth0):

vnstat -u -i eth0

That’s it, wait a few minutes then run vnstat to view a simple console display of the amount of traffic that has traveled though all the interfaces you’re monitoring:

vnstat
 
   eth0 since 01/22/12
 
          rx:  177.59 MiB      tx:  7.78 MiB      total:  185.37 MiB
 
   monthly
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
       Jan '12    177.59 MiB |    7.78 MiB |  185.37 MiB |    0.59 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated       183 MiB |       7 MiB |     190 MiB |
 
   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     yesterday     12.53 MiB |    1.36 MiB |   13.89 MiB |    1.32 kbit/s
         today      8.28 MiB |     127 KiB |    8.40 MiB |    0.88 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated        --     |      --     |      --     |

You can also get vnstat to dump its output in a programming friendly format (semicolon delimited):

vnstat --dumpdb

If you do want a nicer looking interface or one that doesn’t require shell access have a look at: vnstat PHP frontend

If you need a bandwidth monitoring solution that records the utilization of individual protocols instead of just received and transmitted traffic then have a look at bandwidthd