This post should help you get a basic syslog server and client(s) up and running in a virtual environment, It will take you through the implementation of a reasonably secure (using rsyslog’s TLS authentication) yet flexible setup useful to most virtual based server architectures I will assume if you’re reading this that you know what syslog is and what it’s used for. (if not have a quick Google then come back)
Why Setup a Centralized Syslog Server
- For convenience – If you for instance have a large number of web servers and you need to diagnose a problem on one of them (maybe not sure which one) you only have to check in one place, if you wanted to compile some statistics from all of them or check if they had all successfully completed a software upgrade.
- For added security – If someone hacks into one of your servers they will probably try and cover their tracks by erasing any log records created by there presence, however if your logs are also sent to another (hardened) server then the logs will still be available to sysadmins.
- Another very useful reason which only really applies to virtual servers is to help retain the log files from a terminated server (e.g. shut-down due to decreased demand on your application).
rsyslog on AWS
For this post I will be using rsyslog which is an enhanced version of the original syslogd and is very similar to other modern versions of syslog such syslog-ng. I will also be using the Amazon Web Services (AWS) build Amazon Machine Image (AMI) , this will keep things simple and allow you to try it yourself using the same kit.
Installing rsyslog and gnutls
This stage on the AWS AMI is rather easy as rsyslog is already installed and running as the syslog daemon. However if it’s not on your server simply install rsyslog from your package manager (yum, apt-get, …),set it to startup on boot, stop your old syslog daemon from starting on boot and then stop the old daemon and start rsyslog. To tell if rsyslogd is already running:
# ps aux | grep rsyslogd
You will also need to install rsyslog-gnutls which will allow rsyslog to encrypt the traffic to and from your syslog server using TLS (rsyslog-gnutls requires gnutls but this should get dealt with as a dependency by your package manager).
# yum install rsyslog-gnutls
Creating a CA Certificate
In-order to setup encrypted communication between our centralised syslog server (syslog server) and our syslog clients we firstly need to create our own Certificate Authority (CA) private key and certificate. This will then be used to sign our servers certificate and our clients certificate, because both our machine certificates have been signed by our CA we can get syslog to confirm the authenticity of the server it’s trying to connect to (or the client which is trying to connect to it). To create the CA Certificate we are going to use certtool command from the GnuTLS tools package, firstly you will need to install the gnutls-utils package (this only needs to be done on the machine you want to generate the certificates on).
# yum install gnutls-utils
Then we generate the private key:
# certtool --generate-privkey --outfile syslog-ca-key.key
Now we can generate the certificate (as this is self signed you don’t need a certificate signing request) you will probably want to make certificate valid for a couple of years or more:
# certtool --generate-self-signed --load-privkey syslog-ca-key.key --outfile syslog-ca-cert.crt
You will be asked a lot questions by certtool, to go with the default option just leave the response blank (press enter) if you need an idea of what your responses should look like have a look on the rsyslog documentation page. Remember to say yes to “Does the certificate belong to an authority?” and “Will the certificate be used to sign other certificates?”. Make sure you keep your syslog-ca-key.key private, ideally this should not be kept on either your syslog server or any of the client machines, it should be kept on a non-publicly accessible machine/place with only the owner granted read permission.
Generating the Server Certificate
We will generate a certificate for the syslog server and another for use on all the client machines. We will just use a single certificate for all the syslog clients and not generate a new one for each server tied to it’s hostname to allow this setup to work well in a virtual/auto scaling environment where new servers get automatically started up and other ones closed down. If we wanted to generate a new certificate for each one we would either need to have a limited pool of hostnames and have pre-generated the certificates, this would however limit some of the flexibility given by using virtual servers. Or we would need to expose some sort of API to allow a unique certificate to be automatically generated as a new server is provisioned. Neither of which sound ideal so to keep things simple for now we will go with just two certs. To generate the private key:
# certtool --generate-privkey --outfile syslog-server-key.key --bits 2048
Then to generate the csr, set the common name to “syslog server” or something similar:
# certtool --generate-request --load-privkey syslog-server-key.key --outfile syslog-server-request.pem
Follow the prompts on screen to generate the request, you may find the example on the rsyslog site useful. Once you have your signing request you will need to use the syslog CA key from earlier to generate the certificate.
# certtool --generate-certificate --load-request syslog-server-request.pem --outfile syslog-server-cert.crt --load-ca-certificate syslog-ca-cert.crt --load-ca-privkey syslog-ca-key.key
Rememeber to keep the key private and only store it on the syslog server, set the permissions so it can only be read by root.
Generating the Client Certificate
This is the same process as for server certificate. Generate the private key:
# certtool --generate-privkey --outfile syslog-client-key.key --bits 2048
Then to generate the csr, set the common name to “syslog client” or something similar:
# certtool --generate-request --load-privkey syslog-client-key.key --outfile syslog-client-request.pem
Follow the prompts on screen to generate the request, you may find the example on the rsyslog site useful.
# certtool --generate-certificate --load-request syslog-client-request.pem --outfile syslog-client-cert.crt --load-ca-certificate syslog-ca-cert.crt --load-ca-privkey syslog-ca-key.key
Copying the Certificates
You then need to copy the following certificates to your syslog server and put them somewhere like /etc/ssl/mycerts make sure the permissions are set to 400 and root is the owner and group.
For the syslog clients you need:
Rsyslog Config for the Server
The rsyslog config file will probably be found under: /etc/rsyslog.conf We are going to set rsyslog up to listen for TLS protected tcp based syslog messages on the standard TLS syslog port of 10514. The following lines need to be added to your rsyslog.conf file.
$ModLoad imtcp.so # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /etc/ssl/mycerts/syslog-ca-cert.crt $DefaultNetstreamDriverCertFile /etc/ssl/mycerts/syslog-server-cert.crt $DefaultNetstreamDriverKeyFile /etc/ssl/mycerts/syslog-server-key.key # Provides TCP syslog reception $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerStreamDriverAuthMode x509/certvalid # Just validate Cert $InputTCPServerRun 10514
The CA certificate is used here to validate the client(s) certificate when they try to connect. This setup only allows encrypted syslog connections and only validates the client(s) based on their certificate. For the changes to take effect you will need to restart rsyslog:
# /etc/init.d/rsyslog restart
Firewall rules for the Server
As we are expecting syslog tcp packets over port 10514 you will need to open this on your servers firewall, you should ensure this is only open on the local network and not accessible from the internet.
For instance with AWS you can open the port 10514 up to the local network (10.0.0.0/8) on your syslog server and then only allow traffic from other servers within the same AWS Security Group through. This way the port is only open to your virtual servers on AWS and no one else’s.
I tend to use Shorewall to configure iptables, so the following line is added to the Shorewall rules file (followed be restarting Shorewall):
# rsyslog ACCEPT net:10.0.0.0/8 $FW tcp 10514
Then the AWS Security Group is configured:
If your firewall is restricting outbound connections then you will also need to open up the port outbound on the client(s).
Rsyslog Config for the Client(s)
The following is the config required for the syslog clients, this setup again only validates the server based on the certificate and forwards all logs to the central server, it also sets up, upto a 1GB buffer to ensure if the connection to the syslog server is lost temporarily then the logs will still get delivered when the connection is reestablished.
# ### begin forwarding rule ### $DefaultNetstreamDriverCAFile /etc/ssl/mycerts/syslog-ca.crt $DefaultNetstreamDriverKeyFile /etc/ssl/mycerts/syslog-client-key.pem $DefaultNetstreamDriverCertFile /etc/ssl/mycerts/syslog-client-cert.crt $DefaultNetstreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode x509/certvalid # Just validate Cert $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @@10.10.1.2:10514 # ### end of the forwarding rule ###
You will need to replace the IP address(10.10.1.2) used on the second to last line with the IP address of your central syslog server and then restart rsyslog as before.
See if it works
Firstly check on the general logs on your client machine and make sure rsyslog is running without any problems then try tailing the messages log file on the syslog server and hopefully you should see messages from both the server and the client(s) machine(s).
That’s it you now have a working centralised syslog setup, whilst it is a basic configuration it should be a good base to improve on and tweak to meet your needs and security requirements.
A Word about Security
Whilst this setup is good for a cloud based infrastructure, it uses the same TLS certificate for all clients and only validates the certificate against the CA and not against the hostname of the remote server, this means you can’t guarantee the identity of the remote server (although obviously an attacker would need to compromise one of your servers to gain a working certificate and private key).
You must ensure that your private key files are locked down so only root has access (and even then only has read access) and never give the files out to anyone. This also applies to the CA key (even more so).
As your syslog server holds the log files for many other servers and should act as your audit log you must take care to lock this machine down. For instance use different RSA keys for SSH access along with only running the minimum number of services on the machine, i.e. not running a web server. It is advisable to only allow the machine to be accessible from trusted IP addresses (such as your office IPs). You should also make sure the logs on this machine are backed up regularly and preferably to a read only state.
If you look in the log files on your syslog server and see messages similar to the following:
Nov 30 08:49:19 syslog-client.example.com #026#003#002#000V#001#000#000R#003#002NÃ&H<9e>#016<8e>#006Ã¨EÃY<99>Ã»#024AÃ¡Â§Ã¾<8e>L##032<8d>#005#036Â¨#000#000$#0003#000E#0009#000<88>#000#026#0002#000D#000<87>#000#023#000f#000/#000A#0005#000<84> Nov 30 08:51:29 syslog-client.example.com #026#003#002#000V#001#000#000R#003#002NÃ&ÃHhÂ¦Ã#003WÃµÂ¦51Ã #002d<97>Ã¼&:EYÃÃ¼Ã>GÃÃ®Â¾T#000#000$#E#0009#000<88>#000#0002#000D#0008#000<87>#000#023#000f#000/#000A#0005#000<84>
There is a good chance you’ve got the $InputTCPServerRun and $InputTCPServerStreamDriverMode directives in the wrong order, the $InputTCPServerRun should come last.