Setting up a centralised syslog server in the cloud

This post should help you get a basic syslog server and client(s) up and running in a virtual environment, It will take you through the implementation of a reasonably secure (using rsyslog’s TLS authentication) yet flexible setup useful to most virtual based server architectures I will assume if you’re reading this that you know what syslog is and what it’s used for. (if not have a quick Google then come back)

 Why Setup a Centralized Syslog Server

  • For convenience – If you for instance have a large number of web servers and you need to diagnose a problem on one of them (maybe not sure which one) you only have to check in one place, if you wanted to compile some statistics from all of them or check if they had all successfully completed a software upgrade.
  • For added security – If someone hacks into one of your servers they will probably try and cover their tracks by erasing any log records created by there presence, however if your logs are also sent to another (hardened) server then the logs will still be available to sysadmins.
  • Another very useful reason which only really applies to virtual servers is to help retain the log files from a terminated server (e.g. shut-down due to decreased demand on your application).

rsyslog on AWS

For this post I will be using rsyslog which is an enhanced version of the original syslogd and is very similar to other modern versions of syslog such syslog-ng. I will also be using the Amazon Web Services (AWS) build Amazon Machine Image (AMI) , this will keep things simple and allow you to try it yourself using the same kit.

Installing rsyslog and gnutls

This stage on the AWS AMI is rather easy as rsyslog is already installed and running as the syslog daemon. However if it’s not on your server simply install rsyslog from your package manager (yum, apt-get, …),set it to startup on boot, stop your old syslog daemon from starting on boot and then stop the old daemon and start rsyslog. To tell if rsyslogd is already running:

# ps aux | grep rsyslogd

You will also need to install rsyslog-gnutls which will allow rsyslog to encrypt the traffic to and from your syslog server using TLS (rsyslog-gnutls requires gnutls but this should get dealt with as a dependency by your package manager).

# yum install rsyslog-gnutls

Creating a CA Certificate

In-order to setup encrypted communication between our centralised syslog server (syslog server) and our syslog clients we firstly need to create our own Certificate Authority (CA) private key and certificate. This will then be used to sign our servers certificate and our clients certificate, because both our machine certificates have been signed by our CA we can get syslog to confirm the authenticity of the server it’s trying to connect to (or the client which is trying to connect to it). To create the CA Certificate we are going to use certtool command from the GnuTLS tools package, firstly you will need to install the gnutls-utils package (this only needs to be done on the machine you want to generate the certificates on).

# yum install gnutls-utils

Then we generate the private key:

# certtool --generate-privkey --outfile syslog-ca-key.key

Now we can generate the certificate (as this is self signed you don’t need a certificate signing request) you will probably want to make certificate valid for a couple of years or more:

# certtool --generate-self-signed --load-privkey syslog-ca-key.key --outfile syslog-ca-cert.crt

You will be asked a lot questions by certtool, to go with the default option just leave the response blank (press enter) if you need an idea of what your responses should look like have a look on the rsyslog documentation page. Remember to say yes to “Does the certificate belong to an authority?” and “Will the certificate be used to sign other certificates?”. Make sure you keep your syslog-ca-key.key private, ideally this should not be kept on either your syslog server or any of the client machines, it should be kept on a non-publicly accessible machine/place with only the owner granted read permission.

Generating the Server Certificate

We will generate a certificate for the syslog server and another for use on all the client machines. We will just use a single certificate for all the syslog clients and not generate a new one for each server tied to it’s hostname to allow this setup to work well in a virtual/auto scaling environment where new servers get automatically started up and other ones closed down. If we wanted to generate a new certificate for each one we would either need to have a limited pool of hostnames and have pre-generated the certificates, this would however limit some of the flexibility given by using virtual servers. Or we would need to expose some sort of API to allow a unique certificate to be automatically generated as a new server is provisioned. Neither of which sound ideal so to keep things simple for now we will go with just two certs. To generate the private key:

# certtool --generate-privkey --outfile syslog-server-key.key --bits 2048

Then to generate the csr, set the common name to “syslog server” or something similar:

# certtool --generate-request --load-privkey syslog-server-key.key --outfile syslog-server-request.pem

Follow the prompts on screen to generate the request, you may find the example on the rsyslog site useful. Once you have your signing request you will need to use the syslog CA key from earlier to generate the certificate.

# certtool --generate-certificate --load-request syslog-server-request.pem --outfile syslog-server-cert.crt --load-ca-certificate syslog-ca-cert.crt --load-ca-privkey syslog-ca-key.key

Rememeber to keep the key private and only store it on the syslog server, set the permissions so it can only be read by root.

Generating the Client Certificate

This is the same process as for server certificate. Generate the private key:

# certtool --generate-privkey --outfile syslog-client-key.key --bits 2048

Then to generate the csr, set the common name to “syslog client” or something similar:

# certtool --generate-request --load-privkey syslog-client-key.key --outfile syslog-client-request.pem

Follow the prompts on screen to generate the request, you may find the example on the rsyslog site useful.

# certtool --generate-certificate --load-request syslog-client-request.pem --outfile syslog-client-cert.crt --load-ca-certificate syslog-ca-cert.crt --load-ca-privkey syslog-ca-key.key

 Copying the Certificates

You then need to copy the following certificates to your syslog server and put them somewhere like /etc/ssl/mycerts make sure the permissions are set to 400 and root is the owner and group.

  • syslog-ca-cert.crt
  • syslog-server-cert.crt
  • syslog-server-key.key

For the syslog clients you need:

  • syslog-ca-cert.crt
  • syslog-client-cert.crt
  • syslog-client-key.key

Rsyslog Config for the Server

The rsyslog config file will probably be found under: /etc/rsyslog.conf We are going to set rsyslog up to listen for TLS protected tcp based syslog messages on the standard TLS syslog port of 10514. The following lines need to be added to your rsyslog.conf file.

$ModLoad imtcp.so

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/ssl/mycerts/syslog-ca-cert.crt
$DefaultNetstreamDriverCertFile /etc/ssl/mycerts/syslog-server-cert.crt
$DefaultNetstreamDriverKeyFile /etc/ssl/mycerts/syslog-server-key.key

# Provides TCP syslog reception
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputTCPServerStreamDriverAuthMode x509/certvalid # Just validate Cert
$InputTCPServerRun 10514

The CA certificate is used here to validate the client(s) certificate when they try to connect. This setup only allows encrypted syslog connections and only validates the client(s) based on their certificate. For the changes to take effect you will need to restart rsyslog:

# /etc/init.d/rsyslog restart

Firewall rules for the Server

As we are expecting syslog tcp packets over port 10514 you will need to open this on your servers firewall, you should ensure this is only open on the local network and not accessible from the internet.

For instance with AWS you can open the port 10514 up to the local network (10.0.0.0/8) on your syslog server and then only allow traffic from other servers within the same AWS Security Group through. This way the port is only open to your virtual servers on AWS and no one else’s.

I tend to use Shorewall to configure iptables, so the following line is added to the Shorewall rules file (followed be restarting Shorewall):

# rsyslog
ACCEPT         net:10.0.0.0/8          $FW      tcp   10514

Then the AWS Security Group is configured:

AWS Security Group Setup

 

If your firewall is restricting outbound connections then you will also need to open up the port outbound on the client(s).

Rsyslog Config for the Client(s)

The following is the config required for the syslog clients, this setup again only validates the server based on the certificate and forwards all logs to the central server, it also sets up, upto a 1GB buffer to ensure if the connection to the syslog server is lost temporarily then the logs will still get delivered when the connection is reestablished.

# ### begin forwarding rule ###
$DefaultNetstreamDriverCAFile /etc/ssl/mycerts/syslog-ca.crt
$DefaultNetstreamDriverKeyFile /etc/ssl/mycerts/syslog-client-key.pem
$DefaultNetstreamDriverCertFile /etc/ssl/mycerts/syslog-client-cert.crt
$DefaultNetstreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS for the connection
$ActionSendStreamDriverAuthMode x509/certvalid # Just validate Cert

$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@10.10.1.2:10514
# ### end of the forwarding rule ###

You will need to replace the IP address(10.10.1.2) used on the second to last line with the IP address of your central syslog server and then restart rsyslog as before.

See if it works

Firstly check on the general logs on your client machine and make sure rsyslog is running without any problems then try tailing the messages log file on the syslog server and hopefully you should see messages from both the server and the client(s) machine(s).

tail /var/log/messages

That’s it you now have a working centralised syslog setup, whilst it is a basic configuration it should be a good base to improve on and tweak to meet your needs and security requirements.

A Word about Security

Whilst this setup is good for a cloud based infrastructure, it uses the same TLS certificate for all clients and only validates the certificate against the CA and not against the hostname of the remote server, this means you can’t guarantee the identity of the remote server (although obviously an attacker would need to compromise one of your servers to gain a working certificate and private key).

You must ensure that your private key files are locked down so only root has access (and even then only has read access) and never give the files out to anyone. This also applies to the CA key (even more so).

As your syslog server holds the log files for many other servers and should act as your audit log you must take care to lock this machine down. For instance use different RSA keys for SSH access along with only running the minimum number of services on the machine, i.e. not running a web server. It is advisable to only allow the machine to be accessible from trusted IP addresses (such as your office IPs). You should also make sure the logs on this machine are backed up regularly and preferably to a read only state.

Troubleshooting:

If you look in the log files on your syslog server and see messages similar to the following:

Nov 30 08:49:19 syslog-client.example.com #026#003#002#000V#001#000#000R#003#002NÃ&H<9e>#016<8e>#006èEÃY<99>û#024Aá§þ<8e>L##032<8d>#005#036¨#000#000$#0003#000E#0009#000<88>#000#026#0002#000D#000<87>#000#023#000f#000/#000A#0005#000<84>
Nov 30 08:51:29 syslog-client.example.com #026#003#002#000V#001#000#000R#003#002NÃ&ÃHh¦Ã#003Wõ¦51à#002d<97>ü&:EYÃüÃ>GÃî¾T#000#000$#E#0009#000<88>#000#0002#000D#0008#000<87>#000#023#000f#000/#000A#0005#000<84>

There is a good chance you’ve got the $InputTCPServerRun and $InputTCPServerStreamDriverMode directives in the wrong order, the $InputTCPServerRun should come last.

  • Liam Sturgess

    Your troubleshooting section just solved something I’d been stuck on for a day, thankyou!

  • Bob

    In the client configuration should the line be:
    $DefaultNetstreamDriverKeyFile /etc/ssl/mycerts/syslog-client-key.pem

    as written or should it be:
    $DefaultNetstreamDriverKeyFile /etc/ssl/mycerts/syslog-client-key.key

    • Seb Dangerfield

      It should be what ever the name of your private key is, in my example I called the private key “syslog-client-key.pem” but you could have called it “syslog-client-key.key” the extension has no meaning to rsyslog it is just to help you.

      • vacri

        The issue is that it doesn’t match the rest of the how-to. In the client config, ‘syslog-client-key.pem’ is referred to as ‘syslog-client-key.key’ elsewhere in the howto (like the key generation step), and ‘syslog-ca.crt’ should likewise be ‘syslog-ca-cert.crt’. The rsyslog messages just say “couldn’t load TLS module” and it’s not clear what the fault is.

        Apart from this, thanks very much for the article – it made things much clearer than any other documentation I found on using TLS with rsyslog.

  • Olli

    Your troubleshooting section saved my day 😉 Many thanks for the excellent article.

  • eric

    ——————————————————————————————————————
    invalid cert info: peer provided 1 certificate(s). Certificate 1 info: certificate valid from Sat Mar 2 23:45:26 2013 to Tue Feb 28 23:45:30 2023; Certificate public key: RSA; DN: C=CN,O=bizcn,OU=bizcnou,L=xm,ST=CA,CN=syslog server; Issuer DN: C=CN,O=bizcn,OU=bizcn,L=xm,ST=CA,CN=bizcn; SAN:DNSname: syslog server;
    ————————————————————————————————————-
    Dear sir
    In accordance with the the rsyslog documentation, has been unable to successfully.
    Can you tell me how you fill out the certificate information?
    thanks!

  • Great guide, thanks!

    I did a test-setup on a Raspberry pi and one thing that got me stumped for quite a while is that it takes some time before rsyslogd accepts connections on port 10514… This can take up to 20 seconds. Rsyslogd makes it appear that it starts almost instantly but setting up the TLS-fu happens in the background, or so it seems.
    Got me hunting for problems with my configuration that weren’t there, just because I was a bit impatient 🙂

    Thanks again for the clear instructions!

  • Hi,
    Thanks for that. One question and I suppose once Ive followed your instructions I might have my answer.

    If I have 5 servers all sending their log files to my cloud syslog server, How can I tell which log file belongs to which server ? I also want to send all my router and switches log files to the syslog server.

    With the routers and switches they only have the option to enter an IP address for storing the log files, How can I tell which router or switch log file Im looking at ?

    Thanks in advance for any help you can offer

  • Mubashir

    what file on the client should contain this information

    Rsyslog Config for the Client(s)

    # ### begin forwarding rule ###
    $DefaultNetstreamDriverCAFile /etc/ssl/mycerts/syslog-ca.crt
    $DefaultNetstreamDriverKeyFile /etc/ssl/mycerts/syslog-client-key.pem
    $DefaultNetstreamDriverCertFile /etc/ssl/mycerts/syslog-client-cert.crt
    $DefaultNetstreamDriver gtls # use gtls netstream driver
    $ActionSendStreamDriverMode 1 # require TLS for the connection
    $ActionSendStreamDriverAuthMode x509/certvalid # Just validate Cert

    $WorkDirectory /var/spool/rsyslog # where to place spool files
    $ActionQueueFileName fwdRule1 # unique name prefix for spool files
    $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
    $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
    $ActionQueueType LinkedList # run asynchronously
    $ActionResumeRetryCount -1 # infinite retries if host is down
    # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
    *.* @@10.10.1.2:10514
    # ### end of the forwarding rule ###

  • I hope this will save someone many frustrating hours. I followed the guide but the client server was not sending messages. Manually starting rsyslogd -c5 would work, but when the service was restarted via daemon it would stop working. The problem was with the custom port 10516, it is not allowed by default with SELinux, so when the daemon was starting the policy would block the port, but when starting as root it would allow it. The easy was was to switch to already enabled TCP port 6514, and there is a long way to add a policy here: https://logtrust.atlassian.net/wiki/pages/viewpage.action?pageId=1703970