Had this error appear today when trying to send an email using Swift Mailer (on an nginx powered box using PHP FastCGI). It turned out to be caused by the CGI parameter SERVER_NAME, the site was running on a wildcard domain under nginx and as a result the SERVER_NAME param contained the wildcard symbol. (e.g. *.test.example.com) However the SMTP server didn’t like this (and quite rightly so), this is where the Invalid domain name bit comes from.
Adding the following line to my nginx server config for that virtual host fixed the problem.
fastcgi_param SERVER_NAME $host;
The above line just sets the SERVER_NAME parameter to be the same as the host header for the current request. The only problem with this is if the host header in the request is not set then this value can still fall back to the original $server_name one (see nginx manual for more info). However this seemed to fix the problem for me, and I don’t think it should cause any problems as the site is not available directly via an IP address, so host header will always be required.