Why hostname is so important in mail system
When setting up an email system, the first step is to make sure you have a correct hostname for mail server.
For example, your mail server has this IP: 1.2.3.4, and you have this domain: example.com. You should assign a DNS record on the domain for the IP. Generally it's A record with the hostname mail.example.com.
mail.example.com. 300 IN A 1.2.3.4
Then you should request your hosting provider to assign a PTR record for this IP, pointing to the hostname we mentioned. This calls reverse DNS.
4.3.2.1.in-addr.arpa. 3600 IN PTR mail.example.com.
These A and PTR records are so important. Without them you can't send messages to other mail servers.
This is because, in most MTA servers (say it's Postfix) they have these settings:
smtpd_sender_restrictions = reject_unknown_reverse_client_hostname, reject_unknown_sender_domain
The statement "reject_unknown_reverse_client_hostname" requires sender IP has correct PTR record. The statement "reject_unknown_sender_domain" requires sender domain (i.e, mail.example.com) has correct A or MX record.
If the peer MTA has more strict setting like this:
smtpd_sender_restrictions = reject_unknown_client_hostname
It will reject messages when 1) the client IP address->name mapping fails, or 2) the name->address mapping fails, or 3) the name->address mapping does not match the client IP address.
So in general speaking your A and PTR records should keep consistent. PTR points to the hostname who has just the IP in its A record.
Besides DNS records you should also setup the system hostname. In linux OS it's the configuration file /etc/hostname. After putting correct hostname into this file you should run following command manually to make hostname available immediately.
$ sudo hostname mail.example.com
In Postfix's configuration file /etc/postfix/main.cf, you should also have this setting for hostname:
myhostname = mail.example.com
The system hostname is important because when your mail server talks to other mail servers, it will use hostname as HELO host.
If you were using a fake hostname here the HELO session will get rejected by peer MTAs.
More further, if you want to setup SSL for mail server, hostname is even more important. When you issue certificates to mail server, they are differed by hostnames.
Now you would like to setup SSL certificates for mail.example.com. This can be done by Letsencrypt and Certbot.
After deploying certificates to mail.example.com, the clients including either peer MTA or IMAP client like thunderbird can connect to mail server via secure way.
You must issue certificates by providing the correct hostname, otherwise cliet connections will break due to risk existing. Though SMTP session between MTAs may not validate hostname for certificates, but client connections such as thunderbird do verify hostname.
To be additional, besides hostname your IP should keep clean for successful delivery. It should be listed in no RBLs. Most MTAs will check RBL against sending IP. For example, considering Postfix configuration below.
smtpd_recipient_restrictions =
check_policy_service unix:private/policyd-spf,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client bl.spamcop.net
It will check either SPF or RBL for sending IP. So you should make a correct SPF record for your sending IP. And check if your IP is listed in any RBLs. You could check RBLs from this URL.
For more details please reference: