Fighting Spam with ASL
Spam is a complex topic. It can be in your mailbox, your browser, your mobile device. It can be blatant and obvious like a flashing popup, or silent and nefarious, hijacking banners and redirecting sites.
If spam were easy to stop, it would have been done already. The reality is: spam in all its various forms is a complex and myriad beast. It covers multiple services, with multiple technologies — just like any other kind of malicious activity. This means it can be an equally complicated problem to manage… so lets talk about how ASL can help manage spam originating from your system when it comes from a web application.
The system is sending spam from a web application.
This problem can be caused by one of two scenarios:
1) Email Spam is originating from the system, and is being sent from the locally installed Mail server (like postfix, sendmail, or qmail)
2) Email Spam is originating from the system, and is NOT being sent from the locally installed Mail server.
Attack #1: Email Spam is originating from the system, and is being sent from the locally installed Mail server (like postfix, sendmail, or qmail)
This one can be hard. For the sake of this post, we’ll assume the system is mainly PHP-based. It helps to understand how mail from a php application gets into the mail server, otherwise known at the MTA, or Mail Transfer Agent for short. PHP will be using a function like mail() most of the time (in more extreme spam cases it may use something like fsockopen() but lets skip that one for now and focus on abuse of the mail() function).
The PHP mail() function invokes the MUA (Mail User Agent) on the system. The MUA used is defined in the php.ini setting sendmail_path. This will make any message sent from PHP come from localhost. In the best case scenario, you will get the UID of the account used send the message (typically UID 48, apache), but not not what application was used.
The first task is to identify what PHP application is the source. Are you running php-5.3 or above? If so, good! You’re in luck. PHP >= 5.3 can log, or add a header to, all mail() events. We can use this to determine what’s happening. (If you are running older versions of PHP for legacy reasons, I’m afraid there’s no easy answers here. We are working on possible technical solutions for this, stay tuned).
Step 1. Reconfigure PHP to enable mail source logging. If you’re using it like a DSO (dynamic shared object) you just need to modify /etc/php.ini with the following:
mail.log = /var/log/httpd/mail.log
then restart httpd:
$ service httpd restart
This will add the header X-PHP-Originating-Script to every email message sent from the system. This header field will contain the name of the script and the UID it was invoked by. This information will also be written to /var/log/httpd/mail.log.
Step 2. Now you have to be patient. This setting is only going to work on new spam messages sent through the system, so you’ll need to wait for your badguy to come back. An example of the /var/log/httpd/mail.log event you are looking for is:
mail() on [/var/www/vhosts/domain.com/httpdocs/somescript.php:12345]: To: email@example.com — Headers: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed; delsp=yes Content-Transfer-Encoding: 8Bit Sender: firstname.lastname@example.org From: email@example.com
Step 3. Investigate the sending script. In the example above its /var/www/vhosts/domain.com/httpdocs/somescript.php. Is it a legitimate application? Or something malicious? If its a legitimate application, but being abused through a vulnerability, then you’d need to update the application or otherwise disable it. If its malicious, then we have some more work to do, proceed to the next step.
Step 4. Scan the application with clamdscan, to see if it is already known:
$ clamdscan /path/to/file
If its known and identified as malicious by clamav, then you can use this to search the system for that particular malicious code. If its not identified, go to the next step.
Step 5. Send us the code. Email us at firstname.lastname@example.org with either the file in a password-protected zip file, or a link where we can download it. We’ll take it from there.
Step 6. Figure out how it got there. This may be very easy, very very difficult, or impossible. If you had good security controls in place before this happened, you have a decent chance of figuring that out. If you are reading this article after the fact, then odds are this investigation won’t ever lead to anything other than speculation. Start with which user owned the file, then trace the logs from there (ftp logs, /var/log/secure, etc. You may need to go through older/rotated logs). If that goes nowhere, it may mean something more serious, like a root-level compromise.
Attack #2: Email Spam is originating from the system, and is NOT being sent from the locally installed Mail server.
Email spam is originating from the server, and its not going through the MTA. This means that the attacker has written their own MTA and is running it on the server. These malicious code-based MTAs are typically Perl or PHP scripts that use raw socket calls to send spam. The odds of this not being installed by a compromised account / system is virtually nil. The script will send mail as the userid that is invoked by the web server, like apache for a DSO install, or the userid of the domain if you’re using fcgi, or mod_ruid2. The good news here is that ASL has the capability to only allow specific users to send mail, we call this the FW_OUTPUT_MTA policy, and its very easy to enable.
Step 1. Set FW_OUTPUT_MTA to yes in the ASL web configuration, or in /etc/asl/config
Step 2. Add the following users to the “Allowed Users” in the ACL (Access Control List) file /etc/asl/mta-output-acl
Step 3. Reload the firewall policy:
$ service asl-firewall restart
Step 4. User IDs not on the MTA ACL will be blocked. Unlike Attack #1 above, this setting creates a permanent solution. Users attempting to mail outside of the ACL will be logged as:
Sep 11 13:42:25 www2 kernel: ASL_SMTP_OUT IN= OUT=eth0 SRC=220.127.116.11 DST=18.104.22.168 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=20451 DF PROTO=TCP SPT=52640 DPT=25 SEQ=2184330842 ACK=0 WINDOW=14600 RES=0x00 SYN URGP=0 OPT (020405B40402080A711E71B90000000001030305) UID=10045 GID=10045
UID indicates the user id attempting to send on the system. GID indicates the group id of the user.
To verify that this is working correctly on your system, you can use the telnet command:
TL;DR (Too Long, Didn’t Read)
Option 1: Enable X-PHP-Originating-Script in php.ini. Use logs to find the offending script.
Option 2: Enable this ASL policy to block unauthorized user IDs from sending mail in /etc/asl/config:
and update your security policy:
$ asl -s -f