Getting the feet wet with DKIM

DKIM stands for DomainKeys Identified Mail, it’s an anti-phishing / anti-spoofing system for email that relies on private/public key system to authenticate the origin of the email. Kind of SPF records but on steroids. You can find a pretty concise explanation of the DKIM system here.

I needed to figure this out, as my SPAM (sorry let me just correct this), my outgoing SOLICITED bulk email sending box was getting lower deliverability rates. So it seems having this system helps, because as everybody knows all the lower grade spam senders can’t figure this out.

First thing is to create a private/public key pair. You can go all out with OPENSSL, but as I feel a bit lazy, and not in the mood for man pages and the macho man CLI black magic, just used the online tool that the good people of Worx (the PHPMailer people) provide here.

You must provide the filenames for the pair, just click next, and in the next step fill the domain (pretty self explanatory), the identity (just left blank), the selector, and a passphrase (I left that blank also). The selector is mandatory, as the DKIM system allows to have several keys per domain, it uses the selector to identify which key the email was signed with. Even if you are using just one key per domain you need to assign a value here (I used “default”, but it can be anything like “potato” or “balls”.

You will get a zip file with everything you need, and even some instructions to protect the key files with .htaccess directives, because as the good people of Worx know, PHPMailer is PHP, and all PHP programmers are dummies, so they will happily put the key files under the domain website root and probably with a couple of links from the website homepage.

I bet that somewhere in the support forums somebody asked: why I can’t access my key files, always getting those 403 Forbidden errors.

First thing is to publish your public key on DNS. I use DJB tinydns DNS server with VegaDNS management, the old PHP version when the author placed key files in the public website root. Now he is smart, the 2.0 version is in Python and he doesn’t do this anymore… anyway add a new TXT record for the domain, in the hostname put “selector._domainkey” (change the selector part to the string you defined before), and in the address field put in one line only:

v=DKIM1; k=rsa; p=publickey

(replace publickey with the public key that you get previously). If you are using other DNS server check out the specific documentation on creating TXT records.

Now, to sign the email messages you send. Obviously the natural choice is to have your MTA auto magically sign the outgoing emails. To me that means to mess with my power beast Qmail production installation, this means messing with the macho man stuff, patches, libraries and compilers. Not so good to make a quick fix for sending SPAM (sorry, SOLICITED bulk email). So I went the easy route and use the PHPMailer to sign the outgoing emails.

Just have to add a couple of lines:

$email = new PHPMailer();
$email->DKIM_domain = 'domain.com'; // the domain
$email->DKIM_private = '/path/to/private/key'; // please even using PHP keep the file above the webroot
$email->DKIM_selector = 'default'; // the selector you published in DNS

and proceed to testing. There are plenty of online services that provide us this service. I liked this one. But after doing some testing I quickly realized that something was not well in the DKIM land…

All the outgoing emails were sent with some bad header char. Even sending to my own system, the Amavisd was quarantining the emails with virus suspicions due to a bad header. Further looking into this, just realized that between the last header line and the DKIM signature there was a dreadful ^M (a carriage return, the DOS \n\r not compliant with the email spec). This is in a plain vanilla Qmail running on Debian, and with latest stable PHPMailer.

There was just one thing to do, bring out the old quick and fix hammer and change the class.phpmailer.php file. If you have the same problem in line 1305 change the CRLF to just LF:

/*
$this->MIMEHeader = rtrim($this->MIMEHeader, "\r\n ") . self::CRLF .
    str_replace("\r\n", "\n", $header_dkim) . self::CRLF;
*/

$this->MIMEHeader = rtrim($this->MIMEHeader, "\r\n ") . "\n" .
    str_replace("\r\n", "\n", $header_dkim) . self::CRLF;

And everything is running nice and well.

Raspberry PI sending emails

Using the PI as an headless server doing some stuff, it’s a good thing to have reports by email. The best and easy way is to run sSMTP a very simple program that sends outgoing email to a mailhub for delivery.

You can actually use your Gmail account or any other account (I’m using any other account).

Install it:

apt-get install ssmtp

Configure it at /etc/ssmtp/ssmtp.conf. The most relevant option:

mailhub=mail.domain.com (the SMTP agent host, the “outgoing server” in email clients like thunderbird)

If your outgoing server needs authentication for relaying (probably you do need, if you can send emails to your own domain emails, but can’t send to external emails), set the credentials with these two options:
AuthUser=youruser@domain.com
AuthPass=yourpassword

To test it, i executed the good old mail command but get a no-no response:
-bash: mail: command not found

As Raspbian is Debian based, the “user-friendliness” doesn’t provide these stone age commands to protect one from oneself… so I went kind of mental and install it anyway:

apt-get install mailutils


After this operation, 22,8 MB of additional disk space will be used.
Do you want to continue? [Y/n] y

OH BOY, THAT’S A FUCKING LOTS OF MB/PACKAGES/DEPENDENCIES TO GET A CLI COMMAND

mail youremail@domain.com

in the interactive mode, just follow the instructions, write some non-sense to test it and then press CTRL-D to test it. And your email should automagically appear in youremail@domain.com inbox.

If something goes wrong, you can add DEBUG=YES to /etc/ssmtp/ssmtp.conf.

Procmail with Qmail + Vpopmail

Following the qmail threads in this blog, and after a successful experience filtering emails in the server with a php script! time was upon to thinker a bit with the elder of all email server filters, Procmail – the mail processing utility for Unix.

Whe are talking really old stuff, as Wikipedia states the initial release in 1990, so 26 years from this writing, and about a zillion years in computer time (a date so old that it’s closer to the Unix Epoch than it is of today).

As usual, the easy part in FreeBSD is the installation:

cd /usr/ports/mail/procmail
make install

there. Now the tricky part, that is to make it play nicely with Qmail+Vpopmail setup. For the first experiences you probably should setup a couple of test accounts.

The concept is pretty simple, for an account that you want to filter email with Procmail we are going to add/or change the .qmail file that controls the email delivery to a filter script that invokes procmail and throw back a proper qmail exit code according to Procmail result.

It is very important to take into account that in this setup Procmail DOES NOT deliver the email directly, it filters the email, and according with the recipes rules, it can stop the delivery chain, forward the email, invoke an external command, etc.

Without further delays. The customized .qmail that calls the filter script:

| /home/vpopmail/domains/mydomain.com/teste/procmail_filter
/usr/home/vpopmail/domains/mydomain.com/teste/Maildir/

this is pretty simple and standard stuff. The qmail-local parses the .qmail file. The line with a pipe means to feed the message to the specified program. The command is invoked by qmail-command that runs sh -c command in the home directory, makes the email messsage available on standard input and setups some environment variables.

For this thread the most important stuff are the exit codes:

Command’s exit codes are interpreted as follows:
0 means that the delivery was successful;
99 means that the delivery was successful, but that qmail-local should ignore all further delivery instructions;
100 means that the delivery failed permanently (hard error);
111 means that the delivery failed but should be tried again in a little while (soft error).
Currently 64, 65, 70, 76, 77, 78, and 112 are considered hard errors, and all other codes are considered soft errors, but command should avoid relying on this.

With this info it’s pretty straight forward to devise the procmail_filter script.

#!/bin/sh

/var/qmail/bin/preline /usr/local/bin/procmail ./.procmailrc

status=$?

if [ $status -eq 99 ]
then
  status=99
elif [ $status -eq 0 ]
then
  status=0
elif [ $status -le 77 ]
then
  status=111
else
  status=100
fi

exit $status

and mark it executable.

The first line (after the shebang) the script calls qmail preline program, it simply inserts at the top of each message a UUCP-style From_ line, a Return-Path line, and a Delivered-To line because Procmail doesn’t understand the qmail environment variables. Calls procmail and sets Procmail configuration file the .procmailrc in the same directory.

Now for the .procmailrc stripped down to a very simple example:

SHELL = /bin/sh
LOGFILE=./pm.log
LOG="
"
VERBOSE=yes

# Exitcodes
# 0 normal delivery
# 99 silent discard a message
# 100 bounce a message

# Recipes

:0
* ^From: email@domain.com
{
  EXITCODE=99

  :0
  | /usr/local/libexec/dovecot/deliver -d delivertothis@email.com

}

# Avoid duplicates
:0
/dev/null

The top lines are the configuration variables, I would strongly suggest to use a log file for testing, in this example called pm.log that lives in the same directory. The strange LOG directive simple adds a new line to each log entry.

Then the recipes, in this example we match all emails from email@domain.com and deliver it to a local email delivertothis@email.com using Dovecot deliver command (so it takes care of Maildir quota), and set exitcode 99 to discard the message. Exit code 99 means that the delivery was successful, so all .qmail further delivery instructions will be ignored.

We could simply put an email address for a external email address, or even a local address but with less efficiency as it will trigger a new delivery process. This is auto-magical thanks to FreeBSD mailer.conf wrapper.

:0
* ^From: email@domain.com
{
  EXITCODE=99

  :0
  delivertothis@email.com

}

The last lines avoid duplicates, Procmail /dev/nulls the message and gives back the delivery control to the .qmail flow.

That’s it, everything playing nice with each other in old good UNIX tradition.

.qmail delivery filtering by adress and subject

My goal was simple, to filter incoming emails by subject and to address (on a catch-all address…) in the server and move the matches to a specific folder instead of the normal delivery. I’m running a Qmail+Vpopmail system, and these directions should be valid for similar setups.

Probably i could do this with Procmail or Maildrop, but it seemed all so complicated to do just a simple one time task that i opted in for the fun route, doing it myself… for a recurrent email filtering task, multiple accounts, customization, any kind of heavy email filtering i strongly suggest to stop here and go read about Procmail or Maildrop.

Still here? Good. First, add a .qmail (see dot.qmail man) in the user Maildir folder that you want to setup a filter:

| /home/vpopmail/domains/domain.com/user/filter
/usr/home/vpopmail/domains/domain/user/Maildir/

Save it and take care with the permissions, vpopmail user should be able to read it. What we are doing here is really simple, in the first line we pipe incoming emails trough filter, and according to filter exit code qmail execute (or not) the second line and proceed with the normal delivery.

Now setup the filter itself, it’s written in dirty and messy PHP but it gets the job done. Also it depends on PEAR Mail_mimeDecode, so go ahed and install it:

pear install Mail_mimeDecode

Now the filter script itself, it must be customized to your needs:

#! /usr/local/bin/php
<?php

/*
 * QMAIL FILTER EMAILS
 *
 * invoked by .qmail files
 * | /path/to/this/script
 *
 * parses email to address and subject against target strings
 * if BOTH are matched email is saved in $save_matched_dir
 * and qmail is instructed to ignore further .qmail lines
 *
 * CONFIG:
 */

$max_bytes        = 262144;            // mail size > 256Kb is not scanned
$to_address       = 'to@domain.com';   // to address filter
$subjects         = array('subject 1',
                          'match this subject',
                          'other subject');
$save_matched_dir = '/home/vpopmail/domains/domain.com/save-matched-emails/';

/*
 * QMAIL EXIT CODES
 *
 * 0 - Success (go to next .qmail line)
 * 99 - Success and abort (do not execute next lines)
 * 100 - permanent error (bounce)
 * 111 - soft error (retry later)
 */

try {
  function decodeHeader($input) {
    // Remove white space between encoded-words
    $input = preg_replace('/(=\?[^?]+\?(q|b)\?[^?]*\?=)(\s)+=\?/i', '\1=?',
                         $input);

    // For each encoded-word...
    while (preg_match('/(=\?([^?]+)\?(q|b)\?([^?]*)\?=)/i', $input, $matches)) {

      $encoded  = $matches[1];
      $charset  = $matches[2];
      $encoding = $matches[3];
      $text     = $matches[4];
      
      switch (strtolower($encoding)) {
        case 'b':
          $text = base64_decode($text);
          break;

        case 'q':
          $text = str_replace('_', ' ', $text);
          preg_match_all('/=([a-f0-9]{2})/i', $text, $matches);
          foreach($matches[1] as $value)
            $text = str_replace('='.$value, chr(hexdec($value)), $text);
          break;
      }

      $input = str_replace($encoded, $text, $input);
    }

    if (! isset($charset))
      $charset = 'ASCII';

    $input = strtolower(
               preg_replace('/[^a-z ]/i', 
                            '', 
                 iconv($charset, 
                       'ASCII//TRANSLIT//IGNORE', 
                       $input)
               )
             );
    return $input;
  }

  $mail  = '';
  $bytes = 0;

  $fr = fopen("php://stdin", "r");
  while (!feof($fr)) {
    $mail .= fread($fr, 1024);
    $bytes += 1024;
    if ($bytes > $max_bytes) {
      fclose($fr);
      exit(0);
    }
  }
  fclose($fr);

  require_once 'Mail/mimeDecode.php';
  $decoder   = new Mail_mimeDecode($mail);
  $structure = $decoder->decode(array('decode_headers' => true));

  // check from address
  $patt = '/[a-z0-9]+([_\\.-][a-z0-9]+)*@([a-z0-9]+([\.-][a-z0-9]+)*)+\\.[a-z]{2,}/i';
  preg_match($patt, 
             $structure->headers['to'], $matches);
  if (isset($matches[0]) && $matches[0] == $to_address) {
    // check subject
    $structure->headers['subject'] = decodeHeader($structure->headers['subject']);

    foreach ($subjects as $subject) {
       if (strpos($structure->headers['subject'], $subject) !== false) {
         $fw = fopen($save_matched_dir.
                     time().
                     '.'.
                     rand(1000, 99999).
                     '.'.
                     gethostname().
                     'S='.
                     strlen($mail).
                     ':2', 'w');
         fwrite($fw, $mail);
         fclose($fw);
         exit(99);
         // exit(0);
       }
    }
  }
} catch (Exception $e) {
}


// default, continue normal processing
exit(0);

?>

save, add the php hash bang, mark it executable and vpopmail owned. Also, the $save_matched_dir is not created and should already be present in your system.

Thats it. After this setup you should start seeing a steady flow of matched emails being saved in the matched directory and not delivered in your Inbox. As usually this works like a charm to me but can work incredible bad for you, so use at your own peril.

PHP Email validation function

My PHP email validation function nowadays is:

function isEmail($value, $network_validation = true) {

    // Create the syntactical validation regular expression
    // (broken in 2 lines for better readability)
    $regexp = "/^([_a-z0-9-]+)(\.[_a-z0-9-]+)*@".
              "([a-z0-9-]+)(\.[a-z0-9-]+)*(\.[a-z]{2,4})$/i";

    // Validate the syntax
    if(preg_match($regexp, $value)) {
        if (! $network_validation)
            return true;
  	 
        $tmp = explode('@', $value);
        $username  = $tmp[0];
        $domaintld = $tmp[1];
 
        // Validate the domain
        if (getmxrr($domaintld, $mxrecords) || checkdnsrr($domaintld, 'A'))
            return true;
    }
    return false;
}

This is for my own reference, maybe this is all messed up, so use it at your own risk.