Qmail-Scanner-2.11st (st patch)

1. Introduction

2. Download (See CHANGELOG-st-patch file)

3. To apply the patch

4. Antispam options (All the configuration options)

a) --sa-quarantine [num] (how to requeue a quarantined message)

a.1) --sa-forward <user@domain>

a.2) --sa-fwd-verbose [yes|no]

a.3) --spamdir

b) --sa-delete [num]

c) --sa-reject [yes|no] (examples of the logs)

d) --sa-delta [num]

e) --sa-subject <"some text">

f) --sa-alt [yes|no]

f.1) --sa-debug [yes|no] (SpamAssassin configuration sample)

f.2) --sa-report [yes|no]

g) SA_SKIP_MD

h) --sa-socket <path to spamd socket> (unix-socket performance tests)

i) --sa-remote remote.spamd.host[,port]   (NEW)

5. Miscellaneous options

a) --qs-group <usergroup>

b) --debug [0|1|2|3|4|5]    (minidebug removed)

c) BMC_WHITELIST="on"

d) --virus-to-delete [yes|no]

e) --settings-per-domain [yes|no] (file settings_per_domain.txt)

f) --setuidgid-path <path to setuidgid program>

6. Cosmetic options

a) --dscr-hdrs-text <"Descrip-Headers-Text">

b) log-report.sh

c) qs_config.sh

7. Examples

1. Introduction

This is a patched version (ergo unofficial) of qmail-scanner-2.11, that adds some options focused in deal with spam and others features.

Qmail-Scanner (by Jason Haar) is an excellent add-on for Qmail, that enables a Qmail server to scan all gateway-ed email searching for virus and/or Spam. For detailed instructions on how to install and run qmail-scanner visit the home page at http://qmail-scanner.sourceforge.net/, in this page you will only find explanations of the options added by this patch.

I started running qmail-scanner in april 2002, mainly to stop viruses arriving to my users by mail, but since march 2003 the volume of spam mail had increased enormously and my users clamed to block all those messages. So I modified the code of qmail-scanner with the patch from Chris Hine to block (quarantine) spam, based in the score of SpamAssassin, most of my users don't know how to filter messages tagged as spam. And later I added some other little functionalities.

2. Download

It's possible to download the patch (q-s-2.11st-20130319.patch.gz) and apply it yourself, or download a complete distribution (q-s-2.11st-20130319.tgz) already patched. (Older versions).

See the file CHANGELOG-st-patch to know what is new in this version.

3. To apply the patch

(Skip this step if you have downloaded the distribution already patched)

Untar the file "qmail-scanner-2.11.tgz", cd to the parent directory of the directory "qmail-scanner-2.11" and copy the patch there and gunzip it. (Ok... just do this..)

tar xzf qmail-scanner-2.11.tgz -C /var/tmp/ cp q-s-2.11st-20130319.patch.gz /var/tmp/ cd /var/tmp gunzip q-s-2.11st-20130319.patch.gz

Apply the patch

patch -p0 < q-s-2.11st-20130319.patch

4. Antispam options

You can read in this separate page all the configuration-options of this patched version.
(For detailed instructions on how to install and run qmail-scanner visit the home page at http://qmail-scanner.sourceforge.net/, in this page you will only find information about the specific options of the patch)

./configure ...your options... --sa-quarantine [num] --sa-delete [num] --sa-reject [yes|no]

a) --sa-quarantine [num]    (default 0, no message will be quarantined)

sa-quarantine is a relative value to the SpamAssassin required_hits.

You can set a score in /etc/mail/spamassasin/local.cf (for example 6.5) and SpamAssassin will tag as spam all messages over this score, messages that exceed the "required_hits + sa-quarantine" are quarantined.
If you enable sql settings in Spamassassin the required_hits will be read from the database for each recipient.

Basically what it does is extend the spam checking, so that if a message exceeds a certain configurable spam threshold, the message is quarantined as though it had a virus. Obviously this is only relevant if SpamAssassin is detected.

I check every day the subject of the quarantine messages and I have never seen a false positive over 8 points until 24th december 2003, I got three in one day, two with a score of 8.1 and one with 8.4. And very democratic, one in english, one in italian and one in spanish. Really some people writes down all sort of silly things in Christmas Greetings and SpamAssassin was confused. Since that I have only a few false positives, less than one a month. So, better be ready (how to requeue a quarantined message to the recipient)

The string 'spam' have been added to the "@silent_viruses_array", so no notify will be sent to the sender, as usually is a faked sender. If you don't want this option, edit qmail-scanner-queue.pl and remove 'spam' from the array.

It is possible to set a wide site sa-quarantine value (and also the following options) and a per user/domain value enabling the feature settings-per-domain (see below)

a.1) --sa-forward <user@domain>   (defaults to nothing)

Used to redirect spam mails 'being quarantined' for admin purposes...
The mails are redirected almost unmodified to the address set in this option, (an Ip.Guy suggestion) so you can use sa-learn with them.
(i.e. --sa-forward antispam@mydomain.com ).

a.2) --sa-fwd-verbose [yes|no]   (default: no)

Whether to add the X-Spam headers to the forwarded message. Obviously sa-forward must be defined.

a.3) --spamdir

Actually the official version already quarantines spam messages into a different maildir folder than viruses, so it is easier to run sa-learn over the quarantine spam messages.
The default value is: "spam", that splits to /var/spool/qscand/quarantine/spam. This is slightly different from the precedent version, so review your settings if you have changed it to a custom place. You can set this value per user/domain enabling the feature settings-per-domain (see below)
WARNING: if your $smaildir_site it is not in the same file system (partition) than $wmaildir (working directory) you have to change the routine sub email_quarantine_report, you will find the code commented in that routine.

b) --sa-delete [num]    (default 0, no message will be deleted)

sa-delete is a relative value to the SpamAssassin required_hits.

Similar at sa-quarantine but the messages will be deleted. Messages that exceed the "required_hits + sa-delete" will be deleted.

If sa-quarantine is set, sa-delete must be greater.

It is possible to use both, sa-quarantine and sa-delete. For example you can set "required-hits" of spamassassin to 6.5, sa-quarantine to 2.1 and sa-delete to 4.2. Mails with a score over 6.5 will be tagged as spam, over 8.6 will be quarantined and over 10.7 will be deleted (these are my actual settings, but you have to choose your by your experience).

No notify mail will be sent, neither to the admin.

As sa-quarantine and sa-delete are relative values you will be able to do a pseudo per user configuration (never tested). The user can set his own required_hits settings, then the admin (you) sets sa-quarantine and sa-delete, so the user could know at what score over his required_hits the mails are quarantined or deleted.
See FAQ n.20 in the official page for details.

NEW: Now you can set sa-quarantine and sa-delete (and other values) per user/domain enabling the feature settings-per-domain (see below)

c) --sa-reject [yes|no]    (default no)

If you enable sa-reject and sa-delete is properly set, messages with a score higher than sa-delete will be rejected before the smtp session is closed. Otherwise they are just dropped silently. Messages from the LOCALHOST are never rejected. This feature is not really the same that has been implemented in the official version from release 2.02.

Be aware that there is no bandwidth saving, but at least the remote smtp server will have to deal with the rejected messages instead of your server.

From version 2.01st the qmail-queue-custom-error.patch (also included in the distribution) is supported and the configure script will detect if you have installed it. Then the remote server will receive a smtp permanent error code:

554 We have reasons to believe this mail is SPAM (#5.7.1)

If you haven't the qmail-queue-custom-error.patch, the remote smtp server will receive standard qmail smtp permanent error code:

554 mail server permanently rejected message (#5.3.0)

If you want to customize the messages to the remote server (and the remote user, if there is one) you can edit the source of qmail-1.03 and modify the file qmail.c, it is a short file. Just search for the line

   case 31: return "Dmail server permanently rejected message (#5.3.0)";

and change it to what ever you want, for example (Don't remove the first D):

   case 31: return "DWe have reasons to believe this mail is SPAM (#5.7.1)";

and then recompile qmail (make clean ; make setup check ; strip /var/qmail/bin/*).

qmail-smtpd receives an exitcode 31 from qmail-scanner, but you can use one of the exitcodes that you see in the file qmail.c. Be cautious...

NOTE: I've had noticed that a qmail server always close the connection when it receives a 5xx code, but other servers keep the connection open for 10-15 seconds. To avoid this situation, that leaves my servers busy, I wrote a patch for qmail that outs the 553 code and drops the connection immediately, you can find it here.

You can see some examples of the logs in the mail server and the message that is sent to the remote user (if he is real...) when a mail is rejected.

Stefano Pasquini has pointed me to a little odd situation, he is using this feature and his server is rejecting several mails from his secondary server, which is running by another ISP, this is really no good. To avoid this embarrassing situation you can add a rule in the tcp.smtp file with the enviroment variable SA_ONLYDELETE_HOST, if this variable is defined, spam mails coming through your secondary server will be deleted instead of rejected.

   your.secondary.server.ip:allow,QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl",SA_ONLYDELETE_HOST="on"

Don't forget to rebuild the tcp.smtp.cdb database.

127.:allow,RELAYCLIENT="",QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl",QS_SPAMASSASSIN="on"

d) --sa-delta [num]    (default 0, no tag will be added to spamc_subject)

If you enabled this feature (only works in FAST_SPAMASSASSIN mode) and set $spamc_subject to some text, your users will recieve an indication (HIGH, MEDIUM, LOW) about the score spamassassin gives to the message, in the subject.

If the message has reach a score minor than required_hits (sa_max) plus sa_delta, the messages will be tagged as LOW, in other words the subject will be somethig like this "SPAM *** LOW", assuming that $spamc_subject="SPAM *** ".

A score between sa_max+sa_delta and sa_max+2*sa_delta will be tagged as MEDIUM, and if the score is higher than sa_max+2*sa_delta as HIGH.

Be aware that sa_max+2*sa_delta must be lower than sa-quarantine, otherwise it won't never catch any message.

You can edit qmail-scanner-queue.pl and change this in the sub spamassassin to whatever you want.

e) --sa-subject <"some text">    (default to nothing)

This is an alternative way to set $spamc_subject to some text, for example "SPAM *** ". Be sure that is better to tag the subject, of spam messages (only works in FAST_SPAMASSASSIN mode), through qmail-scanner than with the rewrite_subject of SpamAssassin.The input must be quoted.

f) --sa-alt [yes|no]    (default no)

This is an alternative subroutine to call SpamAssassin. It ALWAYS works in FAST_SPAMASSASSIN mode, and it would be a little bit faster because it doesn't create a tmp_file and pass the '-u' option to spamc only if you are running spamassassin sql per user settings.

It also allows you to add the X-Spam-Report header enabling the option sa-report.

When I said above ALWAYS I mean ALWAYS, sa-alt sets the spamc_options by itself so if you want to run in VERBOSE_SPAMASSASSIN mode (bad choice) you have to disable this option and run the 'standard spamassassin' routine.

This option should be used with the following two options.

f.1) --sa-debug [yes|no]    (default no)

If sa-alt is enabled and you enable this option, you will have a log of the tests and scores from SpamAssassin in qmail-queue.log. And these score and tests will be also added to the notifies sent to the admin.
I was looking for a way to control how SpamAssassin was working, and this is the reason for that I wrote the alternative subroutine to connect to SpamAssassin.
If you enable add-dscr-hdrs you will get the process number and then you can do a grep by the process number in qmail-queue.log and debug what happened with a message.
Don't worry, you don't need to reconfigure qmail-scanner to switch from one subroutine to the other, just edit qmail-scanner-queue.pl and disable/enable sa-alt (sa_alt).
Want to see the configuration of SpamAssassin and a sample of the logs?

f.2) --sa-report [yes|no]    (default no)

If sa-alt is enabled the X-Spam-Report header will be added to the messages enabling this option.
Notice that you are still running in FAST_SPAMASSASSIN mode...
You can set this option per user/domain enabling the feature settings-per-domain (see below)

g) SA_SKIP_MD   ( SA skip MAILER-DAEMON )

This is not a configuration option (yet another option for Stefano Pasquini), this is a switch inside the code that you can enable or disable when you need it. Set to something different from zero to enable it.

Supposing that a spammer drops in the net several tens of thousands mails with a random from address like abxtyicj@yourdomain.com, and then in a few minutes your mail server will receive something like 3.800 messages from MAILER-DAEMON because some user from some server is unknown.. Well, you have to deal with all those messages quickly but SpamAssassin spends some seconds for each message, so your server will be on his knees. In this case you can edit qmail-scanner-queue.pl and set SA_SKIP_MD to '1', qmail-scanner will skip SpamAssassin for messages 'From: MAILER-DAEMON' and empty Return-Path, but the antivirus scanners will always check the messages.

I don't think that is a good idea to have it always enabled.

h) --sa-socket <path to spamd socket>   (defaults to nothing)

Actually the configure script can automatically discover if spamd is running in unix-socket mode, but, if for some reasson the socket couldn't be found properly you can set the path with this option. (i.e. --sa-socket /var/run/spamd).

From my test over ten thousand mails, spamd is 7,8% faster running with unix-socket.

h) --sa-remote remote.spamd.host[,port]   (defaults to nothing)

You can use the hostname or the ip address, if not specified the default port is 783. Set it if you want connect to spamd in another host, this option is incompatible with --sa-socket.

5. Miscellaneous options

a) --qs-group <usergroup>    (default: same as qs-user)

Group that Qmail-Scanner runs as, beaware that user has to belong to this group, this is a design choice to allow some checks in the configure script, I know that it won't be strictly necessary, but the configure script will check for it.

This option allows you to install qmail-scanner-2.11 over an old installation (v1.1x, trust me: it is better a fresh install for 2.10 ..) where the user was "qmailq" and the group "qmail" (or if you want to use a group that it is not the same as the initial_group of qs-user, but qs-user MUST be member of qs-group). This will decrease the security level, but qmail itself is already heavily compartmented. (This option is only used during the install process).

b) --debug [0|1|2|3|4|5]   (default: 1)    (minidebug removed)

There are 5 possibles level of logging:
# 1 - only important info is logged
# 2-3-4 more info is logged...
# 5 - all the info is logged (same as set debug to 1 in the official version)
# >100 the temporary files are not removed

c) BMC_WHITELIST="on"

Set this enviroment variable in tcp.smtp to disable BAD_MIME_CHECKS for some servers. It's a little bit hard to mantain...

d) --virus-to-delete [yes|no]   (defaults to "no")

Enable this option if you want to delete some viruses (i.e. mydoom) without notifying anyone. If you don't enable it now, you can later edit qmail-scanner-queue.pl and add the virus you want to the list virus_to_delete.

e) --settings-per-domain [yes|no]   (defaults to "no")

Enable or disable the domain-wise mode, each user/domain will have a customized @scanner_array and sa_settings. If the user/domain haven't a custom @scanner_array and sa_settings, qmail-scanner will fall to the @scanner_default array and sa_settings_site.

You have to edit the file 'settings_per_domain.txt' and configure there your domains, you will find some instructions inside the file. More info...

--setuidgid-path <path to setuidgid program>

Defaults to nothing, the configure script will search for it, this option is only necessary if setuidgid from daemontools packet is installed in an unusual path.

6. Cosmetic options

a) --dscr-hdrs-text <"Descrip-Headers-Text">   (defaults: "X-Qmail-Scanner")

Name of the descriptive header qmail-scanner, if enabled, will add. This header name must start with an 'X-' and it can only contain letters, numbers (means [0-9A-Za-z]) and the 'minus' symbol, no accented characters or spaces are allowed.

b) log-report.sh

This script is installed in the qscand directory and does a quick statistic from the qmail-queue.log files, you can send a mail after rotating the logs....

  ./log-report.sh qmail-queue.log.1.gz

   5965 Messages processed
   3182 Spam rejected
    387 Virus W32/Netsky-P
    213 Virus W32/Mytob-C
     81 Spam tagged
     77 Virus W32/Mytob-Fam
     43 Virus W32/Mytob-E
     27 Spam quarantined
     22 Policy blocked
     16 Virus W32/Mytob-L
      9 Virus HTML.Phishing.Bank-362
      8 Virus W32/Mytob-AR
      8 Virus W32/MyDoom-H
      7 Virus HTML.Phishing.Bank-346
      6 Virus W32/MyDoom-O
      5 Virus JS.Feebs.Z
      4 Virus W32/Netsky-Q
      3 Virus W32/Mytob-D
      3 Virus W32/MyDoom-N
      3 Virus Exp/MS04011-A
      2 Virus W32/Netsky-N
      2 Virus W32/Mytob-DI
      2 Virus W32/Flcss
      1 Virus W32/Mytob-BW
      1 Virus W32/Mytob-A
  

The script qs-scanner-report.sh (by Aizudin Ali Yeon), that could be found in the crontib directory, is an example about how to email the log-reports with a cronjob.

b) qs_config.sh

This script in the distribution (in the official version is inside the contrib directory) outputs the CMDLINE of an installed qmail-scanner-queue.pl in a format suitable to use it for the new installation. You can redirect it to a file, edit this file a change/add the options for the new installation.

Don't forget that the spooldir from version 2.00 is different from older version so it will be better for you to change the output of the script, although it is not strictly necessary. And there are some option that have disappear with the new version...

   ./qs_config.sh /var/qmail/bin/qmail-scanner-queue.pl

     ./configure \
     --qs-user qscand \
     --qs-group qscand \
     --spooldir /var/spool/qscan \
     --qmaildir /var/qmail \
     --bindir /var/qmail/bin \
     --qmail-queue-binary /var/qmail/bin/qmail-queue  \
     --virusdir viruses \
     --admin antivirus \
     --domain domain.com \
     --admin-description "Antivirus MYDOMAIN" \
     --notify psender,admin \
     --local-domains domain.com \
     --silent-viruses auto \
     --virus-to-delete 1 \
     --skip-text-msgs 1 \
     --lang en_GB \
     --debug 3 \
     --add-dscr-hdrs 1 \
     --dscr-hdrs-text "X-Antivirus-MYDOMAIN" \
     --normalize yes \
     --archive 0  \
     --settings-per-domain 1 \
     --max-scan-size 100000000 \
     --unzip 1 \
     --max-zip-size 80000000 \
     --max-unpacked-files 2000 \
     --redundant 1 \
     --log-details 0 \
     --log-crypto 0 \
     --fix-mime 2  \
     --ignore-eol-check 1 \
     --sa-maxsize 512000 \
     --sa-socket /var/spool/spamd/spamd \
     --sa-subject "SPAM *** " \
     --sa-delta 0.5 \
     --sa-alt 1 \
     --sa-debug 1 \
     --sa-report 0 \
     --sa-quarantine 1.8 \
     --sa-delete 3.6 \
     --sa-reject 1 \
     --scanners "auto" \
     --install 1

  

7. Examples

For an standard installation (new or upgrade) with the user qscand, (first create the user) and then the options below would be enough (you can omit the option --sa-socket as the configure script might detect it):

groupadd qscand useradd -c "Qmail-Scanner Account" -g qscand -d /var/spool/qscan -s /bin/false qscand

./configure --domain mydomain.com \ --admin antivirus \ --admin-description "Antivirus MYDOMAIN" \ --add-dscr-hdrs yes \ --dscr-hdrs-text "X-Antivirus-MYDOMAIN" \ --ignore-eol-check yes \ --redundant yes \ --max-zip-size 80000000 \ --max-unpacked-files 2000 \ --virus-to-delete yes \ --settings-per-domain yes \ --debug 2 \ --sa-maxsize 512000 \ --sa-quarantine 2.1 \ --sa-delete 4.2 \ --sa-reject yes \ --sa-subject "SPAM *** " \ --sa-delta 0.5 \ --sa-alt yes \ --sa-debug yes \ --sa-report yes \ --sa-socket /var/spool/spamd/spamd [ --install yes ]

This will be an example of installing over a previous 1.1x installation (or a installation with a user different from the group), obviously the mailbox "antivirus@mydomain.com" should exist... The required_hits in the file /etc/mail/spamassassin/local.cf is '6.5'. If you're upgrading from 1.1x, don't try the manual installation, lets the configure script do its job. But I don't think it is wise to install over a 1.1x (even over a 1.2x), better do a fresh install.

./configure --qs-user qmailq \ --qs-group qmail \ --domain mydomain.com \ --admin antivirus \ --admin-description "Antivirus MYDOMAIN" \ --add-dscr-hdrs yes \ --dscr-hdrs-text "X-Antivirus-MYDOMAIN" \ --ignore-eol-check yes \ --redundant yes \ --max-zip-size 80000000 \ --max-unpacked-files 2000 \ --virus-to-delete yes \ --settings-per-domain yes \ --debug 2 \ --sa-maxsize 512000 \ --sa-quarantine 2.1 \ --sa-delete 4.2 \ --sa-reject yes \ --sa-subject "SPAM *** " \ --sa-delta 0.5 \ --sa-alt yes \ --sa-debug yes \ --sa-report yes \ --sa-socket /var/spool/spamd/spamd [ --install yes ]

I hope these options will be useful for you as they are for me. There isn't a specific mailing-list for this version, you can reach the official qmail-scanner-general mailing-list, you will find a lot of good stuff there.

Thanks to Jason for this very very good tool.

Thanks to Chris for the spamassasin quarantine patch, all my users are very happy since the patch was installed blocking tons of spam.

Salvatore Toribio

20130319

No warranty, expressed or implied, etc, etc, etc...

.