I finally decided to write a little shell script to read the mail log file and extract the last minute of log to count the number of rejected attempts to send mail to my server by the same host and ban this host using iptables if a defined limit is reached.

Here is a version of my script:
#!/bin/bash
IPT=/sbin/iptables
LIMIT=10
cd /root/Filters
# first get one minute of log
grep "`date +"%b %d %H:%M:" --date="1 minutes ago"`" /var/log/mail.info > minutelog
# now extract the rejected attempts, sort and count uniq ip
cat minutelog | grep "reject:" | cut -d" " -f10 | cut -d"[" -f2 | cut -d"]" -f 1 | sort | uniq -c | sort -n | sed 's/^[ \t]*//' > tmp1
# for each line in result
while read line
do
MYCOUNT=`echo $line | cut -d" " -f1`
MYIP=`echo $line | cut -d" " -f2`
if [ $MYCOUNT -lt $LIMIT ] ;
then
echo $MYIP is ok: $MYCOUNT attempts
else
echo blocking the spammer at $MYIP with $MYCOUNT attempts
$IPT -I INPUT -i eth0 --proto tcp -s $MYIP --destination-port 25 -j DROP
echo $MYIP >> blocked.smtp
fi
done < tmp1
rm -f minutelog
rm -f tmp1
You can also download this script here: filter_smtp.zip

This shell script is running for about 12 hours on my mail server and it already stopped 701 spamming machines. Any feedback on the script is welcome !