Email Security Techniques:
Filtering the Future

2005 O'Reilly OSCon, Portland, Oregon

Kees Cook

OSDL

or..

How I deal with email I don't want to get

http://outflux.net/oscon-2005/

Creative Commons Attribution-NonCommercial-ShareAlike License

Who am I?

Kees Cook

@ work:

kees@osdl.org

http://developer.osdl.org/kees/

@ home:

kees@outflux.net

http://outflux.net/

http://outflux.net/blog/

on IRC

“drb” on FreeNode

What do I do?

Senior Systems Administrator for OSDL

http://osdl.org/lab_activities/lab_hardware/server_room.html

Kernel.org mirror/user administration

Author of Sendpage, GOPchop

http://sendpage.org/

http://gopchop.org/

Part-time contributor to Inkscape

http://inkscape.org/

Part-time commercial software consultant

http://outflux.net/prof/

Email Filtering at OSDL.org and Outflux.net

What am I trying to stop?

How can I stop it?

How did I stop it?

Did it work?

What's next?

What am I trying to stop?

Hormel wants it called “Unsolicited Bulk Email”

Viruses (Worms, Trojans, Zombies)

Phishing

Bounced Forgeries

Hormel wants it called
“Unsolicited Bulk Email”

UBE, defined

http://www.spamhaus.org/definition.html

Unsolicited email is okay

Bulk email is okay

Unsolicited Bulk email is Spam

The issue is “consent” not “content”

Traditionally tries to advertise or scam

Viruses (Worms, Trojans, Zombies)

Probably qualifies as UBE

I'll say “Virus”, even if it's inaccurate

Spread

Worms don't need human action

Viruses and Trojans need humans

Impact

High volume

Automated

Phishing

Qualifies as “Trojan”

Still automated, but content tends to differ pretty heavily

Different due to identity/financial theft

Bounced Forgeries

Called “Joe Jobbing”

Due to Viruses and UBE forging return addresses

My biggest pet peeve

Thousands per day and still a big chunk of rejections

(Fun with gnuplot)

set xdata time

set timefmt "%Y%m%d"

set format x "%b\n%Y"

set xrange ["20010101":"20030720"]

set term png small

set title "Junkmail per Week"

set grid

set output "junkreport-oscon-pre.png"

set key top left

plot \

"weekly-junkmail.data" using 1:2 title \

"junkmail" with lines linewidth 5, \

\

"spamassassin.data" using 1:2 title \

"added SpamAssassin" with impulse linewidth 5, \

\

"outflux.data" using 1:2 title \

"created outflux.net" with impulse linewidth 5, \

\

"bogofilter.data" using 1:2 title \

"added bogofilter" with impulse linewidth 5

20010520	60

20010603	53

20010610	38

20010617	41

...

20020225	1000

How can I stop it?

Network evaluation: HELO, IP, DNS, RBLs

Sender validation: SAV, SPF, Domainkeys

Recipient validation: tables, spamtrap, SRS

Content scanning: regex, Bayes, SURBLs, DCC

Virus scanning: ClamAV

Session management: Quarantine, greylisting, tarpits

Network Evaluation

Local Network Policies

HELO validation, delay, high-MX testing

IP/DNS validation/firewalling

External Network/Behavioral Policies

IP-based RBLs

relays.ordb.org

opm.blitzed.org

list.dsbl.org

cbl.abuseat.org

sbl.spamhaus.org

DNS-based RBLs

bogusmx.rfc-ignorant.org, dsn.rfc-ignorant.org

Sender Validation

Sender Address Verification

Not really a common standard

Sender Policy Framework

http://spf.pobox.com/

Domainkeys

http://antispam.yahoo.com/domainkeys

Recipient Validation

List of valid inbound addresses

Good for managing your relay

Spamtrap addresses

dangerous since ISPs could be forwarding it on

Sender Rewrite Scheme

http://spf.pobox.com/srs.html

SRS0=G8cU9RJv=VM=outflux.net=kees@outflux.net

Content Scanning

regex

http://spamassassin.apache.org/

http://www.nuclearelephant.com/projects/dspam/

Bayes

SURBLs

http://www.surbl.org/

DCC

http://razor.sourceforge.net/

Virus Scanning

ClamAV is my hero

http://www.clamav.net/

File::Scan is nice and quick too

http://search.cpan.org/dist/File-Scan

Session Management

Tracking

Sender IP, Envelope To, Envelope From

http://search.cpan.org/dist/File-Cache/

Uses

Greylisting

http://projects.puremagic.com/greylisting/

Tarpits

http://www.spamcannibal.org/

Quarantine

How did I stop it?

Philosophy

sendmail

MIMEDefang

SpamAssassin

ClamAV

Custom code

Philosophy

Email filtering is a very “religious” subject

false positives

censorship

Scoring vs Rejecting

Use of RBLs

No one is “right”, but I've got my opinion

Stick to technical policy, not social policy

Don't want to change my “publishing” behavior

sendmail

Does MTA matter?

everyone has support for RBLs now

everyone has support for various DNS/IP checks

I chose sendmail

ability to reject email instead of accept/bounce

milter is very powerful

http://outflux.net/sendmail.mc.txt

MIMEDefang

Set up milter in sendmail.mc

INPUT_MAIL_FILTER(`mimedefang', `S=unix:/var/spool/MIMEDefang/mimedefang.sock, F=T, T=C:2m;S:5m;R:5m;E:8m')

Learn to use Perl

http://www.mickeyhill.com/mimedefang-howto/#s9

Example of Outflux mimedefang-filter

http://outflux.net/mimedefang-filter.txt

Email passes through various soft states (“temp fail”, “greylist”) until it hits a hard state (“reject”) or processing finishes.

SpamAssassin

Granddaddy of content filtering

Keep your rules up to date

http://www.rulesemporium.com/

http://www.rulesemporium.com/rules/70_sare_genlsubj_eng.cf

http://www.rulesemporium.com/rules/70_sare_header0.cf

http://www.rulesemporium.com/rules/70_sare_html0.cf

http://www.rulesemporium.com/rules/70_sare_specific.cf

http://www.rulesemporium.com/rules/70_sare_spoof.cf

http://www.rulesemporium.com/rules/99_sare_fraud_post25x.cf

ClamAV

My hero

Virus list is auto updated

Many ways to hook in ClamAV

milter directly

MIMEDefang

SpamAssassin

amavisd

No excuse not to use it, there's even ClamWin

http://www.clamwin.net/

Custom code

http://outflux.net/mimedefang-filter.txt

SAV

Inspired by exim

SRS

Needed to stop bounces

SPF

More useful for other domains

Greylisting

Handy for iffy rules

Did it work?

Lies, Damn Lies, and Statistics

Whatever you do first will have the biggest impact

I don't manage AOL, but it made quite a difference

User Impressions

Subjective experiences are probably the best indicator of success in email filtering.

Lies, Damn Lies, and Statistics

http://outflux.net/junkreport-weekly.png

http://outflux.net/junkreport-daily-zoom.png

http://outflux.net/spam/graphs/

(If you want to make your own graphs: GD::Graph::pie is great! Here's how I parsed the logs: http://outflux.net/sm-syslog-scan.txt)

User Impressions

I think it's great

So does Linus Torvalds

Linus went from 1GB/week to less than 1MB/week

Date: Thu, 13 Jan 2005 08:48:27 -0800 (PST)

From: Linus Torvalds <torvalds@osdl.org>

To: ...

Cc: Kees Cook <kees@osdl.org>

Subject: Re: SPF

...

But I'll forward your thoughts to The Most Holy Master of Email at OSDL,

aka Kees Cook. I bow down before him, since he makes my job so much

easier, and he's the one who makes the decisions.

Linus

What's next?

Bot networks

Propaganda UBE

Improved logging/statistics

User-controlled filtering

Just add it to mimedefang-filter as another stage

Questions/Comments

If I run out of time, or you're too shy, send me email or find me on IRC!

relaxed filters: kees@osdl.org

heavy filters: kees@outflux.net

IRC: drb on FreeNode

Let me know how I did with an OSCON Survey:

http://oreilly.com/go/spkreval

A copy of this presentation can be found here:

http://outflux.net/oscon-2005/

Watch out, my web pages a bit out of date.

Why?

- resource drain

- time sucker

- annoying

2005-05-25T11:48:44