Opened 11 years ago

Closed 11 years ago

#547 closed Question/How do I...? (fixed)

How do I filter Spam in Thunderbird?

Reported by: https://id.mayfirst.org/lindabr Owned by: https://id.mayfirst.org/jamie
Priority: Medium Component: Tech
Keywords: spamfilter thunderbird email Cc:
Sensitive: no

Description

Do you have any suggestions? I am using Thunderbird for my email. I have a folder into which Spam goes automatically and then it is my job to choose "Mark/as junk" which my understanding is will "teach" Thunderbird not to permit those emails to even reach me in the future. I always mark the entire spam folder contents as junk which puts them in the trash.

In addition, there are other emails that reach my "in box" that I also "Mark/as junk."

But the spam keeps coming and I can't see any evidence that Thunderbird is learning anything. Is there something I'm doing wrong, or is there something more Mayfirst can do to prevent all this spam?

I'd appreciate any suggestions you may have about this.

Thanks! Linda

Change History (17)

comment:1 Changed 11 years ago by https://id.mayfirst.org/dkg

  • Keywords spamfilter thunderbird email added
  • Summary changed from Spam to How do I filter Spam in Thunderbird?

Hi Linda--

is this a followup to #216? Do any of the messages that show up in your inbox start off being automatically marked as Junk by Thunderbird?

Also, what Thunderbird configuration choices have you made concerning the junk filter? In Thunderbird's Preferences dialog box, if you click the Privacy icon, one of the tabs should be "Junk"; the settings will be there.

comment:2 Changed 11 years ago by https://id.mayfirst.org/alfredo

I find that the absolute most effective thing is to activate the filter for *SPAM flag as per the instructions in ticket #216. But that can be a bit tricky at first so it needs checking.

1 -- Are you filtering any spam at all into your new spam box (that would mean that Spam Assassin and your filters are both working)?

2 -- Are you receiving email into your box that has "spam" in the subject header (that would indicate that Spam Assassin is working but your filters aren't)?

3 -- The filter should be set up as that the "subject" "contains" "*SPAM " Right? and the "Match Any of the Following" is clicked (if you have option in your version of Thunderbird"

4 -- Thunderbird evaluates filters from top to bottom so put this one at the top. Otherwise other filters do stuff to the email that might not allow spam filter to work properly on all of it.

I'm just interested in knowing whether there's identified spam getting through. If not, and you're seeing an increase, it might mean that we need to do something on this side. Spammers are forever upgrading their programs to do stuff with. But,I haven't picked up on that personally; in fact, I've been experiencing a drop in spam lately and had figured the spam upgraders had taken the holidays off.

Let us know.

Alfredo

comment:3 Changed 11 years ago by https://id.mayfirst.org/jamie

From an email Linda sent to support:


Hi, Alfredo,

My apologies for replying to email rather than on the site. Unfortunately I have just switched from Firefox to Safari because I was so tired of having Firefox freeze up/crash. I tried to get into your support site, but my Safari doesn't know the password and unfortunately neither do I so I was unable to either log in or change the password. So, now I need to know how to fix that problem. Sorry!

OK - regarding the Spam. I'll answer the questions by number below as best I can:

  1. I can't get into the MF web site so don't know what ticket #216 said. However, I do have SPAM going into my spam box many times a day. Contrary to your experience, both Neil and I thought we were getting inundated over the holidays and I can't say I've noticed it decreasing since.
  1. No - no email comes into my in box labeled as spam, though things that are spam and should be so labeled do come into my in box. I then "mark them as junk" and that causes them to go into the trash. Then when I open the Detected Spam folder and highlight everything in it and mark it all as junk, it is put into the trash. Somehow I thought this process was supposed to teach Thunderbird to reject all those things so I didn't see them at all!
  1. This may have nothing to do with what you ask - but when I look at Message Filters in the TOOLS menu, and click on the filter called "detected spam" (which is the only filter apparently) and ask to edit it, what I see is this:

X-Spam-Flag IS Yes Move messages to Detected-Spam On Local Folder

Does that answer any of your questions? I'm sorry I don't understand this stuff better.

Thanks! Linda

comment:4 Changed 11 years ago by https://id.mayfirst.org/jamie

I'm sorry you are having such a bad experience with Firefox!

You can recover your password from Firefox by following the following steps:

  • Open Firefox
  • Access your preferences (on a Mac I think you should click the Apple Menu -> Preferences)
  • Click the security tab
  • Click Show passwords...
  • Click Show passwords (again - the button in the bottom right corner)
  • Scroll down to the site support.mayfirst.org - and you should see your password.

From your message - the fact that you are not seeing any messages marked with SPAM in the subject line and based on your description of what you have in the Message Filters - I think we can safely conclude that you do have a filter in place that works with our Server to filter messages that we think our spam.

That's an important step that you've already taken care of.

I think the problem - is that you are getting messages that our detection system is not detecting.

Unfortunately, there is less that we can do about that problem.

Can you please let us know (approximately) how many of these undetected spam messages you are getting per day?

Also - it would be really helpful to get see the "full headers" of a few sample messages.

You can view the headers of a message in Thunderbird by opening the message, then clicking View -> Message Source.

Here is a sample bunch of message headers:

From - Fri Feb 23 09:08:42 2007
X-Account-Key: account2
X-UIDL: UID10-1169645766
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <chavez-admin@virtual.chavez.mayfirst.org>
X-Spam-Checker-Version: SpamAssassin 3.1.7-deb (2006-10-05) on 
	chavez.mayfirst.org
X-Spam-Level: *
X-Spam-Status: No, score=2.0 required=5.0 tests=RCVD_IN_SORBS_DUL 
	autolearn=no version=3.1.7-deb
X-Original-To: chavez-admin@chavez.mayfirst.org
Delivered-To: chavez-admin@chavez.mayfirst.org
Received: from localhost (localhost.localdomain [127.0.0.1])
	by chavez.mayfirst.org (Postfix) with ESMTP id 19B07855EB
	for <chavez-admin@chavez.mayfirst.org>; Fri, 23 Feb 2007 09:09:19 -0500 (EST)
X-Virus-Scanned: Debian amavisd-new at chavez.mayfirst.org
Received: from chavez.mayfirst.org ([127.0.0.1])
	by localhost (chavez.mayfirst.org [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id ZG4nPLyQ0kNw for <chavez-admin@chavez.mayfirst.org>;
	Fri, 23 Feb 2007 09:09:19 -0500 (EST)
Received: from [192.168.69.104] (cpe-74-64-42-251.nyc.res.rr.com [74.64.42.251])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by chavez.mayfirst.org (Postfix) with ESMTP id E2CA7855E1
	for <chavez-admin@virtual.chavez.mayfirst.org>; Fri, 23 Feb 2007 09:09:18 -0500 (EST)
Message-ID: <45DEF559.20107@virtual.chavez.mayfirst.org>
Date: Fri, 23 Feb 2007 09:08:25 -0500
From: Jamie McClelland <chavez-admin@virtual.chavez.mayfirst.org>
User-Agent: Icedove 1.5.0.9 (X11/20061220)
MIME-Version: 1.0
To:  chavez-admin@virtual.chavez.mayfirst.org
Subject: test
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

If you enter the headers in between these curly braces:

{{{
Your headers go here
}}}

They will be displayed like the headers I submitted to this ticket and be easier to read.

Thanks Linda!

Jamie

comment:5 Changed 11 years ago by https://id.mayfirst.org/lindabr

Thanks! I managed to find the password using Firefox as you explained and used it to get into your support site in Safari, which I am now using. As you can see, I got to your support page successfully and to your listing of all my tickets!!! Hooray!!

I tried using your Firefox directions to find the password within Safari and the security preference only offers a list of cookies, not a list of passwords. Wonder why that difference?

We're going out of town tomorrow for a couple of days and then I'll be without my computer for a couple of days after we return, but as soon as I can I'll send you the headers from some of the spam that is not being marked as spam.

Thanks again for helping!

Linda

comment:6 Changed 11 years ago by https://id.mayfirst.org/jamie

Hooray! Nice work Linda. We'll look forward to hearing from you when you get back.

comment:7 Changed 11 years ago by https://id.mayfirst.org/jamie

Thanks for sending the three sample spam messages with headers.

Those are messages that are spam filters are missing. Two of them are what I would consider typical spam messages that our system should detect but did not (the one about the $2 million dollar investment and the one advertising the business directory).

With these messages, the key header that I am interested is the one that starts with X-Spam-Status. Here's the line for the $2 million investment email:

X-Spam-Status: No, score=2.2 required=5.0 tests=ADVANCE_FEE_2, \
ADVANCE_FEE_3,BAYES_00,HTML_MESSAGE,MILLION_USD,US_DOLLARS_3 \
autolearn=noversion=3.2.3

This example means that our spam system gave this message a score of 2.2 and it lists (in shorthand) the tests that the message matched. Our system will tag a message as spam if it gets a score of 5 or more.

The test called BAYES_00 refers to an "self-learning" test. If our spam system gives a message a spam score lower than .1, it will study the message and give future messages that look the same a lower spam score. BAYES_00 means that this message looks to spamassassin very similar to messages that it has previously tagged as legitimate email, which lowered the score.

This is bit disturbing to me - I'm not sure why it did that.

The third one looks to be like a legitimate pay pal email message. If paypal says tells you they don't email their customers they are lying! This doesn't mean it's not spam - but it does not look like a scam to me. I would expect our spam system to not tag messages like these.

The number of undetected spam messages that I get every day ranges from 0 to 10 or so. This number is in comparison to about 150 detected spam messages or so I get per day. If you are getting much higher ratios of undetected spam let us know.

Unfortunately, we may not be able to get lower ratios. One of the reasons is because we are very cautious about false positives, meaning tagging a message as spam that you want to receive.

I would recommend that you teach thunderbird about the messages that our systems miss - that's a good way to get a double spam test which might help reduce that ratio. Just be sure to periodically check your folders in which these messages are being checked.

comment:8 follow-up: Changed 11 years ago by https://id.mayfirst.org/lindabr

Thanks, Jamie, for this information. I think the number of spam we're getting is comparable to yours. If you could be a little more specific about what you say in your last paragraph, I would know better what to do.

You say: "I would recommend that you teach thunderbird about the messages that our systems miss - that's a good way to get a double spam test which might help reduce that ratio. Just be sure to periodically check your folders in which these messages are being checked."

Everything that is missed by the spam filter I highlight (which unfortunately also opens that email) and under the Mail menu choose Mark/As Junk. I have thought that was "teaching" Thunderbird but maybe I'm wrong about that.

I have no idea how to periodically check the folders in which the messages are being checked. Can you be more specific about what you mean - what should I be doing?

Thanks!

comment:9 in reply to: ↑ 8 Changed 11 years ago by https://id.mayfirst.org/jamie

Replying to https://id.mayfirst.org/lindabr:

Thanks, Jamie, for this information. I think the number of spam we're getting is comparable to yours. If you could be a little more specific about what you say in your last paragraph, I would know better what to do.

Sure - thanks for the following up with the questions.

Everything that is missed by the spam filter I highlight (which unfortunately also opens that email) and under the Mail menu choose Mark/As Junk. I have thought that was "teaching" Thunderbird but maybe I'm wrong about that.

Yes - that is what I meant - I should have said: I would suggest continuing what you seem to already be doing of teaching Thunderbird about spam. I think you are doing that right thing.

I have no idea how to periodically check the folders in which the messages are being checked. Can you be more specific about what you mean - what should I be doing?

Thanks for asking this question as well. This question prompted me to look more closely at how Thunderbird does Junk mail fitlering and I and am a bit shocked by how non-intuitive it is.

To help both you and other members, I wrote a help page on the topic. If you click on the link in the previous sentence, you can make sure you are properly setup with Thunderbird junk filtering.

To answer your question, based on your comment above, messages that our server detects as spam are being moved into a folder in your Thunderbird that is called "Detected-Spam." You should see that folder listed under your Inbox folder. If you click on it, you will see all the messages are server tags as spam. By clicking on that folder and doing a quick scan, you can periodically ensure we are not mis-tagging your messages.

Thunderbird will move messages into a folder called "Junk." If you don't see a folder called "Junk" you may need to create that folder (the new help page explains how to do that. If you do see it, then you should be able to scan for messages that Thunderbird mis-tags as well.

Let us know if you have more questions about this. Jamie

comment:10 Changed 11 years ago by https://id.mayfirst.org/lindabr

Hi, Jamie, Thanks so much for all your help with this! You folks are great at supporting folks like me who don't know much about this stuff.

I really appreciate the posting of the help page for Thunderbird, but I find the situation on the Mac is a little different. First of all, the Preferences are under the "Thunderbird" menu which is just to the right of the Apple menu. After that everything you said about clicking the Privacy button, etc., is just right.

Then as far as setting the account behavior, it works like this: I found "account settings" under the Tools menu. There is no "Server Settings" heading, but there is a heading on the left side called "Local Folders" and under that is a subheading of "junk settings." Everything else was just as you described.

I think my last question may be this: Now what happens to the Detected-Spam folder. Is that still being used or has it been replaced by the new Junk folder?

Also, will this setup automatically trash the junk folder contents when Thunderbird is shut down? That's the way it's working with my Detected-Spam folder and I'd like that to continue if possible.

Thanks again! Linda

comment:11 Changed 11 years ago by https://id.mayfirst.org/jamie

Thank you for giving back Linda. I just updated the page with your changes (and for future reference - you can change the page without waiting for me by clicking the Edit button underneath the help page).

As for what happens to the Detected-Spam folder - that's a good question and I don't know the answer. If you check "Trust Junk mail headers set by SpamAssassin" then I really don't know whether your filter rule will put our server-detected messages into the Detected-Spam or whether the Junk Mail control will put it in the Junk folder.

As for mail in the Junk folder getting deleted - please see the Tools -> Account Settings -> Local Folders -> Junk Settings. You should see an option allowing you to "Automatically delete junk mail oder than... 14 days". And you can change the number of days to suit your needs.

comment:12 follow-up: Changed 11 years ago by https://id.mayfirst.org/lindabr

What if I just delete the Detected-Spam folder. Is it serving any purpose? Would the junk just go to the junk folder in that case?

Also - I just noticed a slight error in what I had told you about how the Mac Thunderbird works. There are actually two different folders for "junk settings" on the left side of the "account settings window. One is under, in my case, a heading called "Linda's Personal Account" and the other is under "local folders." I changed one under "local folders" a while back based on your instructions, and strangely the other junk settings folder (which has exactly the same options) did not change automatically. So I have now changed it to agree with the other one. Strange that there are two and they are apparently different folders with the same contents that don't "talk" to each other!

comment:13 in reply to: ↑ 12 Changed 11 years ago by https://id.mayfirst.org/jamie

Replying to https://id.mayfirst.org/lindabr:

What if I just delete the Detected-Spam folder. Is it serving any purpose? Would the junk just go to the junk folder in that case?

Hi Linda,

Before deleting the Detected-Spam folder, I would recommend that you remove the filter rule that uses it.

In comment three above you said:

  1. This may have nothing to do with what you ask - but when I look at Message Filters in the TOOLS menu, and click on the filter called "detected spam" (which is the only filter apparently) and ask to edit it, what I see is this:

X-Spam-Flag IS Yes Move messages to Detected-Spam On Local Folder

That's the rule that moves detected spam into your Detected-Spam folder. You should delete that rule. Then, you can delete the Detected-Spam folder.

As long as you have checked the "Trust Junk mail headers set by SpamAssassin" when you set up your Junk Settings, then the May First/People Link detected spam will end up in the Junk folder as well.

This is all very cutting edge :) - so please report back if it does work as advertised.

Also - I just noticed a slight error in what I had told you about how the Mac Thunderbird works. There are actually two different folders for "junk settings" on the left side of the "account settings window. One is under, in my case, a heading called "Linda's Personal Account" and the other is under "local folders." I changed one under "local folders" a while back based on your instructions, and strangely the other junk settings folder (which has exactly the same options) did not change automatically. So I have now changed it to agree with the other one. Strange that there are two and they are apparently different folders with the same contents that don't "talk" to each other!

That's really confusing! I'm not sure I fully understand why Thunderbird is designed that way.

comment:14 Changed 11 years ago by https://id.mayfirst.org/lindabr

Hi - I see this morning that there is nothing at all in the Detected Spam folder, but there are quite a few emails in the Junk folder - but not the junk folder that I created yesterday. That is also empty. But there is a JUNK folder down at the bottom of the list of folders - a unique folder with a green icon like the one that says JUNK in the menu bar. That folder came with Thunderbird as far as I know. That's where everything is.

So - - I wonder if I need either of the folders I created now that the filter has been set to mark things as Junk. If everything is going into the folder with the green icon, maybe I can get rid of both the Detected Spam and the Junk folders in my In box?

What do you think?

Linda

comment:15 Changed 11 years ago by https://id.mayfirst.org/jamie

Hi Linda,

Sounds like you can safely delete those two extra folders that are empty - although I'd recommend that you delete the rule as I mentioned in my comment above just to be safe.

Jamie

comment:16 Changed 11 years ago by https://id.mayfirst.org/lindabr

Hi, Jamie, I did delete the two folders. I wasn't sure what a "rule" was but I did go back as you said and found the place where "Deleted-spam" was listed as a filter, I think, and I deleted that.

In any case, all the spam seems to be going into the junk folder with the green icon - well, at least most of it is going there and the rest I'm putting there. I'm not sure Thunderbird is such a fast learner but we'll keep working at it.

Thanks for your help. I think we can consider this closed.

Linda

comment:17 Changed 11 years ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from new to closed

Thanks for posting the follow up Linda.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.