Opened 6 years ago

Closed 6 years ago

#5283 closed Bug/Something is broken (fixed)

Critpath Webmail is extremely slow

Reported by: https://id.mayfirst.org/lclement.fight.org Owned by: https://id.mayfirst.org/jamie
Priority: Urgent Component: Tech
Keywords: webmail didier.mayfirst.org cpu-contention Cc: lclement@…, https://id.mayfirst.org/lclement.fight.org
Sensitive: no

Description

Cannot log into any account on the Critpath webmail. The page will stall and say it's connecting after entering the login credentials. If it does log in, it is extremely slow and then will ask for the Mail Maintenance confirmation, choosing either Perform or Skip will put the page into a stall again.

Change History (17)

comment:1 Changed 6 years ago by https://id.mayfirst.org/jamie

Hi Lansgston,

There seems to be a lot of contention for CPU time on didier right now (I checked by running vmstat 1 and noticed higher numbers in the first column).

I ran top and saw that hoursys is once again consuming a lot of CPU time. I killed the hoursys perl scripts (see this comment for more information - I can disable that account if you like, but one way or another, someone has to debug that script because it seems to run for ever and it sucks up a lot of the CPU cycles).

I'm also seeing a lot of use from the user ebbie - both receiving a lot of email and using a lot of CPU time via the IMAP process. ebbie seems to be using webmail and might be moving or searching or sorting a large mailbox. I'm still investigating.

jamie

comment:2 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

Ok, I recently changed the password around 9:30AM this morning and attempted to help ebbie get into their account and that's when I saw how slow it was running. The user was saying they could not login and was also complaining on receiving a ton of spam so I was going to help them with a filter but I'm not sure what the source of all that spam is from. That's as much info as I have as of right now.

comment:3 Changed 6 years ago by https://id.mayfirst.org/jamie

Oh - great, thanks for the update. I see from the horde database, that ebbie's account is configured to apply filters on login. Have you successfully logged in? With your permission, I would recommend:

  • You log out as ebbie
  • I update ebbie's pref so that filters are not applied
  • I kill the currently running IMAP processes under ebbie's name
  • You try to login again as ebbie

I suspect that ebbie's account starting trying to filter thousands of spam messages and something went south.

jamie

comment:4 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

Ok, great. Except I haven't successfully logged in, so could you get in there and update ebbie's pref and kill the IMAP under ebbie's name? Then I can try to jump back in and see if it's fixed.

comment:5 Changed 6 years ago by https://id.mayfirst.org/jamie

Ok - done!

comment:6 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

Ok great it logged in extremely fast now! Now, what do you think we can do about the spam? Do you think the spam filter was set up incorrectly by the user and maybe just has to be set up one more time?

comment:7 Changed 6 years ago by https://id.mayfirst.org/ross

  • Owner set to https://id.mayfirst.org/jamie
  • Status changed from new to assigned

comment:8 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

Also please disable hoursys account for now. I'm doing some investigating into the website and getting some background info on it so that I can see if it's still being monitored by anyone. I don't want it to be a source of trouble and I will do my best to contact the admins.

comment:9 Changed 6 years ago by https://id.mayfirst.org/jamie

I'm addressing the hoursys issue in #5291.

As for the spam - I think we might have better luck if we add a spam filter at the server level. This way, mail gets filtered as it arrives rather than when the user logs into webmail (Tech details are spelled out in ticket #1481 for how to do it). Let me know if you think this would work and I can put the filter in place.

jamie

comment:10 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

That looks and sounds good jamie. Let's give it a try and see how well it improves the interface with people who are being hit with a ton of spam.

comment:11 Changed 6 years ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from assigned to closed

Ok. The filter is in place. It will put detected spam in a box called "detected-spam". For the record, I did the following:

su - ebbie
maildirmake Maildir/.detected-spam

Then, I created a file called .mailfilter with the contents:

logfile "$HOME/mailfilter.log"

# get rid of spam
if (/^X-Spam-Flag: YES/)
{
        to "Maildir/.detected-spam/"
}

You can look at mailfitler.log to ensure things are working properly. Once they are working, we should remove that line.

I think this should take care of things Langston - feel free to re-open if you still have trouble.

jamie

comment:12 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

Thanks jamie. So this is just for the ebbie account? Or is it server wide? If it is server wide I forgot to ask you if this will affect other mail programs besides webmail. For example, since we use Outlook, will this pre-filter the mail before it downloads to our workstations? I'm just worried that some of the mail that is incorrectly labeled Spam might not make it to our inbox and that may cause some confusion amongst employees at FIGHT and other organizations. But if it is just for ebbie then that is great and ignore my previous questions.

comment:13 Changed 6 years ago by https://id.mayfirst.org/jamie

This is just for ebbie - and yes, your concerns are valid! If you are using POP to just download your email to a desktop client, you will never get the messages marked as spam, so it might not be for everyone.

However, I recommend IMAP for all desktop clients. Most desktop IMAP clients have features that behave the way most people are accustomed to POP behaving (like removing messages from the server and saving local copies).

jamie

comment:14 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

Ok, awesome! Thanks jamie, that is super helpful. I will notify the user and I'm sure they will be very happy.

comment:15 Changed 6 years ago by https://id.mayfirst.org/lclement.fight.org

  • Cc https://id.mayfirst.org/lclement.fight.org added
  • Resolution fixed deleted
  • Status changed from closed to assigned

It's happening again. Webmail is running extremely slow.

comment:16 Changed 6 years ago by https://id.mayfirst.org/jamie

It looks again like it's contention for CPU.

I ran mf-find-resource-hog, which does a rough estimate of which users are using the most CPU cycles, I here are the top numbers between 4:00 pm and 4:30 pm:

0.71475 honeyeth
0.7425 mact
0.77675 vincey
0.78175 galaei
1.245 postfix
1.32 fight
24.786 root
47.097 bell

It looks like the user bell was really monopolizing CPU for this half hour period. I also noticed, via top, at 4:30 pm when I logged in, that spamd was using significant CPU time.

I'm not sure if bell's usage is related to filtering or not (I currently don't see a lot of messages in bell's inbox.

Still investigating...

jamie

comment:17 Changed 6 years ago by https://id.mayfirst.org/dkg

  • Keywords didier.mayfirst.org cpu-contention added; down removed
  • Resolution set to fixed
  • Status changed from assigned to closed

didier's disk, CPU, and RAM all seem to be fine right now. I think this delay has been resolved.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.