Opened 7 years ago

Closed 6 years ago

#5283 closed Bug/Something is broken (fixed)

Critpath Webmail is extremely slow

Reported by: Owned by:
Priority: Urgent Component: Tech
Keywords: webmail cpu-contention Cc: lclement@…,
Sensitive: no


Cannot log into any account on the Critpath webmail. The page will stall and say it's connecting after entering the login credentials. If it does log in, it is extremely slow and then will ask for the Mail Maintenance confirmation, choosing either Perform or Skip will put the page into a stall again.

Change History (17)

comment:1 Changed 7 years ago by

Hi Lansgston,

There seems to be a lot of contention for CPU time on didier right now (I checked by running vmstat 1 and noticed higher numbers in the first column).

I ran top and saw that hoursys is once again consuming a lot of CPU time. I killed the hoursys perl scripts (see this comment for more information - I can disable that account if you like, but one way or another, someone has to debug that script because it seems to run for ever and it sucks up a lot of the CPU cycles).

I'm also seeing a lot of use from the user ebbie - both receiving a lot of email and using a lot of CPU time via the IMAP process. ebbie seems to be using webmail and might be moving or searching or sorting a large mailbox. I'm still investigating.


comment:2 Changed 7 years ago by

Ok, I recently changed the password around 9:30AM this morning and attempted to help ebbie get into their account and that's when I saw how slow it was running. The user was saying they could not login and was also complaining on receiving a ton of spam so I was going to help them with a filter but I'm not sure what the source of all that spam is from. That's as much info as I have as of right now.

comment:3 Changed 7 years ago by

Oh - great, thanks for the update. I see from the horde database, that ebbie's account is configured to apply filters on login. Have you successfully logged in? With your permission, I would recommend:

  • You log out as ebbie
  • I update ebbie's pref so that filters are not applied
  • I kill the currently running IMAP processes under ebbie's name
  • You try to login again as ebbie

I suspect that ebbie's account starting trying to filter thousands of spam messages and something went south.


comment:4 Changed 7 years ago by

Ok, great. Except I haven't successfully logged in, so could you get in there and update ebbie's pref and kill the IMAP under ebbie's name? Then I can try to jump back in and see if it's fixed.

comment:5 Changed 7 years ago by

Ok - done!

comment:6 Changed 7 years ago by

Ok great it logged in extremely fast now! Now, what do you think we can do about the spam? Do you think the spam filter was set up incorrectly by the user and maybe just has to be set up one more time?

comment:7 Changed 7 years ago by

  • Owner set to
  • Status changed from new to assigned

comment:8 Changed 7 years ago by

Also please disable hoursys account for now. I'm doing some investigating into the website and getting some background info on it so that I can see if it's still being monitored by anyone. I don't want it to be a source of trouble and I will do my best to contact the admins.

comment:9 Changed 7 years ago by

I'm addressing the hoursys issue in #5291.

As for the spam - I think we might have better luck if we add a spam filter at the server level. This way, mail gets filtered as it arrives rather than when the user logs into webmail (Tech details are spelled out in ticket #1481 for how to do it). Let me know if you think this would work and I can put the filter in place.


comment:10 Changed 7 years ago by

That looks and sounds good jamie. Let's give it a try and see how well it improves the interface with people who are being hit with a ton of spam.

comment:11 Changed 7 years ago by

  • Resolution set to fixed
  • Status changed from assigned to closed

Ok. The filter is in place. It will put detected spam in a box called "detected-spam". For the record, I did the following:

su - ebbie
maildirmake Maildir/.detected-spam

Then, I created a file called .mailfilter with the contents:

logfile "$HOME/mailfilter.log"

# get rid of spam
if (/^X-Spam-Flag: YES/)
        to "Maildir/.detected-spam/"

You can look at mailfitler.log to ensure things are working properly. Once they are working, we should remove that line.

I think this should take care of things Langston - feel free to re-open if you still have trouble.


comment:12 Changed 7 years ago by

Thanks jamie. So this is just for the ebbie account? Or is it server wide? If it is server wide I forgot to ask you if this will affect other mail programs besides webmail. For example, since we use Outlook, will this pre-filter the mail before it downloads to our workstations? I'm just worried that some of the mail that is incorrectly labeled Spam might not make it to our inbox and that may cause some confusion amongst employees at FIGHT and other organizations. But if it is just for ebbie then that is great and ignore my previous questions.

comment:13 Changed 7 years ago by

This is just for ebbie - and yes, your concerns are valid! If you are using POP to just download your email to a desktop client, you will never get the messages marked as spam, so it might not be for everyone.

However, I recommend IMAP for all desktop clients. Most desktop IMAP clients have features that behave the way most people are accustomed to POP behaving (like removing messages from the server and saving local copies).


comment:14 Changed 7 years ago by

Ok, awesome! Thanks jamie, that is super helpful. I will notify the user and I'm sure they will be very happy.

comment:15 Changed 7 years ago by

  • Cc added
  • Resolution fixed deleted
  • Status changed from closed to assigned

It's happening again. Webmail is running extremely slow.

comment:16 Changed 7 years ago by

It looks again like it's contention for CPU.

I ran mf-find-resource-hog, which does a rough estimate of which users are using the most CPU cycles, I here are the top numbers between 4:00 pm and 4:30 pm:

0.71475 honeyeth
0.7425 mact
0.77675 vincey
0.78175 galaei
1.245 postfix
1.32 fight
24.786 root
47.097 bell

It looks like the user bell was really monopolizing CPU for this half hour period. I also noticed, via top, at 4:30 pm when I logged in, that spamd was using significant CPU time.

I'm not sure if bell's usage is related to filtering or not (I currently don't see a lot of messages in bell's inbox.

Still investigating...


comment:17 Changed 6 years ago by

  • Keywords cpu-contention added; down removed
  • Resolution set to fixed
  • Status changed from assigned to closed

didier's disk, CPU, and RAM all seem to be fine right now. I think this delay has been resolved.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.