Opened 6 years ago

Last modified 6 years ago

#6259 assigned Bug/Something is broken

Very slow message delivery on albizu

Reported by: https://id.mayfirst.org/takethestreets Owned by: https://id.mayfirst.org/jamie
Priority: Medium Component: Tech
Keywords: email albizu.mayfirst.org Cc:
Sensitive: no

Description

I've been experiencing it all day, and it seems to be getting worse. Load numbers are in the area of 12 - by comparison, on chavez it's closer to 3.2.

Here are message headers demoing the problem:

Return-Path: <REDACTED>
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on chavez.mayfirst.org
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=AWL,HTML_MESSAGE
	autolearn=disabled version=3.3.2
X-Original-To: REDACTED@takethestreets.org
Delivered-To: REDACTED@chavez.mayfirst.org
Received: from chavez.mayfirst.org (localhost [127.0.0.1])
	by chavez.mayfirst.org (Postfix) with ESMTP id B6BB65E249
	for <REDACTED@takethestreets.org>; Thu,  4 Oct 2012 21:05:01 -0400 (EDT)
Received: from albizu.mayfirst.org (albizu.mayfirst.org [209.51.163.7])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by chavez.mayfirst.org (Postfix) with ESMTPS id 908585E242
	for <REDACTED@takethestreets.org>; Thu,  4 Oct 2012 21:05:01 -0400 (EDT)
Received: from albizu.mayfirst.org (localhost [127.0.0.1])
	by albizu.mayfirst.org (Postfix) with ESMTP id 1D401216CA;
	Thu,  4 Oct 2012 19:14:25 -0400 (EDT)
Received: from palante2.palantetech.com (bella.legalmomentum.org [38.121.134.109])
	by albizu.mayfirst.org (Postfix) with ESMTP id EEFF423836;
	Thu,  4 Oct 2012 17:36:39 -0400 (EDT)
Received: from redmine.palantetech.com (localhost [127.0.0.1])
	by palante2.palantetech.com (Postfix) with ESMTP id A30DE3A16B;
	Thu,  4 Oct 2012 17:36:39 -0400 (EDT)
Date: Thu, 4 Oct 2012 17:36:39 -0400
From: tickets@palantetech.com
To: REDACTED
Cc: REDACTED
Message-Id: <redmine.issue-4632.20121004173639@palantetech.com>
Subject: REDACTED
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary=mimepart_506e0167a01fe_70cb..fdba518dc7030
X-Redmine-Issue-Author: JooHyunKang
X-Auto-Response-Suppress: OOF
X-Redmine-Issue-Id: 4632
X-Mailer: Redmine
X-Redmine-Project: cpr
Auto-Submitted: auto-generated
X-Redmine-Site: Palante Tech Redmine
X-Redmine-Sender: REDACTED
X-Redmine-Host: hq.palantetech.com
List-Id: <tickets.palantetech.com>
X-Virus-Scanned: ClamAV using ClamSMTP
X-Virus-Scanned: ClamAV using ClamSMTP

Change History (8)

comment:1 Changed 6 years ago by https://id.mayfirst.org/ross

  • Keywords email albizu.mayfirst.org added
  • Owner set to https://id.mayfirst.org/ross
  • Status changed from new to assigned

Hi Jon,

I'm looking into it. My current theory is that an account has been high-jacked and is being used for spam, but it's taking so long for mailq to run that I haven't even gotten clear numbers. However, the /var logical volume has gone from 3G to 4.4G in the course of a few hours, so I suspect it's something along these lines.

~/ross

comment:2 Changed 6 years ago by https://id.mayfirst.org/ross

I have disabled the offending email accounts, spam was the cause of this. The offending account was pixelspace-ny.com. I am in the process of following this page https://support.mayfirst.org/wiki/clearing-spam-backscatter-from-mailq to restore all good emails. I suspect that albizu has been put on a few spam lists at this point, unfortunately. I see quite a few deferred emails that were rejected by the host.

comment:3 Changed 6 years ago by https://id.mayfirst.org/ross

  • Owner changed from https://id.mayfirst.org/ross to https://id.mayfirst.org/jamie

All right, the general problem is fixed and all non-spam emails should be restored, but we still have the lingering problem of defer emails which I'm not sure how to deal with. I ran the script to clean out emails in /var/spool/postfix/deferred/ and all of them seemed to be related to the problem account.

However, I cannot figure out what to do about /var/spool/postfix/defer/ . There doesn't seem to be a regex that is meaningful for files in that directory.

Due to the timing of my moving the defer* directories, I ended up with defer*.spamfull directories and defer* directories that each had files in them. I'm pretty sure that the deferred directory is acceptably dealt with but the /var/spool/postfix/defer directory remains a problem. I'm not sure how to deal with what currently exists related to the /var/spool/postfix/defer and /var/spool/postfix/defer.spamfull directories.

Jamie please look this over.

Note:

The current file structure looks like this:

0 albizu:/var/spool/postfix# ls -l
total 3408
drwx------  2 postfix root      106496 Oct  5 00:41 active
drwxr-xr-x  2 root    root        4096 Oct  4 22:04 active.name-collisions
drwx------  2 postfix root      966656 Oct  4 23:07 active.spamfull
drwx------  2 postfix root        4096 Oct  5 00:37 bounce
drwx------  2 postfix root        4096 May 20  2008 corrupt
drwx------ 18 postfix root        4096 Oct  4 22:23 defer
drwx------ 10 postfix root        4096 Oct  5 00:35 deferred
drwxr-xr-x 18 root    root        4096 Oct  5 00:08 deferred.name-collisions
drwx------ 18 postfix root        4096 Oct  4 22:24 deferred.spamfull
drwx------ 18 postfix root        4096 May 24  2008 deferred.spamfull.orig
drwx------ 18 postfix root        4096 May 24  2008 defer.spamfull
drwxr-xr-x  2 root    root        4096 Sep 14  2008 dev
drwxr-xr-x  3 root    root        4096 Oct  5 00:22 etc
drwx------  2 postfix root        4096 Sep 21 15:04 flush
drwx------  2 postfix root        4096 May 20  2008 hold
drwx------  2 postfix root       77824 Oct  5 00:41 incoming
drwxr-xr-x  2 root    root        4096 Oct  4 22:04 incoming.name-collisions
drwx------  2 postfix root     2211840 Oct  4 22:33 incoming.spamfull
drwxr-xr-x  2 root    root        4096 Oct  5 00:22 lib
drwx-wx--T  2 postfix postdrop    4096 Oct  5 00:41 maildrop
drwxr-xr-x  2 root    root        4096 Jun  4  2011 pid
drwx------  2 postfix root        4096 Oct  5 00:22 private
-rw-------  1 root    root        1024 May 30  2010 prng_exch
drwx--s---  2 postfix postdrop    4096 Oct  5 00:22 public
drwx------  2 postfix root        4096 May 20  2008 saved
-rw-------  1 root    root        8192 May 30  2010 smtpd_scache.db
-rw-------  1 root    root        8192 May 30  2010 smtp_scache.db
drwx------  2 postfix root        4096 Oct  4 02:10 trace
drwxr-xr-x  3 root    root        4096 May 20  2008 usr
0 albizu:/var/spool/postfix# 

Also note that there are many name collisions in /var/spool/postfix/deferred.name-collisions. These all seem to be related to the offending account.

Finally, I'm not sure what we should do with the offending account tyler [at] pixelspace-ny.com (as well as a bunch of other email addresses that forward to that account. Can you please also figure out what to do with that?

~/ross

comment:4 Changed 6 years ago by https://id.mayfirst.org/ross

Also note I extended the /var logical volume on albizu to 10G.

comment:5 Changed 6 years ago by https://id.mayfirst.org/pixelspaceny

So I now have info@, and tyler@ forwards to it? But won't all that backsplatter just forward to info@ now?

comment:6 Changed 6 years ago by https://id.mayfirst.org/ross

Hi tyler,

No tyler@ has been disabled, but the tyler user account is still active. So emails delivered to tyler@ still should fail, only the user account is active, i.e. no forwarding.

comment:7 Changed 6 years ago by https://id.mayfirst.org/ross

Tyler,

Could you also make sure to change your password on the tyler account, just for safety's sake. You can do that here:

https://members.mayfirst.org/cp/index.php?area=hosting_order&service_id=1&hosting_order_id=745&action=edit&item_id=16116

~/ross

comment:8 Changed 6 years ago by https://id.mayfirst.org/jamie

Ross and I just confirmed - the tyler user account looks to be compromised - it relayed nearly 200,000 messages yesterday. I just disabled it again - you can re-enable it yourself, but please change the password after you re-enable it!!

thanks, jamie

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.