Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#1029 closed Bug/Something is broken (fixed)

mail from malcolm bouncing because of missing reverse dns record

Reported by: Jamie McClelland Owned by: Jamie McClelland
Priority: Urgent Component: Tech
Keywords: malcolm.mayfirst.org routing Cc:
Sensitive: no

Description

Linda alerted us to the problem.

In looking through malcolm's email logs, I'm seeing a lot of entries like:

May 23 07:37:10 malcolm postfix/smtp[376]: 66AF48DE50: host mx.speakeasy.net[69.17.117.60] said: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [209.5
1.163.206] (in reply to RCPT TO command)

When I first discovered this problem, I saw that 209.51.163.206 does not have a reverse DNS entry. That's a common form of spam control with mail server. I just added one for it. That IP address belongs to IBEW Local 90.

The strange thing is that malcolm's canonical IP address is not 209.51.163.206. Instead, it's 209.51.163.199. And, Malcolm's routing tables seem properly setup to indicate it's canonical IP address as the source address:

0 jm@malcolm:temp$ ip route
209.51.163.192/28 dev eth0  proto kernel  scope link  src 209.51.163.199 
209.51.180.16/28 dev eth0  scope link  src 209.51.163.199 
209.51.172.0/28 dev eth0  scope link  src 209.51.163.199 
209.51.169.80/28 dev eth0  scope link  src 209.51.163.199 
209.51.163.0/27 dev eth0  scope link  src 209.51.163.199 
default via 209.51.163.193 dev eth0 
0 jm@malcolm:temp$

Change History (5)

comment:1 Changed 11 years ago by Jamie McClelland

Resolution: fixed
Status: newclosed

This is now fixed.

I ended up bouncing the network connection a few times and even in a moment of desperation I rebooted malcolm :(.

I'm not sure what caused the problem, however, here are some of the experiences I had:

  • I ran telnet chun.mayfirst.org 25 from malcolm, and then tailed chun's mail log. Even when the ip routing table as displayed by ip route showed the src address as 209.51.163.199, chun still reported the connection as coming from 209.51.163.206.
  • malcolm has a lot of alias IP address on eth0 (in the form eth0:ibew90, etc.). When I ran ifdown eth0; ifup eth0, the ip command reported the src address for the default route to be 209.51.163.207 (the IP for eth0:ibew90), rather than 209.51.163.199 (the IP address for eth0).
  • I could remove that route and re-create it with the src 209.51.163.199, however, the chun test still reported 209.51.163.206.
  • I finally solved the problem by running ifdown eth0, then commenting out the ibew90 stanza, running ifup eth0 (proper src address is reported), uncommenting ibew90, ifup eth0:ibew90.
  • Now, running ifdown eth0; ifup eth0 brings up the proper src address.

comment:2 Changed 11 years ago by Daniel Kahn Gillmor

Keywords: malcolm.mayfirst.org routing added

Maybe i'm misunderstanding the above description, but if ifdown eth0; ifup eth0 could cause it to come back in the wrong state, this doesn't sound really fixed to me. Can you explain more?

comment:3 Changed 11 years ago by Jamie McClelland

Well... the problem described in the summary of the ticket is fixed, however, the mystery as to why it happened in the first place (and the fact that it could happen again) is not resolved.

I don't understand how the kernel (or ifup ?) chooses the src address when there are multiple aliased addresses listed in /etc/network/interfaces.

comment:4 Changed 11 years ago by Jamie McClelland

This problem has come back (#1441). I'm working on changing our script for adding our routes.

Our current script doesn't add routes that are added already.

The problem seems to be that if a route is included in the /etc/network/interfaces file it will get added automatically to the routing table, but not with the canonical IP address of the server as the src IP.

For example, here is our current routing table:

0 malcolm:/etc/network/if-up.d# ip route
209.51.163.192/28 dev eth0  proto kernel  scope link  src 209.51.163.206 
209.51.180.16/28 dev eth0  proto kernel  scope link  src 209.51.180.26 
209.51.172.0/28 dev eth0  scope link  src 209.51.163.199 
209.51.169.80/28 dev eth0  scope link  src 209.51.163.199 
209.51.171.160/27 dev eth0  proto kernel  scope link  src 209.51.171.162 
209.51.163.0/27 dev eth0  proto kernel  scope link  src 209.51.163.3 
default via 209.51.163.193 dev eth0 
0 malcolm:/etc/network/if-up.d#

The two entries added properly with the canonical source IP address are the only two that are not represented in the /etc/network/interfaces file.

I think the answer is to re-write script for adding our routes to simply delete all existing routes and recreate them all with the proper canonical IP address.

comment:5 Changed 11 years ago by Jamie McClelland

I just commited changeset:816 to resolve this issue and I'm now copying the new script to all our telehouse servers.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.