#12572 closed Bug/Something is broken (fixed)

DNS records deleted

Reported by: https://id.mayfirst.org/jaimev Owned by:
Priority: Medium Component: Tech
Keywords: dns, rancierre Cc: https://id.mayfirst.org/jamie
Sensitive: no

Description

After receiving several tickets and e-mails from members who were unable to access their mail on gaspar I investigated and found that the domain was not resolving because DNS records no longer existed for gaspar.mayfirst.org

I was able to track down the deleted records and the time they were modified.

mysql> select red_item_dns.item_id,red_item_dns.dns_fqdn,red_item_dns.dns_type,red_item_dns.dns_ip,red_item_dns.dns_server_name FROM red_item_dns INNER JOIN red_item ON red_item_dns.item_id=red_item.item_id  WHERE dns_fqdn LIKE '%gaspar.mayfirst.org%' AND item_status='deleted';
+---------+---------------------+----------+----------------+-----------------+
| item_id | dns_fqdn            | dns_type | dns_ip         | dns_server_name |
+---------+---------------------+----------+----------------+-----------------+
|   78538 | gaspar.mayfirst.org | a        | 162.247.75.129 |                 |
|  159812 | gaspar.mayfirst.org | ptr      | 162.247.75.129 |                 |
+---------+---------------------+----------+----------------+-----------------+

mysql> select * from red_item WHERE item_id IN (78538,159812);
+---------+------------------+------------+-----------------------+-------------+---------------------+
| item_id | hosting_order_id | service_id | item_host             | item_status | item_modified       |
+---------+------------------+------------+-----------------------+-------------+---------------------+
|   78538 |          1000652 |          9 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:33 |
|  159812 |          1000652 |          9 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:34 |
+---------+------------------+------------+-----------------------+-------------+---------------------+

Based on the modification time this corresponds to changes I myself was making to DNS records for domains that previously pointed to mx1.mayfirst.org on Friday afternoon. I first assumed I'd made a mistake however looking through the history on hay I saw that while I did explicitly delete by item_id some dns records I did not make any modifications to the above item_ids nor did I use any expressions that could have inadvertently included those item_ids.

I searched for the details of other items modified at the same time as these two ids and found several DNS records deleted from ranciere, none of which correspond to the item_ids I intentionally manipulated on Friday.

mysql> select red_item.item_id, red_item.item_host,red_item.item_status,red_item.item_modified,red_item_dns.dns_fqdn,red_item_dns.dns_type,red_item_dns.dns_ip,red_item_dns.dns_server_name FROM red_item INNER JOIN red_item_dns ON red_item.item_id=red_item_dns.item_id WHERE item_modified LIKE '2017-03-10 16:0%';
+---------+-----------------------+-------------+---------------------+------------------------------+----------+----------------+-------------------------+
| item_id | item_host             | item_status | item_modified       | dns_fqdn                     | dns_type | dns_ip         | dns_server_name         |
+---------+-----------------------+-------------+---------------------+------------------------------+----------+----------------+-------------------------+
|   39939 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:32 | 100reporters.com             | a        | 216.70.70.100  |                         |
|   39940 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:33 | www.100reporters.com         | a        | 162.247.75.106 |                         |
|   39941 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:33 | mail.100reporters.com        | a        | 162.247.75.106 |                         |
|   39942 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:34 | 100reportersweb.mayfirst.org | a        | 162.247.75.106 |                         |
|   39943 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:34 | 100reporters.com             | mx       |                | julia.mayfirst.org      |
|   62663 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:58 | dev.100reporters.com         | a        | 162.247.75.106 |                         |
|   62664 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:58 | www.dev.100reporters.com     | a        | 162.247.75.106 |                         |
|   62665 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:58 | mail.dev.100reporters.com    | a        | 162.247.75.106 |                         |
|   62666 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:58 | devohrep.mayfirst.org        | a        | 162.247.75.106 |                         |
|   62667 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:58 | dev.100reporters.com         | mx       |                | julia.mayfirst.org      |
|   66775 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:30 | 100r.org                     | a        | 162.247.75.106 |                         |
|   66776 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:30 | 100R.org                     | a        | 216.70.70.100  |                         |
|   66777 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:31 | www.100r.org                 | a        | 216.70.70.100  |                         |
|   66778 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:31 | mail.100r.org                | a        | 162.247.75.106 |                         |
|   66827 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:31 | 100r.org                     | mx       |                | julia.mayfirst.org      |
|   67351 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:31 | 100reporters.com             | mx       |                | ASPMX.L.GOOGLE.COM      |
|   67352 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:32 | 100reporters.com             | mx       |                | ALT1.ASPMX.L.GOOGLE.COM |
|   67353 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:32 | 100reporters.com             | mx       |                | ALT2.ASPMX.L.GOOGLE.COM |
|   67354 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:32 | 100reporters.com             | mx       |                | ASPMX2.GOOGLEMAIL.COM   |
|   67355 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:32 | 100reporters.com             | mx       |                | ASPMX2.GOOGLEMAIL.COM   |
|   67990 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:33 | 100reporters.com             | text     |                |                         |
|   78538 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:33 | gaspar.mayfirst.org          | a        | 162.247.75.129 |                         |
|  159755 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:34 | 100r.org                     | ptr      | 162.247.75.106 |                         |
|  159812 | ranciere.mayfirst.org | deleted     | 2017-03-10 16:07:34 | gaspar.mayfirst.org          | ptr      | 162.247.75.129 |                         |
+---------+-----------------------+-------------+---------------------+------------------------------+----------+----------------+-------------------------+

At this point my only guess is running red-node-update on ranciere somehow triggered a list of pending deletions that had not been previously processed. I am not sure if or how that is possible though.

Change History (1)

comment:1 Changed 17 months ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from new to closed

Thanks Jaime for tracking this down over the weekend and fixing it.

I don't have an explanation as to the root cause of the problem, but I do have enough information to trace it to a single human error.

Friday afternoon, I removed the 100 Reporters membership (after receiving notification that they no longer wished to be members and have removed all of their data).

So, that was me that triggered all of those deletions at 2017-03-10 16:07:34.

The problem is that the gaspar DNS record was coded to their hosting order. That was the ultimate mistake that led to this problem. It should have been coded to the mayfirst.org hosting order (as it is now that you have properly re-created it).

So.. not sure how or why it was incorrectly added, but at least that points to a single human error rather than a more systemic DNS problem

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.