Opened 4 years ago

Closed 3 years ago

#10825 closed Question/How do I...? (fixed)

network problems when rebooting rodolpho

Reported by: JaimeV Owned by: Jamie McClelland
Priority: Low Component: Tech
Keywords: Cc: Daniel Kahn Gillmor
Sensitive: no

Description

After extending the disk for rodolpho in clr and rebooting rodolpho i was not able to access the vm again over the network. I stopped and started the runit process for rodolpho again and was was able to watch the output of rodolpho's console like this.

tailf /home/rodolpho/vms/rodolpho/console | cat -v

I noticed immediately the network failure.

[....] Configuring network interfaces...RTNETLINK answers: Network is down
RTNETLINK answers: Network is down
RTNETLINK answers: Network is down
RTNETLINK answers: Network is down
RTNETLINK answers: Network is down
RTNETLINK answers: Network is down

Searching I found ticket:9649 and ticket:7293 that documented a possible cause and solution editing /etc/network/interfaces.

Unfortunately I wasn't sure how to login vía rodolpho's console. This has problem has come up for me before and I think this is what I really want help with.

I finally got around this by mounting rodolpho's fileysytem to a temporary mount point in clr vía the following steps

kpartx -av /dev/mapper/vg_clr0-rodolpho
vgchange -ay
mount /dev/mapper/vg_rodolpho0-root /mnt/rodolpho

At this point i was able to access /mnt/rodolpho/etc/network/interfaces and replace "allow-hotplug eth0" with "auto eth0". Then I unmounted the filesystem again.

umount /mnt/rodolpho
vgchange -an vg_rodolpho0
kpartx -dv /dev/mapper/vg_clr0-rodolpho

I started rodolpho again and the network interface came up as expected. I'm documenting here in case I run into his again. Mounting the filesystem is a workaround but I'd really like to know to access a VM's console terminal.

Change History (7)

comment:1 Changed 4 years ago by JaimeV

Owner: set to Jamie McClelland
Status: newassigned

comment:2 Changed 4 years ago by Jamie McClelland

Wow - nice work on the fly Jaime under pressure to get rodolpho back up again.

I'm really sorry you had to deal with that stress due to a lack of documentation. I'm adding the documentation right now to this ticket so it doesn't happen again:

Adding access to the console of a virtual guest:

  • Login as root to the host server, e.g. ssh root@clr.mayfirst.org
  • Add your OpenPGP User Id to the authorized_user_ids file for the user account used by the virtual guest, e.g. echo 'Jamie McClelland <jamie@mayfirst.org>' >> /home/rodolpho/.monkeysphere/authorized_user_ids
  • Update the keys for the user, e.g.: monkeysphere-authentication update-users rodolpho
  • Login from your computer: ssh rodolpho@clr.mayfirst.org
  • Join the screen session: screen -x rodolpho

comment:3 Changed 4 years ago by Jamie McClelland

I think we should check all /etc/networking/interfaces files to ensure it has auto eth0 right?

comment:4 Changed 4 years ago by JaimeV

Well it seems that 56 out of 65 of the mosh's use allow-hotplug eth0 in their /etc/network/interfaces config. I would guess most of these guests have been rebooted without problems on different occasions. So I am curious why has it only been a problem on a few mosh's ? Does that matter? Should we just change them all anyway?

comment:5 Changed 4 years ago by Jamie McClelland

What's the default on new installs? I would be inclined to standardize on whatever is being installed by default. Maybe floriberto is the best test case as the most recent guest. Although, it might be different with Debian Jessie.

comment:6 Changed 4 years ago by JaimeV

Cc: Daniel Kahn Gillmor added

As it turns out floriberto, which is the newest shared mosh to be installed with wheezy uses "allow-hotplug eth0" as well. I don't see any reference to this in our puppet manifests so I'm pretty sure this is the default config for a standard debian install. In my experiments with vm's in my own personal machine this also appears to be the default for jessie.

So... what happened to rodolpho and the other mosh's that had problems with this? One thing they seem to have in common is that they all have multiple interfaces defined. Maybe that's a clue?

I'm ccing dkg as he came up with the "auto eth0" fix originally.

comment:7 Changed 3 years ago by JaimeV

Resolution: fixed
Status: assignedclosed

We've since made a point of replacing allow-hotplug eth0 with auto eth0 on all interfaces.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.