Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#702 closed Task/To do item (fixed)

Transfer viewsic to fred

Reported by: Jamie McClelland Owned by: Jamie McClelland
Priority: Urgent Component: Tech
Keywords: viewsic.mayfirst.org fred.mayfirst.org electricity transfer-viewsic-2008-05 Cc:
Sensitive: no

Description

Now that fred is setup we need to transfer viewsic to fred to save on energy and resolve the ongoing hard disk shortage problem.

Change History (15)

comment:1 Changed 11 years ago by Jamie McClelland

Keywords: viewsic.mayfirst.org fred.mayfirst.org added
Type: Bug/Something is brokenTask/To do item

comment:2 Changed 11 years ago by Jamie McClelland

This is also an important move to cut down on electricity costs.

comment:3 Changed 11 years ago by Daniel Kahn Gillmor

Keywords: electricity added

What's the timeline for this transition? What is the proposed sequence of steps?

I would imagine it being something like (this is grossly oversimplified):

  • build a new domU on fred, temporarily named, say, newsic ;)
  • bring the package list up-to-date (matching viewsic) with something like:
    0 newsic:~# aptitude install $(ssh viewsic dpkg --get-selections | grep '[[:space:]]install$' | cut -f1 -d\  )
    
  • rsync over /etc/ and /home/
  • shutdown outside-facing services on viewsic
  • dump the mysql databases, and load them into the mysql db on viewsic
  • rsync /etc/ and /home/ again
  • turn off viewsic
  • adopt viewsic's IP address in newsic, and change the hostname.

We could practice everything up to the last step (without shutting down services on viewsic) and make notes, and then do the actual transition during a typical maintenance window (wee hours, weekend).

An advisory about the downtime should be published as soon as we schedule the actual transition.

comment:4 Changed 11 years ago by Jamie McClelland

That seems reasonable. A couple points and modifications:

  • Viewsic isn't running LVM and we ran out of room in /var so /var/lib/mysql is a link to /home/mysql. We can fix this on newsic by creating a separate /var/lib/mysql partition. So - our rsync scripts should be set with --exclude /home/mysql to avoid a needless copy.
  • I think we should rsync /usr/local as well.
  • I already allocated a new IP for temp.viewsic.mayfirst.org

As for time line - if you have time to start this it would be great. Since I'm going out of town this week - I probably won't have time to really work on it before Sunday. I'd propose that we shoot for a transition on Sunday April 6. And we announce it after we have down a practice (to make sure we don't run into unexpected trouble which would require us to push back the transition).

comment:5 Changed 11 years ago by Daniel Kahn Gillmor

I'm starting the creation of the new domU on fred now. it is named tempviewsic.

comment:6 Changed 11 years ago by Daniel Kahn Gillmor

This seems like an opportunity to filter out unneeded packages from viewsic as well. For example, viewsic contains turba, horde2, imp3, gnu-efi, and mediawiki1.5 packages, which don't appear to be in etch at all.

Do we really want to try to transfer them?

Also, there are dozens of libraries and X11 components that we might not actually need. Wouldn't it be better to get rid of them at this point rather than transferring them and then trying to clean up later?

I've started by transferring just the debconf selections at least, though:

ssh viewsic debconf-get-selections | ssh root@temp.viewsic.mayfirst.org debconf-set-selections

comment:7 Changed 11 years ago by Jamie McClelland

Yes - that sounds fine to me. Getting rid of turba, horde2, imp3, gnu-efi and mediawiki5 is great. And ditching any orphaned libraries is also great. Thanks Daniel!

comment:8 Changed 11 years ago by Daniel Kahn Gillmor

Are we sure that no members are relying on these packages? why were they still on viewsic in the first place?

comment:9 Changed 11 years ago by Jamie McClelland

I'm not sure why gnu-efi is in there at all. It's pretty specific. turba, horde2, imp3 are all part of the time when we ran these applications on each shared server. They should have been removed after we switched to stallman and felt confident about the move. mediawiki5 - as far as I know - was installed to power http://wiki.mayfirst.org/ which I think should be retired since https://support.mayfirst.org/ is now available.

comment:10 Changed 11 years ago by Jamie McClelland

I spent a bit of time on setting up the new temp.viewsic today. I completed the following:

  • Added viewsic-home and viewsic-var.lib.mysql logical volumes. Currently viewsic has home directory of 100 GB. On the new temp.viewsic, I allocated 150 GB. In addition, the viewsic-var.lib.mysql is dedicated just to the MySQL database. Currently, viewsic is using 3.8 GB. On the new temp.viewsic I allocated 6 GB.
  • I rebooted temp.viewsic so the drives would be available and discovered network weirdness (eth0 device does not exist).
  • I discovered that it was caused by udev (used for auto detecting hardware devices) - which was probably installed automatically when dkg exported and imported packages from the regular viewsic. Since temp.viewsic is a virtual server, it does not need udev.
  • I purged udev (and initramfs-tools)
  • I also noticed that proftpd was there (a relic of days past). I purged it as well.
  • I created filesystems for the new logical volumes and added them to /etc/fstab so they will be mounted automatically (I also tested them /etc/fstab entries)

comment:11 Changed 11 years ago by Jamie McClelland

I just wrote a service advisory to move viewsic sunday night/monday morning.

I'm doing a dump now of all files and databases (it will probably run most of tonight).

It's a lot of data! It takes about a half hour just to generate an rsync file list. If the rsync runs all night, I think it should take about an hour to re-run it tomorrow.

The database is another issue though - since it can't by rsync'ed - it can just be dumped. I'm going to dump all databases tonight as well. Then, I plan to write a script that will only re-dump database tables that have changed in the last two days (since I imagine there are a lot of database tables without changes, so we can save time by not redumping a table without changes).

comment:12 Changed 11 years ago by Jamie McClelland

Keywords: viewsic-transfer added
Resolution: fixed
Status: newclosed

Done!

I also added a new directory of scripts to help the next time we do this.

Everything went relatively smoothly. I accidentally tried to boot the new viewsic with the old viewsic's fstab. That didn't work.

In addition - djbdns and daemontools were not installed.

Otherwise, it went pretty smoothly (so far at least - I expect a few tickets this week).

comment:13 Changed 11 years ago by Jamie McClelland

Keywords: viewsic-transfer-2008-05 added; viewsic-transfer removed

comment:14 Changed 11 years ago by alfredo

Great job Jamie! Is the new viewsic accessible with the same passwords, etc. as the old? In short, has everything come over just as it was?

Also, since Viewsic hosts the sites of some of our "oldest" members, do you think there are any areas in which they'll have problems -- sftp, etc.?

comment:15 Changed 11 years ago by Jamie McClelland

Keywords: transfer-viewsic-2008-05 added; viewsic-transfer-2008-05 removed

Everything, in theory, should be the same. Alas - there's been at least one hiccup. I ve added a keyword to track related problems.

Also - I didn't add users to the wheel group. So, that means you cannot use sudo to gain root privileges. Instead, you can ssh directly into the machine as root:

ssh root@viewsic.mayfirst.org

Using your ssh key.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.