Opened 6 years ago

Closed 5 years ago

Last modified 5 years ago

#7046 closed Task/To do item (fixed)

convert MF/PL svn repository to git repo(s)

Reported by: https://id.mayfirst.org/dkg Owned by: https://id.mayfirst.org/jamie
Priority: High Component: Tech
Keywords: git svn infrastructure f2f svn-to-git Cc:
Sensitive: no

Description

#7035 documents that we now have a functional way to use git repositories here on SMO, and they can be used in parallel with svn.

The main MF/PL SVN repository is quite large, and covers several parts. I would like to see that start to get broken out into individual project-based git repositories.

For example, i would like to see the red code converted entirely to red to ease collaboration. When that conversion is complete, the directory could be removed from the svn trunk (it would still remain in older versions of svn, of course).

If we do such a conversion for each project in the MF/PL red, we could easily tell what remains to be converted.

Change History (26)

comment:1 Changed 6 years ago by https://id.mayfirst.org/ross

  • Keywords f2f added
  • Owner set to https://id.mayfirst.org/dkg
  • Status changed from new to assigned

I've successfully made a git svn copy of red using the following command:

git svn clone https://svn.mayfirst.org/mfpl/trunk/red -A svn-authors-for-git --no-metadata --no-minimize-url

The svn-authors-for-git file looks like this:

bdk = Ben Dean-Kawamura <ben@…> dkg = Daniel Kahn Gillmor <dkg@…> jamie = Jamie McClelland <jm@…> joseph = Joseph Lacey <joseph@…> nat = Nat Meysenburg <nat@…> ross = Ross Glover <ross@…> ajl = Alfredo Lopez <alfredo@…> jsm = Jamie McClelland <jm@…> josue = Josue Guillen <josue@…> mallory = Mallory Knodel <mallory@…> greg = Greg Lyle <greg@…>

Next I copied the svn ignore file to .gitignore and committed:

git svn show-ignore > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

After running this command I created a bare repository using

git init --bare

And added the bare repo to the git-svn repo and pushed the git-svn repo to the bare repository.

git remote add bare ~/new-bare.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare

I needed to change the bare branch from git-svn to master. From the bare repo I did:

git branch -m git-svn master

Finally, in case there were lingering tags (which I don't think there were since the tags were not copied into the original clone), I did:

git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 |
while read ref
do
  git tag "$ref" "refs/heads/tags/$ref";
  git branch -D "tags/$ref";
done

Finally I added the bare repo to another git repository and pulled in the contents. This seems to have worked and given me a full copy of the repository in git format.

Much of this process I borrowed from here http://john.albin.net/git/convert-subversion-to-git

If you see any issues with this process, dkg, let me know. Otherwise, I think this does what we want.

~/ross

comment:2 Changed 6 years ago by https://id.mayfirst.org/dkg

ross, this is great work! Is this git repository visible somewhere i can poke around in it?

I'm not sure what the reasoning is for creating the separate bare repo -- what advantage does this give us?

One thing that might be missing is importing the red tags and converting them to git tags. did you try the above process with the -t or --tags option to git-svn ?

comment:3 Changed 6 years ago by https://id.mayfirst.org/dkg

i just read up on all the options you used. I'm not sure i understand why we'd want --no-metadata and --no-minimize-url -- can you help me understand? It seems to me like having the metadata in the old commit logs is a reasonable way to remember where they came from initially, and --no-minimize-url seems like it only makes a difference for very large repos (we have less than 3K commits, which is quite small)

I just tried a slightly different workflow, using your svn-authors-for-git file (thanks! doing that mapping manually the first time is always a pain!), and after a false start that involved running into the same bug i apparently ran into before, i sorted out this pattern:

git init red
cd red
git svn init https://svn.mayfirst.org/mfpl --trunk trunk/red --tags tags/red
git svn fetch --authors-file=../svn-authors-for-git
../convert-git-svn-tags

where that last step is this thing i just hacked up:

#!/bin/bash

# convert-git-svn-tags
# Author: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
# derived from John Albin's work at http://john.albin.net/git/convert-subversion-to-git
# and from Ross Glover's work at https://support.mayfirst.org/ticket/7046#comment:1

PREFIX="${1:-$(basename $(pwd))}"

old_user_name="$(git config --local user.name)"
old_user_email="$(git config --local user.email)"

git for-each-ref --format='%(refname)' refs/remotes/tags |
cut -d / -f 4 |
while read ref
do
    if git diff refs/remotes/tags/$ref{,^} --quiet; then
        targ="refs/remotes/tags/$ref^"
    else
        # this appears to be git-svn failing to place the tag properly, but 
        # i don't care enough to find the right commit to apply it to.
        targ="refs/remotes/tags/$ref"
    fi
    git config --local user.name "$(git log -n1 refs/remotes/tags/$ref --format=format:'%an')"
    git config --local user.email "$(git log -n1 refs/remotes/tags/$ref --format=format:'%ae')"
    GIT_COMMITTER_DATE="$(git log -n1 refs/remotes/tags/$ref --format=format:'%ai')" \
        git tag -m "$(git log -n1 refs/remotes/tags/$ref --format=format:"%s
%b")" "$PREFIX-$ref" "$targ"
    git branch -D --remotes "tags/$ref";
done

if [ "$old_user_name" ]; then
    git config --local user.name "$old_user_name"
else
    git config --local --unset user.name
fi
if [ "$old_user_email" ]; then
    git config --local user.email "$old_user_email"
else
    git config --local --unset user.email
fi

It seems to yield a repository that also has tags, which we want to keep.

I pushed this to a temporary git repo that you can poke around at here:

git://lair.fifthhorseman.net/~dkg/red

let me know what you think!

comment:4 Changed 6 years ago by https://id.mayfirst.org/dkg

once we settle on a good version of the git repository, we need to think about how to deploy it. i think we have a bunch of code that does updates based on what's in the svn repository, and it would need to now look elsewhere.

comment:5 Changed 6 years ago by https://id.mayfirst.org/dkg

It also occurs to me that maybe starting with red as the first project to convert is a little on the ambitious side.

Maybe it's worth cutting our teeth on one of the other top-level projects first? How about village or openid-login?

comment:6 Changed 6 years ago by https://id.mayfirst.org/jamie

I think working on red is not a bad idea because once you figure out this one, you will have figured out most of the hard parts.

And, I don't think there's much more to do.

The bits to change in puppet should not be that difficult. The file to change is red.pp. The class m_red is the only thing that needs changing I believe:

  exec { "svn co https://svn.mayfirst.org/mfpl/trunk/red":
    cwd => "/usr/local/share",
    user => root,
    unless => "test -d /usr/local/share/red",
    require => [ Exec["update-ca-certificates-mfpl-crt"], Package["subversion"] ]
  }

  $tag = "https://svn.mayfirst.org/mfpl/tags/red/269"

  exec { "svn switch $tag":
    unless => "svn info | grep 'URL: $tag' > /dev/null",
    cwd => "/usr/local/share/red",
    require => Exec["svn co https://svn.mayfirst.org/mfpl/trunk/red"]
  }

For hints on how to change that to git, check out the kvm.pp file. Specifically, this:

  $kvm_manager_commit = 'f54717210eb74bc076cd6f76f0593b7ff0be920e'

  m_git::commit{ "/usr/local/share/kvm-manager": 
    remote => 'git://lair.fifthhorseman.net/~dkg/kvm-manager',
    commit => $kvm_manager_commit
  }

So - we will be switching from updating the code based on svn tags to git commits.

jamie

comment:7 Changed 6 years ago by https://id.mayfirst.org/dkg

You can now reference the puppet files directly on SMO, since #7035 was closed. For example: kvm.pp and red.pp

comment:8 Changed 6 years ago by https://id.mayfirst.org/dkg

  • Keywords svn-to-git added

comment:9 Changed 6 years ago by https://id.mayfirst.org/dkg

take a look at the work joseph and i just did to transition ir from svn to git. it wasn't a trivial transition, but it wasn't terrible either.

Drew's work on #6770 is now using the proposed git tree as a basis also, fwiw.

comment:10 Changed 6 years ago by https://id.mayfirst.org/dkg

I've just set up the git repo on our internal infrastructure, so you can pull it with:

git clone git://git.mayfirst.org/mfpl/red

it doesn't yet contain drew's changes, but we'll get there.

comment:11 Changed 6 years ago by https://id.mayfirst.org/dkg

I've just tagged and pushed mfpl-puppet-1.93 but it is not being pulled in on most hosts due to some rather silly issues i'm hoping to fix shortly.

comment:12 Changed 6 years ago by https://id.mayfirst.org/dkg

I've just tagged and pushed mfpl-puppet-1.94 on the same commit as 1.93 in an attempt to work around the silliness.

comment:13 Changed 6 years ago by https://id.mayfirst.org/ross

I've just tagged and pushed mfpl-puppet-1.96. This release should set red repos to /usr/local/share/red and remove /usr/local/share/red.git and /usr/local/share/red.svn.

comment:14 Changed 6 years ago by https://id.mayfirst.org/ross

dkg,

If you could remove red from our svn trunk, I think we're ready to go with red in git. Looks like the changes went through for all servers.

~/ross

comment:15 Changed 6 years ago by https://id.mayfirst.org/ross

Well, it worked for everything except for red itself (on hay). I was not aware of the significant number of modifications to hay's version of the red repository. So when I created the red repository on hay, it was just the repository without any of the modifications. This rendered red unusable for a period of time.

I fixed it by restoring from backup and then I made a red.bak directory in '/user/local/share'. I don't think the same thing will happen again based on the current configuration. I've run freepuppet-run and it completes without foobarring red. I think it's safe, though I'll feel better once we eliminate the transition script.

comment:16 Changed 5 years ago by https://id.mayfirst.org/dkg

r2571 removes the red svn repo.

the remaining source is:

Can anyone take on one of these pieces, create a ticket for it, figure out where it is in use, and figure out a way to get it out of svn and into something more maintainable?

comment:17 Changed 5 years ago by https://id.mayfirst.org/jamie

This looks like something I should do... thanks dkg for pushing it along.

comment:18 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Owner changed from https://id.mayfirst.org/dkg to https://id.mayfirst.org/jamie

There was an instance of openid-login on moses that looked like a development instance. i backed it up into moses:/root/tickets/7046/ and moved it out of the way.

I think mayfirst_theme and openid-login both have live instances on hay (in /usr/local/share/openid-login and /usr/local/share/mayfirst_theme). Maybe we can get rid of them or convert them to something more maintainable if we need to keep them?

I still don't know where (or if?) mailman2rss is deployed in mf/pl infrastructure.

If you want a new git repository set up, SVN migrated to it, and/or to have it integrated with SMO here, please let me know the details and i'll set it up.

In the meantime, i'm reassigning this to jamie since it sounds like he's up for this last bit.

comment:19 Changed 5 years ago by https://id.mayfirst.org/jamie

  • Owner changed from https://id.mayfirst.org/jamie to https://id.mayfirst.org/dkg

Thanks dkg for keeping on this and for killing hte openid-login instance on moses.

I just copied the various parts of mayfirst_theme used by other repos directly into those repos and then I deleted /usr/local/share/mayfirst_theme. You can now delete that svn repository. One down!

Would you mind making a git repo for openid-login? support.mayfirst.org login depends on it.

And the same with mailman2rss (which is documented).

comment:20 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Priority changed from Medium to High

I've just removed the mayfirst_theme svn subdir (they're not separate repositories).

after i get some shut-eye, i'll see if i can set up git repos for both openid-login and mailman2rss.

comment:21 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Owner changed from https://id.mayfirst.org/dkg to https://id.mayfirst.org/jamie

OK, we now have git repos for both openid-login and mailman2rss, with svn history and tags imported (there were no tags on mailman2rss).

Jamie, i'm turning this back over to you to clean up anything that might have been relying on the svn repos; feel free to turn it back over to me once you're confident they're no longer in use.

comment:22 Changed 5 years ago by https://id.mayfirst.org/jamie

Thanks dkg.

Are these repos supposed to be publicly available? I'm having trouble checking them out:

0 jamie@animal:cdtemp.uRCTJx$ git clone git://git.mayfirst.org/mfpl/mailman2rss
Cloning into 'mailman2rss'...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
128 jamie@animal:cdtemp.uRCTJx$

I can check them out using git@git.mayfirst.org:mfpl/mailman2rss.

comment:23 Changed 5 years ago by https://id.mayfirst.org/dkg

Thanks for the heads-up, i had missed the touch git-daemon-export-ok step for these repos. i've done that now, they should be fetchable anonymously.

comment:24 Changed 5 years ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from assigned to closed

Finished at last! I've just converted those two repos from svn to git where they live on our live servers.

comment:25 Changed 5 years ago by https://id.mayfirst.org/dkg

Hm, the directories in the svn repo still exists, though.

I've just removed the directories in the svn repo. should we remove source:trunk and source:tags as well?

comment:26 Changed 5 years ago by https://id.mayfirst.org/dkg

I got no answer, so i'm going to go ahead and remove them.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.