Opened 6 years ago

Closed 6 years ago

Last modified 5 years ago

#7046 closed Task/To do item (fixed)

convert MF/PL svn repository to git repo(s)

Reported by: Daniel Kahn Gillmor Owned by: Jamie McClelland
Priority: High Component: Tech
Keywords: git svn infrastructure f2f svn-to-git Cc:
Sensitive: no


#7035 documents that we now have a functional way to use git repositories here on SMO, and they can be used in parallel with svn.

The main MF/PL SVN repository is quite large, and covers several parts. I would like to see that start to get broken out into individual project-based git repositories.

For example, i would like to see the red code converted entirely to red to ease collaboration. When that conversion is complete, the directory could be removed from the svn trunk (it would still remain in older versions of svn, of course).

If we do such a conversion for each project in the MF/PL red, we could easily tell what remains to be converted.

Change History (26)

comment:1 Changed 6 years ago by Ross

Keywords: f2f added
Owner: set to Daniel Kahn Gillmor
Status: newassigned

I've successfully made a git svn copy of red using the following command:

git svn clone -A svn-authors-for-git --no-metadata --no-minimize-url

The svn-authors-for-git file looks like this:

bdk = Ben Dean-Kawamura <ben@…> dkg = Daniel Kahn Gillmor <dkg@…> jamie = Jamie McClelland <jm@…> joseph = Joseph Lacey <joseph@…> nat = Nat Meysenburg <nat@…> ross = Ross Glover <ross@…> ajl = Alfredo Lopez <alfredo@…> jsm = Jamie McClelland <jm@…> josue = Josue Guillen <josue@…> mallory = Mallory Knodel <mallory@…> greg = Greg Lyle <greg@…>

Next I copied the svn ignore file to .gitignore and committed:

git svn show-ignore > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

After running this command I created a bare repository using

git init --bare

And added the bare repo to the git-svn repo and pushed the git-svn repo to the bare repository.

git remote add bare ~/new-bare.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare

I needed to change the bare branch from git-svn to master. From the bare repo I did:

git branch -m git-svn master

Finally, in case there were lingering tags (which I don't think there were since the tags were not copied into the original clone), I did:

git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 |
while read ref
  git tag "$ref" "refs/heads/tags/$ref";
  git branch -D "tags/$ref";

Finally I added the bare repo to another git repository and pulled in the contents. This seems to have worked and given me a full copy of the repository in git format.

Much of this process I borrowed from here

If you see any issues with this process, dkg, let me know. Otherwise, I think this does what we want.


comment:2 Changed 6 years ago by Daniel Kahn Gillmor

ross, this is great work! Is this git repository visible somewhere i can poke around in it?

I'm not sure what the reasoning is for creating the separate bare repo -- what advantage does this give us?

One thing that might be missing is importing the red tags and converting them to git tags. did you try the above process with the -t or --tags option to git-svn ?

comment:3 Changed 6 years ago by Daniel Kahn Gillmor

i just read up on all the options you used. I'm not sure i understand why we'd want --no-metadata and --no-minimize-url -- can you help me understand? It seems to me like having the metadata in the old commit logs is a reasonable way to remember where they came from initially, and --no-minimize-url seems like it only makes a difference for very large repos (we have less than 3K commits, which is quite small)

I just tried a slightly different workflow, using your svn-authors-for-git file (thanks! doing that mapping manually the first time is always a pain!), and after a false start that involved running into the same bug i apparently ran into before, i sorted out this pattern:

git init red
cd red
git svn init --trunk trunk/red --tags tags/red
git svn fetch --authors-file=../svn-authors-for-git

where that last step is this thing i just hacked up:


# convert-git-svn-tags
# Author: Daniel Kahn Gillmor <>
# derived from John Albin's work at
# and from Ross Glover's work at

PREFIX="${1:-$(basename $(pwd))}"

old_user_name="$(git config --local"
old_user_email="$(git config --local"

git for-each-ref --format='%(refname)' refs/remotes/tags |
cut -d / -f 4 |
while read ref
    if git diff refs/remotes/tags/$ref{,^} --quiet; then
        # this appears to be git-svn failing to place the tag properly, but 
        # i don't care enough to find the right commit to apply it to.
    git config --local "$(git log -n1 refs/remotes/tags/$ref --format=format:'%an')"
    git config --local "$(git log -n1 refs/remotes/tags/$ref --format=format:'%ae')"
    GIT_COMMITTER_DATE="$(git log -n1 refs/remotes/tags/$ref --format=format:'%ai')" \
        git tag -m "$(git log -n1 refs/remotes/tags/$ref --format=format:"%s
%b")" "$PREFIX-$ref" "$targ"
    git branch -D --remotes "tags/$ref";

if [ "$old_user_name" ]; then
    git config --local "$old_user_name"
    git config --local --unset
if [ "$old_user_email" ]; then
    git config --local "$old_user_email"
    git config --local --unset

It seems to yield a repository that also has tags, which we want to keep.

I pushed this to a temporary git repo that you can poke around at here:


let me know what you think!

comment:4 Changed 6 years ago by Daniel Kahn Gillmor

once we settle on a good version of the git repository, we need to think about how to deploy it. i think we have a bunch of code that does updates based on what's in the svn repository, and it would need to now look elsewhere.

comment:5 Changed 6 years ago by Daniel Kahn Gillmor

It also occurs to me that maybe starting with red as the first project to convert is a little on the ambitious side.

Maybe it's worth cutting our teeth on one of the other top-level projects first? How about village or openid-login?

comment:6 Changed 6 years ago by Jamie McClelland

I think working on red is not a bad idea because once you figure out this one, you will have figured out most of the hard parts.

And, I don't think there's much more to do.

The bits to change in puppet should not be that difficult. The file to change is red.pp. The class m_red is the only thing that needs changing I believe:

  exec { "svn co":
    cwd => "/usr/local/share",
    user => root,
    unless => "test -d /usr/local/share/red",
    require => [ Exec["update-ca-certificates-mfpl-crt"], Package["subversion"] ]

  $tag = ""

  exec { "svn switch $tag":
    unless => "svn info | grep 'URL: $tag' > /dev/null",
    cwd => "/usr/local/share/red",
    require => Exec["svn co"]

For hints on how to change that to git, check out the kvm.pp file. Specifically, this:

  $kvm_manager_commit = 'f54717210eb74bc076cd6f76f0593b7ff0be920e'

  m_git::commit{ "/usr/local/share/kvm-manager": 
    remote => 'git://',
    commit => $kvm_manager_commit

So - we will be switching from updating the code based on svn tags to git commits.


comment:7 Changed 6 years ago by Daniel Kahn Gillmor

You can now reference the puppet files directly on SMO, since #7035 was closed. For example: kvm.pp and red.pp

comment:8 Changed 6 years ago by Daniel Kahn Gillmor

Keywords: svn-to-git added

comment:9 Changed 6 years ago by Daniel Kahn Gillmor

take a look at the work joseph and i just did to transition ir from svn to git. it wasn't a trivial transition, but it wasn't terrible either.

Drew's work on #6770 is now using the proposed git tree as a basis also, fwiw.

comment:10 Changed 6 years ago by Daniel Kahn Gillmor

I've just set up the git repo on our internal infrastructure, so you can pull it with:

git clone git://

it doesn't yet contain drew's changes, but we'll get there.

comment:11 Changed 6 years ago by Daniel Kahn Gillmor

I've just tagged and pushed mfpl-puppet-1.93 but it is not being pulled in on most hosts due to some rather silly issues i'm hoping to fix shortly.

comment:12 Changed 6 years ago by Daniel Kahn Gillmor

I've just tagged and pushed mfpl-puppet-1.94 on the same commit as 1.93 in an attempt to work around the silliness.

comment:13 Changed 6 years ago by Ross

I've just tagged and pushed mfpl-puppet-1.96. This release should set red repos to /usr/local/share/red and remove /usr/local/share/red.git and /usr/local/share/red.svn.

comment:14 Changed 6 years ago by Ross


If you could remove red from our svn trunk, I think we're ready to go with red in git. Looks like the changes went through for all servers.


comment:15 Changed 6 years ago by Ross

Well, it worked for everything except for red itself (on hay). I was not aware of the significant number of modifications to hay's version of the red repository. So when I created the red repository on hay, it was just the repository without any of the modifications. This rendered red unusable for a period of time.

I fixed it by restoring from backup and then I made a red.bak directory in '/user/local/share'. I don't think the same thing will happen again based on the current configuration. I've run freepuppet-run and it completes without foobarring red. I think it's safe, though I'll feel better once we eliminate the transition script.

comment:16 Changed 6 years ago by Daniel Kahn Gillmor

r2571 removes the red svn repo.

the remaining source is:

Can anyone take on one of these pieces, create a ticket for it, figure out where it is in use, and figure out a way to get it out of svn and into something more maintainable?

comment:17 Changed 6 years ago by Jamie McClelland

This looks like something I should do... thanks dkg for pushing it along.

comment:18 Changed 6 years ago by Daniel Kahn Gillmor

Owner: changed from Daniel Kahn Gillmor to Jamie McClelland

There was an instance of openid-login on moses that looked like a development instance. i backed it up into moses:/root/tickets/7046/ and moved it out of the way.

I think mayfirst_theme and openid-login both have live instances on hay (in /usr/local/share/openid-login and /usr/local/share/mayfirst_theme). Maybe we can get rid of them or convert them to something more maintainable if we need to keep them?

I still don't know where (or if?) mailman2rss is deployed in mf/pl infrastructure.

If you want a new git repository set up, SVN migrated to it, and/or to have it integrated with SMO here, please let me know the details and i'll set it up.

In the meantime, i'm reassigning this to jamie since it sounds like he's up for this last bit.

comment:19 Changed 6 years ago by Jamie McClelland

Owner: changed from Jamie McClelland to Daniel Kahn Gillmor

Thanks dkg for keeping on this and for killing hte openid-login instance on moses.

I just copied the various parts of mayfirst_theme used by other repos directly into those repos and then I deleted /usr/local/share/mayfirst_theme. You can now delete that svn repository. One down!

Would you mind making a git repo for openid-login? login depends on it.

And the same with mailman2rss (which is documented).

comment:20 Changed 6 years ago by Daniel Kahn Gillmor

Priority: MediumHigh

I've just removed the mayfirst_theme svn subdir (they're not separate repositories).

after i get some shut-eye, i'll see if i can set up git repos for both openid-login and mailman2rss.

comment:21 Changed 6 years ago by Daniel Kahn Gillmor

Owner: changed from Daniel Kahn Gillmor to Jamie McClelland

OK, we now have git repos for both openid-login and mailman2rss, with svn history and tags imported (there were no tags on mailman2rss).

Jamie, i'm turning this back over to you to clean up anything that might have been relying on the svn repos; feel free to turn it back over to me once you're confident they're no longer in use.

comment:22 Changed 6 years ago by Jamie McClelland

Thanks dkg.

Are these repos supposed to be publicly available? I'm having trouble checking them out:

0 jamie@animal:cdtemp.uRCTJx$ git clone git://
Cloning into 'mailman2rss'...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
128 jamie@animal:cdtemp.uRCTJx$

I can check them out using

comment:23 Changed 6 years ago by Daniel Kahn Gillmor

Thanks for the heads-up, i had missed the touch git-daemon-export-ok step for these repos. i've done that now, they should be fetchable anonymously.

comment:24 Changed 6 years ago by Jamie McClelland

Resolution: fixed
Status: assignedclosed

Finished at last! I've just converted those two repos from svn to git where they live on our live servers.

comment:25 Changed 6 years ago by Daniel Kahn Gillmor

Hm, the directories in the svn repo still exists, though.

I've just removed the directories in the svn repo. should we remove source:trunk and source:tags as well?

comment:26 Changed 5 years ago by Daniel Kahn Gillmor

I got no answer, so i'm going to go ahead and remove them.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.