clone and pull all of your company's repositories in a single command

20 Mar 2017

I recently started to work for a new company ( Globality ).

Since we have a very complex setup, it is spread across a large number of
repositories.

During my first days, I wanted to get all of the repositories to my computer.

This might bring some ugly memories, however it is purely for good.

One of my first tasks was to make our docker build faster and I wanted to get a
list of all of our services that are based on python in order to start with
them.

I asked one of my co-workers if he has a script to do that. Sure enough he had
a make file that I tweaked a bit.

Here it is

all:
force:

token=<your-github-token>
user=<your-github-username>
org=<your-github-organization>

auth_opt=-u $(user):$(token)

# note: not all repos are in there!  hitting the page limit of 30.
x.repos.json:  force
                curl -Ks $(auth_opt) https://api.github.com/orgs/$(org)/repos?per_page=100 \
                > $@

clone-all: x.repos.json
          cat x.repos.json \
          | jq -M -r '.[]|.ssh_url' \
          | while read url; do (test -e $$(echo $$url | sed 's|.*/\(.*\)\.git|\1|') || git clone $$url ); done

list-all: x.repos.json
          cat x.repos.json \
          | jq -M -r '.[]|.ssh_url' \
          | while read url; do (echo $$url); done

pull-all:
          for d in ./*; do echo $$d; (cd $$d; git pull); done

Usage

• <your-github-token>: Get a token on github.
• <your-github-username>
• <your-github-org>: Most likely company name for us it’s globality-corp

What I have is a directory called ~/Code/globality, in it’s root I have this
Makefile and I do make clone-all in order to clone all repositories and I can
also do make pull-all to pull all the recent changes across the repositories.