Git Directory Tree Operations

Published 2021-04-10. Last modified 2025-10-10.
Time to read: 7 minutes.

This page is part of the git collection.

I have several trees of Git repositories, grouped into subdirectories. The total number of repositories is in the hundreds. Here is a sanitized depiction of one of my Git directory trees:

Directory tree

├── cadenzaHome
│ ├── cadenzaAssets
│ ├── cadenzaCode
│ │ ├── cadenzaClient
│ │ ├── cadenzaCourseCode
│ │ ├── cadenzaDependencies
│ │ ├── cadenzaLibs
│ │ ├── cadenzaServer
│ │ ├── cadenzaServerNext
│ │ └── cadenzaSupport
│ ├── cadenzaCreative
│ │ └── cadenzaCreativeTemplates
│ ├── cadenzaCreativeBackup
│ └── cadenzaCurriculum
├── django
│ ├── django
│ ├── django-oscar
│ ├── frobshop
│ ├── main
│ └── oscar
├── jekyll
│ ├── jekyllTemplate
│ └── jekyll-flexible-include-plugin

Some Git repos are forks, and I defined upstream Git remotes for them, in addition to the usual origin remote.

This article discusses git_tree_go, a Go package I wrote with the help of Claude, Grok and Gemini Code Assist to help me work efficiently with multiple trees of Git projects.

This Go package installs commands that walk through one or more git directory trees (breadth-first) and act on each repository. Directories containing a file called .ignore are ignored, as well as all subdirectories. Multiple goroutines are normally used to dramatically boost performance, but serial processing is available for deterministic results.

Command Summary

The git-commitAll command commits and pushes all changes to each repository in the tree. Repositories that are in a detached HEAD state are skipped.
The git-evars command writes a script that defines environment variables pointing to each git repository.
The git-exec command executes an arbitrary shell command for each repository.
The git-list-executables command lists all the executables created by this package.
The git-replicate command writes a script that clones the repositories in the tree, and adds any defined remotes.
- Any git repos that have already been cloned into the target directory tree are skipped. This means you can rerun git-replicate as many times as you want, without ill effects.
- All remotes in each repository are replicated.
The git-update command updates each repository in the trees.

The above command names are subject to change. Feedback is encouraged through issues on the GitHub project page.

Installing git_tree_go

Install the Go language if you need to. You need Go 1.24 or later installed on your system.
Shell
```
$ go version
go version go1.24.2 linux/amd64 
```

To install the command-line programs from releases on the GitHub repository without manually cloning, use go install. The following provides the most recently released version:

Shell

$ go install github.com/mslinn/git_tree_go/cmd/...@latest
go: downloading github.com/mslinn/git_tree_go v0.1.9
go: downloading github.com/MakeNowJust/heredoc v1.0.0
go: downloading github.com/ProtonMail/go-crypto v1.3.0
go: downloading dario.cat/mergo v1.0.2
go: downloading github.com/sergi/go-diff v1.4.0
go: downloading github.com/pjbgf/sha1cd v0.5.0
go: downloading golang.org/x/sys v0.37.0
go: downloading github.com/kevinburke/ssh_config v1.4.0
go: downloading github.com/skeema/knownhosts v1.3.2
go: downloading golang.org/x/crypto v0.43.0
go: downloading golang.org/x/net v0.46.0
go: downloading github.com/cyphar/filepath-securejoin v0.5.0
go: downloading github.com/klauspost/cpuid/v2 v2.3.0

Propellerheads Only

Download, build and install to $HOME/go/bin/ like this:

Shell

$ git clone https://github.com/mslinn/git_tree_go.git

$ cd git_tree_go

$ make install

Configuration

The git_tree_go commands can be configured to suit your preferences. Settings are resolved in the following order of precedence, where items higher in the list override those lower down:

Environment variables
User configuration file (~/.treeconfig.yml)
Default values built into the gem.

This allows for flexible customization of the gem’s behavior.

Environment Variables

For temporary overrides or use in CI/CD environments, you can use environment variables. They must be prefixed with GIT_TREE_ and be in uppercase.

export GIT_TREE_GIT_TIMEOUT=900
export GIT_TREE_VERBOSITY=2
export GIT_TREE_DEFAULT_ROOTS="dev projects personal" (space-separated string)

You can set these and use all of them like this:

Shell

$ GIT_TREE_GIT_TIMEOUT=900 \
GIT_TREE_VERBOSITY=2 \
GIT_TREE_DEFAULT_ROOTS="dev projects personal" \
git evars --zowee

This might be more typical:

Shell

$ GIT_TREE_GIT_TIMEOUT=900 git evars --zowee

Configuration File

The git-treeconfig command generates a YAML file (~/.treeconfig.yml) that you can also edit manually. Here is an example:

~/.treeconfig.yml

git_timeout: 900
verbosity: 1
default_roots:
- projects
- personal

Interactive Setup: git-treeconfig

The easiest way to get started is to use the git-treeconfig command. This interactive tool will ask you a few questions and create a configuration file for you at ~/.treeconfig.yml.

Shell

$ git-treeconfig
Welcome to git-tree configuration.
This utility will help you create a configuration file at: /home/user/.treeconfig.yml
Press Enter to accept the default value in brackets.
Git command timeout in seconds? |300| 600Enter
Default verbosity level (0=quiet, 1=normal, 2=verbose)? |1| Enter
Default root directories (space-separated)? |sites sitesUbuntu work| projects personalEnter

Configuration saved to /home/user/.treeconfig.yml

Use Cases

Dependent Gem Maintenance

One of my directory trees holds Jekyll plugins, packaged as 25 gems. They depend on one another and must be built in a particular order. Sometimes an operation must be performed on all the plugins, and then they must all be rebuilt.

Most operations do not require that the projects be processed in any particular order; however, the build process must be invoked on the dependencies first. It is quite tedious to do this 25 times, over and over.

Several years ago I wrote a Bash script to perform this task, but as its requirements became more complex, the Bash script proved difficult to maintain. This use case is now fulfilled by the git-exec command provided by the git_tree_go gem when used with the --serial option. See below for further details.

Replicating Trees of Git Repositories

Whenever I set up an operating system for a new development computer, one of the tedious tasks that must be performed is to replicate the directory trees of Git repositories.

It is a bad idea to attempt to copy an entire Git repository between computers because the .git directories within them can be huge. So large, in fact, that it might take much more time to copy than re-cloning.

The reason is that copying the entire Git repository actually means copying the same information twice: first the .git hidden directory, complete with all the history for the project, and then again for the files in the currently checked-out branch. Git repos store the entire development history of the project in their .git directories, so as they accumulate history, they eventually become much larger than the code that is checked out at any given time.

This use case is fulfilled by the git-replicate and git-evars commands provided by git_tree_go.

History

One morning I found myself facing the boring task of doing this manually once again. Instead, I wrote a Bash script that scanned a Git directory tree and wrote out another Bash script that clones the repos in the tree. Any additional remote references are replicated.

Two years later, I decided to add new features to the script. Bash is great for short scripts, but it is not conducive to debugging or structured programming. I rewrote the bash script in Ruby, using the rugged gem. Much better!

Two years after that, I used Google Gemini Code Assist, Grok, and Claude Console to rewrite it again in Ruby, this time as a multithreaded program. Performance was lightning-fast for most use cases, but the LLMs were unable to get all the moving parts to move correctly at the same time. Fixing one bug would cause another bug. Fixing the second bug would bring back the first bug. LLMs have difficulty with untyped languages such as Ruby, Python and JavaScript.

Still determined as ever, I used Google Gemini Code Assist and Claude Console to rewrite it yet again in Go. The core logic for the various Git-related scripts I had written over the years has evolved. The result is this git_tree_go package.

Usage

All of these commands are default to multi-processing mode using goroutines. You may notice that your computer's fan gets louder when you run these commands on large numbers of Git repositories.

For builds and other sequential tasks, however, multiprocessing is inappropriate. Instead, it is necessary to build components in the proper order. Doing all the work as a single process is a straightforward way of ensuring proper task ordering.

Use the -s/--serial option when the order that Git projects are processed matters. All of the commands support this option. Execution will take much longer than without the option, because performing most tasks take longer to perform in sequence than performing them via multiprocessing.

git-commitAll

The git-commitAll command commits and pushes all changes to each repository in the tree. Repositories in a detached HEAD state are skipped.

git-commitAll help message

$ git commitAll -h
git-commitAll - Recursively commits and pushes changes in all git repositories
under the specified DIRECTORY roots.
If no directories are given, it uses default environment variables
('sites', 'sitesUbuntu', 'work') as roots.
Skips directories containing a .ignore file.
Repositories in a detached HEAD state are skipped.

Options:
-h, --help Show this help message and exit.
-m, --message MESSAGE Use the given string as the commit message.
(default: "-")
-q, --quiet Suppress normal output, only show errors.
-s, --serial Run tasks serially in a single thread in the order specified.
-v, --verbose Increase verbosity. Can be used multiple times (e.g., -v, -vv).

git-evars

The git-evars command writes a script that defines environment variables pointing to each Git repository. This command should be run on the target computer.

Only one parameter is required: an environment variable reference pointing to the top-level directory to replicate. The environment variable reference must be contained within single quotes to prevent expansion by the shell.

The following appends to any script in the $work directory called .evars. The script defines environment variables that point to each Git repository pointed to by $work:

Shell

$ git-evars '$work' >> $work/.evars

Generated Script

Following is a sample of environment variable definitions. The -z/--zowee option generates intermediate environment variable definitions, making them much easier to work with.

Shell

$ git-evars -z '$sites'
export mnt=/mnt
export c=$mnt/c
export _6of26=$sites/6of26
export computers=$sites/computers.mslinn.com
export ebooks=$sites/ebooks
export expert=$sites/expert
export fonts=$sites/fonts
export intranet=$sites/intranet.ancientwarmth.com
export intranet_mslinn=$sites/intranet.mslinn.com
export jekyllTemplate=$sites/jekyllTemplate
export lyrics=$sites/lyrics
export metamusic=$sites/metamusic
export music=$sites/music.mslinn.com
export photos=$sites/photos
export supportingLiterature=$sites/supportingLiterature
export www=$sites/www.scalacourses.com

The environment variable definitions are meant to be saved into a file that is sourced upon boot. While you could place them in a file like ~/.bashrc, the author’s preference is to instead place them in $work/.evars and add the following to ~/.bashrc:

~/.bashrc snippet

source "$work/.evars"

Thus each time you log in, the environment variable definitions will have been re-established. You can therefore change directory to any of the cloned projects, like this:

Shell

$ cd $git_root

$ cd $my_project

git-exec

The git-exec command can be run on any computer. The command requires two parameters. The first parameter indicates the directory or directories to process. Three forms are accepted:

A directory name, which may be relative or absolute.
An environment variable reference, which must be contained within single quotes to prevent expansion by the shell.
A list of directory names, which may be relative or absolute and may contain environment variables.

Several use cases for git-exec follow.

Count Repositories

Display the number of active repositories (those which do not have a .ignore file), and the length of time it took to run the command.

Shell

$ time git exec '$sites $work $sitesUbuntu' 'echo 1' | \
  awk '{s+=$1} END {print s}'
121

real 0m0.513s
user 0m0.064s
sys 0m0.129s

😎

It took about half a second to scan three Git directory trees containing 121 active Git repositories (and many more ignored Git repositories.)

Update, Build, Install

For all specified Git repositories (which are the source of varies Ruby gems that I publish), update dependencies and install a local copy of the gem. The --serial option ensures that the gems will be built in the specified order.

Shell

$ git-exec --serial \
  '$jekyll_plugin_logger
   $jekyll_draft
   $jekyll_plugin_support
   $jekyll_all_collections
   $jekyll_plugin_template
   $jekyll_flexible_include_plugin
   $jekyll_href
   $jekyll_img
   $jekyll_outline
   $jekyll_plugin_template
   $jekyll_pre
   $jekyll_quote' \
  'bundle && bundle update && rake install'

Display Locations

Display the full path of every Git repository under the given root:

Shell

$ git exec '$my_plugins' pwd
Processing $my_plugins
/mnt/f/work/jekyll/my_plugins/jekyll_archive_create
/mnt/f/work/jekyll/my_plugins/jekyll_archive_display
/mnt/f/work/jekyll/my_plugins/jekyll_auto_redirect
/mnt/f/work/jekyll/my_plugins/jekyll_badge
/mnt/f/work/jekyll/my_plugins/jekyll_basename_dirname
/mnt/f/work/jekyll/my_plugins/jekyll_begin_end
/mnt/f/work/jekyll/my_plugins/jekyll_bootstrap5_tabs
/mnt/f/work/jekyll/my_plugins/jekyll_download_link
/mnt/f/work/jekyll/my_plugins/jekyll_draft
/mnt/f/work/jekyll/my_plugins/jekyll_emoji_tag
/mnt/f/work/jekyll/my_plugins/jekyll_eval_filter
/mnt/f/work/jekyll/my_plugins/jekyll_flexible_include_plugin
/mnt/f/work/jekyll/my_plugins/jekyll_from_to_until
/mnt/f/work/jekyll/my_plugins/jekyll_google_translate
/mnt/f/work/jekyll/my_plugins/jekyll_href
/mnt/f/work/jekyll/my_plugins/jekyll_img
/mnt/f/work/jekyll/my_plugins/jekyll_nth
/mnt/f/work/jekyll/my_plugins/jekyll_outline
/mnt/f/work/jekyll/my_plugins/jekyll_plugin_logger
/mnt/f/work/jekyll/my_plugins/jekyll_plugin_support
/mnt/f/work/jekyll/my_plugins/jekyll_pre
/mnt/f/work/jekyll/my_plugins/jekyll_qr_generator
/mnt/f/work/jekyll/my_plugins/jekyll_quote
/mnt/f/work/jekyll/my_plugins/jekyll_random_hex
/mnt/f/work/jekyll/my_plugins/jekyll_reading_time
/mnt/f/work/jekyll/my_plugins/jekyll_run
/mnt/f/work/jekyll/my_plugins/jekyll_site_inspector
/mnt/f/work/jekyll/my_plugins/jekyll_sort_natural
/mnt/f/work/jekyll/my_plugins/jekyll_time_since
/mnt/f/work/jekyll/my_plugins/jekyll_todo
/mnt/f/work/jekyll/my_plugins/jekyll_video

All work is complete.

Display Versions

This example shows how to display the version of projects that create gems under the directory pointed to by $my_plugins.

An executable script is required on the PATH, so git-exec can invoke it as it loops through the subdirectories. I call this script version, and it is written in bash, although the language used is not significant:

version

#!/bin/bash

x="$( ls lib/**/version.rb 2> /dev/null )"
if [ -f "$x" ]; then
  v="$(
    cat "$x" | \
    grep '=' | \
    sed -e s/.freeze// | \
    tr -d 'VERSION =\"' | \
    tr -d \'
  )"
  echo "$(basename $PWD) v$v"
fi

Call it like this:

Shell

$ git-exec '$my_plugins' version
jekyll_all_collections v0.3.3
jekyll_archive_create v1.0.2
jekyll_archive_display v1.0.1
jekyll_auto_redirect v0.1.0
jekyll_basename_dirname v1.0.3
jekyll_begin_end v1.0.1
jekyll_bootstrap5_tabs v1.1.2
jekyll_context_inspector v1.0.1
jekyll_download_link v1.0.1
jekyll_draft v1.1.2
jekyll_flexible_include_plugin v2.0.20
jekyll_from_to_until v1.0.3
jekyll_href v1.2.5
jekyll_img v0.1.5
jekyll_nth v1.1.0
jekyll_outline v1.2.0
jekyll_pdf v0.1.0
jekyll_plugin_logger v2.1.1
jekyll_plugin_support v0.7.0
jekyll_plugin_template v0.3.0
jekyll_pre v1.4.1
jekyll_quote v0.4.0
jekyll_random_hex v1.0.0
jekyll_reading_time v1.0.0
jekyll_revision v0.1.0
jekyll_run v1.0.1
jekyll_site_inspector v1.0.0
jekyll_sort_natural v1.0.0
jekyll_time_since v0.1.3

Display Projects With a Demo

List the projects under the directory pointed to by $my_plugins that have a demo/ subdirectory:

Shell

$ git-exec '$my_plugins' \
'if [ -d demo ]; then realpath demo; fi'
/mnt/c/work/jekyll/my_plugins/jekyll-hello/demo
/mnt/c/work/jekyll/my_plugins/jekyll_all_collections/demo
/mnt/c/work/jekyll/my_plugins/jekyll_archive_create/demo
/mnt/c/work/jekyll/my_plugins/jekyll_download_link/demo
/mnt/c/work/jekyll/my_plugins/jekyll_draft/demo
/mnt/c/work/jekyll/my_plugins/jekyll_flexible_include_plugin/demo
/mnt/c/work/jekyll/my_plugins/jekyll_from_to_until/demo
/mnt/c/work/jekyll/my_plugins/jekyll_href/demo
/mnt/c/work/jekyll/my_plugins/jekyll_img/demo
/mnt/c/work/jekyll/my_plugins/jekyll_outline/demo
/mnt/c/work/jekyll/my_plugins/jekyll_pdf/demo
/mnt/c/work/jekyll/my_plugins/jekyll_plugin_support/demo
/mnt/c/work/jekyll/my_plugins/jekyll_plugin_template/demo
/mnt/c/work/jekyll/my_plugins/jekyll_pre/demo
/mnt/c/work/jekyll/my_plugins/jekyll_quote/demo
/mnt/c/work/jekyll/my_plugins/jekyll_revision/demo
/mnt/c/work/jekyll/my_plugins/jekyll_time_since/demo

git-list-executables

The git-list-executables command lists all the executables created by this package.

git-list-executables help message

$ git-list-executables -h
git-list-executables - Lists executables installed by git-tree-go.

Usage: git-list-executables [OPTIONS]

OPTIONS:
-h, --help Show this help message and exit.

Shell

$ git-list-executables
Executables installed by git-tree-go in: /home/mslinn/go/bin

git-commitAll: Commit all changes in the current repository.
git-evars: Lists all environment variables used by git.
git-exec: Execute a command in each repository of the tree.
git-list-executables: Lists executables installed by git-tree-go.
git-replicate: Replicate a git repository.
git-treeconfig: Manage the git-tree configuration.
git-update: Update all repositories in the tree.

git-replicate

This command generates a shell script to replicate a tree of Git repositories. ROOTS can be directory names or environment variable references (e.g., '$work'). Multiple roots can be specified in a single quoted string.

Shell

$ git-replicate '$work' > work.sh # Replicate repos under $work

$ git-replicate '$work $sites' > replicate.sh # Replicate repos under $work and $sites

The generated environment variables will all be relative to the path pointed to by the expanded environment variable that you provided. You will understand what this means once you look at the generated script, following.

When git-replicate completes, edit the generated script to suit, then copy it to the target machine and run it. The following example copies the script to machine2 and runs it there:

Shell

$ scp work.sh machine2:

$ ssh machine2 work.sh

Generated Script

Following is a sample of one section, which is repeated for every Git repository that is processed.

Shell

if [ ! -d "sinatra/sinatras-skeleton/.git" ]; then
  mkdir -p 'sinatra'
  pushd 'sinatra' > /dev/null
  git clone git@github.com:mslinn/sinatras-skeleton.git
  git remote add upstream 'https://github.com/simonneutert/sinatras-skeleton.git'
  popd > /dev/null
fi

git-update

The git-update command updates each repository in the tree.

This command has its own article.

Development

See DEVELOPMENT.md.

Contributing

Fork the project
Create a descriptively named feature branch
Add your feature
Submit a pull request

License

The package is available as open source under the terms of the MIT License.

Mainframe image; Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License by PekoeBlaze

© Copyright 1994-2025 Michael Slinn. All rights reserved.
For requests to use this copyright-protected work in any manner, email mslinn@mslinn.com.

This website was made using Jekyll and Mike Slinn’s Jekyll Plugins.