Working With Git LFS

Published 2025-01-06.
Time to read: 7 minutes.

This page is part of the git collection.

I have published 5 articles about the Git large file system (LFS). They are meant to be read in order.

7 articles are still in process.

Instructions for typing along are given for Ubuntu and WSL/Ubuntu. If you have a Mac, most of this information should be helpful.

If you have been reading along from the start of this miniseries, this page discusses various unrelated concepts that you should know at this point in your journey.

Recovering From Data Loss After Git LFS Import

Calm down; your data is not lost. The git history on the git server was rewritten; that made the remote history incompatible with your pristine copy of the git repository.

The current version of the Git User Manual explains the technical details of the problem. Older versions of the Git User Manual are much more alarmist.

You just need to fuss around a bit to incorporate the work you did since you last committed and pushed to orgin. You will have to go through this procedure for every git repository that was converted to Git LFS that you cloned to date. I will walk you through the steps so you can type along and resolve your problem. I will also give you a bash script that performs the same steps.

Caveat: This procedure only preserves the most recent version of your work. The commit history since your last push to origin is not also preserved by this procedure, just the version that you last ended up with.

Rescuing Your Changes Since Your Last Push to Origin

In this scenario, you have a git repository in $my_repo; these instructions assume that this is an absolute path, not a relative path.

Someone else, who is not currently your favorite person, has rewritten the git history on the git server. You can no longer git pull, git fetch or git push against that repository on the server. This scenario assumes that you are either working on the main, master, or one branch of your own.

If you were working on multiple branches simultaneously, you could adapt these directions, but you will not enjoy the process. Rescuing several branches can be tedious.

Move to your git repository.

Shell

$ cd $my_repo

Save the git `origin` remote in an environment variable called GIT_ORIGIN

Shell

$ GIT_ORIGIN="$( git config get remote.origin.url )"%}

Now move up to the parent directory.

Shell

$ cd ..

Now initialize Git LFS on your computer for your OS user account.

Shell

$ git lfs install

Re-clone the problem repository into $my_repo.new.

Shell

$ git clone ${GIT_ORIGIN} "$my_repo.new"

Now move into the newly cloned directory.

Shell

$ cd "$my_repo.new"

If you were working on your own branch, create it if it does not exist on origin. If it does exist on origin, make that branch current.

Shell

$ git checkout -b my_branch

This next command was written assuming you are still in the newly cloned directory. The following makes a copy of your existing work, including all changes, into the current directory. Your existing repository will not be modified in any way. If a problem occurs, you can rerun this command without damaging anything.

Shell

$ cd ..
$ rsync -a --progress "$my_repo" . --exclude .git
$

Now verify that the latest version of your work is here.

Shell

$ git status

Now you can commit your changes as usual, with git add, git commit and git push.

Detecting Files Not Managed By LFS

The following script shows files that are NOT managed by Git LFS in the current repository:

nonlfs

#!/bin/bash

function cleanup {
  if [ "$ALL_FILES" ]; then rm "$ALL_FILES"; fi
  if [ "$LSF_FILES" ]; then rm "$LSF_FILES"; fi
}

unset ALL_FILES, LSF_FILES
trap cleanup 0 # cleanup will always be called at exit, even for exit 0

ALL_FILES="$( mktemp )"
find . -type f -not -path "./.git/*" | sed s^\./^^ > "$ALL_FILES"

LSF_FILES="$( mktemp )"
while IFS=' ' read -r f _; do
  find . -name "$f" | sed s^\./^^
done < .gitattributes > "$LSF_FILES"

grep -f "$LSF_FILES" -F -w -v "$ALL_FILES"

Save the above script to a directory on your PATH, like /usr/local/bin, and make it executable:

Shell

$ chmod a+x /usr/local/bin/nonlfs

Now run it:

Shell

$ nonlfs
.gitattributes
normal_file.txt

No output will be shown if you run nonlfs from a non-LFS repository. You will only see output from this incantation if:

The current repository has Git LFS enabled and configured.
At least one file that matches one of the wildcard patterns stored in .gitattributes.

In contrast, the following shows files that ARE managed by Git LFS in the current repository:

Shell

$ git lfs ls-files

File locking

Git LFS includes a file locking feature into Git LFS repositories, a feature which git does not normally provide.

I have no need of this feature. The normal git conflict resolution mechanism works for me.

Add Locking Ability

To grant the files that match a wildcard pattern the ability to be locked, specify the ‑‑lockable parameter when invoking git lfs track. For example:

Shell

$ git lfs track --lockable "*.zip" "*.ZIP"
Tracking ".zip"
Tracking ".ZIP"

$ cat .gitattributes
*.zip filter=lfs diff=lfs merge=lfs -text lockable
*.ZIP filter=lfs diff=lfs merge=lfs -text lockable

You can now use the git lfs lock command to make files read-only for other team members.

Shell

$ git lfs lock outputs/stems/i_love_chocolate.zip

$ git lfs unlock outputs/stems/i_love_chocolate.zip

$ # Unlock someone else's files:
$ git lfs unlock outputs/stems/i_love_chocolate.zip --force

You can make the files that match an existing wildcard pattern lockable:

Shell

$ cat .gitattributes
*.big filter=lfs diff=lfs merge=lfs -text

$ git lfs track --lockable "*.big"
Tracking "*.big"

$ cat .gitattributes
*.big filter=lfs diff=lfs merge=lfs -text lockable

Remove Locking Ability

Conversely, you can remove the ability to be locked from a wildcard pattern.

Shell

$ cat .gitattributes
*.big filter=lfs diff=lfs merge=lfs -text lockable

$ git lfs track --unlockable "*.big"
Tracking "*.big"

$ cat .gitattributes
*.big filter=lfs diff=lfs merge=lfs -text

Caveat

I was unable to make file locking work using the local protocol.

This is the error message I got:

$ git lfs lock *

hint: The remote resolves to a file:// URL, which can only work with a
hint: standalone transfer agent. See section "Using a Custom Transfer Type
hint: without the API server" in custom-transfers.md for details.
Locking i_am.big failed: missing protocol: "file:///tmp/my_bare_repo"

Solving Problems

Error Logs

If you encounter an error and want to see more detail, type:

Shell

$ git lfs logs last

Trace Output

If you need more detail to debug a problem, you can enable tracing. The most common setting recommended by the git-lfs development team is: GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1.

Want to really know what Git is up to? Git has a fairly complete set of traces embedded, and all you need to do is turn them on. The possible values of these variables are as follows:

“true”, “1”, or “2” – the trace category is written to stderr.
An absolute path starting with / – the trace output will be written to that file.

GIT_TRACE controls general traces, which don’t fit into any specific category. This includes the expansion of aliases, and delegation to other sub-programs.

GIT_TRACE_PACK_ACCESS controls tracing of packfile access. The first field is the packfile being accessed, the second is the offset within that file:

GIT_TRACE_PACKET enables packet-level tracing for network operations.

GIT_TRACE_PERFORMANCE controls logging of performance data. The output shows how long each particular git invocation takes.

GIT_TRACE_SETUP shows information about what Git is discovering about the repository and environment it’s interacting with.

– Pro Git, chapter 10.8 Git Internals - Environment Variables

Let's trace the git push command by setting GIT_TRACE to a truthy value.

Shell

$ GIT_TRACE=true git push
10:08:19.388977 git.c:465               trace: built-in: git push
10:08:19.402310 run-command.c:657       trace: run_command: unset GIT_PREFIX; ssh git@github.com 'git-receive-pack '\''mslinn/song_test.git'\'''
10:08:19.884364 run-command.c:657       trace: run_command: .git/hooks/pre-push origin git@github.com:mslinn/song_test.git
10:08:19.893046 git.c:750               trace: exec: git-lfs pre-push origin git@github.com:mslinn/song_test.git
10:08:19.893074 run-command.c:657       trace: run_command: git-lfs pre-push origin git@github.com:mslinn/song_test.git
10:08:19.913859 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
10:08:19.921764 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
10:08:19.928421 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
10:08:19.947331 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
10:08:19.952187 trace git-lfs: exec: git 'config' '--includes' '-l'
10:08:19.956986 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
10:08:19.961434 trace git-lfs: exec: git 'config' '--includes' '-l' '-f' '/mnt/e/media/songs/test/.lfsconfig'
10:08:19.965132 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
10:08:19.969711 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
10:08:19.977203 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
10:08:19.994035 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '-q' 'origin'
10:08:20.631688 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
10:08:20.631716 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
10:08:20.632463 trace git-lfs: filepathfilter: accepting "tmp"
Everything up-to-date

If the mystery continues, you can enable verbose tracing by also setting GIT_CURL_VERBOSE to a truthy value.

Shell

$ GIT_TRACE=true GIT_CURL_VERBOSE=true git push
10:05:19.703116 git.c:465               trace: built-in: git push
10:05:19.722155 run-command.c:657       trace: run_command: unset GIT_PREFIX; ssh git@github.com 'git-receive-pack '\''mslinn/song_test.git'\'''
10:05:20.151523 run-command.c:657       trace: run_command: .git/hooks/pre-push origin git@github.com:mslinn/song_test.git
10:05:20.157969 git.c:750               trace: exec: git-lfs pre-push origin git@github.com:mslinn/song_test.git
10:05:20.158017 run-command.c:657       trace: run_command: git-lfs pre-push origin git@github.com:mslinn/song_test.git
10:05:20.182156 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
10:05:20.193179 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
10:05:20.200669 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
10:05:20.220094 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
10:05:20.225163 trace git-lfs: exec: git 'config' '--includes' '-l'
10:05:20.229139 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
10:05:20.232487 trace git-lfs: exec: git 'config' '--includes' '-l' '-f' '/mnt/e/media/songs/test/.lfsconfig'
10:05:20.236417 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
10:05:20.240127 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
10:05:20.246355 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
10:05:20.262021 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '-q' 'origin'
10:05:20.702973 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
10:05:20.703008 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
10:05:20.703918 trace git-lfs: filepathfilter: accepting "tmp"
Everything up-to-date

Trace Output

You can debug Git LFS by setting certain environment variables before you push. The GIT_TRACE variable enables trace output when set to a truthy value:

Shell

$ GIT_TRACE=1 git push origin master

Network problems can be detected by defining environment variables GIT_CURL_VERBOSE and GIT_TRANSFER_TRACE to a truthy value. You could define them all on the same line that launches git, like this:

Shell

$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 \
  git push origin master

Or you could make the environment variables last until the end of the console session, or until they are unset, like this:

Shell

$ GIT_TRACE=1

$ GIT_TRANSFER_TRACE=1

$ GIT_CURL_VERBOSE=1

$ git push origin master

$ # Do work, then ...

$ unset GIT_TRACE GIT_TRANSFER_TRACE GIT_CURL_VERBOSE

Setting the environment variables to truthy values will cause the headers and their responses to be displayed, which should help you troubleshoot problems that arise.

The man page for git-lfs-config mentions an additional environment variable, GIT_LFS_PROGRESS:

This environment variable causes Git LFS to emit progress updates to an absolute file-path on disk when cleaning, smudging, or fetching. Progress is reported periodically in the form of a new line being appended to the end of the file. Each new line will take the following format:

<direction> <current>/<total files> <downloaded>/<total> <name>

Each field is described below:

direction: The direction of transfer, either "checkout", "download", or "upload".
current: The index of the currently transferring file.
total files: The estimated count of all files to be transferred.
downloaded: The number of bytes already downloaded.
total: The entire size of the file, in bytes.
name: The name of the file.

– man git-lfs-config

Additional environment variables are available, for which I have not yet found documentation:

GIT_TRACE_PACKET=1
GIT_TRACE_PERFORMANCE=1
GIT_TRACE_SETUP=1

All Trace Environment Variables

The complete list of environment variables for tracing Git LFS client activity, in alphabetical order, is:

GIT_CURL_VERBOSE=1
GIT_LFS_PROGRESS=1
GIT_TRACE=1
GIT_TRACE_PACKET=1
GIT_TRACE_PERFORMANCE=1
GIT_TRACE_SETUP=1
GIT_TRANSFER_TRACE=1

Timeouts

There's a 30-second activity timeout before git retries each object up to 8 times. Output displaying that tells you it's attempting to download only starts after the headers are received, so git might appear to be hung when it is just timing out.

There are timeouts and other limits, which you can try lowering when debugging Git LFS.

Lowering Timeouts

The timeout-related configuration variables, listed next, must be stored in the default git configuration files, not in .lfsconfig. In other words, set and get them this way:

Shell

$ git config lfs.dialtimeout 1

$ git config lfs.dialtimeout
1

$ git config --global lfs.tlstimeout 1

$ git config --global lfs.tlstimeout
1

Timeout-Related Configuration Values

These configuration values are described in man git-lfs-config.

lfs.dialtimeout: Sets the maximum time, in seconds, that the HTTP client will wait to initiate a connection. This does not include the time to send a request and wait for a response. Default: 30 seconds.
lfs.tlstimeout: Sets the maximum time, in seconds, that the HTTP client will wait for a TLS handshake. Default: 30 seconds.
lfs.activitytimeout / lfs.https://<host>.activitytimeout: Sets the maximum time, in seconds, that the HTTP client will wait for the next tcp read or write. If < 1, no activity timeout is used at all. Default: 30 seconds.
lfs.keepalive: Sets the maximum time, in seconds, for the HTTP client to maintain keepalive connections. Default: 30 minutes.
lfs.ssh.retries: Specifies the number of times Git LFS will attempt to obtain authorization via SSH before aborting. Default: 5.
lfs.concurrenttransfers: The number of concurrent uploads/downloads. Default 8.
lfs.transfer.maxretries: Specifies how many retries LFS will attempt per OID before marking the transfer as failed. Must be an integer which is at least one, which means at least one retry will always be necessary. If the value is not an integer, is less than one, or is not given, a value of eight will be used instead.
lfs.transfer.maxretrydelay: Specifies the maximum time in seconds LFS will wait between each retry attempt. LFS uses exponential backoff for retries, doubling the time between each retry until reaching this limit. If a server requests a delay using the Retry-After header, the header value overrides the exponential delay for that attempt and is not limited by this option.

Must be an integer which is not negative. Use zero to disable delays between retries unless requested by a server. If the value is not an integer, is negative, or is not given, a value of ten will be used instead.
lfs.transfer.maxverifies: Specifies how many verification requests LFS will attempt per OID before marking the transfer as failed, if the object has a verification action associated with it. Must be an integer which is at least one. If the value is not an integer, is less than one, or is not given, a default value of three will be used instead.

I have published 5 articles about the Git large file system (LFS). They are meant to be read in order.

7 articles are still in process.

Instructions for typing along are given for Ubuntu and WSL/Ubuntu. If you have a Mac, most of this information should be helpful.

Mainframe image; Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License by PekoeBlaze

© Copyright 1994-2025 Michael Slinn. All rights reserved.
For requests to use this copyright-protected work in any manner, email mslinn@mslinn.com.

This website was made using Jekyll and Mike Slinn’s Jekyll Plugins.