Mike Slinn
Mike Slinn

Python Dependency Management With Pip-Tools

Published 2021-04-05. Last modified 2021-04-14.
Time to read: about 2 minutes.

This article is categorized under Django, Django-Oscar, Python.

Django-oscar defines PIP dependencies with a setting called install_requires.

Django-oscar settings
install_requires = [
    # PIL is required for image fields, Pillow is the "friendly" PIL fork
    # We use the ModelFormSetView from django-extra-views for the basket page
    # Search support
    # Treebeard is used for categories
    # Babel is used for currency formatting
    # For manipulating search URLs
    # For phone number field
    # Used for oscar.test.newfactories
    # Used for automatically building larger HTML tables
    # Used for manipulating form field attributes in templates (eg: add
    # a css class)

According to the documentation, pip-tools/ uses install_requires to maintain requirements.txt.

$ pip install pip-tools

Layer 1: requirements.in

I could not get the install_requires setting to work with pip-tools. Instead, I was able to create a file called requirements.in to hold top-level dependencies, and pip-tools happily used it:


Notice that pip-tools is an unpinned requirement :)

With requirements.in in place, a new requirements.txt can be generated using the pip-compile command provided by pip-tools. Here is the pip-compile help message:

(aw) $ pip-compile -h
Usage: pip-compile [OPTIONS] [SRC_FILES]...

  Compiles requirements.txt from requirements.in specs.

  --version                       Show the version and exit.
  -v, --verbose                   Show more output
  -q, --quiet                     Give less output
  -n, --dry-run                   Only show what would happen, don't change

  -p, --pre                       Allow resolving to prereleases (default is

  -r, --rebuild                   Clear any caches upfront, rebuild from

  -f, --find-links TEXT           Look for archives in this directory or on
                                  this HTML page

  -i, --index-url TEXT            Change index URL (defaults to

  --extra-index-url TEXT          Add additional index URL to search
  --cert TEXT                     Path to alternate CA bundle.
  --client-cert TEXT              Path to SSL client certificate, a single
                                  file containing the private key and the
                                  certificate in PEM format.

  --trusted-host TEXT             Mark this host as trusted, even though it
                                  does not have valid or any HTTPS.

  --header / --no-header          Add header to generated file
  --emit-trusted-host / --no-emit-trusted-host
                                  Add trusted host option to generated file
  --annotate / --no-annotate      Annotate results, indicating where
                                  dependencies come from

  -U, --upgrade                   Try to upgrade all dependencies to their
                                  latest versions

  -P, --upgrade-package TEXT      Specify particular packages to upgrade.
  -o, --output-file FILENAME      Output file name. Required if more than one
                                  input file is given. Will be derived from
                                  input file otherwise.

  --allow-unsafe / --no-allow-unsafe
                                  Pin packages considered unsafe: distribute,
                                  pip, setuptools.

                                  WARNING: Future versions of pip-tools will
                                  enable this behavior by default. Use --no-
                                  allow-unsafe to keep the old behavior. It is
                                  recommended to pass the --allow-unsafe now
                                  to adapt to the upcoming change.

  --generate-hashes               Generate pip 8 style hashes in the resulting
                                  requirements file.

  --reuse-hashes / --no-reuse-hashes
                                  Improve the speed of --generate-hashes by
                                  reusing the hashes from an existing output

  --max-rounds INTEGER            Maximum number of rounds before resolving
                                  the requirements aborts.

  --build-isolation / --no-build-isolation
                                  Enable isolation when building a modern
                                  source distribution. Build dependencies
                                  specified by PEP 518 must be already
                                  installed if build isolation is disabled.

  --emit-find-links / --no-emit-find-links
                                  Add the find-links option to generated file
  --cache-dir DIRECTORY           Store the cache data in DIRECTORY.
                                  [default: /home/mslinn/.cache/pip-tools]

  --pip-args TEXT                 Arguments to pass directly to the pip

  --emit-index-url / --no-emit-index-url
                                  Add index URL to generated file
  -h, --help                      Show this message and exit. 

Now I was able to update requirements.txt from requirements.in, and then upgrade all PIP packages like this:

(aw) $ pip-compile -U

(aw) $ pip install --upgrade -r requirements.txt

This could be written as one line.

(aw) $ pip-compile -U && \
pip install --upgrade -r requirements.txt

Layer 2

I wanted to take advantange of the pip-tools layered requirements feature. Overtop the basic dependencies listed in requirements.in, I also wanted to manage development dependencies in dev.requirements.in and deployment dependencies in prod.requirements.in. The dev and prod layers are siblings.

Layer dev

There is no need to pin django-debug-toolbar because it is constrained by the django dependency in the lower layer. Jack Cushman, a pip-tools contributor, explains why the --generate-hashes option is important.

-c requirements.txt
(aw) $ pip-compile dev.requirements.in --generate-hashes --allow-unsafe

This produces dev.requirements.txt:

# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile dev.requirements.in
    # via
    #   -c requirements.txt
    #   django
    # via
    #   -c requirements.txt
    #   -r dev.requirements.in
    # via
    #   -c requirements.txt
    #   django-debug-toolbar
    # via
    #   -c requirements.txt
    #   django
    # via
    #   -c requirements.txt
    #   django
    #   django-debug-toolbar

Jack Cushman had this to say about --allow-unsafe:

First, the warning to use --allow-unsafe seems unnecessary — I believe that --allow-unsafe should be the default behavior for pip-compile. I spent some time digging into the reasons that pip-tools considers some packages “unsafe,” and as best I can tell it is because it was thought that pinning those packages could potentially break pip itself, and thus break the user's ability to recover from a mistake. This seems to no longer be true, if it ever was. Instead, failing to use --allow-unsafe is unsafe, as it means different environments will end up with different versions of key packages despite installing from identical requirements.txt files.

Layer prod

I added gunicorn as a production dependency, and was surprised to find that it declares lists a specific version of the Pyton setuptools as a transitive dependency.

-c requirements.txt
(aw) $ pip-compile prod.requirements.in --generate-hashes --allow-unsafe

This produces prod.requirements.txt:

# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile --allow-unsafe --generate-hashes prod.requirements.in
gunicorn==20.1.0 \
    # via -r prod.requirements.in

# The following packages are considered to be unsafe in a requirements file:
setuptools==56.0.0 \
    --hash=sha256:08a1c0f99455307c48690f00d5c2ac2c1ccfab04df00454fef854ec145b81302 \
    # via gunicorn