Django and Oscar

Python Dependency Management With Pip-Tools

Published 2021-04-05. Last modified 2021-04-14.
Time to read: 2 minutes.

This page is part of the django collection.

Django-oscar defines PIP dependencies with a setting called install_requires.

Django-oscar settings
install_requires = [
    # PIL is required for image fields, Pillow is the "friendly" PIL fork
    # We use the ModelFormSetView from django-extra-views for the basket page
    # Search support
    # Treebeard is used for categories
    # Babel is used for currency formatting
    # For manipulating search URLs
    # For phone number field
    # Used for oscar.test.newfactories
    # Used for automatically building larger HTML tables
    # Used for manipulating form field attributes in templates (eg: add
    # a css class)

According to the documentation, pip-tools/ uses install_requires to maintain requirements.txt.

$ pip install pip-tools

Layer 1:

I could not get the install_requires setting to work with pip-tools. Instead, I was able to create a file called to hold top-level dependencies, and pip-tools happily used it:


Notice that pip-tools is an unpinned requirement :)

With in place, a new requirements.txt can be generated using the pip-compile command provided by pip-tools. Here is the pip-compile help message:

(aw) $ pip-compile -h
Usage: pip-compile [OPTIONS] [SRC_FILES]...

  Compiles requirements.txt from specs.

  --version                       Show the version and exit.
  -v, --verbose                   Show more output
  -q, --quiet                     Give less output
  -n, --dry-run                   Only show what would happen, don't change

  -p, --pre                       Allow resolving to prereleases (default is

  -r, --rebuild                   Clear any caches upfront, rebuild from

  -f, --find-links TEXT           Look for archives in this directory or on
                                  this HTML page

  -i, --index-url TEXT            Change index URL (defaults to

  --extra-index-url TEXT          Add additional index URL to search
  --cert TEXT                     Path to alternate CA bundle.
  --client-cert TEXT              Path to SSL client certificate, a single
                                  file containing the private key and the
                                  certificate in PEM format.

  --trusted-host TEXT             Mark this host as trusted, even though it
                                  does not have valid or any HTTPS.

  --header / --no-header          Add header to generated file
  --emit-trusted-host / --no-emit-trusted-host
                                  Add trusted host option to generated file
  --annotate / --no-annotate      Annotate results, indicating where
                                  dependencies come from

  -U, --upgrade                   Try to upgrade all dependencies to their
                                  latest versions

  -P, --upgrade-package TEXT      Specify particular packages to upgrade.
  -o, --output-file FILENAME      Output file name. Required if more than one
                                  input file is given. Will be derived from
                                  input file otherwise.

  --allow-unsafe / --no-allow-unsafe
                                  Pin packages considered unsafe: distribute,
                                  pip, setuptools.

                                  WARNING: Future versions of pip-tools will
                                  enable this behavior by default. Use --no-
                                  allow-unsafe to keep the old behavior. It is
                                  recommended to pass the --allow-unsafe now
                                  to adapt to the upcoming change.

  --generate-hashes               Generate pip 8 style hashes in the resulting
                                  requirements file.

  --reuse-hashes / --no-reuse-hashes
                                  Improve the speed of --generate-hashes by
                                  reusing the hashes from an existing output

  --max-rounds INTEGER            Maximum number of rounds before resolving
                                  the requirements aborts.

  --build-isolation / --no-build-isolation
                                  Enable isolation when building a modern
                                  source distribution. Build dependencies
                                  specified by PEP 518 must be already
                                  installed if build isolation is disabled.

  --emit-find-links / --no-emit-find-links
                                  Add the find-links option to generated file
  --cache-dir DIRECTORY           Store the cache data in DIRECTORY.
                                  [default: /home/mslinn/.cache/pip-tools]

  --pip-args TEXT                 Arguments to pass directly to the pip

  --emit-index-url / --no-emit-index-url
                                  Add index URL to generated file
  -h, --help                      Show this message and exit. 

Now I was able to update requirements.txt from, and then upgrade all PIP packages like this:

(aw) $ pip-compile -U

(aw) $ pip install --upgrade -r requirements.txt

This could be written as one line.

(aw) $ pip-compile -U && \
pip install --upgrade -r requirements.txt

Layer 2

I wanted to take advantange of the pip-tools layered requirements feature. Overtop the basic dependencies listed in, I also wanted to manage development dependencies in and deployment dependencies in The dev and prod layers are siblings.

Layer dev

There is no need to pin django-debug-toolbar because it is constrained by the django dependency in the lower layer. Jack Cushman, a pip-tools contributor, explains why the --generate-hashes option is important.

-c requirements.txt
(aw) $ pip-compile --generate-hashes --allow-unsafe

This produces dev.requirements.txt:

# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile
    # via
    #   -c requirements.txt
    #   django
    # via
    #   -c requirements.txt
    #   -r
    # via
    #   -c requirements.txt
    #   django-debug-toolbar
    # via
    #   -c requirements.txt
    #   django
    # via
    #   -c requirements.txt
    #   django
    #   django-debug-toolbar
The warning to use --allow-unsafe seems unnecessary – I believe that --allow-unsafe should be the default behavior for pip-compile.

I spent some time digging into the reasons that pip-tools considers some packages “unsafe,” and as best I can tell it is because it was thought that pinning those packages could potentially break pip itself, and thus break the user's ability to recover from a mistake.

This seems to no longer be true, if it ever was. Instead, failing to use --allow-unsafe is unsafe, as it means different environments will end up with different versions of key packages despite installing from identical requirements.txt files.

Layer prod

I added gunicorn as a production dependency, and was surprised to find that it declares lists a specific version of the Pyton setuptools as a transitive dependency.

-c requirements.txt
(aw) $ pip-compile --generate-hashes --allow-unsafe

This produces prod.requirements.txt:

# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile --allow-unsafe --generate-hashes
gunicorn==20.1.0 \
    # via -r

# The following packages are considered to be unsafe in a requirements file:
setuptools==56.0.0 \
    --hash=sha256:08a1c0f99455307c48690f00d5c2ac2c1ccfab04df00454fef854ec145b81302 \
    # via gunicorn

* indicates a required field.

Please select the following to receive Mike Slinn’s newsletter:

You can unsubscribe at any time by clicking the link in the footer of emails.

Mike Slinn uses Mailchimp as his marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp’s privacy practices.