Published 2021-04-05.
Last modified 2021-04-14.
Time to read: 2 minutes.
Django-oscar defines PIP dependencies with a
setting called install_requires.
install_requires = [
'django>=2.2,<3.2',
# PIL is required for image fields, Pillow is the "friendly" PIL fork
'pillow>=6.0',
# We use the ModelFormSetView from django-extra-views for the basket page
'django-extra-views>=0.13,<0.14',
# Search support
'django-haystack>=3.0b1',
# Treebeard is used for categories
'django-treebeard>=4.3,<4.5',
# Babel is used for currency formatting
'Babel>=1.0,<3.0',
# For manipulating search URLs
'purl>=0.7',
# For phone number field
'phonenumbers',
'django-phonenumber-field>=3.0.0,<4.0.0',
# Used for oscar.test.newfactories
'factory-boy>=2.4.1,<3.0',
# Used for automatically building larger HTML tables
'django-tables2>=2.3,<2.4',
# Used for manipulating form field attributes in templates (eg: add
# a css class)
'django-widget-tweaks>=1.4.1',
]
According to the documentation,
pip-tools/ uses install_requires
to maintain requirements.txt.
$ pip install pip-tools
Layer 1: requirements.in
I could not get the
install_requires
setting to work with pip-tools.
Instead, I was able to create a file called requirements.in to hold top-level dependencies,
and pip-tools happily used it:
boto3==1.17.27 django django-cors-headers==3.7.0 django-extensions django-oscar>=3.0.2,<4.0.0 django_storages==1.11.1 django-grappelli==2.14.3 pip pip-tools psycopg2-binary==2.8.6 pycountry==20.7.3 python-decouple==3.4 sorl-thumbnail==12.6.3
Notice that pip-tools is an unpinned requirement :)
With requirements.in in place, a new requirements.txt can be generated using the pip-compile
command provided by pip-tools.
Here is the pip-compile help message:
(aw) $ pip-compile -h Usage: pip-compile [OPTIONS] [SRC_FILES]... Compiles requirements.txt from requirements.in specs. Options: --version Show the version and exit. -v, --verbose Show more output -q, --quiet Give less output -n, --dry-run Only show what would happen, don't change anything -p, --pre Allow resolving to prereleases (default is not) -r, --rebuild Clear any caches upfront, rebuild from scratch -f, --find-links TEXT Look for archives in this directory or on this HTML page -i, --index-url TEXT Change index URL (defaults to https://pypi.org/simple) --extra-index-url TEXT Add additional index URL to search --cert TEXT Path to alternate CA bundle. --client-cert TEXT Path to SSL client certificate, a single file containing the private key and the certificate in PEM format. --trusted-host TEXT Mark this host as trusted, even though it does not have valid or any HTTPS. --header / --no-header Add header to generated file --emit-trusted-host / --no-emit-trusted-host Add trusted host option to generated file --annotate / --no-annotate Annotate results, indicating where dependencies come from -U, --upgrade Try to upgrade all dependencies to their latest versions -P, --upgrade-package TEXT Specify particular packages to upgrade. -o, --output-file FILENAME Output file name. Required if more than one input file is given. Will be derived from input file otherwise. --allow-unsafe / --no-allow-unsafe Pin packages considered unsafe: distribute, pip, setuptools. WARNING: Future versions of pip-tools will enable this behavior by default. Use --no- allow-unsafe to keep the old behavior. It is recommended to pass the --allow-unsafe now to adapt to the upcoming change. --generate-hashes Generate pip 8 style hashes in the resulting requirements file. --reuse-hashes / --no-reuse-hashes Improve the speed of --generate-hashes by reusing the hashes from an existing output file. --max-rounds INTEGER Maximum number of rounds before resolving the requirements aborts. --build-isolation / --no-build-isolation Enable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if build isolation is disabled. --emit-find-links / --no-emit-find-links Add the find-links option to generated file --cache-dir DIRECTORY Store the cache data in DIRECTORY. [default: /home/mslinn/.cache/pip-tools] --pip-args TEXT Arguments to pass directly to the pip command. --emit-index-url / --no-emit-index-url Add index URL to generated file -h, --help Show this message and exit.
Now I was able to update requirements.txt from requirements.in,
and then upgrade all PIP packages like this:
(aw) $ pip-compile -U (aw) $ pip install --upgrade -r requirements.txt
This could be written as one line.
(aw) $ pip-compile -U && \
pip install --upgrade -r requirements.txt
Layer 2
I wanted to take advantange of the pip-tools
layered requirements feature.
Overtop the basic dependencies listed in requirements.in,
I also wanted to manage development dependencies in dev.requirements.in
and deployment dependencies in prod.requirements.in.
The dev and prod layers are siblings.
Layer dev
There is no need to pin django-debug-toolbar because it is constrained by the
django dependency in the lower layer.
Jack Cushman, a pip-tools contributor,
explains why the --generate-hashes option is important.
-c requirements.txt django-debug-toolbar docutils json5 pytest-django PyYAML
(aw) $ pip-compile dev.requirements.in --generate-hashes --allow-unsafe
This produces dev.requirements.txt:
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile dev.requirements.in
#
asgiref==3.3.4
# via
# -c requirements.txt
# django
django-debug-toolbar==3.2
# via
# -c requirements.txt
# -r dev.requirements.in
django==3.1.8
# via
# -c requirements.txt
# django-debug-toolbar
pytz==2021.1
# via
# -c requirements.txt
# django
sqlparse==0.4.1
# via
# -c requirements.txt
# django
# django-debug-toolbar
--allow-unsafe seems unnecessary –
I believe that --allow-unsafe should be the default behavior for pip-compile.
I spent some time digging into the reasons that
pip-tools considers some packages “unsafe,”
and as best I can tell it is because it was thought that pinning those packages could potentially break pip itself,
and thus break the user's ability to recover from a mistake.
This seems to no longer be true, if it ever was. Instead, failing to use
--allow-unsafe is unsafe,
as it means different environments will end up with different versions of key packages despite installing from identical
requirements.txt files.
Layer prod
I added gunicorn as a production dependency, and was surprised to find that it declares
lists a specific version of the Pyton setuptools as a transitive dependency.
-c requirements.txt gunicorn json5
(aw) $ pip-compile prod.requirements.in --generate-hashes --allow-unsafe
This produces prod.requirements.txt:
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --allow-unsafe --generate-hashes prod.requirements.in
#
gunicorn==20.1.0 \
--hash=sha256:e0a968b5ba15f8a328fdfd7ab1fcb5af4470c28aaf7e55df02a99bc13138e6e8
# via -r prod.requirements.in
# The following packages are considered to be unsafe in a requirements file:
setuptools==56.0.0 \
--hash=sha256:08a1c0f99455307c48690f00d5c2ac2c1ccfab04df00454fef854ec145b81302 \
--hash=sha256:7430499900e443375ba9449a9cc5d78506b801e929fef4a186496012f93683b5
# via gunicorn