Index ¦ Archives ¦ Atom

Python: Using specific git commits of third party packages

After a fair number of furious impatient attempts to try and use specific git commits of third party packages in my Python software, I finally have been able to make it work. I went back to the drawing board - basically reading setup-vs-requirement.


This is what I did:

  • If you have a, add the library name to (abstract dependency)
  • Add the git URL in a requirements.txt file (concrete dependency)
  • When you create your deployment artifact, do pip install -r requirements.txt.

An example requirements.txt file would look like:

git+https://<git repo>@master

You can replace the master by a specific commit/git tag.

Libraries/End-user focused tools

Now, let's say you are publishing a package to PyPI and this package has a dependency on a version of a package which is not in PyPi or in a git repo. This is what we do:

In our, we add the package name in install_requires and add dependency_links as follows:


However if the third party package is already in pypi, we have a problem. So, we fool pip like, so:

# I fool `pip` by specifying the version number which
# is greater than the one released in PyPi and force
# it to look at the dependency_links where i wrongly specify
# that i have a version which is greater than 0.1.2

See an example project

Now, if we install pip install . --process-dependency-links, we will see:

Could not find a tag or branch '9bff9d01ce16589201f57ffef27ea84744951c11', assuming commit.
  Requested fire>0.1.2 from git+ (from my-awesome-cli==0.1), but installing version None

To then distribute this to PyPI, we need to make sure that we distribute this as a source tarball, not a wheel:

 $ python sdist
 $ TWINE_REPOSITORY_URL= TWINE_USERNAME=echorand TWINE_PASSWORD="secret" twine upload dist/*

Once we have done that, we can install it, like so:

$ pip install my-awesome-cli==0.2 --process-dependency-links -i
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting fire>0.1.2 (from my-awesome-cli==0.2)
  Cloning (to 9bff9d01ce16589201f57ffef27ea84744951c11) to /tmp/pip-build-SykxjY/fire
  Could not find a tag or branch '9bff9d01ce16589201f57ffef27ea84744951c11', assuming commit.
  Requested fire>0.1.2 from git+ (from my-awesome-cli==0.2), but installing version None
Collecting six (from fire>0.1.2->my-awesome-cli==0.2)

Successfully installed fire-0.1.2 my-awesome-cli-0.2 six-1.10.0

We can then run our application:

$ my-awesome-cli
Type:        Calculator
String form: <my_awesome_cli.main.Calculator object at 0x7feecae69850>
Docstring:   A simple calculator class.

Usage:       my-awesome-cli
             my-awesome-cli double

Helpful links

Share on: Diaspora*TwitterFacebookGoogle+Email

© Amit Saha. Built using Pelican. Customised theme based on the one by Giulio Fidente on github.