How to make a Python package in 2021

Comments on this reply seem to have forgotten two key things about python.

1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies.

2) “There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.py, pypi, and so on. Python is not Rust, or Node, or Clojure, or anything else. Do it Pythonicly and good things will happen.

In the history of Python, it’s only because of the Zen’s guiding principles and the “come join us” attitude of the community that Python accidentally became “enterprise grade”. It succeeded in spite of the lack of Cargo or NPM, or (shudder) Mavin[, or CI/CD, or 3D game engines, or easy binary package productization, or everything else Python is obviously bad at]; almost precisely because it’s not those things. Python encourages me to get my task done, and all it asks is that I leave the flame wars at the door and embrace it for what it is.

It saddens me to see so many people jump down zomglings throat for making the daring suggestion that Pythonistas use the language as designed. Because, he is absolutely right about this: “Complex is better than complicated.”

I actually think one should do the exact opposite to what you're suggesting. I've experienced modern package management through Cargo and anything below that level now seems like returning to stone age.

In the python ecosystem, poorly defined packages is a wide problem. You never know what you get in the next version upgrade.

So my suggestion: burn all the "simple" python packaging tools in a big fire and move eveything to Poetry today. It will be painful at start but will increase the quality of the ecosystem by a huge margin if just 50% of the projects do this.

Using setup.py does not mean "not using extra tools". It depends on setuptools, which is an "extra tools" just like flit (used in the article) or any other tool. In fact, with using only setuptools one will need a whole additional set of tools to manage things like:

    * virtual environments (granted, venv is now part of the stdlib, but it's still an "extra tool")
    * publishing to PyPI or another index (twine)
    * dependency management (both for development and actual dependencies)

    * unit tests (pytests)
    * static typing (mypy)
    * linting (flake8, pylint)
    * styling (black)

Finally, tools like Poetry [2] offer a whole set of functionalities in one place (dependency management, virtual environments, publishing), which means that they need fewer "extra tools" than just setuptools.

[0] https://www.python.org/dev/peps/pep-0518/

[1] https://setuptools.readthedocs.io/en/latest/build_meta.html

[2] https://python-poetry.org/

How do you handle version pinning? hash checking? CI? testing on multiple platforms? multiple python versions? deployment? credential management? package data? version bumps?

Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.

This is bad advice. Do not create a setup.py for a new package. (Keeping setup.py for an old package can be okay.)

The author is correct that you want a tool such as flit or poetry which will work with pyproject.toml. Setting up a basic package will be no harder than using setuptools, and it is much more future-proof. You won't have to copy-paste some other crufty setup config either.

It is fair that you don't need all the external tools in this tutorial. In particular, using make is very silly since you can configure different linting and testing workflows directly in pyproject.toml, rather than pull in a whole other system which only works decently on *nix. Poetry also removes the need for tox.

I would probably recommend just following https://packaging.python.org/tutorials/packaging-projects/ instead.

For all the hate nodejs gets, it solved the software packaging problem. That is the sole reason npm ecosystem is so big.

I have to say that it probably isn't a fare comparison because python is much older than nodejs. Python package management might very well have been state of the art in 1995.

Maybe the reason that the library itself is simple in the article is that that’s just an example and the author wants to show how to do it properly, end to end.

Huge thumbs up to Poetry. It's drastically simplified package management for me and replaced Pipenv (which I simply dreaded working with due to performance and DX issues).

I no longer start Python projects without Poetry. It's really good.

EDIT: Also, it's being integrated to most PaaS as well. I deploy to Render.com with Poetry now.

That's a 72MB download, and yet another way to fragment the ecosystem. Not something I'd get just to make a package when a default Python setup has recommended tools and everything I need to make and/or install packages.

There are at least 5 comments in this thread saying "just use the tool I use"

+1 for poetry. It also includes deterministic dependency resolution with a lock file.

I just published a repo today[0] using Poetry and it didn’t take me more than 5 minutes. Poetry build && poetry publish

[0] https://github.com/santiagobasulto/hyper-inspector

Yup, was surprised to see no mention of Poetry. Hands down, the best package manager for Python.

Holy crap, for some reason I never thought to consider another package manager than Pip, which I loathe.

`poetry` and `pipenv` are both so much slower than `pip-compile` for me (there are many open issues for both complaining about lock speed), and manage to update locked dependencies half the time despite me asking them not to with e.g. `poetry lock --no-update`.

Do any big projects actually use it?

Thomas is well know as one of the maintainer of IPython and Jupyter, and developed flit while working on the pep for pyproject.toml and the pip backend allowing things like python -m build.

Though `python -m build` only works _if_ you use something like flit or setup.py in the backend to do build the package and hence why you can set flit as a build-backend.

So yes, flit is one of the latest tool, and yes it is one of the things that push for the ability to use pyproject.toml+python3 -m build, you just seem to miss some subtleties of the toolchain.

Flit is definitely opinionated, and not suitable for every use case. As Carreau hinted, I think its bigger impact will be from the specifications, especially PEP 517, which it helped to prompt, rather than people using Flit directly. The specifications mean it's practical to make new tools which interoperate nicely, without having to either wrap setuptools or carefully imitate its behaviour.

The PyPA recommendation is old and out of date. Using a system that doesn't manage constraints and hash-checked lockfiles is bad practice.

I don't see how your version is easier than the sequence of commands in the first few steps of the article, which is basically `pip install flit; flit init; flit publish`. Flit is just as easy to install as twine, but you save yourself the hassle of having to write a setup.py.

Also automatcially handles the issue of public vs private etc. Note this [1] recommends using zip files for speed, especially for larger repositories:

    pip install https://github.com/django/django/archive/master.zip

    pip install https://github.com/django/django/archive/stable/1.7.x.zip

I do that where I work because we have an internal git that is wide open, but packages from the outside world need to get whitelisted.

More controversially: With a few (<10) lines of code in your setup file, you can make an install-able module out of jupyter notebooks.

But now packages on PyPI can't depend on it.

How does this work for dependencies?

And what if some of the dependencies are incompatible with the versions already on your system?

I agree.

What I actually like is using the system package manager to install stuff. pacman -s or apt-get install

letting multiple packaging systems muck with your system is a recipe for hurt somewhere down the line.

The only constant in the python community over the last twenty years are the criticisms about its packaging

I think for the vast majority of at least pure-Python projects you could just use poetry and upload your packages to PyPI or a private index. You can go from an empty directory to publishing a package within minutes with poetry (although, of course, you probably shouldn't).

It's fragmented, but it doesn't need to be error prone if people use the good tools instead of the old low-level tools.

I’d argue the counterpoint actually: Writing Makefile targets for common commands significantly improves usability and ergonomics, especially when they follow common idioms (make test, make build, make install, ...).

The recipes for each target describe not only how a project intends to run each tool, but which tools it intends to run. Instead of having to know that this project runs tests under tox, while that one runs only under pytest, we can run ‘make test’ in each, and count on the recipe doing the Right Thing.

That consistency across projects makes it much easier for someone to get started on a new project (or to remember how the pieces fit together on your own project from a few months ago)

IMO as long as there is a healthy CI pipeline which people can check to see how to build/test things, ultimately it doesn't matter much.

I rather just write a bash script. It's the lowest common denominator. Make may not be installed by default in many places and it has some weird syntax quirks that make it annoying to use IMO.

Here's an example of how I like to do my "bash scripts that sorta work like make": https://github.com/francislavoie/laravel-websockets-example/... basically each function is a "command", so I do like "./utils start" or whatever.

I would strongly discourage using make on new projects, make syntax is full of footguns and quirks (not being able to pass multiple args to subcommands is an easy example).

Bash, Python, or even Typescript are much easier, safer, and more widely standardized environments to maintain and grow your scripts once you get past a few lines.

It's programming language that distinguishes between spaces and tabs in a way that changes behavior. It's also the only PL that I know of that's outright incompatible with expand-all-tabs-to-spaces editing policy, which is what the vast majority of coders use in practice.

If you want to see a powerful and actually simple tool that does the job, take a look at DJB redo: https://redo.readthedocs.io/en/latest/

I like the idea of Make, but it's far too hacky. Even using it for a static blog site (turning foo.md -> foo.html, which is pretty close to the usual foo.c -> foo.o examples) ended up with recursive invocations, rule-creation macros, double-escaped sigils, eval, etc.

There are a bunch of lightweight alternatives to Make out there (I hear Ninja is pretty good). My personal preference is Nix these days (although that's quite heavyweight).

The biggest problem with make appears to be that people refuse to spend a little time learning how it works, and instead charge off to reimplement it, poorly, instead.

cookiecutter is great. I recently moved from click to typer, which I so far really have enjoyed. I probably should make a cc template for that one day...