Hacker News

a day ago by zomglings

The number of extra tools used in this article boggles my mind.

Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package.

Want to publish to PyPI? Use twine. It's standard and it's simple.

You don't need complicated tooling for simple projects.

13 hours ago by vonwoodson

Comments on this reply seem to have forgotten two key things about python.

1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies.

2) “There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.py, pypi, and so on. Python is not Rust, or Node, or Clojure, or anything else. Do it Pythonicly and good things will happen.

In the history of Python, it’s only because of the Zen’s guiding principles and the “come join us” attitude of the community that Python accidentally became “enterprise grade”. It succeeded in spite of the lack of Cargo or NPM, or (shudder) Mavin[, or CI/CD, or 3D game engines, or easy binary package productization, or everything else Python is obviously bad at]; almost precisely because it’s not those things. Python encourages me to get my task done, and all it asks is that I leave the flame wars at the door and embrace it for what it is.

It saddens me to see so many people jump down zomglings throat for making the daring suggestion that Pythonistas use the language as designed. Because, he is absolutely right about this: “Complex is better than complicated.”

11 hours ago by patagonia

Unpopular opinion: Python is bait and switch. Naive, aspiring programmers are told to learn Python because it’s the perfect beginner language, only to quickly hit a brick wall of complication they are unable to foresee (because they’re completely new to all this) and ill-equipped navigate.

Like it or not Python is now “enterprise grade”. The sooner it either grows up and figures out packages, ci/cd, binaries etc, or it looses its reputation at a great beginner and general programming language to something like Julia or Crystal, the better.

11 hours ago by cixinlui

The thing is that using setuptools is neither standard nor Pythonic. It isn't part of the standard library. It's a way of doing things that is broken, and has been specifically called out by the Python developer community as something that people should stop doing.

18 hours ago by darthrupert

I actually think one should do the exact opposite to what you're suggesting. I've experienced modern package management through Cargo and anything below that level now seems like returning to stone age.

In the python ecosystem, poorly defined packages is a wide problem. You never know what you get in the next version upgrade.

So my suggestion: burn all the "simple" python packaging tools in a big fire and move eveything to Poetry today. It will be painful at start but will increase the quality of the ecosystem by a huge margin if just 50% of the projects do this.

16 hours ago by hardwaresofton

Bit of a side note but remember cargo learned a lot from generations of package managers, across languages. It essentially represents some of the best that package managers have to offer. You only get that kind of result by starting from scratch every few years (as in, rust started from scratch when the Node/NPM ecosystem had been around for years to steal ideas from, Haskell had been around for years to steal language design from, etc.).

Rust is incredibly lucky to have been created when it was because it benefits immensely from these things (I think it's the best new-to-middle-aged production-usable systems language out there today).

I agree with the idea but languages like Python and their ecosystems are really hard to move (remember python 2->3? is that even over?) -- it's a herculean and often impossible task.

14 hours ago by lmm

Cargo is not hugely different from Maven which has been working fine for over a decade. Yes, it takes some polished ideas from other systems, but Python has had more packaging tools come and go than several other ecosystems put together.

16 hours ago by afarrell

Migrating python2 -> python3 is hard because it requires rewriting code and because you can't really run python3 code if you depend upon python2.

If poetry create consume non-poetry packages and create packages which other package management systems can consume? If so, then:

1. Projects can move more effectively independently.

2. Projects can re-package as a single task rather than a massive rewrite effort.

9 hours ago by alexgmcm

Like the other commenter, I was also expecting them to use Poetry.

It's the best one I've tried so far.

8 hours ago by avhception

Came here to about how it would be better to use poetry, thanks for spelling it out for me.

14 hours ago by say_it_as_it_is

I was hoping to see this blog post highlight poetry usage. Any good resource for this? I've already seen the official page.

5 hours ago by raphar

I learned with this article. And it teaches how to combine Poetry with Pyenv effectively.

https://blog.jayway.com/2019/12/28/pyenv-poetry-saviours-in-...

9 hours ago by alexgmcm

I quite liked the series of blog posts called Hypermodern Python: https://cjolowicz.github.io/posts/hypermodern-python-01-setu...

16 hours ago by BerislavLopac

Using setup.py does not mean "not using extra tools". It depends on setuptools, which is an "extra tools" just like flit (used in the article) or any other tool. In fact, with using only setuptools one will need a whole additional set of tools to manage things like:

    * virtual environments (granted, venv is now part of the stdlib, but it's still an "extra tool")
    * publishing to PyPI or another index (twine)
    * dependency management (both for development and actual dependencies)
Plus the tools that are needed anyway, to manage common development actions like:

    * unit tests (pytests)
    * static typing (mypy)
    * linting (flake8, pylint)
    * styling (black)
The article is correct in using pyproject.toml, which has become the standard way to specify build mechanism for your package [0]. Even setuptools supports it in the latest versions [1], meaning that setup.py is becoming obsolete, or at least unnecessary.

Finally, tools like Poetry [2] offer a whole set of functionalities in one place (dependency management, virtual environments, publishing), which means that they need fewer "extra tools" than just setuptools.

[0] https://www.python.org/dev/peps/pep-0518/

[1] https://setuptools.readthedocs.io/en/latest/build_meta.html

[2] https://python-poetry.org/

14 hours ago by say_it_as_it_is

Where is there a comprehensive poetry packaging tutorial?

9 hours ago by BerislavLopac

Right here:

    poetry build -f wheel
    poetry publish
You asked for packaging, and that is pretty much it. Of course, setting up a project and its dependencies take a bit more work; the basic intro for that is here: https://python-poetry.org/docs/basic-usage/

21 hours ago by neolog

How do you handle version pinning? hash checking? CI? testing on multiple platforms? multiple python versions? deployment? credential management? package data? version bumps?

Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.

21 hours ago by zomglings

Iteratively. You don't need to solve all those problems at once.

Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though.

You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnable scripts.

I typically bump version directly in a version file and move on with my life.

This stuff usually builds up iteratively and at least for me has never been the starting point. Starting point should be a library worth sharing. It is not the end of the world of you release the first few versions manually.

21 hours ago by staticassertion

TBH as someone trying to use Python professionally it is extremely frustrating that basic things with regards to package management are something you have to iterate towards, as opposed to just being obvious and default.

21 hours ago by ehsankia

To be fair, while this is a single article, if you only look at step 1, 2 and 3, you get a fully published package with only one tool used (flit) and not much extra.

It's the succeeding sections (A, B, C, D, E) that get more advanced, but they're all optional. You should definitely do A, but the rest I'd say it's a lot more opinionated and definitely not needed.

20 hours ago by dagenix

> Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file

The problem with this approach is that it doesn't handle transitive dependencies well. Say you depend on version 1.4.6 of a particular library. And then that library depends on version >= 2 of some other library. When you install your package, you know that you'll get version 1.4.6 of the first library but have no idea what version you'll get of the second library. You can of course pin all the transitive dependencies - except that clutters up the setup.py and is a massive pain to keep up to date as you bump individual dependency versions.

15 hours ago by BerislavLopac

> You should be very conservative when pinning versions in a library, though

No; you should be very conservative when pinning versions in an application, not in a library. Check this article for the explanation: https://caremad.io/posts/2013/07/setup-vs-requirement/

10 hours ago by asdf123wtf

I generally reach for pip-tools, when I need to pin versions in a requirements for a deployable app, like an api.

It's by far the simplest option I've found.

If your project is a library, just use setup.py and express your deps abstractly (min, max or version range). Don't pin to specific versions at all, if you can help it.

17 hours ago by 1337shadow

I usually don't pin, I'd rather deal with upstream BC breaks as they are published instead of accumulating tech debt. I call this "continuously integrating upstream code", because Continuous Integration is a practice, not a tool.

17 hours ago by globular-toast

You don't pin anything for a package. I'm not aware of any "standard" CI that a package tool could set up. I guess you mean testing on multiple versions, in which case tox will help. Deployment for a package is handled by twine. Package data? What about it? Version bumps should always be manual but I recommend setuptools-scm.

You seem to be confusing packages with "apps". It's very important to understand the clear distinction between these.

5 hours ago by neolog

> You seem to be confusing packages with "apps".

I'm not confusing libraries with applications. Pinning dependency versions enables a repeatable test environment.

a day ago by neolog

Just use Poetry [1]. It's popular and works well.

[1] https://python-poetry.org/

21 hours ago by sergiomattei

Huge thumbs up to Poetry. It's drastically simplified package management for me and replaced Pipenv (which I simply dreaded working with due to performance and DX issues).

I no longer start Python projects without Poetry. It's really good.

EDIT: Also, it's being integrated to most PaaS as well. I deploy to Render.com with Poetry now.

15 hours ago by fctorial

There are at least 5 comments in this thread saying "just use the tool I use"

12 hours ago by kuu

20 hours ago by guggle

That's a 72MB download, and yet another way to fragment the ecosystem. Not something I'd get just to make a package when a default Python setup has recommended tools and everything I need to make and/or install packages.

16 hours ago by BerislavLopac

> a default Python setup has recommended tools and everything I need to make and/or install packages

No it doesn't. Neither setuptools nor pip are part of the standard library. Yes, they are installed by default in many cases, but they are still "extra tools".

13 hours ago by oblio

On the list of priorities for any package manager, download size should be #100 on the list.

If you've solved 1-99 and download size is an issue, you're in heaven already.

15 hours ago by cixinlui

72MB? On a development machine? Are you on dialup?

Poetry doesn't fragment the ecosystem. Unlike setuptools it uses pyproject.toml, which can be read by other tools, and is the correct way of storing package configuration.

A package built using Poetry is installable without Poetry in the exact same way as one built using setuptools.

10 hours ago by guggle

Last time I tried poetry it was so broken that it was not even usable. I may try again later.

10 hours ago by remram

I think it's the right direction, and I look forward to Poetry maturing. But right now it has a lot of gotchas, and I would only recommend it for people who are serious about dependency management/compliance/reproducibility.

See details upthread https://news.ycombinator.com/item?id=26739234

a day ago by santiagobasulto

+1 for poetry. It also includes deterministic dependency resolution with a lock file.

I just published a repo today[0] using Poetry and it didn’t take me more than 5 minutes. Poetry build && poetry publish

[0] https://github.com/santiagobasulto/hyper-inspector

21 hours ago by rahimnathwani

This looks cool. I had never come across rich[0] before.

[0] https://github.com/willmcgugan/rich

17 hours ago by ThePhysicist

The author mixes different things like linting and testing into the packaging process, which (IMHO) are not really part of making a package. The process is really much easier than this article makes it seem:

- Write a simple setup.py file.

- Generate a source or binary release by e.g. running "python setup.py sdist"

- You're done!

Adding a setup.py file is already enough to make your library pip-installable, so you could argue that this is a package already. The files generated by "setup.py" can also be pip-imported, so they are also packages. Now you might want to upload your package to a repository, for which there are different tools available. The simplest one being twine. Again, you just install it and run "twine upload -r dist/*" and your packages get uploaded to PyPi (it will ask for a username and password). So why complicate things?

17 hours ago by phodge

I don't see how your version is easier than the sequence of commands in the first few steps of the article, which is basically `pip install flit; flit init; flit publish`. Flit is just as easy to install as twine, but you save yourself the hassle of having to write a setup.py.

17 hours ago by ThePhysicist

Maybe I'm too old-fashioned then. But I like that you don't have any dependencies when using distutils/setuputils with a `setup.py` file, so if you don't distribute your code you're already done. I'm also not a fan of tools that are just wrappers around other tools.

16 hours ago by takluyver

Flit isn't (mostly) a wrapper around other tools - it has its own code to create and upload packages. This was one of the motivating cases for the PEPs (517, 518) defining a standard interface for build tools, so it's practical to make tools like this without wrapping setuptools.

(flit install does wrap pip, however)

15 hours ago by cixinlui

How does setuptools not count as a dependency?

If you've never run into setuptools compatibility problems, you've either been much luckier than me, or you haven't done much with Python packages.

Vanilla ubuntu used to come with a version of setuptools which didn't work for installing many recent packages.

a day ago by mikepurvis

I hadn't heard of flit, it does seem like it's not brand new on the scene, however it is primarily a single author, so expect a tool which is opinionated and for which the opinions may not necessarily reflect a broad consensus:

https://github.com/takluyver/flit/graphs/contributors

With a title like this, I'd be expecting to see an article describing the latest tools and recommendations from the PyPA, which are here:

https://packaging.python.org/tutorials/packaging-projects/

(In short, it's setup.cfg + pyproject.toml, `python3 -m build` to build, twine to upload.)

a day ago by carreau

Thomas is well know as one of the maintainer of IPython and Jupyter, and developed flit while working on the pep for pyproject.toml and the pip backend allowing things like python -m build.

Though `python -m build` only works _if_ you use something like flit or setup.py in the backend to do build the package and hence why you can set flit as a build-backend.

So yes, flit is one of the latest tool, and yes it is one of the things that push for the ability to use pyproject.toml+python3 -m build, you just seem to miss some subtleties of the toolchain.

20 hours ago by mikepurvis

The additional context is appreciated— it sounds like this tool is something which is likely to be supported long term, so that at least is good.

19 hours ago by erezsh

How does flit compare to poetry? The seem to be both doing the same thing, and in a very similar way.

16 hours ago by takluyver

Poetry does much more than Flit, like resolving dependencies, creating a lock file, and managing an environment where you can run your code. In particular, Poetry is meant to support application development (where you want to have a fixed version of your dependencies) as well as library development.

Flit is more aimed at being the simplest possible thing to put a package on PyPI, if that's all you want to do. It expects you to list dependencies in pyproject.toml manually.

16 hours ago by takluyver

Flit is definitely opinionated, and not suitable for every use case. As Carreau hinted, I think its bigger impact will be from the specifications, especially PEP 517, which it helped to prompt, rather than people using Flit directly. The specifications mean it's practical to make new tools which interoperate nicely, without having to either wrap setuptools or carefully imitate its behaviour.

a day ago by neolog

The PyPA recommendation is old and out of date. Using a system that doesn't manage constraints and hash-checked lockfiles is bad practice.

21 hours ago by icy

Honestly, I don't even bother "packaging" Python tools anymore. Just put it in all in a git repo, and pip can install using

    pip install git+https://myg.it/repo.git

18 hours ago by stinos

Also automatcially handles the issue of public vs private etc. Note this [1] recommends using zip files for speed, especially for larger repositories:

    pip install https://github.com/django/django/archive/master.zip

    pip install https://github.com/django/django/archive/stable/1.7.x.zip

[1] https://stackoverflow.com/questions/20101834/pip-install-fro...

20 hours ago by bloaf

I do that where I work because we have an internal git that is wide open, but packages from the outside world need to get whitelisted.

More controversially: With a few (<10) lines of code in your setup file, you can make an install-able module out of jupyter notebooks.

14 hours ago by amelius

How does this work for dependencies?

And what if some of the dependencies are incompatible with the versions already on your system?

19 hours ago by anewhnaccount2

But now packages on PyPI can't depend on it.

15 hours ago by nxpnsv

But very often, that's ok...

21 hours ago by perrygeo

Why not to make a Python package in 2021.

Even as a long time Python user, the packaging ecosystem feels fragmented and error-prone at best. Honestly, it sours the experience of writing Python code knowing you might eventually need to make it work on another computer.

13 hours ago by m463

I agree.

What I actually like is using the system package manager to install stuff. pacman -s or apt-get install

letting multiple packaging systems muck with your system is a recipe for hurt somewhere down the line.

11 hours ago by NavinF

Even more fragmentation? As a dev I'm not going to make packages for anything other than 'pip install' and maybe Ubuntu if you're lucky. I would also heavily discourage distros from shipping ancient buggy versions of the package, which is all distros are good for these days.

Packaging sucks.

18 hours ago by saila

I think for the vast majority of at least pure-Python projects you could just use poetry and upload your packages to PyPI or a private index. You can go from an empty directory to publishing a package within minutes with poetry (although, of course, you probably shouldn't).

21 hours ago by neolog

It's fragmented, but it doesn't need to be error prone if people use the good tools instead of the old low-level tools.

21 hours ago by jimmaswell

There would be no room for error if we just put the libraries in with the project as files instead of adding all these extra steps. Nobody seems to like this simple, bulletproof method anymore for some reason though.

17 hours ago by 1337shadow

That's exactly what a package manager does

14 hours ago by say_it_as_it_is

The only constant in the python community over the last twenty years are the criticisms about its packaging

18 hours ago by shadycuz

After reading this guide I would still recommend people to use this guide.

https://cjolowicz.github.io/posts/hypermodern-python-01-setu...

It's great for beginners and experts. As a long time python veteran it completely changed how I work with python for the better. It is lengthy with lots of optional steps. Just skip the ones you don't find relevant.

a day ago by rtpg

The recommendation to set up a Makefile on top of tox is a bit odd to be honest. Tox basically "just works", and you can do things like pass stuff to `pytest` by setting up `{posargs}` in the tox config (see [0])

I do feel like tox gets a bad rap despite having a complete feature set. I think a part of it is that the documentation is complete but not organized in the "Tox user"'s perspective, so for someone who shows up on a project using it, it's hard to figure out the quickstart( though the "general tips and tricks" page gets somewhere [1])

Anyways yeah, would not recommend Make over just leaning into tox more here.

EDIT: also, this article reminded me of how much I really dislike Github Action's configuration syntax. Just balls of mud on top of the Docker "ball of mud" strategy. I will re-iterate my belief that a CI system where the configuration system isn't declarative but just like..... procedural Lua will be a billion dollar business. CI is about running commands one after another! If you want declarative DAGs use Bazel

[0]: https://tox.readthedocs.io/en/latest/example/pytest.html?hig... [1]: https://tox.readthedocs.io/en/latest/example/general.html

19 hours ago by kristjansson

I’d argue the counterpoint actually: Writing Makefile targets for common commands significantly improves usability and ergonomics, especially when they follow common idioms (make test, make build, make install, ...).

The recipes for each target describe not only how a project intends to run each tool, but which tools it intends to run. Instead of having to know that this project runs tests under tox, while that one runs only under pytest, we can run ‘make test’ in each, and count on the recipe doing the Right Thing.

That consistency across projects makes it much easier for someone to get started on a new project (or to remember how the pieces fit together on your own project from a few months ago)

16 hours ago by rtpg

For me it’s important for a testing command to be able to receive parameters at runtime (for example tox test —- —pdb) , is it possible to do that with make in general? I never knew how.

I generally agree with your sentiment, though. I’m usually limiting myself to Python stuff so don’t have much exposure to make, it’s always felt like a less powerful task runner than other stuff

16 hours ago by karlicoss

IMO as long as there is a healthy CI pipeline which people can check to see how to build/test things, ultimately it doesn't matter much.

Daily digest email

Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.