Please drop setuptools_scm

This kind of release process is a huge nuisance for those of us packaging software downstream. Not being able to use PyPI tarballs or even GitHub tarballs is requires us to go well out of our way to accomodate your package. Please just conform to normal Python packaging norms.

Just set the SETUPTOOLS_SCM_PRETEND_VERSION environment variable.

https://github.com/archlinux/svntogit-community/blob/19f4a531a1cc0b1dedc7aa2a2911d79a610b3caf/trunk/PKGBUILD#L18

I don't think that's a very good workaround for not having good packaging practices.

Respectfully, I disagree. I understand that it is annoying but that environment variable is there for just that, to override the setuptools_scm version. We could document this better, but overall I don't really see an issue with this approach besides being slightly annoying to packagers, it has absolutely nothing to do with good or bad packaging practices.

being slightly annoying to packagers

has absolutely nothing to do with good or bad packaging practices

Is this not a contradiction? There's little reason to rock the boat. Just ship with standard Python packaging tools and it'll work better for everyone.

How is something being slightly annoying to packagers a bad packaging practice??
First of all, "packaging practices" are executed by packagers, not by upstreams. All you can argue is that the upstream is making the packagers' lives a harder, which can be a fair point, but generally is a tradeoff, so it is up to the upstream to choose how much of their workflow they are willing to sacrifice to make your life easier.

Upstreams are free to manage their projects as they want, but I think they should be careful to not make the workflow incompatible, or unreasonable, with popular packaging workflow requirements. This definitely not the case here, I would agree with you if this did actually make it impossible or unreasonable for you to package the project, but that is just not true.

And please note, I am a packager myself. There's little point trying to argue in name of packagers(TM) here.

Just ship with standard Python packaging tools and it'll work better for everyone.

Just wait until you hear about PEP 517....

Just wait until you hear about PEP 517....

Oh, we all know Python is a nightmare. We should all do our part by doing our best to deal with it by making things as uncomplicated and predictable as possible.

And please note, I am a packager myself.

Debian agrees with me:

https://wiki.debian.org/UpstreamGuide#Out-of-VCS_Builds

So if packagers are concerned, my argument has a majority. I've said my piece. I think that you are making a poor packaging choice as an upstream. If you won't change it, there's nothing I can do about it.

I would hardly call SETUPTOOLS_SCM_PRETEND_VERSION "ergonomic to use", so it would be nice if we didn't need this sort of awkward workaround.

On that note, in search of a solution I wrote to the git mailing list fairly recently: https://public-inbox.org/git/4f65f02c-1d16-aa2c-3e7b-28d807b9ebe9@archlinux.org/T/

It's highly annoying that github archives are rendered either entirely useless or obscure and difficult to work with, depending on which particular variety of broken setuptools plugin any given project is using. I'll admit that one could do much worse than setuptools_scm, but that still isn't especially ideal IMO.

And PyPI sdist are often not an option as they more often than not don't distribute tests... (This is not a comment on this specific project.)

Oh, we all know Python is a nightmare. We should all do our part by doing our best to deal with it by making things as uncomplicated and predictable as possible.

We do, but things are very complicated.

Debian agrees with me:

wiki.debian.org/UpstreamGuide#Out-of-VCS_Builds

So if packagers are concerned, my argument has a majority. I've said my piece. I think that you are making a poor packaging choice as an upstream. If you won't change it, there's nothing I can do about it.

That at most only partially agrees with you. The VCS or code repository is not required to be present, it is the one of the resorts when detecting the version.
I understand your annoyance, but it is not really not up to you, or me for that matter, to decide what they are willing to compromise at. As a packager you need to accept when upstream are not willing to make the compromise you are looking for.
If you want to see my position as a lead maintainer, you can check out pypa/build#170. I get your sentiment here, I really do, I just don't think your attitude of "just conform", "this is incompatible" and "it'll work better for everyone" is correct.

I'll also add that one small annoyance for you as an upstream maintainer (e.g. having to bump the version in a file somewhere) is N, but package maintainers have to deal with N² annoying packages. It adds up for us, and doesn't add up for you.

pypa/build#170

Funny that you went from -2 to being a part of the problem.

@jaraco

I would hardly call SETUPTOOLS_SCM_PRETEND_VERSION "ergonomic to use", so it would be nice if we didn't need this sort of awkward workaround.

It sure would, but it is up to the maintainers, not you. As a packager I find this solution annoying, but acceptable, I just fix my PKGBUILD and go around my way.

I'll also add that one small annoyance for you as an upstream maintainer (e.g. having to bump the version in a file somewhere) is N, but package maintainers have to deal with N² annoying packages. It adds up for us, and doesn't add up for you.

I am very aware of the impact of this. It adds up, but it's a one time fix.

Funny that you went from -2 to being a part of the problem.

I am not the main maintainer of this repo, I have only tagged one release, it is not my workflow that this will be breaking.

You can wait for @jaraco to weigh in, but I doubt he'll agree to this, hence my first comment.

@jaraco

I don't think @jaraco expected to receive pushback for this, and I would hope that the knee-jerk reaction to dismiss these complaints out of hand isn't what we'll get. But @jaraco, please weigh carefully whether or not the perceived benefits of using scmtools-scm, which is almost universally hated among packagers, are worth the trade-off.

I was never trying to dismiss complaints here, my first comment should have been worded better, sorry. I was trying to provide you an immediate solution, you then escalated saying how this is bad packaging practices.
I am +1 on dropping setuptools_scm, but as I said, it is definitely not my call.

I appreciate you bringing this concern to bear. Unfortunately, I'll probably need more information to understand what's causing you problems. Since the request here applies to the hundreds of projects I maintain, I'd wish to move it to jaraco/skeleton, where these practices are maintained across those projects, but that's another org, so I can't and I'll just address the concerns here.

Not being able to use PyPI tarballs

You should be able to use PyPI tarballs.

~ $ pip-run --no-binary importlib_resources importlib_resources
Collecting importlib_resources
  Downloading importlib_resources-5.1.0.tar.gz (31 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: importlib-resources
  Building wheel for importlib-resources (PEP 517) ... done
  Created wheel for importlib-resources: filename=importlib_resources-5.1.0-py3-none-any.whl size=24554 sha256=9e6c5526f11d7b45fb761fa597bd2990d973c2dfecfb79bfaffbb7712e654521
  Stored in directory: /Users/jaraco/Library/Caches/pip/wheels/a0/82/c0/6455ddb59a10e136e92e902c63b8864c52c8a8e8146c377766
Successfully built importlib-resources
Installing collected packages: importlib-resources
Successfully installed importlib-resources-5.1.0
WARNING: You are using pip version 21.0; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/jaraco/.local/homebrew/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.
Python 3.9.1 (default, Jan  5 2021, 11:10:38) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Under what conditions are you unable to install PyPI tarballs?

Please just conform to normal Python packaging norms.

The setuptools_scm approach is a Python packaging norm that's evolved from best practices that go back to core features in setuptools from 2006 and that have been consolidated in the setuptools_scm under the PyPA umbrella as the blessed approach for leveraging SCM metadata in Python project development. I believe other PyPA projects include this behavior, even built-in (though I'm unfamiliar). I acknowledge that it has some shortcomings (the main one being that Github tarballs don't bring enough metadata), but Github checkouts are still supported:

~ $ pip-run git+https://github.com/python/importlib_resources@v5.0.1 -- -c pass
Collecting git+https://github.com/python/importlib_resources@v5.0.1
  Cloning https://github.com/python/importlib_resources (to revision v5.0.1) to /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-req-build-pmqcuum9
  Running command git clone -q https://github.com/python/importlib_resources /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-req-build-pmqcuum9
  Running command git checkout -q ecc2a24af948c55ecc88f0c3cfc4c3900c8f6246
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: importlib-resources
  Building wheel for importlib-resources (PEP 517) ... done
  Created wheel for importlib-resources: filename=importlib_resources-5.0.1-py3-none-any.whl size=23364 sha256=5fff94d19749d20910aeee76f8ee5e39504e530060591bc3191e1c6b1b5d3b8d
  Stored in directory: /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-ephem-wheel-cache-2oypcf6_/wheels/0f/33/ae/fa8051a526161d3dfd5c2f824a228aea20468ce132c34a3dfa
Successfully built importlib-resources
Installing collected packages: importlib-resources
Successfully installed importlib-resources-5.0.1

I'm heavily invested in setuptools_scm and the value it provides, and in my experience, the value far outweighs the limitations it imposes. The benefits include:

No need to maintain a manifest.in. Files included in the repo are included in the sdist. No opportunity to add to one and not the other.
Possible to create releases without a terminal. Simply tagging a commit in the repo, even through the web, creates a release. That means I can accept a pull request, tag it, and cut a release of the changes from my mobile while in bed. Super low friction.
Repo state and published releases state is kept in sync (it's not possible to have one version declared in the repository and a different version in the tags).
Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.
There's no need for any additional development tooling (bump2version, ...), tooling which itself needs to be installed, maintained, etc.

Removing setuptools_scm support from this project and the hundreds of others I maintain would add toilsome steps, redundant data, noisy commits, inconsistent states, and reduce the fidelity of the intermediate versions produced.

Can you explain more about what workflows are unmet by the current approach? Perhaps there are other approaches that would satisfy the need.

Under what conditions are you unable to install PyPI tarballs?

The issue is building from github tarballs, as is common for packagers to do. It is usually desired to fetch from the original source rather than a 3rd party service, but I'll let Drew elaborate on this if he wants.

Lot to unpack here.

You should be able to use PyPI tarballs.

Kind of, though I ran into issues. Crucially, you can't use the PyPI tarballs combined with a standard Python packaging procedure, nor the GitHub tarballs.

Note as well that no one uses pip for packaging Python stuff downstream. Working around the problem in pip definitely doesn't work for us. For one, we have a hard constraint on not accessing the network during the build process.

No need to maintain a manifest.in. Files included in the repo are included in the sdist. No opportunity to add to one and not the other.

Not that hard. I can offer some advice if you find this difficult.

Possible to create releases without a terminal. Simply tagging a commit in the repo, even through the web, creates a release. That means I can accept a pull request, tag it, and cut a release of the changes from my mobile while in bed. Super low friction.

Honestly this sounds irresponsible as hell. This leads to broken releases. Surely you've broken a release doing it this way and had to hastily ship a fix? Strong NACK to this as an advantage. I don't want you releasing things into my production systems from the comfort of your bed.

Repo state and published releases state is kept in sync (it's not possible to have one version declared in the repository and a different version in the tags).

This can also be dealt with. I have a script for doing releases that deals with this stuff.

Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.

???

There's no need for any additional development tooling (bump2version, ...), tooling which itself needs to be installed, maintained, etc.

No, just downstream workarounds which also have to be maintained.

I don't really want to engage with you on this discussion further. I'm not going to waste my time when I'm up against the effort of getting you to change your approach for "hundreds of projects". I don't think I could possibly make an argument which would overcome your change resistance when the change involved requires that much work.

@eli-schwartz

@eli-schwartz

It's highly annoying that github archives are rendered either entirely useless or obscure and difficult to work with, depending on which particular variety of broken setuptools plugin any given project is using.

In fact, there's a package that does exactly that — setuptools-scm-git-archive. This project doesn't use it but most of the other setuptools-scm-based libs do. Personally, I do this in all my projects and it works like a charm. @jaraco I recommend you add this into the skeleton, we already have it in the CherryPy repos if you want to see examples.

I'd like to draw attention once more to my proposal on the git mailing list: public-inbox.org/git/4f65f02c-1d16-aa2c-3e7b-28d807b9ebe9@archlinux.org/T

Just to clarify: setuptools-scm-git-archive does this but in a standardized fashion that's been a de-facto standard for many years already. So it'd be only right to keep using it. Also, many projects have an if-conditional or a try/except detecting the version similarly to what you've described — they attempt to get a specific build version from the metadata (via pkg_resources/importlib_metadata/static file/etc). Please also understand that version = attr: foomod.__version__ is something that is used by setuptools on build (when you run things pip install ... or setup.py sdist) but then in runtime one can grab the version a bit differently which is where you'd normally see those try/excepts.

I do think it would be nicer if the tool used to calculate git versions would override the existing version checked into git only when run from a git repo -- that is how non-python projects tend to do such things -- but the existing tools don't seem to support that.

Well, that's what setuptools-scm basically does. Once the project is installed or built, the metadata is static.

Thanks, the git-archive plugin does look very neat. I think it would work even better if the disclaimer

Note that it only works for archives of tagged commits (because git currently lacks a format option equivalent to git describe --tags).

got solved via the git mailing list proposal in question.

@missa112

@missa112 that is not related to this repo at all. But I think what you are looking for is https://cryptography.io/en/latest/installation.html#building-cryptography-on-linux. Also, for any further guidance, please ask in the PyCA community channels: https://cryptography.io/en/latest/community.html.

@FFY00 maybe this should be locked...

Drew, these kinds of comments are not helpful in any way. Please re-evaluate your attitude in this issue, and frankly OSS in general. Remember no-one here is required to give you support or adapt to your needs, maybe you need a refresher on the Apache 2 license... we try to that but it's best-effort and done in our free time, and this kind of behavior honestly only demotivates me, and I am pretty sure the other maintainers too.

With that said, I am sure if someone opens a pull request on https://github.com/jaraco/skeleton introducing setuptools-scm-git-archive, that will help this issue get resolved faster.

Sorry? I don't understand how what I said was rude or unhelpful or entitled. I was just suggesting that this issue wouldn't be locked on the basis that someone commented in the wrong place.

This was the culmination of all your behavior during this whole issue, your last comment was the last drop. I am now opting out of this issue, take care.

Suit yourself. OSS doesn't imply agreement.

@jaraco

@jaraco I recommend you add this into the skeleton

I am sure if someone opens a pull request on https://github.com/jaraco/skeleton introducing setuptools-scm-git-archive, that will help this issue get resolved faster.

This sounds like a reasonable proposal. I want to investigate what downsides, if any, there are to adopting that feature. The only other thing I'd like to see is an articulation of the motivation behind adopting it - what use-cases does it solve and why aren't those use-cases solved by another preferable technique (using sdists as published on PyPI or a repo clone). I requested this information above, but only got indications of problems downstream, not an actual description of what the problems are. I'm fairly confident there are real problems - I want to be sure to capture what they are and what conditions need to be maintained to alleviate them.

Considering some particular github users are prone to spam projects with their comments, I would have locked the thread long time ago. Lucky I am not a core here.

Such kind of tickets is why I enabled github discussions and moved some tickets there, allowing people to continue to engage in long conversations without adding noise to the issue tracker (or having to lock issues).

The underhanded personal insults are not called for. I came here with a legitimate interest in the project seeking a solution as a package maintainer. Your comment is out of line. Maybe this thread should be locked if that's what it's going to devolve into.

@jaraco

@jaraco

At the very least, it's an extra bit of metadata that prevents people from messing up if they use github archives.

I don't know what particular error @ddevault might have hit trying to use the sdist, maybe a traceback would help with forensics on that count... whatever the issue was, it was no doubt frustrating that trying to use github archives instead (a VERY common workaround for packaging problems) did not solve the issue and instead made it worse/more confusing.

So I'd posit that reducing the number of ways a user could end up with a problematic copy of the code, is at least one reason to go ahead with the git archive plugin.

reason to go ahead with the git archive plugin.

UPD: setuptools-scm v7+ bundles that plugin's behavior. But the text file template + gitattributes changes are still needed for it to function.

Navigation Menu

Repository navigation

Please drop setuptools_scm #213

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Please drop setuptools_scm #213

Description

Activity

FFY00 commented on Feb 9, 2021

ddevault commented on Feb 9, 2021

FFY00 commented on Feb 9, 2021

ddevault commented on Feb 9, 2021

FFY00 commented on Feb 9, 2021

ddevault commented on Feb 9, 2021

eli-schwartz commented on Feb 9, 2021

FFY00 commented on Feb 9, 2021

ddevault commented on Feb 9, 2021

FFY00 commented on Feb 9, 2021

ddevault commented on Feb 9, 2021

FFY00 commented on Feb 9, 2021

jaraco commented on Feb 9, 2021

FFY00 commented on Feb 9, 2021

ddevault commented on Feb 9, 2021

6 remaining items

webknjaz commented on Feb 10, 2021

eli-schwartz commented on Feb 10, 2021

missa112 commented on Feb 12, 2021

FFY00 commented on Feb 12, 2021

webknjaz commented on Feb 12, 2021

ddevault commented on Feb 12, 2021

FFY00 commented on Feb 12, 2021

ddevault commented on Feb 12, 2021

FFY00 commented on Feb 12, 2021

ddevault commented on Feb 12, 2021

jaraco commented on Feb 12, 2021

ssbarnea commented on Feb 12, 2021

ddevault commented on Feb 12, 2021

eli-schwartz commented on Feb 12, 2021

webknjaz commented on Dec 5, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions