Run pants under free threaded python

[tobni]

Switches Pants' own runtime to free-threaded CPython and makes the engine and
Pants' Python code thread-safe without the GIL.

This has been validated as running without error in my companies CI, and nets great speedups.

Most important callouts:

• Keep a persistent per-thread PyThreadState in the executor. Without
  this, every Python::attach cycle tears down the thread state, and on 3.14t
  that abandons the thread's mimalloc heaps. Created per tokio thread, torn down in on_thread_stop (blocking
  threads are recycled, so the state must not leak per thread creation).
• Makes run-scoped internals thread-safe: rule_visitor, exception_sink, logging, build_root, run_tracker.

I used #23437 as a launching point.

I have on speculation set this as shipping in 2.34.x, as I believe 2.33 is too far along the dev-cycles. But I also do not see any reason to delay further.

Blocks pantsbuild/scie-pants#541

Edit:
This branch now serves as proof of green CI of the accumulation of changes, and will be rebased as its parts are accepted incrementally:

• Free-threaded: Upgrade psutil #23487
• Free-threaded: Export free threaded site packages #23488
• Free-threaded: Engine groundwork for performant free-threading #23489
• Free-threaded: Drop hdrhistorgram python dependency #23490
• Free-threaded: Thread safe internals #23491
• Free-threaded: Native, thread-safe instance memoization #23492
• Free-threaded: Abiflags aware build artifacts #23497
• Free-threaded: Pex-style ABI interpreter constraints via extras #23514

Edit2:

Now that all pre-requisites are in, this PR contains CI/CD changes and the new PANTS_FREE_THREADED option.
Current proposal for consumer UX is

Consumer	pants_free_threaded	Resolves to	Bootstrap warning
scie-pants - not updated	absent / false	cp314 (GIL 3.14)	-
scie-pants - not updated	true	cp314 (GIL 3.14)	"update scie-pants". Old launcher can't honor the field
scie-pants - updated	absent (default)	cp314t (free-threaded)	-
scie-pants - updated	false	cp314 (GIL 3.14)	- (escape hatch)
scie-pants - updated	true	cp314t (free-threaded)	-
Fat scie - …-cp314	true	GIL 3.14	"no effect on a fat scie; download the binary you want"
Fat scie - …-cp314t	false	free-threaded	"no effect on a fat scie; download the binary you want"

This PR is coupled to pantsbuild/scie-pants#541.

[cburroughs]

(Some adjacent or related issues #22790 #22946 #15840 #21518 so I don't need to search for them again.)

@tobni As I understand it there isn't a stable ABI for free threaded until 3.15. Having dug in to do this work, do you have a sense of how much of a problem or risk that is (or is not)?

[tobni]

@cburroughs Risk is close to none afaik, it is mostly a pyo3 compat (meaning we couldnt build it) issue. If we are allowed to assume pants is ran as a scie we defined, the internal 3rd party lockfile does the rest. I think that is ok, building a pants dist and targeting a different python would be highly special usecase.

If a plugin uses native deps in rule code that could bite them iff there is no 3.14t wheels for them.

[sureshjoshi]

Pretty substantial (and consequential) change - so I tacked on a bunch of reviewers

[tobni]

Once #23482 is in this will be rebased and the pex binary will again be stripped.

Edit: Done.

[tdyas]

General review comments:

1. This PR should probably be split into distinct components to make review easier. For example, the LockMap change and the related memoization changes can be a standalone PR.

2. Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

[tobni]

Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

We could vendor release versions with t-suffixes to make it target free-threaded? I'm doubtful of the value to be honest, but that'd be simple enough implementation wise (but requires CI/CD work).

I'd prefer opt-out though. I dont want to treat this as a scary change when it isnt, but I will also not claim 0 impact across all plugin code.

For the first comment: Yes, easily split. Each commit ~stands on its own already so I can do that branch-ectomy with ~0 effort.

[tdyas]

We could vendor release versions with t-suffixes to make it target free-threaded? I'm doubtful of the value to be honest, but that'd be simple enough implementation wise (but requires CI/CD work).

So like everything in software engineering, my view is "it depends." My preference here is that we not introduce a one way version gate on users when there is no one dedicated to fixing issues for them in a timely manner. I'd rather avoid users being potentially stuck on 2.32 or lower versions with no way to consume all of the other fixes Pants has in 2.33 unrelated to free-threading.

So how much more complex would it be to release a free-threading version of Pants alongside the GIL version? The main artifact is the engine's shared library (native_engine.so), can we just build it twice? (once with 3.14 GIL and once with 3.14 free-threaded)?

My view really depends on just how much additional work the CI/CD & release changes would be.

I'd prefer opt-out though. I dont want to treat this as a scary change when it isnt, but I will also not claim 0 impact across all plugin code.

Opt-out is fine. The concern for me is having the ability to control adoption of the change with some sort of flag versus forcing users to stay on a prior versions because there is no escape hatch from this change. I submit that Incremental adoption with an escape hatch makes for a better user experience.

[tobni]

So like everything in software engineering, my view is "it depends." My preference here is that we not introduce a one way version gate on users when there is no one dedicated to fixing issues for them in a timely manner. I'd rather avoid users being potentially stuck on 2.32 or lower versions with no way to consume all of the other fixes Pants has in 2.33 unrelated to free-threading.

You used specific version numbers so I'll just clarify that I do not think this is a 2.33.x change but rather a 2.34.x feature. The point is of course still valid. I'm not sure I agree that this is a strong case for pants and free threading though, we have a patch-version series for fixes and we vendor a binary. We can choose to only release free threading and address eventual bugs. Because it will be bugs, users being "stuck" will be on some flaky CI-runs surfacing races we missed, most likely. Unless of course there is ABI-incompatible user plugins out there.

So how much more complex would it be to release a free-threading version of Pants alongside the GIL version? The main artifact is the engine's shared library (native_engine.so), can we just build it twice? (once with 3.14 GIL and once with 3.14 free-threaded)?

I dont know exactly, the main effort is figuring that out, and is the counterweight in terms of value I was refering to.

[tobni]

Now I know some more - It'd have to be a separate pants.toml field to not mess with PEP440 compat, but otherwise it goes:

1. One release, two asset sets. The GitHub release has to publish both scies per platform eg: pants.2.34.0-linux-x86_64.scie and a free threaded sibling whose naming is TBD.
2. Selection is a scie-pants config knob. scie-pants reads from pants.toml. scie-pants maps the pants.toml field to the asset-name suffix and fetches.

Essentially a question of if we're okay to publish 2x scies.

[tobni]

General review comments:

1. This PR should probably be split into distinct components to make review easier. For example, the LockMap change and the related memoization changes can be a standalone PR.
2. Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

@tdyas 1 is now done. See the list in the pull request description.

[svenier]

I just want to point out that not everyone uses a scie. We use the big monolithic binaries that are shipped in the release. So please make sure to build an ft version of that as well.

[cburroughs]

I just want to point out that not everyone uses a scie. We use the big monolithic binaries that are shipped in the release. So please make sure to build an ft version of that as well.

To make sure I have this right, what from https://github.com/pantsbuild/pants/releases/tag/release_2.32.1 do you use? Colloquially If I head "big monolithic binaries" I would think of the scies there (ex: pants.2.32.1-cp314-linux_x86_64), while scie-pants today uses the .pex files (ex : pants.2.32.1-cp314-linux_x86_64.pex ).

[ndellosa95]

We actually don't need to ship multiple scies/versions I think, the free-threaded Python builds support re-enabling the GIL with the PYTHON_GIL environment variable https://docs.python.org/3/howto/free-threading-python.html#the-global-interpreter-lock-in-free-threaded-python

[tobni]

We actually don't need to ship multiple scies/versions I think, the free-threaded Python builds support re-enabling the GIL with the PYTHON_GIL environment variable https://docs.python.org/3/howto/free-threading-python.html#the-global-interpreter-lock-in-free-threaded-python

That doesnt cover ABI mismatch in user-written plugins third party deps, which is imo the strongest argument for a non free-threaded variant.

That said, PYTHON_GIL does work and will launch the pants daemon with GIL enabled under free-threaded. It's a bit fickle though, the daemon doesnt pick it up mid run.

[svenier]

To make sure I have this right, what from https://github.com/pantsbuild/pants/releases/tag/release_2.32.1 do you use? Colloquially If I head "big monolithic binaries" I would think of the scies there (ex: pants.2.32.1-cp314-linux_x86_64), while scie-pants today uses the .pex files (ex : pants.2.32.1-cp314-linux_x86_64.pex ).

Yes, sorry. I mis-spoke. We don't use scie-pants.

[tobni]

All pre-requisite pull requests are now in. That means this PR now contains the switch and CI/CD change. Since @cburroughs cut 2.33.x I believe there's not much else than deciding on how we ship it.

I don't think publishing multiple scies is a problem. But historically changing the minimum scie-pants version has been perceived as disruptive to users and we have tried not to do so "often". (Making it not so disruptive is the impetuous for #22946) So however we decide on opt in/out, I think it is important than users on old scie-pants versions get something that works (that is, one of the scies follows the current naming scheme).

So the UX I am proposing herein is thus:

• scie-pants consumers will have to update to get the free-threaded variant. This has to be the case, because the scie-pants manifest need to include the "t" abiflag to resolve correctly, and the launcher needs to pick it. If they dont, the launcher will land on GIL-bound 3.14. If free-threaded turns out to not work for them, they can set pants_free_threaded=false in either toml or env to get scie-pants to download the gil bound pex + interpreter.
• For fat scies, nothing needs doing. You download the binary you want, we publish both.

We can then decide which future version we stop building wheels and binaries, because that is when backwards-compat breaks. We can choose to do that never.

Since skimming release notes might lead to users thinking PANTS_FREE_THREADED=true is the only thing to configure to get free-threaded I propose we warn at bootstrap time for scie environs.

Consumer	pants_free_threaded	Resolves to	Bootstrap warning
scie-pants - not updated	absent / false	cp314 (GIL 3.14)	-
scie-pants - not updated	true	cp314 (GIL 3.14)	"update scie-pants". Old launcher can't honor the field
scie-pants - updated	absent (default)	cp314t (free-threaded)	-
scie-pants - updated	false	cp314 (GIL 3.14)	-
scie-pants - updated	true	cp314t (free-threaded)	-
Fat scie - …-cp314	true	GIL 3.14	"no effect on a fat scie; download the binary you want"
Fat scie - …-cp314t	false	free-threaded	"no effect on a fat scie; download the binary you want"

Off-by-default when scie-pants is updated is the alternative. I like it less because I want to shed the GIL and keep the performance landscape un-parametrized. Free-threaded is so much faster.

[ndellosa95]

That doesnt cover ABI mismatch in user-written plugins third party deps, which is imo the strongest argument for a non free-threaded variant.

Don't those plugins need to be updated anyway whenever the Python version Pants uses is upgraded? Though I'm guessing most third-party deps with native components provide wheels for multiple GIL-bound Python versions but not necessarily for free-threaded Python. So, fair!

[tobni]

I've decided to additionally open #23514, which should be merged before this.