Publish cp313t (free-threaded) wheels

[ngoldbaum]

Currently pip install libcst will do a from-source build on free-threaded 3.13.0:

goldbaum at Nathans-MBP in ~/Documents/cryptography on main!
± pip install libcst
Collecting libcst
  Downloading libcst-1.5.0.tar.gz (772 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 773.0/773.0 kB 22.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyyaml>=5.2 (from libcst)
  Using cached PyYAML-6.0.2-cp313-cp313t-macosx_14_0_arm64.whl
Building wheels for collected packages: libcst
  Building wheel for libcst (pyproject.toml) ... done
  Created wheel for libcst: filename=libcst-1.5.0-cp313-cp313t-macosx_14_0_arm64.whl size=2027094 sha256=ec7f1a08268640717734e28b8454ca23043112fb2f002b46694494c82cf6b6bf
  Stored in directory: /Users/goldbaum/Library/Caches/pip/wheels/af/db/82/f05ed1829e60f978a9af19520c78cb52df1bed762e579d7f84
Successfully built libcst
Installing collected packages: pyyaml, libcst
Successfully installed libcst-1.5.0 pyyaml-6.0.2

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip

goldbaum at Nathans-MBP in ~/Documents/cryptography on main!
± python -VV
Python 3.13.0 experimental free-threading build (main, Nov  5 2024, 16:45:19) [Clang 16.0.0 (clang-1600.0.26.3)]

The build succeeds, but it probably shouldn't, and immediately segfaults if I try to import anything that imports the native module:

$  python
Python 3.13.0 experimental free-threading build (main, Nov  5 2024, 16:45:19) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import libcst.codegen.gen_matcher_classes as matcher_codegen
[1]    33127 segmentation fault  python

I think the first step is likely going to be updating the PyO3 dependency to 0.23 once that comes out (hopefully this week). Since the crate is at PyO3 0.20 right now, updating to 0.21 and 0.22 first will likely help.

[zsol]

#1180 already updates pyo3 to 0.22, so anyone taking on the work for 0.23 should use that as the base probably

[ngoldbaum]

Thanks! After building with that PR I'm able to get some more useful compiler errors and warnings:

warning: unused imports: `DeflatedAsynchronous as Asynchronous` and `DeflatedFormattedStringText as FormattedStringText`
  --> libcst/src/nodes/mod.rs:55:29
   |
55 | ...   DeflatedArg as Arg, DeflatedAsynchronous as Asynchronous, Deflated...
   |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
66 | ...   DeflatedFormattedStringText as FormattedStringText, DeflatedFrom a...
   |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: unused import: `DeflatedImportStar as ImportStar`
  --> libcst/src/nodes/mod.rs:89:9
   |
89 |         DeflatedImportStar as ImportStar, DeflatedSemicolon as Semicolon,
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: use of deprecated trait `pyo3::IntoPy`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
   --> libcst/src/nodes/traits.rs:121:46
    |
121 |     use pyo3::{types::PyAny, types::PyTuple, IntoPy, PyObject, PyResult...
    |                                              ^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: use of deprecated trait `pyo3::IntoPy`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:32:16
   |
32 |     use pyo3::{IntoPy, PyErr, PyErrArguments, Python};
   |                ^^^^^^

warning: use of deprecated associated function `pyo3::types::PyModule::import_bound`: renamed to `PyModule::import`
    --> libcst/src/nodes/expression.rs:2539:44
     |
2539 |                     let libcst = PyModule::import_bound(py, "libcst")?;
     |                                            ^^^^^^^^^^^^

warning: use of deprecated method `pyo3::types::IntoPyDict::into_py_dict_bound`: renamed to `IntoPyDict::into_py_dict`
    --> libcst/src/nodes/expression.rs:2551:22
     |
2551 |                     .into_py_dict_bound(py);
     |                      ^^^^^^^^^^^^^^^^^^

warning: use of deprecated associated function `pyo3::types::PyModule::import_bound`: renamed to `PyModule::import`
    --> libcst/src/nodes/expression.rs:2575:44
     |
2575 |                     let libcst = PyModule::import_bound(py, "libcst")?;
     |                                            ^^^^^^^^^^^^

warning: use of deprecated method `pyo3::types::IntoPyDict::into_py_dict_bound`: renamed to `IntoPyDict::into_py_dict`
    --> libcst/src/nodes/expression.rs:2596:22
     |
2596 |                     .into_py_dict_bound(py);
     |                      ^^^^^^^^^^^^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
   --> libcst/src/nodes/traits.rs:138:21
    |
138 |             Ok(self.into_py(py))
    |                     ^^^^^^^

warning: use of deprecated associated function `pyo3::types::PyTuple::new_bound`: renamed to `PyTuple::new`
   --> libcst/src/nodes/traits.rs:173:25
    |
173 |             Ok(PyTuple::new_bound(py, converted).into())
    |                         ^^^^^^^^^

error[E0599]: no method named `into_py` found for struct `PyTuple` in the current scope
   --> libcst/src/nodes/traits.rs:179:21
    |
179 |             Ok(self.into_py(py))
    |                     ^^^^^^^
    |
help: there is a method `into` with a similar name, but with different arguments
   --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/convert/mod.rs:450:5
    |
450 |     fn into(self) -> T;
    |     ^^^^^^^^^^^^^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
   --> libcst/src/nodes/traits.rs:185:21
    |
185 |             Ok(self.into_py(py))
    |                     ^^^^^^^

error[E0599]: no method named `into_py` found for reference `&T` in the current scope
   --> libcst/src/nodes/traits.rs:194:21
    |
194 |             Ok(self.into_py(py))
    |                     ^^^^^^^
    |
    = help: items from traits can only be used if the type parameter is bounded by the trait
help: there is a method `into` with a similar name, but with different arguments
   --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/convert/mod.rs:450:5
    |
450 |     fn into(self) -> T;
    |     ^^^^^^^^^^^^^^^^^^^
help: the following trait defines an item `into_py`, perhaps you need to restrict type parameter `T` with it:
    |
191 |         T: AsRef<PyAny> + IntoPy,
    |                         ++++++++

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:63:47
   |
63 |                     ("message", e.to_string().into_py(py)),
   |                                               ^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:64:37
   |
64 |                     ("lines", lines.into_py(py)),
   |                                     ^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:65:45
   |
65 |                     ("raw_line", (line + 1).into_py(py)),
   |                                             ^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:66:40
   |
66 |                     ("raw_column", col.into_py(py)),
   |                                        ^^^^^^^

warning: use of deprecated method `pyo3::types::IntoPyDict::into_py_dict_bound`: renamed to `IntoPyDict::into_py_dict`
  --> libcst/src/parser/errors.rs:68:18
   |
68 |                 .into_py_dict_bound(py);
   |                  ^^^^^^^^^^^^^^^^^^

warning: use of deprecated associated function `pyo3::types::PyModule::import_bound`: renamed to `PyModule::import`
  --> libcst/src/parser/errors.rs:70:31
   |
70 | ...   PyModule::import_bound(py, "libcst").expect("libcst cannot be impo...
   |                 ^^^^^^^^^^^^

warning: use of deprecated associated function `pyo3::PyErr::from_value_bound`: renamed to `PyErr::from_value`
  --> libcst/src/parser/errors.rs:71:24
   |
71 |                 PyErr::from_value_bound(
   |                        ^^^^^^^^^^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:85:42
   |
85 |                 ("message", self.message.into_py(py)),
   |                                          ^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:86:38
   |
86 |                 ("lines", self.lines.into_py(py)),
   |                                      ^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:87:44
   |
87 |                 ("raw_line", self.raw_line.into_py(py)),
   |                                            ^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:88:48
   |
88 |                 ("raw_column", self.raw_column.into_py(py)),
   |                                                ^^^^^^^

warning: use of deprecated method `pyo3::types::IntoPyDict::into_py_dict_bound`: renamed to `IntoPyDict::into_py_dict`
  --> libcst/src/parser/errors.rs:90:14
   |
90 |             .into_py_dict_bound(py)
   |              ^^^^^^^^^^^^^^^^^^

warning: use of deprecated method `pyo3::IntoPy::into_py`: `IntoPy` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23/migration) for more information.
  --> libcst/src/parser/errors.rs:91:14
   |
91 |             .into_py(py)
   |              ^^^^^^^

For more information about this error, try `rustc --explain E0599`.
warning: `libcst` (lib) generated 24 warnings

It looks like almost everything is about the introduction of IntoPyObject, which should hopefully be straightforward to handle.

Anyone trying to do this with a free-threaded Python may run into the fact that both pip install hatch and pip install jupyter are broken right now for free-threaded Python, so you'll need to install a slightly hacked version of hatch that doesn't bring in python-zstandard at install time and comment out jupyter and nbsphinx in the libcst pyproject.toml.

[wwnew]

Hello，is cp313t available now？

[ngoldbaum]

Hello，is cp313t available now?

Not yet but it looks like the update to PyO3 0.22 was merged so this is unblocked. I'll take a look at this soon.

[ngoldbaum]

Now that PyO3 has IntoPyObject, the TryIntoPy trait that LibCST uses can probably be ripped out. It might require less surgery to just rework it to use IntoPyObject under the hood and then leave it to future work to remove TryIntoPy in a piecemeal manner.

[ngoldbaum]

Unfortunately because of pypa/hatch#1801 both CI and contributors who want to build libcst will need to install a patched version of python-zstandard to get past a build issue. This will only happen on the free-threaded build so we can add some documentation to help out. Please let me know if that is problematic for some reason.

[ngoldbaum]

Now that #1289 is merged it looks like work on officially adding support and tests is continuing in #1295.

[ngoldbaum]

I realized today that previous attempts to run the LibCST tests using hatch run tests were using a GIL-enabled interpreter and hatch doesn't yet support creating environments with the free-threaded build (pypa/hatch#1931).

I managed to build LibCST in a virtualenv and run the tests without using hatch with the following commands:

pip install git+https://github.com/hynek/argon2-cffi-bindings@refs/pull/70/head --no-build-isolation
pip install setuptools setuptools_rust setuptools_scm wheel
python -m pip install -e ".[dev]" --no-build-isolation
python -m libcst.tests

This works around a build error in argon2-cffi-bindings, otherwise we could still use build isolation.

@zsol are you ok with adding a CI job that runs the libcst tests using a free-threaded interpreter set up like that, without using hatch?

[zsol]

Yeah sure, let's do it. We could also just have the CI job not install jupyter and then we won't need the argon PR and can keep build isolation