Switching to modules causes LoadLibrary to fail in some python extensions due to MAX_PATH length

[carpenterjc]

🐞 bug report

Affected Rule

Its within the pip extension.

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse( ...
)

Is this a regression?

This works without modules, but doesn't now modules are enabled.

Description

Using the module extension to download whl files adds significantly to the path.
rules_python~~pip~example_pip_312_charset_normalizer
vs:
example_pip_charset_normalizer
when running on windows with --nobuild_python_zip you can easily exceed MAX_PATH, and any library which uses extensions will fail to load the extensions.

🔬 Minimal Reproduction

Reproduction attached:
pip_path_length.zip

Run: bazel build //example/... you get the following error:

INFO: Invocation ID: 19b20afe-a6fa-4573-bac9-f74bccb40807
INFO: Analyzed 3 targets (85 packages loaded, 3896 targets configured).
ERROR: D:/bazel_bugs/pip_path_length/example/BUILD.bazel:19:8: Executing genrule //example:run failed: (Exit 1): sh.exe failed: error executing Genrule command (from target //example:run) C:\Program Files\Git\bin\sh.exe -c ... (remaining 1 argument skipped)
Traceback (most recent call last):
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\rules_python~~pip~example_pip_312_requests\site-packages\requests\compat.py", line 11, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\_main\example\test.py", line 2, in <module>
    import requests
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\rules_python~~pip~example_pip_312_requests\site-packages\requests\__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\rules_python~~pip~example_pip_312_requests\site-packages\requests\exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\rules_python~~pip~example_pip_312_requests\site-packages\requests\compat.py", line 13, in <module>
    import charset_normalizer as chardet
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\rules_python~~pip~example_pip_312_charset_normalizer\site-packages\charset_normalizer\__init__.py", line 26, in <module>
    from .api import from_bytes, from_fp, from_path, is_binary
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\rules_python~~pip~example_pip_312_charset_normalizer\site-packages\charset_normalizer\api.py", line 7, in <module>
    from .cd import (
  File "c:\dev\_bazel\x3mr6iom\execroot\_main\bazel-out\x64_windows-opt-exec-st-d57f47055a04\bin\example\a_moderatly_long_tool_path.exe.runfiles\rules_python~~pip~example_pip_312_charset_normalizer\site-packages\charset_normalizer\cd.py", line 16, in <module>
    from .md import is_suspiciously_successive_range
ImportError: DLL load failed while importing md__mypyc: The filename or extension is too long.
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.440s, Critical Path: 0.93s
INFO: 7 processes: 6 internal, 1 local.
ERROR: Build did NOT complete successfully

You can open up the directories and you can see its trying to open a path greater than MAX_PATH.

To see what its old behaviour: bazel build //example/... --noenable_bzlmod

🌍 Your Environment

Operating System:

  
WINDOWS 11
  

Output of bazel version:

  
7.5.0
  

Rules_python version:

  
1.1.0
  

Anything else relevant?

We enable build --nobuild_python_zip on windows in our repo because the performance without is enabled it too slow. We use python to process many GB of assets using the data construct, zipping that all up is unfeasible.

python/cpython#126929

Note: if you comment out:
https://github.com/bazelbuild/rules_python/blob/e7d2f09394dd14816310c4c661d2fefab33b2b1b/python/private/python_bootstrap_template.txt#L55-L56
then we are presented with the UNC paths and LoadLibrary in python works.

[aignas]

Ah, this is Windows and the bootstrap=script does not work yet. The plan with #2521 is to first stabilize it on UNIX and then we could potentially look into reimplementing it in native code - #2500. Or maybe it can be done in powershell or something that all Windowses can understand?

However, it is likely quite far out into the future, I think.

The work in the #2156 could also help with this, but so far we haven't got a good solution for that yet that would cover all cases and it would depend on the bootstrap=script work being ported to Windows as well.