Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f518072
fix(windows): don't require symlinks for the runtime venv
aignas Nov 15, 2026
056ae8c
ci: test examples/bzlmod on Windows without symlinks
aignas Nov 15, 2026
0a739bf
ci: temporarily run only the Windows jobs
aignas Nov 15, 2026
644e4be
Revert "ci: temporarily run only the Windows jobs"
aignas Nov 15, 2026
656966d
test: check the venv without symlinks in Python for Windows paths
aignas Nov 15, 2026
fdc4cb1
ci: fix the flags of the Windows job without symlinks
aignas Nov 15, 2026
e5d01cc
fix(windows): keep the read-only attribute of hard linked outputs
aignas Nov 15, 2026
22b7ee5
fix(windows): delete read-only files without clearing the attribute
aignas Nov 15, 2026
52e600e
refactor(windows): define the symlink fallback globals with the other…
aignas Nov 16, 2026
29437d6
fix(windows): link to the target of a symlink in the fallback
aignas Nov 16, 2026
c8a319a
fix(windows): don't copy over a venv entry that another process created
aignas Nov 16, 2026
0c86987
refactor(windows): set up the kernel32 prototypes only once
aignas Nov 16, 2026
a9d0b11
fix(windows): remove a dangling junction left by a cached venv
aignas Nov 16, 2026
8a43e51
test(windows): make the test without symlinks check what it claims
aignas Nov 16, 2026
e073dab
test: use the Windows constraint directly
aignas Nov 16, 2026
aadea93
docs: link the rules in the news entry
aignas Nov 16, 2026
4fb4ca9
test(windows): check that a dangling junction of a cached venv is rem…
aignas Nov 16, 2026
688af3e
docs(windows): note that copying a directory can still race
aignas Nov 16, 2026
52dc7ef
test(windows): don't hard link to the running interpreter in the cach…
aignas Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .bazelci/presubmit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,30 @@ tasks:
- "bazel build --vendor_dir=./vendor //..."
- "rm -rf ./vendor"

integration_test_bzlmod_windows_no_symlinks:
<<: *reusable_build_test_all
name: "examples/bzlmod: Windows, without symlinks"
working_directory: examples/bzlmod
platform: windows
bazel: 8.x

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

examples/bzlmod/.bazelversion specifies 9.x and sibling integration_test_*
jobs omit bazel:, whereas this job pins bazel: 8.x without a comment
explaining why.

Suggested fix: Remove bazel: 8.x so the job uses
examples/bzlmod/.bazelversion, or add an inline comment explaining why Bazel
8.x is needed here.

This is an agent-generated comment reviewed and approved by the user.

batch_commands:
# The last value wins, so this overrides the setting in .bazelrc to
# check that binaries work without the privilege to create symlinks.
- "echo startup --nowindows_enable_symlinks >> .bazelrc"
build_flags:
? "--keep_going"
# The CI user may be able to create symlinks, so force the fallback.
? "--action_env=RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1"
# Without symlinks, Bazel creates a junction instead of a copy in the
# runfiles tree for an output that isn't downloaded from the remote
# cache yet, and a junction to a file doesn't work.
? "--remote_download_all"
test_flags:
? "--keep_going"
? "--action_env=RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1"
? "--remote_download_all"
? "--test_env=RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1"
Comment on lines +407 to +419

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting --test_env only in test_flags invalidates the analysis cache
between bazel build and bazel test, forcing Buildkite to re-analyze all
targets.

Suggested fix: Add --test_env=RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1 to
build_flags and share the mapping with a YAML anchor:

    build_flags: &windows_no_symlinks_flags
      ? "--keep_going"
      # The CI user may be able to create symlinks, so force the fallback.
      ? "--action_env=RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1"
      # Without symlinks, Bazel creates a junction instead of a copy in the
      # runfiles tree for an output that isn't downloaded from the remote
      # cache yet, and a junction to a file doesn't work.
      ? "--remote_download_all"
      ? "--test_env=RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1"
    test_flags: *windows_no_symlinks_flags

This is an agent-generated comment reviewed and approved by the user.


integration_test_multi_python_versions_ubuntu_workspace:
<<: *reusable_build_test_all
<<: *common_workspace_flags
Expand Down
5 changes: 5 additions & 0 deletions news/3792.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
(binaries/tests) (Windows) {obj}`py_binary` and {obj}`py_test` no longer need
the privilege to create symlinks to run on Windows 10 1809, Windows Server
2019 or later. If creating a symlink fails, the venv that is created at runtime
uses hard links, junctions or copies instead
([#3792](https://github.com/bazel-contrib/rules_python/issues/3792)).
172 changes: 161 additions & 11 deletions python/private/python_bootstrap_template.txt
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,19 @@ IS_WINDOWS = os.name == "nt"
BIN_DIR_NAME = "bin" if not IS_WINDOWS else "Scripts"
LIB_DIR_NAME = "lib" if not IS_WINDOWS else "Lib"

# Windows only: set to False once creating a symlink fails, e.g. without the
# privilege to create symlinks, so that it isn't attempted for every file.
# Tests set the environment variable to use the fallback even when symlinks
# can be created.
_symlinks_supported = not (
IS_WINDOWS and os.environ.get("RULES_PYTHON_TESTING_DISABLE_SYMLINKS")
)

# The build of Windows 10 1809 and Windows Server 2019, which added
# FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE that the fallback for when
# symlinks can't be created needs to clean up.
_WINDOWS_FALLBACK_MIN_BUILD = 17763

# Windows APIs can be picky about slashes depending on the context,
# so convert to backslashes to avoid any issues.
if IS_WINDOWS:
Expand Down Expand Up @@ -441,11 +454,10 @@ print(site.getsitepackages(["{venv_src}"])[-1])
link_to = normpath(join(runfiles_root, link_to_rf_path))
venv_parent = dirname(venv_abs_path)
# A cached venv can leave this parent as a dangling link to an old EXTRACT_ROOT.
if os.path.islink(venv_parent):
try:
os.stat(venv_parent)
except FileNotFoundError:
os.unlink(venv_parent)
# islink() is False for a junction on Windows, but lexists() doesn't
# follow it, and unlink() removes it.
if os.path.lexists(venv_parent) and not os.path.exists(venv_parent):
os.unlink(venv_parent)
os.makedirs(venv_parent, exist_ok=True)
_symlink_exist_ok(from_=venv_abs_path, to=link_to)

Expand Down Expand Up @@ -599,7 +611,7 @@ def execute_file(python_program, main_filename, args, env, runfiles_root,
if delete_dirs:
for delete_dir in delete_dirs:
print_verbose("cleanup: rmtree:", delete_dir)
shutil.rmtree(delete_dir, True)
_rmtree(delete_dir)
sys.exit(ret_code)

def _run_execv(python_program, argv, env):
Expand All @@ -617,13 +629,151 @@ def _run_execv(python_program, argv, env):
raise

def _symlink_exist_ok(*, from_, to):
"""Creates a symlink, or on Windows, a hard link, junction or copy if that's not possible."""
global _symlinks_supported
# On Windows, symlinks have to be told whether they're
# pointing to a file or directory. Python is supposed to auto-detect
# this, but in practice, this doesn't reliably happen.
is_dir = os.path.isdir(to)
if _symlinks_supported:
try:
os.symlink(to, from_, target_is_directory=is_dir)
return
except FileExistsError:
return
except OSError as e:
if not IS_WINDOWS:
raise
if sys.getwindowsversion().build < _WINDOWS_FALLBACK_MIN_BUILD:
raise OSError(
"Creating symlinks failed, which is required before Windows 10 "
"1809 and Windows Server 2019: {}".format(e)
) from e
print_verbose("symlink failed, falling back to links or copies:", e)
_symlinks_supported = False

if os.path.lexists(from_):
return
# The target can itself be a symlink, e.g. when Bazel could create the
# runfiles symlinks. A hard link to a symlink is also a symlink, so link to
# the file or directory that it points to instead.
to = os.path.realpath(to)
if is_dir:
_link_dir_windows(from_, to)
else:
_link_file_windows(from_, to)

def _link_file_windows(from_, to):
# Hard links don't need any privileges, but only work within the same
# volume, so fall back to copying the file.
try:
# On Windows, symlinks have to be told whether they're
# pointing to a file or directory. Python is supposed to auto-detect
# this, but in practice, this doesn't reliably happen.
os.symlink(to, from_, target_is_directory=os.path.isdir(to))
os.link(to, from_)
# Another process sharing RULES_PYTHON_EXTRACT_ROOT created it first.
except FileExistsError:
pass
return
except OSError as e:
print_verbose("hard link failed, copying:", to, "error:", e)
shutil.copy2(to, from_)
Comment on lines +661 to +676

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Windows, os.link(to, from_) calls CreateHardLinkW without dereferencing
symlinks, hard-linking the reparse point itself when to is a .runfiles
symlink. That leaves sys.executable as a symlink, failing
run_binary_without_symlinks_test on the Default: Windows CI jobs and
breaking relative symlinks.

Suggested fix: Resolve to with os.path.realpath(to) before dispatching,
and pass to directly to CreateJunction in _link_dir_windows:

  if os.path.lexists(from_):
    return
  to = os.path.realpath(to)
  if is_dir:
    _link_dir_windows(from_, to)
  else:
    _link_file_windows(from_, to)

Where: Line 676 in python/private/python_bootstrap_template.txt.

This is an agent-generated comment reviewed and approved by the user.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when to is a runfiles symlink ... leaves sys.executable as a symlink

If symlinks aren't enabled then they wouldn't be a symlink. I think the comment is mistaken unless there's some edge case that I'm not seeing


def _link_dir_windows(from_, to):
import _winapi
try:
# Junctions don't need any privileges, but need an absolute target.
_winapi.CreateJunction(to, from_)
# Another process sharing RULES_PYTHON_EXTRACT_ROOT created it first.
except FileExistsError:
return
except OSError as e:
print_verbose("junction failed, copying:", to, "error:", e)
# This can still fail with FileExistsError if another process sharing
# RULES_PYTHON_EXTRACT_ROOT creates the directory while copying. Both
# creating the junction and a concurrent run have to fail for that, so
# this isn't handled.
shutil.copytree(to, from_)

Comment on lines +666 to +693

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.lexists(from_) is non-atomic when concurrent processes share
RULES_PYTHON_EXTRACT_ROOT. If another process creates from_ before os.link
or CreateJunction runs, FileExistsError hits except OSError and calls
shutil.copy2 or shutil.copytree onto the existing destination.

Suggested fix: Catch FileExistsError and return before except OSError in
both _link_file_windows and _link_dir_windows:

  except FileExistsError:
    return

This is an agent-generated comment reviewed and approved by the user.

_kernel32 = None

def _get_kernel32():
"""Returns kernel32 with the prototypes that cleaning up the venv needs."""
global _kernel32
if _kernel32 is not None:
return _kernel32

import ctypes
from ctypes import wintypes

kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
kernel32.CreateFileW.restype = wintypes.HANDLE
kernel32.CreateFileW.argtypes = [
wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD, wintypes.LPVOID,
wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE,
]
kernel32.SetFileInformationByHandle.restype = wintypes.BOOL
kernel32.SetFileInformationByHandle.argtypes = [
wintypes.HANDLE, ctypes.c_int, wintypes.LPVOID, wintypes.DWORD,
]
kernel32.CloseHandle.restype = wintypes.BOOL
kernel32.CloseHandle.argtypes = [wintypes.HANDLE]
_kernel32 = kernel32
return kernel32

def _remove_read_only_file_windows(path):
"""Removes a read-only file without changing its read-only attribute.

Windows refuses to delete a read-only file, such as a hard link to or a
copy of a Bazel output. Clearing the attribute would also clear it for the
other hard links of the file, i.e. the Bazel output, so the file is deleted
with FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE instead.
"""
import ctypes
from ctypes import wintypes

kernel32 = _get_kernel32()

Comment on lines +702 to +732

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_remove_read_only_file_windows constructs
ctypes.WinDLL("kernel32", use_last_error=True) and rebinds prototypes on
every read-only file removed by _rmtree, while omitting
CloseHandle.restype = wintypes.BOOL.

Suggested fix: Lazily initialize and cache the kernel32 WinDLL handle and
prototypes once in a _get_kernel32() helper, and set
kernel32.CloseHandle.restype = wintypes.BOOL.

This is an agent-generated comment reviewed and approved by the user.

DELETE = 0x00010000
FILE_SHARE_READ_WRITE_DELETE = 0x7
OPEN_EXISTING = 3
FILE_FLAG_OPEN_REPARSE_POINT = 0x00200000
FILE_DISPOSITION_INFO_EX = 21
FILE_DISPOSITION_FLAG_DELETE = 0x1
FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE = 0x10
INVALID_HANDLE_VALUE = wintypes.HANDLE(-1).value

handle = kernel32.CreateFileW(
path, DELETE, FILE_SHARE_READ_WRITE_DELETE, None, OPEN_EXISTING,
FILE_FLAG_OPEN_REPARSE_POINT, None,
)
if handle == INVALID_HANDLE_VALUE:
raise ctypes.WinError(ctypes.get_last_error())
try:
flags = wintypes.DWORD(
FILE_DISPOSITION_FLAG_DELETE | FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE
)
if not kernel32.SetFileInformationByHandle(
handle, FILE_DISPOSITION_INFO_EX, ctypes.byref(flags), ctypes.sizeof(flags)
):
raise ctypes.WinError(ctypes.get_last_error())
finally:
# The file is deleted when the handle is closed.
kernel32.CloseHandle(handle)

def _rmtree(path):
if not IS_WINDOWS:
shutil.rmtree(path, True)
return

def _on_error(func, error_path, _):
if func not in (os.unlink, os.remove):
return
try:
_remove_read_only_file_windows(error_path)
except OSError as e:
print_verbose("cleanup: failed to remove:", error_path, "error:", e)

if sys.version_info >= (3, 12):
shutil.rmtree(path, onexc=_on_error)
else:
shutil.rmtree(path, onerror=_on_error)


def main():
Expand Down
9 changes: 9 additions & 0 deletions tests/bootstrap_impls/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,15 @@ sh_py_run_test(
venvs_use_declare_symlink = "no",
)

sh_py_run_test(
name = "run_binary_without_symlinks_test",
bootstrap_impl = "system_python",
py_src = "bin.py",
sh_src = "run_binary_without_symlinks_test.sh",
# The fallback for when symlinks can't be created is Windows only.
target_compatible_with = ["@platforms//os:windows"],
)

sh_py_run_test(
name = "run_binary_find_runfiles_test",
py_src = "bin.py",
Expand Down
2 changes: 2 additions & 0 deletions tests/bootstrap_impls/bin.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,6 @@ def handle(signum, _frame):
print("sys.flags.safe_path:", sys.flags.safe_path)
print("file:", __file__)
print("sys.executable:", sys.executable)
print("sys.executable is symlink:", os.path.islink(sys.executable))
print("sys.prefix:", sys.prefix)
print("sys._base_executable:", getattr(sys, "_base_executable", None))
44 changes: 42 additions & 2 deletions tests/bootstrap_impls/cached_venv_dangling_symlink_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,17 +43,55 @@ def _load_bootstrap() -> dict:

class CachedVenvDanglingSymlinkTest(unittest.TestCase):
def test_replaces_dangling_package_link_before_creating_child_link(self) -> None:
self._test_replaces_dangling_package_link(symlinks_supported=True)

@unittest.skipUnless(os.name == "nt", "junctions are Windows only")
def test_replaces_dangling_package_junction_before_creating_child_link(
self,
) -> None:
self._test_replaces_dangling_package_link(symlinks_supported=False)

@unittest.skipUnless(os.name == "nt", "junctions are Windows only")
def test_dangling_junction_is_detected_and_removed(self) -> None:
# The bootstrap relies on these to remove a dangling junction, which
# islink() doesn't detect.
import _winapi

with tempfile.TemporaryDirectory() as temp:
root = pathlib.Path(temp)
target = root / "target"
target.mkdir()
junction = root / "junction"
_winapi.CreateJunction(str(target), str(junction)) # pyrefly: ignore[missing-attribute]
target.rmdir()

self.assertFalse(os.path.islink(junction))
self.assertTrue(os.path.lexists(junction))
self.assertFalse(os.path.exists(junction))
os.unlink(junction)
self.assertFalse(os.path.lexists(junction))

def _test_replaces_dangling_package_link(self, *, symlinks_supported) -> None:
bootstrap = _load_bootstrap()
# False uses the fallback for when symlinks can't be created, which
# creates a junction for a directory.
bootstrap["_symlinks_supported"] = symlinks_supported
with tempfile.TemporaryDirectory() as temp:
root = pathlib.Path(temp)
runfiles = root / "runfiles"
old_package = runfiles / "old/annotated_types"
old_package.mkdir(parents=True)
(old_package / "__init__.py").touch()
# The venv isn't run, and Windows can't delete a hard link to the
# running interpreter, e.g. when cleaning up the temporary
# directory, so link to a stand-in instead.
python = root / "python/python.exe"
python.parent.mkdir()
python.touch()
bootstrap.update(
EXTRACT_ROOT=str(root / "cache"),
PYTHON_BINARY="_main/_process.venv/Scripts/python.exe",
PYTHON_BINARY_ACTUAL=sys.executable,
PYTHON_BINARY_ACTUAL=str(python),
RESOLVE_PYTHON_BINARY_AT_RUNTIME=False,
RUNTIME_VENV_SYMLINKS={
"Lib/site-packages/annotated_types": "old/annotated_types",
Expand All @@ -78,7 +116,9 @@ def test_replaces_dangling_package_link_before_creating_child_link(self) -> None
package_init = package / "__init__.py"
self.assertTrue(package.is_dir())
self.assertFalse(package.is_symlink())
self.assertTrue(package_init.is_symlink())
self.assertFalse(getattr(os.path, "isjunction", lambda _: False)(package))
# The fallback creates a hard link for a file.
self.assertEqual(package_init.is_symlink(), symlinks_supported)
self.assertTrue(os.path.samefile(package_init, new_package / "__init__.py"))


Expand Down
Loading
Loading