Skip to content

fix(windows): don't require symlinks for the runtime venv - #4509

Open
aignas wants to merge 19 commits into
bazel-contrib:mainfrom
aignas:fix.aignas.windows-no-symlinks
Open

aignas wants to merge 19 commits into
bazel-contrib:mainfrom
aignas:fix.aignas.windows-no-symlinks

Conversation

@aignas

@aignas aignas commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

On Windows, py_binary and py_test create a venv at runtime by
symlinking the interpreter, its .dll files and the site-packages files
into a temporary directory. Without the privilege to create symlinks, this
fails with WinError 1314, which is the default for users that can't enable
Developer Mode.

On Windows, if creating a symlink fails, the bootstrap now uses hard links
for files, junctions for directories, and copies if those aren't possible,
e.g. when the temporary directory is on a different volume. The runfiles
files are already copies when Bazel can't create symlinks, so they aren't
moved, as that would break the next run that uses the same runfiles tree.
Other platforms are unchanged.

Windows refuses to delete read-only files, such as hard links to Bazel
outputs, and the read-only attribute is shared by all the hard links of a
file, so the venv's read-only files are deleted with
FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE instead of clearing the
attribute. This needs Windows 10 1809 or Windows Server 2019, so older
versions still require the privilege to create symlinks.

The RULES_PYTHON_TESTING_DISABLE_SYMLINKS environment variable forces the
fallback on Windows, which a new test uses, and a new CI job uses to build
and test examples/bzlmod with --nowindows_enable_symlinks.

Fixes #3792

If creating a symlink fails, e.g. on Windows without the privilege to
create symlinks, the venv that is created at runtime uses hard links,
junctions or copies instead, and its pyvenv.cfg sets home, because
Python can't find it from the interpreter otherwise.

The RULES_PYTHON_TESTING_DISABLE_SYMLINKS environment variable forces
the fallback, so that it can be tested where symlinks work.
@aignas

aignas commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

This PR was written by an AI agent (Claude Code) working for @aignas. Review comments are read and addressed by the agent, and every reply and change is reviewed and approved by @aignas before it is posted or pushed.

The new Windows CI job is experimental: it checks whether examples/bzlmod builds and passes without symlinks, so this stays a draft until it's green.

This comment was generated by an AI agent and approved by @aignas.

Revert this before the PR leaves draft.
On Windows, the venv paths are Windows paths, which the bash checks
couldn't handle.
Use the `? "--flag"` keys that presubmit_test requires, and download
the remote cache outputs, because without symlinks Bazel creates a
junction in the runfiles tree for an output that isn't downloaded yet.
@aignas

aignas commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

I am not sure if it is the smallest possible change to fix the issue, but it is pretty small and if it works, I'll be chuffed.

The fallback for when symlinks can't be created is now Windows only.

Windows refuses to delete a read-only file and the attribute is shared
by all the hard links of a file, so removing the venv made the hard
linked Bazel outputs writable. The hard links are now removed first, and
the attribute is restored on their originals.
Instead of restoring the read-only attribute of hard linked Bazel
outputs after clearing it, delete the read-only files of the venv with
FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE, which doesn't change
the other hard links of a file. It needs Windows 10 1809 or Windows
Server 2019, so older versions still require symlinks.

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several lines in the PR description reach 73–76 columns, exceeding the
72-column limit for merge commit messages in .editorconfig.

Suggested fix: Re-wrap the PR description body at 72 columns while preserving
the existing wording.

This is an agent-generated comment reviewed and approved by the user.

Comment on lines +658 to +670
if is_dir:
_link_dir_windows(from_, to)
else:
_link_file_windows(from_, to)

def _link_file_windows(from_, to):
# Hard links don't need any privileges, but only work within the same
# volume, so fall back to copying the file.
try:
os.link(to, from_)
except OSError as e:
print_verbose("hard link failed, copying:", to, "error:", e)
shutil.copy2(to, from_)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Windows, os.link(to, from_) calls CreateHardLinkW without dereferencing
symlinks, hard-linking the reparse point itself when to is a .runfiles
symlink. That leaves sys.executable as a symlink, failing
run_binary_without_symlinks_test on the Default: Windows CI jobs and
breaking relative symlinks.

Suggested fix: Resolve to with os.path.realpath(to) before dispatching,
and pass to directly to CreateJunction in _link_dir_windows:

  if os.path.lexists(from_):
    return
  to = os.path.realpath(to)
  if is_dir:
    _link_dir_windows(from_, to)
  else:
    _link_file_windows(from_, to)

Where: Line 676 in python/private/python_bootstrap_template.txt.

This is an agent-generated comment reviewed and approved by the user.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when to is a runfiles symlink ... leaves sys.executable as a symlink

If symlinks aren't enabled then they wouldn't be a symlink. I think the comment is mistaken unless there's some edge case that I'm not seeing

try:
# Junctions don't need any privileges, but need an absolute target.
import _winapi
_winapi.CreateJunction(abspath(to), from_)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.islink() returns False for NTFS junctions on Windows, so a dangling
parent junction in a cached RULES_PYTHON_EXTRACT_ROOT venv is not unlinked
and os.makedirs(venv_parent, exist_ok=True) raises FileExistsError.

Suggested fix: Check
os.path.lexists(venv_parent) and not os.path.exists(venv_parent) since
os.unlink removes both symlinks and junctions on Windows, and add a test
case with _SYMLINKS_SUPPORTED = False:

        if os.path.lexists(venv_parent) and not os.path.exists(venv_parent):
            os.unlink(venv_parent)

Where: tests/bootstrap_impls/cached_venv_dangling_symlink_test.py

This is an agent-generated comment reviewed and approved by the user.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something to verify is how lexists() behaves on a junction. There was a lot of subtle behaviors when the unix-style understanding of things operated on windows things.

Comment on lines +663 to +680
def _link_file_windows(from_, to):
# Hard links don't need any privileges, but only work within the same
# volume, so fall back to copying the file.
try:
os.link(to, from_)
except OSError as e:
print_verbose("hard link failed, copying:", to, "error:", e)
shutil.copy2(to, from_)

def _link_dir_windows(from_, to):
try:
# Junctions don't need any privileges, but need an absolute target.
import _winapi
_winapi.CreateJunction(abspath(to), from_)
except (ImportError, AttributeError, OSError) as e:
print_verbose("junction failed, copying:", to, "error:", e)
shutil.copytree(to, from_)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.path.lexists(from_) is non-atomic when concurrent processes share
RULES_PYTHON_EXTRACT_ROOT. If another process creates from_ before os.link
or CreateJunction runs, FileExistsError hits except OSError and calls
shutil.copy2 or shutil.copytree onto the existing destination.

Suggested fix: Catch FileExistsError and return before except OSError in
both _link_file_windows and _link_dir_windows:

  except FileExistsError:
    return

This is an agent-generated comment reviewed and approved by the user.

Comment on lines +55 to +66
read_only_before=$(find "$runfiles" -type f ! -writable | sort)
actual=$(RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1 $bin)
expect_match "Hello" "$actual"
expect_match "sys.executable is symlink: False" "$actual"
venv=$(to_unix "$(echo "$actual" | grep "^sys.prefix:" | cut -d' ' -f2-)")
if [[ -z "$venv" || -e "$venv" ]]; then
echo "expected the venv to be removed: $venv"
touch EXPECTATION_FAILED
fi
read_only_after=$(find "$runfiles" -type f ! -writable | sort)
if [[ "$read_only_before" != "$read_only_after" ]]; then
echo "expected the read-only runfiles to be unchanged"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows python.exe writes CRLF line endings, leaving a trailing \r on $venv
so [[ -e "$venv" ]] is always false even when cleanup fails. Running find
without -L also skips symlinked runfiles, letting an empty read_only_before
pass vacuously.

Suggested fix: Strip \r from $actual, pass -L to find for both
read_only_before and read_only_after, and verify read_only_before is
non-empty:

# -L: follow symlinks when Bazel materializes runfiles as symlinks.
read_only_before=$(find -L "$runfiles" -type f ! -writable | sort)
if [[ -z "$read_only_before" ]]; then
  echo "expected read-only files in runfiles: $runfiles"
  touch EXPECTATION_FAILED
fi
actual=$(RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1 "$bin" | tr --delete '\r')

Where: Also strip \r at line 45 and pass -L at line 64 in
tests/bootstrap_impls/run_binary_without_symlinks_test.sh.

This is an agent-generated comment reviewed and approved by the user.

Comment on lines +44 to +45
venvs_root=$(mktemp -d)
actual=$(RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1 RULES_PYTHON_EXTRACT_ROOT=$venvs_root $bin)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bare mktemp -d creates venvs_root in MSYS /tmp outside TEST_TMPDIR, leaking
a populated venv on every run. Lines 13, 45, and 56 also leave
$BIN_RLOCATION, $venvs_root, and $bin unquoted.

Suggested fix: Create venvs_root inside TEST_TMPDIR and quote the path
variables:

bin=$(rlocation "$BIN_RLOCATION")
...
venvs_root=$(mktemp --directory "$(to_unix "$TEST_TMPDIR")/venvs_root.XXXXXX")
actual=$(RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1 RULES_PYTHON_EXTRACT_ROOT="$venvs_root" "$bin")

Where: Lines 13 and 56 in
tests/bootstrap_impls/run_binary_without_symlinks_test.sh.

This is an agent-generated comment reviewed and approved by the user.

Comment on lines +689 to +703
import ctypes
from ctypes import wintypes

kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
kernel32.CreateFileW.restype = wintypes.HANDLE
kernel32.CreateFileW.argtypes = [
wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD, wintypes.LPVOID,
wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE,
]
kernel32.SetFileInformationByHandle.restype = wintypes.BOOL
kernel32.SetFileInformationByHandle.argtypes = [
wintypes.HANDLE, ctypes.c_int, wintypes.LPVOID, wintypes.DWORD,
]
kernel32.CloseHandle.argtypes = [wintypes.HANDLE]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_remove_read_only_file_windows constructs
ctypes.WinDLL("kernel32", use_last_error=True) and rebinds prototypes on
every read-only file removed by _rmtree, while omitting
CloseHandle.restype = wintypes.BOOL.

Suggested fix: Lazily initialize and cache the kernel32 WinDLL handle and
prototypes once in a _get_kernel32() helper, and set
kernel32.CloseHandle.restype = wintypes.BOOL.

This is an agent-generated comment reviewed and approved by the user.

Comment thread tests/bootstrap_impls/BUILD.bazel Outdated
Comment on lines +82 to +85
target_compatible_with = select({
"@platforms//os:windows": [],
"//conditions:default": ["@platforms//:incompatible"],
}),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@platforms//os:windows is already a constraint_value, so wrapping
@platforms//:incompatible in select() isn't needed here.

Suggested fix: Pass the constraint list directly:

    target_compatible_with = ["@platforms//os:windows"],

This is an agent-generated comment reviewed and approved by the user.

@@ -0,0 +1,72 @@
# --- begin runfiles.bash initialization v3 ---

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script has executable mode 100755 and uses Bash-specific syntax
(pipefail, [[ ... ]], and process substitution), but is missing a Bash
shebang on line 1.

Suggested fix: Add #!/usr/bin/env bash at the top of the file:

#!/usr/bin/env bash
# --- begin runfiles.bash initialization v3 ---

This is an agent-generated comment reviewed and approved by the user.

exit 1
fi

function expect_match() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expect_match mutates global test state by creating EXPECTATION_FAILED on
mismatch without documenting that side effect.

Suggested fix: Add a function doc comment with a Modifies: section:

# Checks that actual output matches expected_pattern.
#
# Modifies:
#   Creates EXPECTATION_FAILED in the working directory on mismatch.
function expect_match() {

This is an agent-generated comment reviewed and approved by the user.

# Converts a Windows path to a Unix one under MSYS, so that bash can use it.
function to_unix() {
if command -v cygpath >/dev/null; then
cygpath -u "$1"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cygpath, mktemp, and cut are called with short flags instead of long option
names.

Suggested fix: Use long option names:

  • cygpath --unix "$1"
  • mktemp --directory ...
  • cut --delimiter=' ' --fields=2-

Where: Also at lines 44 and 59 in
tests/bootstrap_impls/run_binary_without_symlinks_test.sh.

This is an agent-generated comment reviewed and approved by the user.

… globals

Move them next to the other top-level globals and use a lower case name
for the one that changes at runtime.
When Bazel could create the runfiles symlinks, but the venv falls back
to hard links, os.link() hard linked the symlink itself, so
sys.executable was still a symlink. Resolve the target first.
When processes share RULES_PYTHON_EXTRACT_ROOT, another one can create
the entry between the existence check and creating the link, which
made the fallback copy onto it. Also only catch OSError from
_winapi.CreateJunction, which every supported Python on Windows has.
They were set up again for every read-only file that is removed. Also
declare the return type of CloseHandle.
The fallback creates junctions for directories, and islink() is False
for them, so a dangling one wasn't removed and makedirs() failed.
Remove the CR of the Windows output, so that the check that the venv was
removed doesn't always pass, follow the runfiles symlinks when looking
for read-only outputs and require that there are some, create the venvs
under TEST_TMPDIR, and quote the paths.
…oved

Check the behavior of lexists(), exists() and unlink() for a dangling
junction, which the bootstrap relies on, and run the cached venv test
with the fallback for when symlinks can't be created.
…ed venv test

Windows can't delete a hard link to a running executable, so cleaning
up the temporary directory failed after the fallback hard linked
sys.executable into the venv. The venv isn't run, so link to a
stand-in file instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

py_binary no longer runs on Windows with rules_python 2.0.0+ using default Windows settings

2 participants