Repository navigation
Stack/htcondor 6 contract suite - #38
Draft
Alexander-Auer wants to merge 87 commits into
Draft
Alexander-Auer wants to merge 87 commits into
Alexander-Auer wants to merge 87 commits into
Conversation
… have them link to each other
…ede into feat/core-contract
…a consumer should choose to use it or not
* feat: Models the core contract needs * Feat: Protocols and Exceptions added for the core contract * Feat: Private protocols for the BatchBackend * feat: a fake backend to run unit tests against * fix: some code cleanup in the helper functions * feat: setup for the unit test suite * chore: style fix * chore: styling, unit test fixture * chore: single letter var `h` refering to hanlde changed to `handle` * fix: unused parameter in `test_injected_failure_is_reported_per_job()` removed * feat: Models the core contract needs * Feat: Protocols and Exceptions added for the core contract * Feat: Private protocols for the BatchBackend * feat: a fake backend to run unit tests against * fix: some code cleanup in the helper functions * feat: setup for the unit test suite * chore: style fix * chore: styling, unit test fixture * chore: single letter var `h` refering to hanlde changed to `handle` * fix: unused parameter in `test_injected_failure_is_reported_per_job()` removed * fix: made the OutputSpec optional as it is not suppoerted by all and a consumer should choose to use it or not --------- Co-authored-by: alexander <alexander.auer@cern.ch>
added 29 commits
October 26, 2026 16:19
Brings in the per-kind configuration models of stack/registry, and moves the credential code onto them: - credential_requirements(config, scopes) in _backends/credentials.py replaces the methods on the removed internal settings classes, with the same rules: a CE asks for its endpoint with the configured token and/or proxy, SSH for user@host:port unless an SSH agent holds the key, and a local batch system for nothing. - registry.get_backend(config, credentials) checks the provider and passes it to the parts that authenticate: single-class backends and transports. - The credential requirement tests move from test_config.py to test_credentials.py and build their configurations through validate_config().
Two tests had moved without changing, which only made the diff harder to read.
# Conflicts: # src/intercede/_backends/fake.py # src/intercede/registry.py # tests/unit/conftest.py # tests/unit/test_registry.py
…, squash-merged as DIRACGrid#29)
A value given by mistake, such as a misspelled field or plain data in a CredentialSet, may be the secret itself. Validation errors now hide their input, as the configuration models already do.
The message comes from the consumer's code and may hold a token or a secret URL. The AuthenticationError names only its type; the exception itself is still available as __cause__.
The fields were described in comments, which the generated reference does not show. They now use attribute docstrings, as the configuration models do.
Name every argument and the return value, and drop the blank lines before closing quotes.
Moves test_fake_backend.py to tests/unit/backends/fake/, ready for the HTCondor tests that follow. The tests themselves do not change. Part of DIRACGrid#23.
Renders a submit description from a plan: one process per copy, per-copy values indexed by ProcId, quoting, and refusing what a description cannot hold. It sets no policy on the job except keeping finished jobs until they are purged, so output can be collected again. Shared with the HTCondor scheduler to come (DIRACGrid#21), so it knows nothing about any backend. Expected values were checked against HTCondor 25. Part of DIRACGrid#23.
Builds the tools' command lines and reads what they print: condor_submit -terse, condor_rm, and the job ads of condor_q and condor_history, mapped to the common JobStatus. A finished job's result comes from its exit, a removed job keeps the result it had, and any hold that interCEde did not cause counts as aborted. Part of DIRACGrid#23.
For one of several CEs on a host, or a CE behind a forwarded address. The endpoint and schedd end up on the tools' command lines, so they are checked when the configuration is read: neither can pass as an option. Part of DIRACGrid#23.
The contract says inputs are read before submit returns, so an unreadable one is a SpecificationRejectedError, not a raw OSError. The conformance suite now asks for it, and the fake does it. Part of DIRACGrid#23.
- process: runs one tool, killing it on a timeout - session: one operation's credentials in private files that exist only while it runs, the tools' environment, and telling a rejected token from an unreachable CE - staging: the private working area, and what a submission puts in it Nothing uses these yet. The backend follows in the next commit. Part of DIRACGrid#23.
Spooled submission to a remote schedd, token authentication, and an optional X.509 proxy for sites that account on VOMS attributes. Copies whose variable names differ get separate clusters, because HTCondor merges each process's environment onto the previous one. Registered as `htcondor-ce`. Kill, purge and output follow in their own branches. Part of DIRACGrid#23.
FakeCondor answers the way HTCondor 25 does (checked against a live schedd), so these run without HTCondor installed: credentials, error classification, staging, secrets, containers and the working area. Part of DIRACGrid#23.
Kill only removes active jobs, because removing a finished one would release output that has not been collected. That is what purge is for. A job removed a moment ago is still queued but already released, so purging twice is harmless. Token scopes follow the operation: compute.cancel here. Part of DIRACGrid#23.
A stack's tests run with pytest-xdist (stacks.yml `workers`, default 4), and `-rs --no-fold-skipped` makes the log name every skipped test and why. The workflow also runs when src/ changes, since that is what it exists to test. Part of DIRACGrid#23.
Whether a purged job's status is then UNKNOWN or a final state depends on whether the resource still remembers it: a schedd's history does, an in-memory fake does not. Both are within the contract. Part of DIRACGrid#23.
An HTCondor-CE in front of a one-machine HTCondor pool. The image refuses package mirrors below 1 MB/s and caches its slow installs per version. No test selects this stack yet, so its CI job only shows that the stack starts (pytest exit 5 is tolerated). The shared suite lands later. Part of DIRACGrid#23.
condor_transfer_data into the private working area, then moved into dest. The job ad comes from the remote side, so the CE can never choose where this host writes: links are refused and the output limits apply. Two collections of one job take turns, and the backend remembers nothing about jobs afterwards. Part of DIRACGrid#23.
Written once against the interCEde protocols and run on every stack: a new backend writes no integration tests of its own. Tests carry the `contract` marker, and a test that needs an optional capability takes its fixture and is skipped on a backend that lacks it. [refusals] in a stack's client config says what its backend cannot honour, and the suite checks each is refused at submit. Part of DIRACGrid#23.
4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test: the shared contract suite, on the htcondor stack
Last PR for #23 (PR 6 of 6).
Important
Depends on #34 (
stack/htcondor-2-integration-stack) and PR 5 (stack/htcondor-5-output). Merge after both.Until then, this diff also shows the earlier PRs. This PR's own change: 799 lines
One set of tests, written against the interCEde protocols, runs on every stack, so a new backend writes no integration tests of its own.
What's added
tests/integration/test_*.py: 22 tests covering submit, status, output, kill, purge and what a consumer sees when things go wrong.contractmarker, which selects these tests whatever a stack'smarkerssay.output_retriever,killable,purgeable), which skip a test on a backend without that capability.[refusals]in a stack's client config, which says what its backend cannot honour. The suite checks each is refused at submit.htcondor-utilsin the pixi testing environment, plus the docs and the stack template.Result
The same content passed on
ltsandlateston my fork (eb0fec6). Three tests skip on this stack, and the log names them: the refusal checks forurl_inputsandurl_outputs(HTCondor supports both), and the test for a backend that cannot collect output.