[dimitri-yatsenko]

Closes #1594. Part of #1577.

Job.refresh(*restrictions) narrowed key_source to add jobs, then reused the narrowed key_source in its cleanup step. Every job older than stale_timeout whose key fell outside the restriction therefore looked vanished and was deleted, including jobs reserved by other workers and error records. populate(*restrictions, reserve_jobs=True) calls refresh this way.

• Cleanup checks keys against the unrestricted key_source. The restrictions still limit which jobs are added and re-pended.
• Records of deleted rows go on every refresh (Jobs: with keep_completed, success and error records outlive the rows they describe #1577). Success and error records whose key has left both key_source and the target (because the upstream row was deleted) are removed on every refresh, whatever their age, instead of lingering until stale_timeout. Records of rows that still exist are kept.
• Unchanged: the age-based check for the other statuses, and stale_timeout=0 skipping it. That is the rest of Jobs: with keep_completed, success and error records outlive the rows they describe #1577, which waits for 2.4.
• One DELETE per cleanup step, instead of an existence query and a DELETE for each key.

Tests: tests/integration/test_jobs_refresh.py, on MySQL and PostgreSQL. It covers a restricted refresh, a restricted populate, records of vanished keys, records of rows that still exist, stale_timeout=0, and the statement count. The jobs and autopopulate suites pass locally.

The same commit goes onto maint/2.3 for the v2.3.5 patch release.

[MilagrosMarin]

Thanks for the clear write-up @dimitri-yatsenko — judging cleanup against the unrestricted key_source is the right fix for #1594, and limiting the restrictions to the add/re-pend steps is the correct split. I read #1577 too; 3a lines up with the stale_timeout=0 complaint there. I haven't run the suite locally — these come from reading the diff.

Requested changes

✅ src/datajoint/jobs.py:343 — the stale_timeout parameter still reads "Set to 0 to skip." With 3a that's no longer true: success and error records of vanished keys are removed whatever stale_timeout is. The *restrictions entry and the Notes section were updated; this one wasn't. Suggested:

stale_timeout : float, optional
    Seconds after which jobs are checked for staleness.
    Jobs older than this are removed if key not in key_source.
    Default from ``config.jobs.stale_timeout``. Set to 0 to skip the
    age-based check; the cleanup of success and error records whose
    keys have left both key_source and the target runs regardless.

✅ Test docstring — see the inline comment on test_jobs_refresh.py.

Follow-up, not for this PR

3a keeps a record whose row still exists (Not(target.proj())); 3b judges on key_source alone, so the same record is removed once it passes stale_timeout. That divergence is pre-existing, and it belongs with the rest of #1577 in 2.4 rather than here — worth a line in the issue so the next reader doesn't take 3a's principle as global.

Before merge

On head f3d44e0 I see no workflow runs or check-runs through the API and the combined status is pending. Worth confirming CI before this lands, given it's one of the two data-loss fixes.

The restricted-refresh and restricted-populate cases, the re-pend ordering ahead of cleanup, and the move to one DELETE per step all read correct to me. Let me know if anything needs adjusting!

[MilagrosMarin]

Suggested change

	"""A success record stays while its row exists, even if its key leaves key_source."""
	"""A fresh success record stays while its row exists, even if its key leaves key_source."""

The job isn't backdated here, so this covers the 3a path only. 3b judges vanished on key_source without 3a's Not(target.proj()) guard, so the same record is removed once it's older than stale_timeout — pre-existing behavior, not something this PR changes. "Fresh" keeps the docstring to what the test actually pins down.

[dimitri-yatsenko]

Thanks, @MilagrosMarin. Pushed 6a1e171 with both suggestions; the same commit is on #1597.

• stale_timeout: the entry now reads as you suggested. 0 skips the age-based check, and the cleanup of success and error records whose keys have left both key_source and the target runs regardless.
• Test docstring: it now says "a fresh success record", since that is all the test covers.
• The 3a/3b difference: I added a note on Jobs: with keep_completed, success and error records outlive the rows they describe #1577, which handles the rest of it in 2.4. fix(jobs): stale_timeout=0 means no grace period #1623, stacked on this PR, is rebased onto the new commit.
• CI: the workflow runs from early to mid October were deleted across the repository. I reopened this PR to run the checks again.