Repository navigation
Support DMS migration during bundle deploy - #6989
Open
shreyas-goenka wants to merge 12 commits into
Open
shreyas-goenka wants to merge 12 commits into
shreyas-goenka wants to merge 12 commits into
Conversation
Collaborator
Integration test reportCommit: 3ddeba6
Top 6 slowest tests (at least 2 minutes):
|
denik
reviewed
Oct 13, 2026
Contributor
This seems a bit narrow? In direct we have env var and config setting. Config setting is consulted first, then env var, then state-derived, then the default. Important also that state is sticky, so if you are on DMS and neither config nor env var are set, you still remain on DMS and not migrated back. Do we have that? The config setting/env var/stickiness is complex enough so that it warrants its own PR, what do you think? |
denik
reviewed
Oct 14, 2026
This was referenced Oct 14, 2026
shreyas-goenka
changed the base branch from
main
to
shreyas-goenka_data/shreyas-goenka/dms-serial-stack
October 14, 2026 12:22
shreyas-goenka
added this pull request to stack #7047
October 14, 2026 12:22
Base automatically changed from
shreyas-goenka_data/shreyas-goenka/dms-serial-stack
to
main
October 19, 2026 11:33
shreyas-goenka
force-pushed
the
shreyas-goenka_data/shreyas-goenka/dms-deploy-migration
branch
from
October 19, 2026 12:30
f90499d to
bc86051
Compare
denik
reviewed
Oct 19, 2026
| musterr trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy | ||
| trace print_requests.py --dms //api/2.0/bundle --oneline | ||
| title "Normal state selection prefers local serial 2 over remote serial 1" | ||
| trace update_file.py .databricks/bundle/default/resources.json '"serial": 1' '"serial": 2' |
Contributor
There was a problem hiding this comment.
I don't understand why we have this serial update here? what its purpose, does migration below not work without it?
Contributor
Author
There was a problem hiding this comment.
No it absolutely does. This just simulates that if local state is ahead, the local state is used and thus the version created us at version 3.
I can see this test being a bit hard to read, let me simplify this just to the core assertions.
| trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-versions "deployments/${deployment_id}" -o json | | ||
| jq 'map(select((.version_id | tonumber) >= 4))' | | ||
| jq -c 'sort_by(.version_id | tonumber) | map({version_id, previous_version_id, version_type, status, completion_reason})' | | ||
| jq -r '.[] | "version \(.version_id): previous=\(.previous_version_id) type=\(.version_type | ltrimstr("VERSION_TYPE_")) status=\(.status | ltrimstr("VERSION_STATUS_")) completion=\(.completion_reason | ltrimstr("VERSION_COMPLETE_"))"' |
Contributor
There was a problem hiding this comment.
what is this post processing, cannot we record original json here?
| // Upgrade from read (opened by process.go, or by Migrate) to write mode. After approval | ||
| // and the migration commit above, so a declined deploy never opens a WAL it must discard, | ||
| // and the migration's tf+1 push lands before this advances the WAL header to tf+2. | ||
| if err := b.DeploymentBundle.StateDB.UpgradeToWrite(); err != nil { |
Contributor
There was a problem hiding this comment.
Do we need this if we're on DMS?
| } | ||
| } | ||
|
|
||
| if b.MigratingToDMS { |
Contributor
There was a problem hiding this comment.
Are we reusing & patching the plan that we've created again resources.json? I thought we discussed following tf-to-direct migration and calculating plan with DMS before comitting?
Contributor
|
Commit: Regression test reportSelected 4 changed tests (limit=20, 0 not selected) 4 tests: regression=4 unreleased=0 coverage=0 failing=0
TestAccept/bundle/dms/existing-state/DMS=: branch ✅ | main (9a01571) ❌ | latest (v1.21.0) ❌main (9a01571): latest (v1.21.0): TestAccept/bundle/dms/migration-before-deploy/DMS=true: branch ✅ | main (9a01571) ❌ | latest (v1.21.0) ❌main (9a01571): latest (v1.21.0): TestAccept/bundle/dms/repeated-migration/DMS=true: branch ✅ | main (9a01571) ❌ | latest (v1.21.0) ❌main (9a01571): latest (v1.21.0): TestAccept/bundle/migrate/auto/dms-migration/DMS=: branch ✅ | main (9a01571) ❌ | latest (v1.21.0) ❌main (9a01571): latest (v1.21.0): |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Enabling
experimental.deployment_historyfor an existing bundle target will automatically migrate the bundle.resources.jsontoresources.json.backupbefore migration.INITIAL_REGISTERin a dedicated version, then apply deployment changes in a subsequent version.max(state serial, latest DMS version) + 1. Re-register when the selected state has no deployment-history marker, including repeated migration.Migration uses the already-selected, open state and acquires its own deployment lock only when migration is needed. Once committed, it remains committed even if the subsequent deployment fails planning or is declined. Normal deployment has no pending-migration object or migration-specific plan patching. Read-only plan commands do not commit migration.
Tests