Repository navigation
KVM: live volume migration on Ceph RBD, and between RBD, NFS, SharedMountPoint and local pools - #14426
KVM: live volume migration on Ceph RBD, and between RBD, NFS, SharedMountPoint and local pools#14426weizhouapache wants to merge 9 commits into
Conversation
Everything that talks to Ceph needs the pool's monitors as a list the client can parse, and the rules are the same wherever that list is built: give each monitor the pool's port unless it names one already, and put a bare IPv6 address in square brackets so its own colons stay distinguishable from the separator before the port. The mon_host option of a disk's rbd: string wants the same list again, with ':' and ';' escaped, because those two separate the options of that string and its monitors. None of it depends on a storage pool, so collect it in RbdUtils as static methods with their unit tests, rather than leaving each caller to grow a copy that is right in a different set of cases.
- enable migrate volume between ceph - regenerate disk xml for virsh blockcopy - use SHALLOW flag instead of REUSE_EXT
test_13_migrate_volume_and_change_offering previously always stopped the VM before migrating an attached volume on KVM. Now that VolumeApiServiceImpl allows KVM to live-migrate a volume between two RBD pools (VolumeApiServiceImpl.isSourceAndDestOnRBD), keep the VM running and pass livemigrate=true for that RBD-to-RBD case, same as already done for VMware/XenServer. Any other KVM source/destination combination keeps the existing stop/migrate/start behavior.
…l pools Migrate a volume of a running Instance between an RBD pool and an NFS, SharedMountPoint or host-local pool, and between those file based pools, with a libvirt block copy on the host the Instance runs on, so that neither the Instance nor its host change. Migrate encrypted volumes between RBD and NFS, online and offline. Management server: - VolumeApiServiceImpl lets a KVM live migration through for these pool pairs, where it asked for migrateVirtualMachineWithVolumes before. A local pool has to be the one of the host the Instance runs on. - AncientDataMotionStrategy keeps the format of the volume right after the move: RAW on RBD, QCOW2 on a file based pool, and luks2 or luks as the encryption format, as the agent encrypts an RBD volume with librbd and any other volume with qemu. KVM agent: - LibvirtMigrateVolumeCommandWrapper builds the disk XML of the destination for the type of the destination pool, an RBD image or a qcow2 file, and copies the whole disk, so a volume cloned from a template does not depend on a backing file of the source pool. SHALLOW is only used from RBD to RBD. The source volume is removed after the pivot and the destination pool refreshed. - An encrypted volume gets its destination created with the encryption header first, and an offline copy goes through qemu-img with both secrets (LibvirtStorageAdaptor, KVMStorageProcessor). - LibvirtStorageVolumeXMLParser read the volume format with its arguments swapped and found none, so a migrated qcow2 file named like an RBD image was taken for a raw one and the Instance failed to start again. It reads the format of the volume now. - The ceph.conf which the copy of an encrypted volume writes gets its mon_host from RbdUtils.getRbdMonitors, which gives each monitor its port, leaves alone one which has it, and brackets a bare IPv6 address. Tests: - Unit tests for the pool pairs which are accepted, the local pool of another host, and the dispatch of the agent to the block copy. - test_volume_migration_rbd_nfs.py migrates ROOT and DATA volumes online, offline and detached, encrypted or not, and stops and starts the Instance after every move to check that it is still reachable. - test_volumes.py: the data disk of the encrypted volume migration test is found by its size.
boring-cyborg
Bot
added
component:api
component:integration-test
component:kvm
component:storage
Python
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14426 +/- ##
============================================
+ Coverage 20.39% 20.42% +0.02%
- Complexity 21283 21335 +52
============================================
Files 6481 6482 +1
Lines 588040 588448 +408
Branches 72021 72109 +88
============================================
+ Hits 119957 120177 +220
- Misses 455052 455208 +156
- Partials 13031 13063 +32
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@winterhazel @bernardodemarco I will revert some of the changes related to live volume migration between NFS, local, and SharedMountPoint pools. These changes use blockcopy for the migrations. Although they worked in my testing, I think it's better to keep the current behavior and leave this out of the scope of the PR for now. We can discuss it separately later. |
|
This is really great! Let me review this soon |
wido
left a comment
wido
left a comment
There was a problem hiding this comment.
Went through this by hand first and then had Claude Fable dig through the paths I wasn't sure about. Really great work, we've wanted this for years!
One thing I think is blocking: in migrateRBDVolume everything after the pivot can still fail the migration. The delete of the source volume in the finally isn't guarded and pool.refresh() is inside the try. If one of those throws the agent reports a failure, mgmt server keeps the volume on the old pool, but qemu is already writing to the new image. Next stop/start the instance boots from the old image or doesn't start at all. Thats your 5th run failure I think. Once the PIVOT returned the migration is done, delete and refresh should just log and not touch the answer anymore. Was the source image still there after that failed run btw?
The LibvirtStorageVolumeXMLParser fix is correct (args were swapped) but it changes format detection for every volume on every libvirt pool, that fallback has been in effect since forever. I'd put that in a seperate PR with its own testing. LXC .tar templates for example get probed as raw now.
Agree with dropping NFS/SMP/local live migration from this PR. The server gate, isFileBasedPool in the wrapper, smoke tests and the migrateRBDVolume name should then go together.
Checked and fine: the encrypted copy in copyPhysicalDisk is only reached from the primary to primary copy with both passphrases set, templates and snapshots are not affected. Endpoint selector change only hits running instances.
Ran the unit tests from the description on the branch with JDK 17, 238 tests, all pass.
I don't have a Ceph test env at the moment. CCC is coming up in November, can you give me a demo there so we can verify this together? Would love to see it live.
|
thanks a lot @wido ! I will address the first issue. the 2nd issue is one of my major concerns too. the fix is small but it could have large impact. I will research |
…y storages A volume which is not live migrated is copied directly between two pools, with qemu-img and its passphrase, only if the data motion strategy can bypass secondary storage for both pools. A SharedMountPoint pool was left out of the pool types it bypasses it for, although it is a file based pool like the NetworkFilesystem and Filesystem ones: its volume was staged on secondary storage, and the copy to and from secondary storage does not handle an encrypted volume, so qemu-img failed with "Parameter 'encrypt.key-secret' is required for cipher" after the migration had been accepted. Bypass secondary storage for a SharedMountPoint pool as well, which the agent already supports: it copies a volume between two file based pools in place, and an encrypted one with its passphrase.
Once the block copy of a running Instance is pivoted, the Instance writes to the destination volume. migrateRBDVolume still let what follows fail the migration: the removal of the source volume in the finally block was not guarded, the refresh of the destination pool was inside the try, and so was the removal of the libvirt secret of the source volume in checkBlockJobStatus. A failure of one of them made the agent report a failed migration, so the management server kept the volume on the source pool while the Instance used the destination one, and the Instance would start from the source volume, or not start, the next time. For an encrypted volume the destination volume was even removed as that of a failed migration. Remember that the block copy was pivoted, and from then on only log the failure to remove the secret, to refresh the pool or to remove the source volume, and never remove the destination volume.
…sh out of the migration The volume format read by LibvirtStorageVolumeXMLParser changes how getPhysicalDisk picks the format of every volume of every libvirt pool, not only of a migrated one: libvirt probes the content of a file, so a tar template is reported as raw instead of tar, and a raw volume without a suffix is raw instead of the QCOW2 default. That is a change of its own, with its own testing, so go back to the parser it had. The migration does not need it. A volume migrated to a file based pool is named after the format of the destination, so the suffix of its name matches its content, and the refresh of the destination pool after the block copy is not needed either: getVolume refreshes the pool when it does not find a volume that was written outside of libvirt. Drop the refresh of the pool, and its test.
…a format copyVolumeFromPrimaryToPrimary named the destination volume <uuid>.<extension of the default format of the destination pool>: <uuid>.raw on an RBD pool and <uuid>.qcow2 on a file based one. The name of an RBD image is not read, but a volume keeps its name when it is migrated online to another pool, where the libvirt block copy writes a qcow2 file under it. A volume copied offline to an RBD pool, and then migrated online to a file based pool, ended up as a qcow2 file named <uuid>.raw. getPhysicalDisk takes a name ending with ".raw" for a raw volume, so the Instance could not be started again: qemu was told to open the file as a block device. Name the volume of an RBD pool after its UUID only. The volume of a file based pool is still named after its format, with the ".qcow2" extension.
…ools Revert the block copy between NFS, SharedMountPoint and host-local pools for now. A volume of a running Instance is copied in place only between two RBD pools, or between an RBD pool and a file based pool (NFS, SharedMountPoint or local), as both pools are then reachable from the host the Instance runs on. Storage answers which pools those are, so that neither caller compares the pool types itself: arePoolsSameType and isAnyPoolFileBased compare two pool types, areBothPoolsFileBased tells two file based pools apart from the rest, and isCopiedInPlaceOnKvm composes the combinations a libvirt block copy reaches. VolumeApiServiceImpl asks it in the live migration check and in the encrypted volume check, LibvirtMigrateVolumeCommandWrapper asks it to choose between a block copy and a copy with qemu-img, and AncientDataMotionStrategy uses the set of file based pool types it keeps. The tests of test_volume_migration_rbd_nfs.py which hop between two file based pools assert the refusal instead of the move: a volume attached to a running Instance is answered by the live migration check whatever the livemigrate flag says, and a detached encrypted volume by the encrypted volume check. An offline hop of a plain volume between two file based pools is still allowed, and is still made.
addressed in abdaa03
reverted in dac5d81
addressed in 3a1a22c
|
sure @wido Actually many cases are covered by a smoke test introduced in this PR: Coverage14 tests in two classes:
A refused hop asserts the message of the check that answers it, the live What is verified after every migrationEvery hop is verified twice: right after the migration, and once more after the
Test resultsRun on a nested Trillian environment: KVM, two management servers, two hosts, On the current branch tip:
|
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19608 |
This branch has not been deployed
3 participants
Description
migrateVolumewithlivemigrate=trueon KVM was refused for every pair of pools, except for storage with its own handling (StorPool, PowerFlex). The error told the operator to usemigrateVirtualMachineWithVolumes, which moves the whole Instance to another host. This PR lets an operator move a single volume of a running Instance, without stopping the Instance and without moving it, between:A volume is copied in place only when the Instance's host can reach both pools, which is the case for these pairs. A live migration between two file based pools (NFS, SharedMountPoint, local) is not part of this PR and keeps being refused as before. A volume which is not attached to a running Instance can still be copied between any two of those pools, and a SharedMountPoint pool is now copied directly like the NFS and local ones.
Encrypted volumes can also be migrated between RBD and NFS, online and offline.
How it works:
VolumeApiServiceImplaccepts these pool pairs for a KVM live migration. A local pool has to be the one of the host the Instance is running on, as the copy runs there.Storageanswers which pool types are copied in place (isMigrationBetweenRbdOrRbdAndFileBased), so that no caller compares pool types itself.AncientDataMotionStrategykeeps the volume's format right after the move (RAW on RBD, QCOW2 on a file pool) and its encryption format (luks2on RBD,lukselsewhere), and bypasses secondary storage for a SharedMountPoint pool, as it does for NFS and local ones.LibvirtMigrateVolumeCommandWrapperbuilds the disk XML of the destination for the type of the destination pool (an RBD image or a qcow2 file) and runs a libvirt block copy on the running domain. The whole disk is copied, so a volume cloned from a template does not depend on a backing file of the source pool.SHALLOWis used only from RBD to RBD. After the pivot the source volume is removed. An encrypted volume gets its destination created with the encryption header first; an offline copy goes throughqemu-imgwith both secrets.<uuid>.raw; a later online migration keeps the name of a volume, so the qcow2 file written by the block copy to an NFS pool ended up named.raw, was taken for a raw volume, and the Instance could not start again (qemu was told to open the file as a block device). A volume of a file based pool keeps being named after its format, with the.qcow2extension.The branch has nine commits:
utils: add RbdUtils for the Ceph monitors of an RBD pool. This is the first commit of utils,kvm: build every Ceph monitor list with RbdUtils #14402, so the encrypted copy can build itsmon_hostwithRbdUtils.getRbdMonitors. It will drop out of this PR when utils,kvm: build every Ceph monitor list with RbdUtils #14402 is merged.KVM: live volume migration on ceph. RBD to RBD.smoke tests: exercise online volume migration for KVM on RBD.KVM: live migrate volumes between RBD, NFS, SharedMountPoint and local pools. The migration of a volume between RBD and file based pools, encrypted or not, with its unit tests andtest_volume_migration_rbd_nfs.py.KVM: copy a volume of a SharedMountPoint pool directly between primary storages. A SharedMountPoint volume was staged on secondary storage, which does not handle an encrypted volume.KVM: do not fail a volume migration whose block copy was pivoted.KVM: leave the format detection of libvirt volumes and the pool refresh out of the migration. A change of the format detection of every libvirt volume is not needed here, and neither is the refresh of the destination pool.KVM: do not name an RBD volume copied between primary storages after a format.KVM: limit live volume migration to RBD pools and RBD to file based pools. The block copy between two file based pools is left out for now.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
How Has This Been Tested?
Unit tests, on this branch:
RbdUtilsTest(17),StorageTest(10),VolumeApiServiceImplTest(164),AncientDataMotionStrategyTest(19),LibvirtMigrateVolumeCommandWrapperTest(14),KVMStorageProcessorTest(53) andLibvirtStorageAdaptorTest(23), all passing. All the tests of thecom.cloud.hypervisor.kvm(961),org.apache.cloudstack.storage.motion(78) andcom.cloud.storagepackages pass, and so does a fullmvn installof theservermodule with its upstream modules (4059 tests inserver, no failure). New cases cover the pool pairs that are accepted, the local pool of another host, the dispatch of the agent to the block copy, the failures after the pivot, and the name of a volume copied to an RBD pool and to a file based pool.Live matrices, on the same changes from my
4.23.0-cephbranch, which carries them together with the commits of my other Ceph PRs. Two KVM hosts; Ceph with two pools; two NFS pools; two SharedMountPoint pools; one local pool per host. An Instance with a ROOT and a DATA volume, moved together. After each leg:Results, from runs made before commit 9: 15 online and 15 offline legs of non-encrypted volumes passed (the online run starts with an offline move to RBD), including RBD to RBD and back, RBD to and from NFS, SharedMountPoint and local, and the moves among the file based pools. Commit 9 came after these runs. It makes the 8 online legs among NFS, SharedMountPoint and local pools refuse, so those no longer apply; the other 7 online legs, which involve RBD, and the 15 offline legs still do. I did not rerun the matrices after commit 9; its refusal is covered by the unit tests and by the smoke test, which asserts it. A run of 10 legs, which copies a volume offline to RBD and then moves it online to a file based pool, passed; that is the sequence which produced the
.rawfile before commit 8. A 5 GiB DATA disk took about 45 to 95 seconds to copy, a small ROOT disk 3 to 9 seconds.Encrypted volumes: 10 online legs passed, and the first 6 offline legs passed before I stopped that run (the 5 others were not run). All of them were between NFS, SharedMountPoint and local pools, as the KVM hosts of the lab cannot encrypt an RBD image (see below). The online ones were among file based pools, so they too predate commit 9 and are now refused.
An early run failed once on an RBD to RBD copy of the DATA disk: the copy finished and the Instance switched to the new image, but the removal of the old image by libvirt failed ("failed to remove volume … No such file or directory"), the migration was reported as failed, and CloudStack kept the old pool while the Instance used the new one. I did not find why the removal failed, and it did not happen again; the agent no longer fails a migration for it (commit 6).
Not done:
test_volume_migration_rbd_nfs.pyhas not been run by Marvin from this branch (Marvin does not work on that setup).migrateVirtualMachineWithVolumeswas not retested.How did you try to break this feature and the system with this change?
.rawname above, because a wrong format on the volume only shows when the Instance starts again.apache/main, which does not yet have kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188 or kvm: remove the partial RBD image when a template copy fails #14393. Two edits of this work tomon_hostlines in methods from those PRs were left out, so expect a small conflict inLibvirtStorageAdaptorwhen they merge.