Skip to content

VMware: wait for NSX DV port group readiness - #13753

Open
Dogface2k wants to merge 1 commit into
apache:4.22from
Dogface2k:agent/wait-for-nsx-dvportgroup
Open

Dogface2k wants to merge 1 commit into
apache:4.22from
Dogface2k:agent/wait-for-nsx-dvportgroup

Conversation

@Dogface2k

@Dogface2k Dogface2k commented Jul 30, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Fixes #13752 by retrying the vCenter lookup while an NSX segment's backing distributed port group becomes visible.

  • Add vmware.nsx.dvportgroup.wait.timeout: default 60 seconds, capped by the VMware operation timeout. Zero performs one lookup; negative values are rejected before saving.
  • Check after every completed sleep, including the final sleep. Preserve lookup errors and interruption.
  • Convert typed configuration values before validation.

Testing

Windows 11, Java 17.0.20.1, Maven 3.9.11, VMware 8.0 SDK. The 37-module reactor passed with zero Checkstyle violations and 101 tests passed:

  • ConfigKeyTest: 8
  • HypervisorHostHelperTest: 46
  • VmwareResourceTest: 47

The final-sleep regression fails on the previous implementation and passes with the fix. It covers 25 ms and 2,500 ms budgets, with wake-ups at the deadline and 1 ms late.

testOvfDomRewriter was excluded after reproducing the same Windows line-ending assertion failure on the previous PR head.

Remaining validation

No live NSX/vCenter validation or full repository test suite was run. Delayed visibility and permanent absence still need validation against a real environment.

@boring-cyborg

boring-cyborg Bot commented Jul 30, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

@Dogface2k
Dogface2k marked this pull request as ready for review July 30, 2026 15:30
@DaanHoogland DaanHoogland added this to the 4.22.2 milestone Jul 31, 2026

@DaanHoogland DaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@codecov

codecov Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.30435% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.05%. Comparing base (74e7105) to head (201ffc7).
⚠️ Report is 56 commits behind head on 4.22.

Files with missing lines Patch % Lines
...oud/hypervisor/vmware/resource/VmwareResource.java 80.00% 1 Missing and 1 partial ⚠️
...cloud/hypervisor/vmware/manager/VmwareManager.java 75.00% 0 Missing and 1 partial ⚠️
...oud/hypervisor/vmware/mo/HypervisorHostHelper.java 96.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               4.22   #13753      +/-   ##
============================================
+ Coverage     17.84%   18.05%   +0.20%     
- Complexity    16025    16316     +291     
============================================
  Files          5928     5937       +9     
  Lines        534457   536806    +2349     
  Branches      65404    65787     +383     
============================================
+ Hits          95388    96909    +1521     
- Misses       428239   428808     +569     
- Partials      10830    11089     +259     
Flag Coverage Δ
uitests 4.04% <ø> (+0.01%) ⬆️
unittests 19.12% <91.30%> (+0.21%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18740

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan help

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland [SL] I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [keepEnv] [qemuEv] [basicZone|securityGroups]
Mgmt OS options: ['suse15', 'alma10', 'ol10', 'rocky10', 'alma9', 'centos7', 'centos6', 'rocky9', 'alma8', 'ubuntu18', 'ol9', 'ol8', 'ubuntu22', 'debian12', 'ubuntu20', 'ubuntu26', 'rocky8', 'ubuntu24']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-rocky8', 'kvm-rocky9', 'kvm-rocky10', 'kvm-ol8', 'kvm-ol9', 'kvm-ol10', 'kvm-alma8', 'kvm-alma9', 'kvm-alma10', 'kvm-ubuntu18', 'kvm-ubuntu20', 'kvm-ubuntu22', 'kvm-ubuntu24', 'kvm-ubuntu26', 'kvm-debian12', 'kvm-suse15', 'vmware-55u3', 'vmware-60u2', 'vmware-65u2', 'vmware-67u3', 'vmware-70u1', 'vmware-70u2', 'vmware-70u3', 'vmware-80', 'vmware-80u1', 'vmware-80u2', 'vmware-80u3', 'vmware-80u3e', 'xenserver-65sp1', 'xenserver-71', 'xenserver-74', 'xenserver-84', 'xcpng74', 'xcpng76', 'xcpng80', 'xcpng81', 'xcpng82', 'xcpng83']
Note: when keepEnv is passed, you need to specify mgmt server os and hypervisor or use the matrix command.
when qemuEv is passed, it will deploy KVM hyperviosr hosts with qemu-kvm-ev, else it will default to stock qemu.
When basicZone and/or securityGroups are passed it will create a zone of the last type specified (default is Advanced)
Package command usage: package [all(default value),kvm,xen,vmware,hyperv,ovm] - a comma separated list can be passed with package command to bundle the required hypervisor's systemVM templates. Not passing any argument will bundle all - kvm,xen and vmware templates.

Blessed contributors for kicking Trillian test jobs: ['rohityadavcloud', 'shwstppr', 'Damans227', 'vishesh92', 'Pearl1594', 'harikrishna-patnala', 'nvazquez', 'DaanHoogland', 'weizhouapache', 'borisstoyanov', 'vladimirpetrov', 'kiranchavala', 'andrijapanicsb', 'NuxRo', 'rajujith', 'sureshanaparti', 'abh1sar', 'sudo87', 'RosiKyu']

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test ol10 vmware-80u3

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol10 mgmt + vmware-80u3) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-16671)

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test ol9 vmware-80u3

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol9 mgmt + vmware-80u3) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-16791)

@Pearl1594 Pearl1594 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code lgtm. I don't have a test env to test this fix. But it looks pretty straightforward. Thanks @Dogface2k

@Pearl1594

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19155

@Pearl1594

Copy link
Copy Markdown
Contributor

@blueorangutan test ol9 vmware-80u3

@blueorangutan

Copy link
Copy Markdown

@Pearl1594 a [SL] Trillian-Jenkins test job (ol9 mgmt + vmware-80u3) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16935)
Environment: vmware-80u3 (x2), zone: Advanced Networking with Mgmt server ol9
Total time taken: 303029 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13753-t16935-vmware-80u3.zip
Smoke tests completed. 137 look OK, 12 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_enableHumanReadableLogs Error 0.17 test_human_readable_logs.py
test_list_vms_metrics_admin Error 3623.52 test_metrics_api.py
test_list_vms_metrics_history Error 5.99 test_metrics_api.py
test_list_volumes_metrics_history Error 3624.95 test_metrics_api.py
test_01_vpn_usage Error 1.12 test_usage.py
test_01_deploy_vm_on_specific_host Error 19.07 test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster Error 3605.68 test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod Error 4.59 test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster Error 4.57 test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod Error 3604.33 test_vm_deployment_planner.py
test_02_offline_migrate_VM_with_two_data_disks Error 3658.75 test_vm_life_cycle.py
test_03_live_migrate_VM_with_two_data_disks Error 45.83 test_vm_life_cycle.py
test_08_migrate_vm Error 3629.38 test_vm_life_cycle.py
test_09_expunge_vm Failure 427.85 test_vm_life_cycle.py
test_10_attachAndDetach_iso Failure 48.74 test_vm_life_cycle.py
test_11_destroy_vm_and_volumes Error 1842.75 test_vm_life_cycle.py
test_13_destroy_and_expunge_vm Error 1973.62 test_vm_life_cycle.py
test_01_migrate_vm_strict_tags_success Error 8.91 test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failure Error 8.83 test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_success Error 8.91 test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failure Error 8.95 test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_success Error 8.96 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Error 8.96 test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tags Error 8.77 test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tags Error 11.05 test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_success Error 8.74 test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_success Error 11.87 test_vm_strict_host_tags.py
test_04_deploy_vnf_appliance Error 2025.08 test_vnf_templates.py
test_05_delete_vnf_template Error 1.11 test_vnf_templates.py
ContextSuite context=TestVnfTemplates>:teardown Error 2.23 test_vnf_templates.py
ContextSuite context=TestCreateVolume>:teardown Error 2234.97 test_volumes.py
test_11_attach_volume_with_unstarted_vm Error 1930.82 test_volumes.py
test_13_migrate_volume_and_change_offering Error 2410.60 test_volumes.py
ContextSuite context=TestVolumes>:teardown Error 1824.06 test_volumes.py
test_01_verify_ipv6_vpc Error 2374.56 test_vpc_ipv6.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Error 3611.54 test_vpc_redundant.py
test_02_redundant_VPC_default_routes Error 3611.65 test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers Error 3609.80 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Error 3610.49 test_vpc_redundant.py
test_05_rvpc_multi_tiers Error 3686.67 test_vpc_redundant.py
test_01_VPC_nics_after_destroy Failure 5771.22 test_vpc_router_nics.py
test_02_VPC_default_routes Error 3609.75 test_vpc_router_nics.py
test_01_redundant_vpc_site2site_vpn Failure 3687.56 test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn Error 3687.82 test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options Failure 3687.51 test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options Error 3687.76 test_vpc_vpn.py
test_01_vpc_remote_access_vpn Error 1957.89 test_vpc_vpn.py
test_01_vpc_site2site_vpn Failure 3614.47 test_vpc_vpn.py
test_01_vpc_site2site_vpn Error 3614.83 test_vpc_vpn.py

@Dogface2k
Dogface2k marked this pull request as draft October 16, 2026 19:41
@Dogface2k
Dogface2k force-pushed the agent/wait-for-nsx-dvportgroup branch from 250f972 to ca6706f Compare October 16, 2026 20:57
@Dogface2k
Dogface2k marked this pull request as ready for review October 17, 2026 17:41
@Dogface2k
Dogface2k marked this pull request as draft October 17, 2026 19:18
Poll vCenter for named NSX backing port groups using a validated timeout,
defaulting to 60 seconds and capped by the VMware operation timeout.
Follow each completed sleep with a lookup, including the final sleep,
and preserve lookup errors and interruption.

Convert typed configuration values before validation. Cover timeout
routing, configuration updates and readiness behavior in the existing
configuration and VMware test suites.

Co-authored-by: dahn <daan.hoogland@gmail.com>
@Dogface2k
Dogface2k force-pushed the agent/wait-for-nsx-dvportgroup branch from ca6706f to 201ffc7 Compare October 17, 2026 19:35
@Dogface2k
Dogface2k marked this pull request as ready for review October 17, 2026 20:10

@Damans227 Damans227 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @Dogface2k, fix looks right. LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Ready

Development

Successfully merging this pull request may close these issues.

VMware NSX network implementation can fail before backing DV port group is visible

6 participants