Skip to content

vmware: retrieve object properties from vCenter in bulk - #635

Open
tyderian1978 wants to merge 2 commits into
bb-Ricardo:developmentfrom
tyderian1978:perf/vmware-bulk-fetch
Open

tyderian1978 wants to merge 2 commits into
bb-Ricardo:developmentfrom
tyderian1978:perf/vmware-bulk-fetch

Conversation

@tyderian1978

@tyderian1978 tyderian1978 commented Nov 18, 2026 •

Copy link
Copy Markdown

Disclaimer: AI generated, based off the work we're doing with Netbox.

This is the second of three pull requests, aimed to significantly reduce both VMware inventory sync time and data usage.

Builds on #634: its commit is the first one here, and the change in this PR is the second commit (6f2f5a7).

What happens

Even with each property read only once (#634), every VM and host still costs several SOAP round trips: name, parent, config, runtime, guest and so on. The number of requests grows with the inventory, and over a WAN or VPN link the round trips dominate the run. In our production dry run the VMware source still spent 10.5 minutes, mostly waiting for vCenter.

Fix

The properties the handlers need are retrieved for many objects at once with the PropertyCollector: RetrievePropertiesEx over a ListView, paged with ContinueRetrievePropertiesEx. A ListView drops objects deleted in the meantime instead of failing the request.

The new PropertyCache (module/sources/vmware/property_cache.py) wraps the session's SoapStubAdapter as its outerStub. As a result, every managed object returned through it (prefetched objects, their parents, a VM's host and resource pool, ...) reads prefetched properties from the cache. Properties that weren't prefetched are still requested from vCenter, so a path missing from the prefetch list costs time but never returns wrong data. The handlers are unchanged apart from reading config.network, config.storageDevice.hostBusAdapter and custom object attributes through get_prefetched_value().

  • Objects are prefetched and parsed in chunks of 250, and the heavy properties of a chunk are dropped afterwards to bound memory (name and parent are kept for the tree walks).
  • Folders and resource pools are prefetched (name and parent) for the walks up the tree: datacenter, VM folder and resource pool paths.
  • Only the parts of a host's ~1 MB config that are used are requested.
  • The second pass over all VMs only requests the VMs not parsed yet.
  • Objects to reevaluate are prefetched again in one go.
  • Attached vCenter tags are requested per chunk with list_attached_tags_on_objects(), falling back to one request per object.
  • Custom object attribute paths that vCenter refuses (InvalidProperty) are dropped from the bulk request for the rest of the run and read object by object.
  • If the session stub isn't a SoapStubAdapter, or bulk retrieval fails, objects read their properties one by one as before.

For review

  • PropertyCache relies on pyVmomi's SoapStubAdapter.InvokeMethod(mo, info, args, outerStub) and StubAdapterAccessorMixin.InvokeAccessor. It was tested with pyvmomi 9.1.1.0, as pinned in requirements.txt.
  • vcsim's tagging API doesn't work with the vcf-sdk (create and list calls return nothing), and vcsim ignores invalid property paths. So the bulk tag call, its per-object fallback and the InvalidProperty retry are covered by tests with fakes, not by vcsim. Against our production vCenters, tag sync ran without errors and gave the same tags as development.

Tests

Added to tests/test_vmware_vcenter_requests.py, against vcsim unless noted:

  • VM and host properties are retrieved in bulk
  • the property cache serves prefetched properties without requests
  • an unknown custom object attribute doesn't break the sync
  • attached tags are retrieved in bulk, and per object if the bulk call fails (fake tagging API)
  • an invalid custom object attribute path is retried without it (fake InvalidProperty)

All of the file's tests fail on development except the unknown custom object attribute test, which guards behaviour development already has. Full suite: 851 passed, 8 skipped.

Numbers

vcsim capture development #634 this PR
vchvr 1469 SOAP calls, 20.1 MB 323 calls, 3.6 MB 68 calls, 1.5 MB
vc001 1707 SOAP calls, 30.1 MB 355 calls, 6.0 MB 66 calls, 2.5 MB

The number of requests no longer grows with the number of VMs: about 4 per 250 VMs, plus one QueryNetworkHint per host. The synced inventory is identical to development's for both captures, with default settings and with the FC adapter, tenant and resource pool settings.

A dry run against our production NetBox and five vCenters (about 1,500 VMs, over VPN, tag sync on):

VMware source SOAP requests transferred time
development 80,654 1.15 GB 48 min
#634 12,435 198 MB 10.5 min
this PR 442 91 MB 2 min

The planned changes are the same as development's. The only differences come from one VM that was removed from vCenter between the runs.

Every read of a property of a pyVmomi managed object is a SOAP round trip
which returns the whole top level property. The VMware source read e.g. a
VM's 'config' about 9 times, its 'guest' once per NIC, the ~1 MB 'config'
of a host 6 times per host, and walked the same folders and resource pools
again for every VM.

* read 'config', 'runtime', 'guest' of a VM and 'summary', 'config' of a
  host once per object
* skip VMs already parsed in the first run over all VMs by their managed
  object id instead of requesting name and config again
* cache names, parents and resource pools of objects for the run; the
  folder and resource pool walks for VM paths, tenants and resource pool
  filters use these caches
* cache the tags attached to an object for the run (the tags of parent
  folders, clusters and datacenters are requested for every VM); tag and
  category definitions are already cached since ec1ba92
* cache custom attribute definitions per object type and only request
  them if the object has custom values
* hand the already retrieved guest info to add_update_interface() through
  the new SourceBase.get_vmware_guest_info() hook

get_parent_object_by_class() kept its depth in an instance attribute which
was not reset when a walk ended at the root, so later lookups could give up
early. The rewrite walks iteratively and has no such state.

Measured against vcsim with the captured inventories, synced inventory
unchanged:
  vchvr: 1469 -> 323 SOAP calls, 20.1 -> 3.6 MB
  vc001: 1707 -> 355 SOAP calls, 30.1 -> 6.0 MB
pyVmomi requests each property read from vCenter separately. Retrieve the
properties the handlers need for many objects at once with the
PropertyCollector (RetrievePropertiesEx over a ListView, paged with
ContinueRetrievePropertiesEx) instead.

The new PropertyCache wraps the session's SoapStubAdapter as its
'outerStub', so every managed object returned through it (prefetched
objects, their parents, the host and resource pool of a VM, ...) reads
prefetched properties from the cache. Properties which were not prefetched
are still requested from vCenter: a path missing from the prefetch list
costs time but never returns wrong data. The handlers are unchanged apart
from reading the nested paths 'config.network',
'config.storageDevice.hostBusAdapter' and custom object attributes through
get_prefetched_value().

* objects are prefetched and parsed in chunks of 250, the heavy properties
  of a chunk are dropped afterwards to bound memory
* folders and resource pools are prefetched (name, parent) for the walks up
  the tree (datacenter, VM folder and resource pool paths)
* only the parts of a host's ~1 MB 'config' which are used are requested
* the second run over all VMs only requests the VMs not parsed yet
* objects to reevaluate are prefetched again in one go
* attached vCenter tags are requested per chunk with
  list_attached_tags_on_objects(), falling back to one request per object
* custom object attribute paths vCenter refuses are dropped from the bulk
  request for the rest of the run and read object by object
* if bulk retrieval fails, objects read their properties one by one as before

Measured against vcsim with the captured inventories, synced inventory
unchanged (relative to the previous commit / to development):
  vchvr: 323 / 1469 -> 68 SOAP calls, 3.6 / 20.1 -> 1.5 MB
  vc001: 355 / 1707 -> 66 SOAP calls, 6.0 / 30.1 -> 2.5 MB
The number of requests no longer grows with the number of VMs (about 4 per
250 VMs plus one QueryNetworkHint per host).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant