Skip to content

vmware: reduce repeated requests for the same vCenter data - #634

Open
tyderian1978 wants to merge 1 commit into
bb-Ricardo:developmentfrom
tyderian1978:perf/vmware-quick-wins
Open

tyderian1978 wants to merge 1 commit into
bb-Ricardo:developmentfrom
tyderian1978:perf/vmware-quick-wins

Conversation

@tyderian1978

Copy link
Copy Markdown

Disclaimer: AI generated, based off the work we're doing with Netbox.

This is one of three pull requests, aimed to significantly reduce both VMware inventory sync time and data usage.

What happens

pyVmomi caches nothing: every attribute read on a managed object is a SOAP Fetch of the whole top-level property, so grab(vm, "config.uuid") and grab(vm, "config.hardware.device") each transfer the VM's complete config. The VMware source read a VM's config about 9 times, its guest once per NIC, a host's ~1 MB config 6 times, and walked the same folders and resource pools up the tree again for every VM. On the vchvr vcsim capture that's 1469 SOAP calls and 20.1 MB per run.

Fix

  • config, runtime and guest of a VM, and summary and config of a host, are read once per object.
  • The second pass over all VMs skips VMs already parsed in the first pass by managed object id, instead of requesting their name and config again.
  • Names, parents and resource pools are cached for the run. The walks for VM folder and resource pool paths, tenants and the resource pool filter use these caches.
  • The tags attached to an object are cached for the run, since the tags of parent folders, clusters and datacenters were requested again for every VM. Tag and category definitions are already cached since vmware: fetch each vCenter tag and category once per run #621.
  • Custom attribute definitions are cached per object type and only requested if the object has custom values.
  • add_update_interface() gets the guest info that was already read through a new SourceBase.get_vmware_guest_info() hook.

This also fixes a bug: get_parent_object_by_class() kept its depth in an instance attribute that wasn't reset when a walk ended at the root, so later lookups could give up early. It now walks iteratively and keeps no state.

Tests

tests/test_vmware_vcenter_requests.py runs the sync against vcsim with the captured inventories and counts the SOAP requests per object and property:

  • no host or VM property is requested more than twice
  • a VM parsed in the first pass isn't requested again in the second
  • tag definitions and attached tags are requested once
  • parents are requested once per object
  • an unsuccessful parent lookup doesn't break later ones (10-level folder chain)
  • custom field definitions are only requested for objects with custom values

All of them fail on development without the change. Full suite: 842 passed, 8 skipped.

Numbers

vcsim capture development this PR
vchvr 1469 SOAP calls, 20.1 MB 323 calls, 3.6 MB
vc001 1707 SOAP calls, 30.1 MB 355 calls, 6.0 MB

The synced inventory is identical to development's for both captures, with default settings and with the FC adapter, tenant and resource pool settings.

A dry run against our production NetBox and five vCenters (about 1,500 VMs, over VPN, tag sync on) shows the same pattern. The VMware source went from 80,654 SOAP requests, 1.15 GB and 48 min on development to 12,435 requests, 198 MB and 10.5 min. Its planned changes were identical to those of runs with all three PRs made directly before and after it. Those runs in turn matched development's, apart from one VM that was removed from vCenter during the runs.

Every read of a property of a pyVmomi managed object is a SOAP round trip
which returns the whole top level property. The VMware source read e.g. a
VM's 'config' about 9 times, its 'guest' once per NIC, the ~1 MB 'config'
of a host 6 times per host, and walked the same folders and resource pools
again for every VM.

* read 'config', 'runtime', 'guest' of a VM and 'summary', 'config' of a
  host once per object
* skip VMs already parsed in the first run over all VMs by their managed
  object id instead of requesting name and config again
* cache names, parents and resource pools of objects for the run; the
  folder and resource pool walks for VM paths, tenants and resource pool
  filters use these caches
* cache the tags attached to an object for the run (the tags of parent
  folders, clusters and datacenters are requested for every VM); tag and
  category definitions are already cached since ec1ba92
* cache custom attribute definitions per object type and only request
  them if the object has custom values
* hand the already retrieved guest info to add_update_interface() through
  the new SourceBase.get_vmware_guest_info() hook

get_parent_object_by_class() kept its depth in an instance attribute which
was not reset when a walk ended at the root, so later lookups could give up
early. The rewrite walks iteratively and has no such state.

Measured against vcsim with the captured inventories, synced inventory
unchanged:
  vchvr: 1469 -> 323 SOAP calls, 20.1 -> 3.6 MB
  vc001: 1707 -> 355 SOAP calls, 30.1 -> 6.0 MB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant