Repository navigation
inventory: index lookups instead of scanning every object - #636
Open
tyderian1978 wants to merge 1 commit into
Open
tyderian1978 wants to merge 1 commit into
tyderian1978 wants to merge 1 commit into
Conversation
Syncing N objects compared each one with every object of its type in
several places, so a sync cost O(N^2): finding a VM/device by name and
cluster or a VM interface/disk by name and VM, listing the interfaces,
IPs, MACs and disks of an object, the longest matching prefix, matching
VMs by MAC, serial or primary IP and resolving NetBox IDs.
The inventory now keeps per object type indexes from an attribute value to
the objects carrying it (NetBox ID, primary key, interface parent, IP
address, assigned object, MAC address, prefix network, serial, asset tag,
primary IPs). An index is built on the first lookup of its type.
Invariants:
* An index only selects candidates. Every lookup still applies its
original comparison to them in inventory order, so it returns exactly
what the full scan returned (first match wins, case-insensitive names).
* Keys depend only on the object's own data. The primary key index files
an object under its lower-cased primary key value, which every display
name starts with; a lookup tries the whole name and every part in front
of a " (". Values without a stable string (i.e. objects) are filed as
volatile and returned for every lookup.
* NetBoxObject.data is an NBDataDict which reports every write, and
assigning data or nb_id is reported too, so an object is re-keyed right
away whatever changes it (update(), resolve_relations(), direct writes).
* An index is rebuilt if its object list was replaced or changed outside
add_object(), as tests do.
SourceBase.ip_address_objects_for() (4f5f686) keeps its interface and
callers but now answers from the inventory's address index. That index
follows address changes of existing objects and a replaced inventory list,
which the append-only per-source index did not, and it is the same index
the primary IP matching uses, so there is one address index instead of two.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer: AI generated, based off the work we're doing with Netbox.
This is the third of three pull requests, aimed to significantly reduce both VMware inventory sync time and data usage.
Independent of the two VMware PRs (#634, #635); this one only touches the inventory and its callers.
What happens
Syncing N objects compares each one with every object of its type in several places, so a sync costs O(N²). Those places are:
This doesn't show while vCenter reads dominate a run, but it does once they're fast. In a benchmark where the inventory already holds N VMs (1.5 NICs, MACs and IPs per VM, N/4 prefixes), running the VMware source's per-VM matching for every VM through the real code paths takes:
developmentThe indexes cost memory: peak RSS at 4000 VMs goes from 102 MB to 130 MB.
Fix
The inventory keeps per-type indexes from an attribute value to the objects carrying it: NetBox ID, primary key, interface parent, IP address, assigned object, MAC address, prefix network, serial, asset tag and primary IPs. An index is built on the first lookup of its type.
" (". Values without a stable string, i.e. objects, are filed as volatile and returned for every lookup.NetBoxObject.datais anNBDataDictthat reports every write, and assigningdataornb_idis reported too. So an object is re-keyed right away, whatever changes it:update(),resolve_relations()or a direct write.add_object(), as tests do.SourceBase.ip_address_objects_for()from #622 keeps its interface and callers but now answers from the inventory's address index. That index follows address changes of existing objects and a replaced inventory list, which the append-only per-source index didn't, and the primary IP matching uses the same index, so there's one address index instead of two.Unchanged:
set_primary_ip = always, each existing primary IP still scans all devices and VMs to remove it from other objects.get_by_data(). An unlabelled cable's display name doesn't start with its primary key, so a future caller would need a look.Tests
tests/test_inventory_indexes.pycompares the indexed lookups with full scans. It covers:tests/test_ip_address_index.pygets a case for an address change after the first lookup.test_inventory_indexes.pychecks the index API itself, so it doesn't run ondevelopment. The newtest_ip_address_index.pycase fails there. Full suite: 850 passed, 8 skipped.Numbers
In a dry run against our production NetBox (about 37,000 IPs, 1,500 VMs) and five vCenters, on top of the bulk fetch PR, the VMware source took 122 s without this PR and 52 s and 70 s in two runs with it. Without the SOAP requests, that's 91 s against 23 s and 28 s. The planned changes were byte-identical with and without this PR. Without bulk fetch, the saving is lost in the 48 min of vCenter requests.