perf(check): flow access and entity-access calls read a flow without decoding it

.claude/skills/fix-issue/findings/mdl-executor/2026-11-12-check-decodes-every-flow-per-run.json

@@ -0,0 +1 @@
+{"area": "mdl/executor", "date": "2026-11-12", "symptom": "`check` of even a five-statement script took about 2 s on a 5,373-unit project with a warm file cache (1 s at 1,373 units), unchanged from v0.28.0 through v0.31.0 to main (mendixlabs/mxcli#1842, ako/mxcli#2160)", "cause": "two whole-script checks run once per check and each read every microflow of the project in full. CheckFlowAccess listed every microflow and nanoflow through ListMicroflows/ListNanoflows, decoding and converting each flow's whole body, for its name, Excluded flag and allowed module roles. CheckEntityAccessCalls unmarshalled every microflow and rule into a bson.D to read the name, two flags and the call activities, including for the flows that hold no call", "file": "mdl/executor/validate_entity_access_calls.go (storedEAFlow, storedEAFlowDecoded), mdl/executor/validate_flow_access.go (flowAccessLister), mdl/backend/modelsdk/microflow.go (ListFlowAccess), mdl/types/infrastructure.go (FlowAccess)", "insight": "Every measurement under #2160 until this one was of exec, or of check with hundreds of statements, where a per-run cost of a second or two is noise; the reporter's batch-size table (check: 3.5 s for 10 statements, 4.5 s for 200) is what showed a cost per run, and rerunning their experiment on three versions showed nobody had ever moved it. Both checks need a few top-level values of every flow, and both got them by decoding the flow. The values are read with bson.Raw lookups instead; the calls, which are anywhere in the body, still need the document decoded, but only when its bytes contain a call type's name - a $Type is stored as its name, so a document with a call always contains it. The flow listing went into the backend rather than the executor's raw listing so that it covers exactly the units ListMicroflows covers, and so mock backends, which serve only the decoded listings, keep working through the fallback. The fixture has 16 stored flows and one with a call, too thin to carry the claim alone: the old and new readers were also compared on 5,632 flows of two generated projects, 301 with calls. Measured at 5,373 units, best of three, warm: check of 5 updates 1.98 s -> 0.91 s, 100 updates 2.91 s -> 1.75 s, 200 updates 3.85 s -> 2.83 s; output identical. Cold file reads, which made the first check after a copy take 4 s, are not changed."}

CHANGELOG.md

@@ -34,6 +34,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - **`use building block` places a block whose text-template parameters are unbound** (ako/mxcli#2181, mendixlabs/mxcli#1834) — the block's widget tree is expanded by describing it as MDL and parsing that back, and `describe` prints a parameter nobody has bound yet as `{1} = <unbound>`, which did not parse. Building blocks are full of them (PlatformDesignSystem's Gallery_List_Divider, Gallery_Card_Grid_Actions and Gallery_List_Panels; Atlas's ListItem_WithImage), so the whole block was refused with "could not expand widget tree: extraneous input '<'". The same line made `describe page` output of any page with an unbound parameter fail to re-execute. `{1} = <unbound>` now parses and writes the parameter back unbound, in its position, as Studio Pro stores it: the block is placed, mxbuild reports CE0402 "No value specified." for each parameter until it is bound, and describe -> exec of such a page is unchanged.
 - **`check` resolves an `alter page` target against the building blocks the page holds** (ako/mxcli#2182, mendixlabs/mxcli#1835) — an `alter page` / `alter snippet` on a document the same script creates was never resolved, so `set … on a_container99` after `use building block PlatformDesignSystem.Alert_Info as a_` passed `check -p` and exec then stopped on `widget "a_container99" not found`, with the page already created. A plain misspelt widget passed the same way. Check now replays the documents the script creates, expanding each `use building block` and `use fragment` as exec does (prefix included), and refuses a target they do not hold at that point as MDL-ALTER02, suggesting the nearest name (`did you mean a_container1`). On a stored page, the names a block inserted by `alter page … insert` are now its expanded names: `insert after s_container2` after inserting a block `as s_` is no longer refused, and `set … on s_text99` is no longer waived. Without a project a block cannot be expanded, so every name under its prefix is accepted (every name, for an unprefixed block), and a page holding DataGrid 2 columns or object-list items, whose addresses are derived, is not judged. The duplicate-name side (ako/mxcli#559) was already fixed.
 - **A pluggable widget's template is prepared once per widget type** (ako/mxcli#2160) — every data grid, filter and other pluggable widget a script builds loaded its template by deep-cloning the built-in one and merging the project's widget package into the clone, the same work each time. The merged template is now made once for each widget type and package and reused; each widget still gets IDs of its own. Creating 600 pages: 29.5 s before, 26.5 s after; re-running them settled: 83.1 s before, 73.9 s after. The pages written are the same: all 1,719 units of the two projects compare equal once IDs are set aside.
+- **`check` no longer decodes every microflow of the project before it starts** (mendixlabs/mxcli#1842, ako/mxcli#2160) — two of its checks, flow access (who may run a flow) and entity-access calls (MDL-SEC22), each read every microflow in full once per run, so a `check` of five statements cost seconds on a large project whatever the script held. They now read the few values they need from each stored flow, and decode a flow's body only when it contains a call. What `check` reports is unchanged. On a project of 5,373 units, with files already read once: `check` of 5 `create or modify microflow` statements 2.0 s before, 0.9 s after; of 200, 3.9 s before, 2.8 s after.
 - **A pluggable widget's own `caption`, `label`, `buttonStyle` and `icon` are the widget's, not the core properties of that name** (ako/mxcli#2180, mendixlabs/mxcli#1833) — PlatformDesignSystem's Button declares property keys named `caption`, `buttonStyle` and `icon`, its Form label one named `label`, and mxcli read those names as the core widget properties first. `alter page … set (caption: 'Save') on btn` failed with "widget has no Caption property"; `set (buttonStyle: 'outline')` printed "Altered page" and wrote nothing; `set (label: …)` wrote a core label beside the widget's own one, and `create page` did the same, so `describe` printed two labels; `check` refused the widget's own `buttonStyle: 'fillBlue'` as an unknown Mendix button style (MDL-WIDGET02) and its `icon: 'bookmark'` as "not a qualified reference"; and `create page` — and a `use building block` copy of a block holding one — dropped the `icon`. Where the widget's definition declares the key (not as a system property, as a combo box declares `Label`), the name now means that key in all three: ALTER writes it, `create page` stores it, and `check` judges the value against the widget's own enumeration. Core buttons, and pluggable widgets without such a key, are unchanged. Measured on Mendix 11.12.1 with a widget package shaped like the PDS Button: before, two `check` errors, a dropped icon and a second LabelTemplate per widget; after, `check` clean, every value stored in the widget's own property, `mx check` 0 errors.
 - **`describe page` loads the page it describes** (ako/mxcli#2160) — it loaded every page with the same local name, in whatever module, before picking the one asked for, and read the page's document twice. `Overview`, `NewEdit` and `Home` repeat across the modules of most projects, and rewriting a page describes the stored one up to three times, so a script that re-runs its pages paid for it per statement. Output is unchanged. Re-running a settled script of 600 `create or modify page` statements: 87.8 s before, 77.2 s after.
 - **The decode cache compares a unit's bytes once per run** (ako/mxcli#2160) — the cache that spares a script decoding the same unit again recognises the unit by its bytes. It kept the buffer of an earlier pass, so in the next pass every listing compared every unit of the type byte for byte: all of a project's pages, twice per page statement. It now keeps the buffer it matched. Re-running a settled script of 600 `create or modify page` statements: 103.7 s before, 86.8 s after.

mdl/backend/modelsdk/microflow.go

@@ -3,6 +3,7 @@
 package modelsdkbackend
 
 import (
+	"go.mongodb.org/mongo-driver/v2/bson"
 	"strconv"
 	"strings"
 
@@ -136,6 +137,50 @@ func (b *Backend) GetNanoflow(id model.ID) (*microflows.Nanoflow, error) {
 	return nanoflowFromGen(u.Element, u.ContainerID), nil
 }
 
+// ListFlowAccess lists, for every microflow and then every nanoflow, what
+// deciding its access takes (types.FlowAccess), in the order ListMicroflows
+// and ListNanoflows list them, and without decoding any flow: the name, the
+// Excluded flag and the allowed module roles are top-level values of the
+// stored document, read where they lie.
+//
+// `check` simulates flow access once per script and needs these three values
+// of every flow in the project. Listing the flows for them decoded and
+// converted every flow's whole body, which is what a check cost per project
+// unit before it had looked at a statement (ako/mxcli#2160,
+// mendixlabs/mxcli#1842).
+func (b *Backend) ListFlowAccess() ([]types.FlowAccess, error) {
+	var out []types.FlowAccess
+	for _, kind := range []struct{ name, typ string }{
+		{"microflow", "Microflows$Microflow"},
+		{"nanoflow", "Microflows$Nanoflow"},
+	} {
+		refs, err := b.reader.ListUnitsByTypeShared(kind.typ, b.unitScope)
+		if err != nil {
+			return nil, err
+		}
+		for _, ref := range refs {
+			if ref.Type != kind.typ {
+				continue
+			}
+			raw := bson.Raw(ref.Contents)
+			fa := types.FlowAccess{Kind: kind.name, ContainerID: model.ID(ref.ContainerID)}
+			fa.Name, _ = raw.Lookup("Name").StringValueOK()
+			fa.Excluded, _ = raw.Lookup("Excluded").BooleanOK()
+			if roles, ok := raw.Lookup("AllowedModuleRoles").ArrayOK(); ok {
+				values, _ := roles.Values()
+				for _, v := range values {
+					// The list's first value is its marker, a number.
+					if qn, ok := v.StringValueOK(); ok {
+						fa.AllowedModuleRoles = append(fa.AllowedModuleRoles, model.ID(qn))
+					}
+				}
+			}
+			out = append(out, fa)
+		}
+	}
+	return out, nil
+}
+
 // ListNanoflowHeaders lists every nanoflow's ID, container, name and Excluded
 // flag without decoding any flow.
 func (b *Backend) ListNanoflowHeaders() ([]types.DocumentHeader, error) {

mdl/executor/flow_access_listing_pedapp_test.go

@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: Apache-2.0
+
+package executor
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/mendixlabs/mxcli/mdl/types"
+)
+
+// ListFlowAccess reads a flow's name, Excluded flag and allowed module roles
+// from the stored document's top level, where the flow listings decode and
+// convert the whole flow for them. It must list what they list, in their
+// order: compared on every microflow and nanoflow of the fixture, after a
+// script has given one flow roles and excluded another, so that neither value
+// is compared only in its empty form.
+func TestListFlowAccess_ListsWhatTheFlowListingsList(t *testing.T) {
+	exec, out, _ := openPedAppCopy(t)
+	const setup = `mdl 1;
+create module role MyFirstModule.FlowAccessProbe;
+create microflow MyFirstModule.FlowAccessGranted () returns boolean begin return true; end;
+grant execute on microflow MyFirstModule.FlowAccessGranted to MyFirstModule.FlowAccessProbe;
+@excluded
+create microflow MyFirstModule.FlowAccessExcluded () returns boolean begin return true; end;
+`
+	if err := runScript(t, exec, setup); err != nil {
+		t.Fatalf("setup: %v\n%s", err, out.String())
+	}
+	lister, ok := exec.Backend().(flowAccessLister)
+	if !ok {
+		t.Fatalf("backend %T does not list flow access", exec.Backend())
+	}
+	got, err := lister.ListFlowAccess()
+	if err != nil {
+		t.Fatalf("ListFlowAccess: %v", err)
+	}
+
+	var want []types.FlowAccess
+	mfs, err := exec.Backend().ListMicroflows()
+	if err != nil {
+		t.Fatalf("ListMicroflows: %v", err)
+	}
+	for _, mf := range mfs {
+		want = append(want, types.FlowAccess{Kind: "microflow", ContainerID: mf.ContainerID, Name: mf.Name, Excluded: mf.Excluded, AllowedModuleRoles: mf.AllowedModuleRoles})
+	}
+	nfs, err := exec.Backend().ListNanoflows()
+	if err != nil {
+		t.Fatalf("ListNanoflows: %v", err)
+	}
+	for _, nf := range nfs {
+		want = append(want, types.FlowAccess{Kind: "nanoflow", ContainerID: nf.ContainerID, Name: nf.Name, Excluded: nf.Excluded, AllowedModuleRoles: nf.AllowedModuleRoles})
+	}
+
+	withRoles, excluded := 0, 0
+	for _, f := range want {
+		if len(f.AllowedModuleRoles) > 0 {
+			withRoles++
+		}
+		if f.Excluded {
+			excluded++
+		}
+	}
+	t.Logf("%d microflows, %d nanoflows; %d with roles, %d excluded", len(mfs), len(nfs), withRoles, excluded)
+	if len(mfs) == 0 || len(nfs) == 0 || withRoles == 0 || excluded == 0 {
+		t.Fatalf("control: the fixture must hold microflows, nanoflows, a flow with roles and an excluded one")
+	}
+	if len(got) != len(want) {
+		t.Fatalf("ListFlowAccess lists %d flows, the flow listings %d", len(got), len(want))
+	}
+	for i := range want {
+		if !reflect.DeepEqual(got[i], want[i]) {
+			t.Errorf("flow %d: ListFlowAccess gives %+v, the flow listing %+v", i, got[i], want[i])
+		}
+	}
+}

mdl/executor/validate_entity_access_calls.go

@@ -3,10 +3,12 @@
 package executor
 
 import (
+	"bytes"
 	"context"
 	"fmt"
 	"sort"
 	"strings"
+	"sync/atomic"
 
 	"go.mongodb.org/mongo-driver/v2/bson"
 
@@ -93,21 +95,15 @@ func CheckEntityAccessCalls(ctx *ExecContext, prog *ast.Program) []linter.Violat
 			if u.Type != typ || len(u.Contents) == 0 {
 				continue
 			}
-			var doc bson.D
-			if err := bson.Unmarshal(u.Contents, &doc); err != nil {
+			st, ok := storedEAFlow(u.Contents, kind == "microflow")
+			if !ok {
 				continue
 			}
-			name, _ := bsonField(doc, "Name").(string)
-			qn := h.GetQualifiedName(u.ContainerID, name)
-			excluded, _ := bsonField(doc, "Excluded").(bool)
+			qn := h.GetQualifiedName(u.ContainerID, st.name)
 			if prev, ok := sim.flows[qn]; ok && !prev.excluded {
 				continue // a live namesake wins over excluded ones (#914)
 			}
-			ea, _ := bsonField(doc, "ApplyEntityAccess").(bool)
-			f := &eaFlow{kind: kind, name: qn, exists: true, excluded: excluded, ea: ea}
-			if kind == "microflow" {
-				f.calls = collectRawEntityAccessCalls(doc)
-			}
+			f := &eaFlow{kind: kind, name: qn, exists: true, excluded: st.excluded, ea: st.ea, calls: st.calls}
 			sim.flows[qn] = f
 		}
 	}
@@ -173,6 +169,75 @@ func scriptTouchesEntityAccessCalls(prog *ast.Program) bool {
 	return false
 }
 
+// storedEAState is what the simulation reads of one stored microflow or rule.
+type storedEAState struct {
+	name         string
+	excluded, ea bool
+	calls        []eaCall
+}
+
+// eaCallMarkers are the $Type values collectRawEntityAccessCalls looks for. A
+// document that holds neither string holds no call.
+var eaCallMarkers = [][]byte{[]byte("Microflows$MicroflowCall"), []byte("Microflows$RuleCall")}
+
+// eaDocumentsDecoded counts the stored documents decoded in full to read
+// their calls. For tests.
+var eaDocumentsDecoded atomic.Int64
+
+// storedEAFlow reads one stored microflow or rule for the simulation. ok is
+// false for contents that are not a document.
+//
+// The check runs once per script and reads every microflow of the project, so
+// what it costs per document is what `check` costs per project unit before it
+// has looked at a statement (ako/mxcli#2160, mendixlabs/mxcli#1842). The name
+// and the two flags are top-level values, read where they lie. The calls are
+// anywhere in the body, and finding them takes the whole document decoded:
+// that is done only for a document whose bytes hold one of the call types'
+// names, which a document with a call always does, since a $Type is stored as
+// its name.
+func storedEAFlow(contents []byte, withCalls bool) (storedEAState, bool) {
+	raw := bson.Raw(contents)
+	if raw.Validate() != nil {
+		return storedEAState{}, false
+	}
+	var st storedEAState
+	st.name, _ = raw.Lookup("Name").StringValueOK()
+	st.excluded, _ = raw.Lookup("Excluded").BooleanOK()
+	st.ea, _ = raw.Lookup("ApplyEntityAccess").BooleanOK()
+	if !withCalls {
+		return st, true
+	}
+	for _, marker := range eaCallMarkers {
+		if bytes.Contains(contents, marker) {
+			var doc bson.D
+			if err := bson.Unmarshal(contents, &doc); err != nil {
+				return storedEAState{}, false
+			}
+			eaDocumentsDecoded.Add(1)
+			st.calls = collectRawEntityAccessCalls(doc)
+			break
+		}
+	}
+	return st, true
+}
+
+// storedEAFlowDecoded is storedEAFlow as it was: the whole document decoded,
+// whatever it holds. It is what storedEAFlow is compared with.
+func storedEAFlowDecoded(contents []byte, withCalls bool) (storedEAState, bool) {
+	var doc bson.D
+	if err := bson.Unmarshal(contents, &doc); err != nil {
+		return storedEAState{}, false
+	}
+	var st storedEAState
+	st.name, _ = bsonField(doc, "Name").(string)
+	st.excluded, _ = bsonField(doc, "Excluded").(bool)
+	st.ea, _ = bsonField(doc, "ApplyEntityAccess").(bool)
+	if withCalls {
+		st.calls = collectRawEntityAccessCalls(doc)
+	}
+	return st, true
+}
+
 // collectRawEntityAccessCalls reads the calls out of a stored microflow: the
 // qualified name under `Microflow` in a Microflows$MicroflowCall (call
 // activities, queued or not) or a Microflows$RuleCall (a rule split).