Skip to content

✨ Support sub-field projections - #774

Closed
Mohith26 wants to merge 2 commits into
share:mainfrom
Mohith26:subfield-projections
Closed

Mohith26 wants to merge 2 commits into
share:mainfrom
Mohith26:subfield-projections

Conversation

@Mohith26

@Mohith26 Mohith26 commented Nov 8, 2026 •

Copy link
Copy Markdown

Fixes #365

addProjection() already accepts dotted keys like 'owner.name', and sharedb-mongo fetches honour them, but op projection only checks the first path segment. So a query or doc subscribed through the projection never sees changes to the projected sub-fields, and the memory DB treats 'owner.name' as a literal key.

This turns the projection fields into a tree and uses it everywhere projections are applied, covering the cases listed in the issue (ops exactly at, below, and above a projected sub-field):

  • Components at or below a projected sub-field pass through unchanged.
  • Components on non-projected siblings (owner.phone) are dropped.
  • oi/od that replace a parent of projected sub-fields (owner, or the root) keep only the projected parts of the old and new values. If the new value isn't an object, the projected op deletes the parent, whether or not the original op had an od.
  • Sub-field paths only go through objects. If an array or scalar is in the way, it's left out of the projection, since projecting inside arrays can't keep list indexes in sync.
  • Submits through a projection may only change values at or below a projected path. Replacing a parent is rejected, because it could overwrite fields the client can't see.
  • addProjection() validates sub-field paths (in projections.js): no empty segments, no numeric segments, and no overlapping paths like {owner: true, 'owner.name': true} (MongoDB 4.4+ rejects those as a path collision). Top-level keys keep their existing rules.

Because Mongo projects dotted paths through arrays of documents, a DB that projects snapshots itself could disagree with the op projection at the edges. For projections that have sub-fields, ShareDB now also projects the snapshot after the DB does (a no-op when they agree), so snapshots always match the ops. Projections without sub-fields are unchanged. skipPoll still gets the unprojected op, so query polling isn't affected.

The field tree is re-derived from the fields on each call (no cache, to stay ES3 and to pick up changes to projection.fields). A quick micro-benchmark of projectOp with flat fields (1M two-component ops, median of 7, Apple Silicon, Node 26) was about 47 ms on main and 150 ms on this branch, so roughly 0.1 µs extra per op.

One behaviour change to flag: a field like {'a.b': true} used to match a literal top-level key 'a.b' (with the memory DB only). It now means the sub-field b of a, which matches what Mongo already did.

Tests

  • Unit tests for snapshot, op and submit projection with sub-fields, including prototype keys, parents replaced with and without od, and field validation.
  • A randomized test that applies random json0 ops (oi/od, sometimes without od, li/ld/lm, na, si/sd, root replace) to random documents and checks that applying the projected op to the projected snapshot always equals projecting the updated snapshot, for several field sets.
  • Client tests in the shared DB suite (fetch, subscribe, queries, op fetch/subscribe, submits), so DB adapters running test/db.js exercise it too.
  • npm test (1147 passing) and npm run lint pass locally. lib/projections.js has 100% line coverage.

Docs are updated in docs/projections.md and the addProjection() API page.

Known limitation: a json0 subtype op (t, eg rich-text) applied to a parent of a projected sub-field can't be projected, so it's dropped. This is noted in the docs.

Fixes #365

`addProjection()` accepted dotted keys like `'owner.name'`, and
sharedb-mongo's fetches honoured them, but op projection only looked at
the first path segment, so subscribers never saw changes to projected
sub-fields (and the memory DB treated the key as a literal field name).

Projection fields are now turned into a tree, which is used to project
snapshots, project ops and check ops submitted to a projection:

- components at or below a projected sub-field pass through unchanged
- components on non-projected siblings are dropped
- object inserts/deletes that replace a parent of projected sub-fields
  keep only the projected parts of the old and new values
- sub-field paths only go through objects; arrays and scalars in the
  way are left out of the projection
- submits may only change values at or below a projected path, since
  replacing a parent could overwrite fields the client can't see

Since a DB that projects snapshots itself (eg MongoDB, which projects
documents inside arrays) may disagree with this at the edges, ShareDB
also projects snapshots for projections with sub-fields, so snapshots
always match the projected ops.

A randomized test applies random json0 ops to random documents and
checks that applying the projected op to the projected snapshot gives
the same result as projecting the updated snapshot.
@alecgibson
alecgibson requested review from alecgibson and a balanced review from Copilot November 10, 2026 15:47
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 98.087% (+0.03%) from 98.057% — Mohith26:subfield-projections into share:main

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved projection bugs discard visible subtype and ancestor-overwrite updates and lose selected __proto__ properties, producing stale subscriber state and incomplete snapshots.

3 open issues
What changed in this PR

Adds dotted-path sub-field projections to ShareDB, addressing #365 by making nested-field updates visible to subscribed documents and queries.

Changes:

  • Uses a cached field tree to project snapshots and operations, and validate submissions.
  • Re-projects sub-field snapshots returned by database adapters to align snapshot and operation behavior.
  • Adds unit, randomized, and client tests, plus documentation for nested-field behavior and restrictions.
File Description
test/​projections.js Tests nested projections and snapshot/operation consistency.
test/​client/​projections.js Tests client reads, subscriptions, and projected submissions.
test/​backend.js Tests path validation and adapter snapshot re-projection.
lib/​projections.js Implements field trees and nested projection logic.
lib/​backend.js Validates dotted paths and enables snapshot re-projection.
docs/​projections.md Explains sub-field semantics and submission restrictions.
docs/​api/​backend.md Documents dotted keys in addProjection().

Review effort: Balanced

Comment thread lib/projections.js
// Only object inserts and deletes can replace an object. Anything else
// (eg a number add or string edit) applies to a value which isn't an
// object, and so isn't in the projection.
if (c.oi === undefined && c.od === undefined) return null;

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, a subtype op on a parent of a projected sub-field can't be projected without knowing the subtype's data model, so it's dropped. I've documented this as a limitation in docs/projections.md rather than guessing. I'm happy to look at it as a follow-up if the maintainers want it.

Comment thread lib/projections.js Resolved
Comment thread lib/projections.js
Comment on lines +240 to 245
if (child === true) {
// Shallow copy of the field
result[key] = data[key];
} else if (isPlainObject(data[key])) {
result[key] = projectData(child, data[key]);
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same thing happens for a top-level __proto__ field on main, since projectData() there also does result[key] = data[key] on a plain object. I've kept it out of this PR to keep it focused, but it could be a small follow-up.

Comment thread lib/projections.js Outdated
Comment on lines +167 to +170
if (isPlainObject(c.od)) newC.od = projectData(node, c.od);
if (isPlainObject(c.oi)) newC.oi = projectData(node, c.oi);
if (newC.od === undefined && newC.oi === undefined) return null;
return newC;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if someone replaces owner with a non-object and doesn't send od? json0 doesn't require od, and plenty of clients don't send it. With {'owner.name': true}, the op {p: ['owner'], oi: 'gone'} projects to []. Subscribers then keep owner: {name: 'jim'} forever, even though the projected snapshot has no owner any more. oi: null does the same.

I think we need to send a delete whenever the new value isn't an object (json0 doesn't mind deleting a key the client doesn't have):

Suggested change
if (isPlainObject(c.od)) newC.od = projectData(node, c.od);
if (isPlainObject(c.oi)) newC.oi = projectData(node, c.oi);
if (newC.od === undefined && newC.oi === undefined) return null;
return newC;
if (isPlainObject(c.oi)) {
newC.oi = projectData(node, c.oi);
if (isPlainObject(c.od)) newC.od = projectData(node, c.od);
} else {
newC.od = isPlainObject(c.od) ? projectData(node, c.od) : {};
}
return newC;

(I fuzzed oi with and without od against this: 73 divergences on this branch, 0 with the change. One existing expectation changes: {od: 'jim', oi: 5} now gives [{p: ['owner'], od: {}}] rather than [].)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied your suggestion in 9c7c748. Replacing a parent with a non-object now always sends od (the projected old value, or {}), and I updated the {od: 'jim', oi: 5} expectation. I added unit tests for oi of a string, null, a number and an array without od, plus client op fetch/subscribe tests for the same case.

Comment thread test/projections.js
return {p: path, ld: clone(value), li: randomValue(random, 2)};
}
if (random() < 0.3) return {p: path, od: clone(value)};
return {p: path, od: clone(value), oi: randomValue(random, 3)};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related: the generator always sends an accurate od whenever there is one, which is why it didn't catch the case above. Could it sometimes leave od out, both here and in the root branch? json0 never checks od, so the projection can't rely on it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 9c7c748. The generator now sometimes leaves od out on nested replaces, top-level replaces and root replaces. As a check, I ran it against the old projectObjectReplace() and all five sub-field field sets fail, and they pass with the fix.

Comment thread lib/projections.js Outdated
var node = tree;
for (var i = 0; i < segments.length; i++) {
var segment = segments[i];
// An ancestor already includes this whole subtree

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually want to support {owner: true, 'owner.name': true}? sharedb-mongo's getProjection() copies the keys straight into the Mongo projection, and MongoDB 4.4+ rejects that with Path collision at owner.name. It would work on the memory DB and then fail in prod. I'd rather reject overlapping paths in addProjection(), which also gets rid of the break and override branches here.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it would only fail later in sharedb-mongo. addProjection() now throws for overlapping paths in either order and at any depth, so the tree builder no longer needs the break or the override branch.

Comment thread lib/backend.js Outdated
throw new Error('Invalid field ' + key + ' - fields must be {somekey: true}. Subfields not currently supported.');
throw new Error('Invalid field ' + key + ' - fields must be {somekey: true} or {\'some.subfield\': true}.');
}
if (key.indexOf('.') !== -1 && key.split('.').indexOf('') !== -1) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this validation live in projections.js, next to the code that actually splits the paths? At the moment the path syntax is spread over two modules, and the rules suggested on projections.js (numeric segments, overlapping paths) would make it longer. Also, is the indexOf('.') guard only there so that '' stays a legal field? That's fine if so, but it reads like an optimisation rather than a deliberate exception.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it into projections.validateFields(), which addProjection() now calls. On indexOf('.'): yes, it was there so top-level keys like '' stay legal. That's now an explicit rule with a comment: keys without a dot keep their existing behaviour, and the sub-field rules (empty segments, numeric segments, overlaps) only apply to dotted keys. There's a test that {'': true, 0: true} is still accepted.

Comment thread lib/projections.js Outdated
Comment on lines 213 to 214
var field = path[0];
if (typeof field !== 'string' && typeof field !== 'number') return false;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these two lines are dead now. projectedNode() already returns null for keys that aren't strings or numbers, and an empty path returns the root tree, which isn't true.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, removed. Empty paths and non-string/number keys both already fail in projectedNode().

Comment thread lib/backend.js Outdated
target: collection,
fields: fields
fields: fields,
hasSubFields: projections.hasSubFields(fields)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should re-rederive this wherever needed instead of storing it here? It's undocumented, but I guess consumers can technically change backend.projections.fields outside of addProjection() (which is just doing some validation anyway).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. It's no longer stored on the projection, and _getSnapshotProjection() calls projections.hasSubFields(projection.fields) each time.

Comment thread lib/backend.js Outdated
Comment on lines +315 to +320
// Databases that project snapshots themselves (eg MongoDB) may treat
// sub-field paths differently to ShareDB (eg when they go through arrays),
// so we always project sub-fields ourselves too. This keeps snapshots
// consistent with the ops we project, and is a no-op on data that the
// database already projected the same way.
if (projection && projection.hasSubFields) return projection;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please expand on this comment? I'm not sure I follow? Is this to deal with projections like a.0.b? If so, shouldn't we just forbid that sort of projection?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expanded the comment. It wasn't about a.0.b, it's about the data: with {'owner.name': true} and {owner: [{name: 'jim'}]}, Mongo projects into the array and returns {owner: [{name: 'jim'}]}, while ShareDB leaves owner out and drops ops on it. If we trusted the DB snapshot, the client would have an owner that never updates. That depends on the data, so addProjection() can't reject it. Separately, numeric segments like a.0.b are now rejected outright.

Comment thread lib/projections.js Outdated
// project every op and snapshot read through a projection. This assumes that
// fields aren't changed after they're first used, which holds for fields
// passed to `backend.addProjection()`.
var fieldTreeCache = new WeakMap();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We technically still target ES3 (I know) -- I'm surprised the lint allowed this to be honest. Do we even need this cache?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the WeakMap cache, so the tree is built on each call. It's measurable but small: a micro-benchmark of 1M two-component ops with flat fields was about 47 ms on main, 57 ms with the cache, and 150 ms without it, so roughly 0.1 µs per op. If you want that back, an ES3-safe option is a fast path for fields with no dots, but I've kept it simple for now.

Comment thread lib/projections.js Outdated
return tree;
}

function buildFieldTree(fields) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please see some discussion around whether we should just define projections directly as trees? ie comparing:

// dot notation
backend.addProjection({'foo', 'bar', {'parent.child': true}})

vs

// tree notation
backend.addProjection({'foo', 'bar', {parent: {child: true}}})

As far as I can tell:

  • dot notation is more consistent with MongoDB notation
  • tree notation more verbose
  • tree notation can be made type safe by consumers
  • tree notation possibly less ambiguous (eg your subtree case below)
  • dot notation can be trivially derived from tree notation, but not so much the other way around(?)
  • other...?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's how I see it:

  • Dot notation matches MongoDB, and sharedb-mongo already copies the keys straight into the Mongo projection, so it works there with no adapter change.
  • Tree notation is backwards compatible in a way dot notation isn't. {'a.b': true} used to mean a literal top-level key a.b on the memory DB, and this PR changes that. A tree never reinterprets existing keys, since {name: true} is already a valid one-level tree.
  • Trees can't express overlaps (owner can only have one value), so most of the new validation goes away. Numeric keys would still need rejecting.
  • Trees are easy for consumers to type, and they're what we build internally anyway.
  • Converting: tree to dots is a simple flatten, as long as no key contains a dot. Dots to tree is also simple (it's what fieldTree() does), but dots can't represent a key that contains a ., so that's the lossy direction.
  • The cost of trees is that sharedb-mongo would need to flatten them into dot paths. MongoDB 4.4+ accepts the nested form directly, but older servers reject it.

I lean slightly toward trees, mainly for backwards compatibility and type safety, with sharedb-mongo flattening them. It's your call though, and I'm happy to switch the PR over. Most of the code already works on the tree, so it would mostly be validation, docs and tests.

Comment thread lib/projections.js Outdated
Comment on lines +151 to +154
// Inside a sub-field projection, we can only descend into objects. A
// number would be an array index, and arrays aren't projected.
if (i > 0 && typeof key !== 'string') return null;
if (typeof key !== 'string' && typeof key !== 'number') return null;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is another hint that we should just outright reject numeric keys.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Dotted paths with numeric segments (tags.0, 0.name) now throw in addProjection(), and the numeric special case in projectedNode() is gone. I left plain top-level keys like '0' alone, since those have always worked and are just keys on the root object. I can reject those too if you'd prefer, but it would be a breaking change.

@alecgibson

Copy link
Copy Markdown
Collaborator

Thanks for submitting this by the way! (Sorry I hit go on my review too fast 😅 )

- Move field validation into projections.js
- Reject overlapping paths, which MongoDB 4.4+ rejects as path collisions
- Reject numeric segments in sub-field paths
- Send a delete when a parent of a projected sub-field is replaced by a
  non-object, even without od
- Sometimes omit od in the randomized test
- Re-derive whether a projection has sub-fields instead of storing it
- Remove the WeakMap field tree cache (ES3), and some dead checks
@Mohith26

Copy link
Copy Markdown
Author

Thanks for the review. I pushed 9c7c748, which:

  • Moves validation into projections.validateFields(). It now also rejects overlapping paths and numeric segments in sub-field paths. Top-level keys keep their existing rules.
  • Always sends a delete when a parent of a projected sub-field is replaced by a non-object, with or without od (your suggestion). The randomized test now sometimes omits od, and it fails on the old code.
  • Removes the dead checks, the stored hasSubFields (re-derived where needed) and the WeakMap cache.
  • Expands the comment on re-projecting DB snapshots.
  • Documents the path rules and the subtype limitation in docs/projections.md, and updates the PR description.

npm test (1147 passing) and npm run lint pass locally. I put a dot vs tree notation comparison in that thread. I lean slightly toward trees, but I'm happy to go either way.

@Mohith26 Mohith26 closed this by deleting the head repository Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Projection's nested fields aren't being updated when subscribing to a query

4 participants