Repository navigation
Conversation
Fixes #365 `addProjection()` accepted dotted keys like `'owner.name'`, and sharedb-mongo's fetches honoured them, but op projection only looked at the first path segment, so subscribers never saw changes to projected sub-fields (and the memory DB treated the key as a literal field name). Projection fields are now turned into a tree, which is used to project snapshots, project ops and check ops submitted to a projection: - components at or below a projected sub-field pass through unchanged - components on non-projected siblings are dropped - object inserts/deletes that replace a parent of projected sub-fields keep only the projected parts of the old and new values - sub-field paths only go through objects; arrays and scalars in the way are left out of the projection - submits may only change values at or below a projected path, since replacing a parent could overwrite fields the client can't see Since a DB that projects snapshots itself (eg MongoDB, which projects documents inside arrays) may disagree with this at the edges, ShareDB also projects snapshots for projections with sub-fields, so snapshots always match the projected ops. A randomized test applies random json0 ops to random documents and checks that applying the projected op to the projected snapshot gives the same result as projecting the updated snapshot.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved projection bugs discard visible subtype and ancestor-overwrite updates and lose selected __proto__ properties, producing stale subscriber state and incomplete snapshots.
3 open issues
What changed in this PR
Adds dotted-path sub-field projections to ShareDB, addressing #365 by making nested-field updates visible to subscribed documents and queries.
Changes:
- Uses a cached field tree to project snapshots and operations, and validate submissions.
- Re-projects sub-field snapshots returned by database adapters to align snapshot and operation behavior.
- Adds unit, randomized, and client tests, plus documentation for nested-field behavior and restrictions.
| File | Description |
|---|---|
test/projections.js |
Tests nested projections and snapshot/operation consistency. |
test/client/projections.js |
Tests client reads, subscriptions, and projected submissions. |
test/backend.js |
Tests path validation and adapter snapshot re-projection. |
lib/projections.js |
Implements field trees and nested projection logic. |
lib/backend.js |
Validates dotted paths and enables snapshot re-projection. |
docs/projections.md |
Explains sub-field semantics and submission restrictions. |
docs/api/backend.md |
Documents dotted keys in addProjection(). |
Review effort: Balanced
| // Only object inserts and deletes can replace an object. Anything else | ||
| // (eg a number add or string edit) applies to a value which isn't an | ||
| // object, and so isn't in the projection. | ||
| if (c.oi === undefined && c.od === undefined) return null; |
There was a problem hiding this comment.
Right, a subtype op on a parent of a projected sub-field can't be projected without knowing the subtype's data model, so it's dropped. I've documented this as a limitation in docs/projections.md rather than guessing. I'm happy to look at it as a follow-up if the maintainers want it.
| if (child === true) { | ||
| // Shallow copy of the field | ||
| result[key] = data[key]; | ||
| } else if (isPlainObject(data[key])) { | ||
| result[key] = projectData(child, data[key]); | ||
| } |
There was a problem hiding this comment.
The same thing happens for a top-level __proto__ field on main, since projectData() there also does result[key] = data[key] on a plain object. I've kept it out of this PR to keep it focused, but it could be a small follow-up.
| if (isPlainObject(c.od)) newC.od = projectData(node, c.od); | ||
| if (isPlainObject(c.oi)) newC.oi = projectData(node, c.oi); | ||
| if (newC.od === undefined && newC.oi === undefined) return null; | ||
| return newC; |
There was a problem hiding this comment.
What happens if someone replaces owner with a non-object and doesn't send od? json0 doesn't require od, and plenty of clients don't send it. With {'owner.name': true}, the op {p: ['owner'], oi: 'gone'} projects to []. Subscribers then keep owner: {name: 'jim'} forever, even though the projected snapshot has no owner any more. oi: null does the same.
I think we need to send a delete whenever the new value isn't an object (json0 doesn't mind deleting a key the client doesn't have):
| if (isPlainObject(c.od)) newC.od = projectData(node, c.od); | |
| if (isPlainObject(c.oi)) newC.oi = projectData(node, c.oi); | |
| if (newC.od === undefined && newC.oi === undefined) return null; | |
| return newC; | |
| if (isPlainObject(c.oi)) { | |
| newC.oi = projectData(node, c.oi); | |
| if (isPlainObject(c.od)) newC.od = projectData(node, c.od); | |
| } else { | |
| newC.od = isPlainObject(c.od) ? projectData(node, c.od) : {}; | |
| } | |
| return newC; |
(I fuzzed oi with and without od against this: 73 divergences on this branch, 0 with the change. One existing expectation changes: {od: 'jim', oi: 5} now gives [{p: ['owner'], od: {}}] rather than [].)
There was a problem hiding this comment.
Applied your suggestion in 9c7c748. Replacing a parent with a non-object now always sends od (the projected old value, or {}), and I updated the {od: 'jim', oi: 5} expectation. I added unit tests for oi of a string, null, a number and an array without od, plus client op fetch/subscribe tests for the same case.
| return {p: path, ld: clone(value), li: randomValue(random, 2)}; | ||
| } | ||
| if (random() < 0.3) return {p: path, od: clone(value)}; | ||
| return {p: path, od: clone(value), oi: randomValue(random, 3)}; |
There was a problem hiding this comment.
Related: the generator always sends an accurate od whenever there is one, which is why it didn't catch the case above. Could it sometimes leave od out, both here and in the root branch? json0 never checks od, so the projection can't rely on it.
There was a problem hiding this comment.
Done in 9c7c748. The generator now sometimes leaves od out on nested replaces, top-level replaces and root replaces. As a check, I ran it against the old projectObjectReplace() and all five sub-field field sets fail, and they pass with the fix.
| var node = tree; | ||
| for (var i = 0; i < segments.length; i++) { | ||
| var segment = segments[i]; | ||
| // An ancestor already includes this whole subtree |
There was a problem hiding this comment.
Do we actually want to support {owner: true, 'owner.name': true}? sharedb-mongo's getProjection() copies the keys straight into the Mongo projection, and MongoDB 4.4+ rejects that with Path collision at owner.name. It would work on the memory DB and then fail in prod. I'd rather reject overlapping paths in addProjection(), which also gets rid of the break and override branches here.
There was a problem hiding this comment.
Agreed, it would only fail later in sharedb-mongo. addProjection() now throws for overlapping paths in either order and at any depth, so the tree builder no longer needs the break or the override branch.
| throw new Error('Invalid field ' + key + ' - fields must be {somekey: true}. Subfields not currently supported.'); | ||
| throw new Error('Invalid field ' + key + ' - fields must be {somekey: true} or {\'some.subfield\': true}.'); | ||
| } | ||
| if (key.indexOf('.') !== -1 && key.split('.').indexOf('') !== -1) { |
There was a problem hiding this comment.
Should this validation live in projections.js, next to the code that actually splits the paths? At the moment the path syntax is spread over two modules, and the rules suggested on projections.js (numeric segments, overlapping paths) would make it longer. Also, is the indexOf('.') guard only there so that '' stays a legal field? That's fine if so, but it reads like an optimisation rather than a deliberate exception.
There was a problem hiding this comment.
Moved it into projections.validateFields(), which addProjection() now calls. On indexOf('.'): yes, it was there so top-level keys like '' stay legal. That's now an explicit rule with a comment: keys without a dot keep their existing behaviour, and the sub-field rules (empty segments, numeric segments, overlaps) only apply to dotted keys. There's a test that {'': true, 0: true} is still accepted.
| var field = path[0]; | ||
| if (typeof field !== 'string' && typeof field !== 'number') return false; |
There was a problem hiding this comment.
I think these two lines are dead now. projectedNode() already returns null for keys that aren't strings or numbers, and an empty path returns the root tree, which isn't true.
There was a problem hiding this comment.
You're right, removed. Empty paths and non-string/number keys both already fail in projectedNode().
| target: collection, | ||
| fields: fields | ||
| fields: fields, | ||
| hasSubFields: projections.hasSubFields(fields) |
There was a problem hiding this comment.
I wonder if we should re-rederive this wherever needed instead of storing it here? It's undocumented, but I guess consumers can technically change backend.projections.fields outside of addProjection() (which is just doing some validation anyway).
There was a problem hiding this comment.
Agreed. It's no longer stored on the projection, and _getSnapshotProjection() calls projections.hasSubFields(projection.fields) each time.
| // Databases that project snapshots themselves (eg MongoDB) may treat | ||
| // sub-field paths differently to ShareDB (eg when they go through arrays), | ||
| // so we always project sub-fields ourselves too. This keeps snapshots | ||
| // consistent with the ops we project, and is a no-op on data that the | ||
| // database already projected the same way. | ||
| if (projection && projection.hasSubFields) return projection; |
There was a problem hiding this comment.
Could you please expand on this comment? I'm not sure I follow? Is this to deal with projections like a.0.b? If so, shouldn't we just forbid that sort of projection?
There was a problem hiding this comment.
Expanded the comment. It wasn't about a.0.b, it's about the data: with {'owner.name': true} and {owner: [{name: 'jim'}]}, Mongo projects into the array and returns {owner: [{name: 'jim'}]}, while ShareDB leaves owner out and drops ops on it. If we trusted the DB snapshot, the client would have an owner that never updates. That depends on the data, so addProjection() can't reject it. Separately, numeric segments like a.0.b are now rejected outright.
| // project every op and snapshot read through a projection. This assumes that | ||
| // fields aren't changed after they're first used, which holds for fields | ||
| // passed to `backend.addProjection()`. | ||
| var fieldTreeCache = new WeakMap(); |
There was a problem hiding this comment.
We technically still target ES3 (I know) -- I'm surprised the lint allowed this to be honest. Do we even need this cache?
There was a problem hiding this comment.
Removed the WeakMap cache, so the tree is built on each call. It's measurable but small: a micro-benchmark of 1M two-component ops with flat fields was about 47 ms on main, 57 ms with the cache, and 150 ms without it, so roughly 0.1 µs per op. If you want that back, an ES3-safe option is a fast path for fields with no dots, but I've kept it simple for now.
| return tree; | ||
| } | ||
|
|
||
| function buildFieldTree(fields) { |
There was a problem hiding this comment.
Can we please see some discussion around whether we should just define projections directly as trees? ie comparing:
// dot notation
backend.addProjection({'foo', 'bar', {'parent.child': true}})vs
// tree notation
backend.addProjection({'foo', 'bar', {parent: {child: true}}})As far as I can tell:
- dot notation is more consistent with MongoDB notation
- tree notation more verbose
- tree notation can be made type safe by consumers
- tree notation possibly less ambiguous (eg your subtree case below)
- dot notation can be trivially derived from tree notation, but not so much the other way around(?)
- other...?
There was a problem hiding this comment.
Here's how I see it:
- Dot notation matches MongoDB, and sharedb-mongo already copies the keys straight into the Mongo projection, so it works there with no adapter change.
- Tree notation is backwards compatible in a way dot notation isn't.
{'a.b': true}used to mean a literal top-level keya.bon the memory DB, and this PR changes that. A tree never reinterprets existing keys, since{name: true}is already a valid one-level tree. - Trees can't express overlaps (
ownercan only have one value), so most of the new validation goes away. Numeric keys would still need rejecting. - Trees are easy for consumers to type, and they're what we build internally anyway.
- Converting: tree to dots is a simple flatten, as long as no key contains a dot. Dots to tree is also simple (it's what
fieldTree()does), but dots can't represent a key that contains a., so that's the lossy direction. - The cost of trees is that sharedb-mongo would need to flatten them into dot paths. MongoDB 4.4+ accepts the nested form directly, but older servers reject it.
I lean slightly toward trees, mainly for backwards compatibility and type safety, with sharedb-mongo flattening them. It's your call though, and I'm happy to switch the PR over. Most of the code already works on the tree, so it would mostly be validation, docs and tests.
| // Inside a sub-field projection, we can only descend into objects. A | ||
| // number would be an array index, and arrays aren't projected. | ||
| if (i > 0 && typeof key !== 'string') return null; | ||
| if (typeof key !== 'string' && typeof key !== 'number') return null; |
There was a problem hiding this comment.
I think this is another hint that we should just outright reject numeric keys.
There was a problem hiding this comment.
Done. Dotted paths with numeric segments (tags.0, 0.name) now throw in addProjection(), and the numeric special case in projectedNode() is gone. I left plain top-level keys like '0' alone, since those have always worked and are just keys on the root object. I can reject those too if you'd prefer, but it would be a breaking change.
|
Thanks for submitting this by the way! (Sorry I hit go on my review too fast 😅 ) |
- Move field validation into projections.js - Reject overlapping paths, which MongoDB 4.4+ rejects as path collisions - Reject numeric segments in sub-field paths - Send a delete when a parent of a projected sub-field is replaced by a non-object, even without od - Sometimes omit od in the randomized test - Re-derive whether a projection has sub-fields instead of storing it - Remove the WeakMap field tree cache (ES3), and some dead checks
|
Thanks for the review. I pushed 9c7c748, which:
|

Fixes #365
addProjection()already accepts dotted keys like'owner.name', and sharedb-mongo fetches honour them, but op projection only checks the first path segment. So a query or doc subscribed through the projection never sees changes to the projected sub-fields, and the memory DB treats'owner.name'as a literal key.This turns the projection fields into a tree and uses it everywhere projections are applied, covering the cases listed in the issue (ops exactly at, below, and above a projected sub-field):
owner.phone) are dropped.oi/odthat replace a parent of projected sub-fields (owner, or the root) keep only the projected parts of the old and new values. If the new value isn't an object, the projected op deletes the parent, whether or not the original op had anod.addProjection()validates sub-field paths (inprojections.js): no empty segments, no numeric segments, and no overlapping paths like{owner: true, 'owner.name': true}(MongoDB 4.4+ rejects those as a path collision). Top-level keys keep their existing rules.Because Mongo projects dotted paths through arrays of documents, a DB that projects snapshots itself could disagree with the op projection at the edges. For projections that have sub-fields, ShareDB now also projects the snapshot after the DB does (a no-op when they agree), so snapshots always match the ops. Projections without sub-fields are unchanged.
skipPollstill gets the unprojected op, so query polling isn't affected.The field tree is re-derived from the fields on each call (no cache, to stay ES3 and to pick up changes to
projection.fields). A quick micro-benchmark ofprojectOpwith flat fields (1M two-component ops, median of 7, Apple Silicon, Node 26) was about 47 ms onmainand 150 ms on this branch, so roughly 0.1 µs extra per op.One behaviour change to flag: a field like
{'a.b': true}used to match a literal top-level key'a.b'(with the memory DB only). It now means the sub-fieldbofa, which matches what Mongo already did.Tests
od, and field validation.oi/od, sometimes withoutod,li/ld/lm,na,si/sd, root replace) to random documents and checks that applying the projected op to the projected snapshot always equals projecting the updated snapshot, for several field sets.test/db.jsexercise it too.npm test(1147 passing) andnpm run lintpass locally.lib/projections.jshas 100% line coverage.Docs are updated in
docs/projections.mdand theaddProjection()API page.Known limitation: a json0 subtype op (
t, eg rich-text) applied to a parent of a projected sub-field can't be projected, so it's dropped. This is noted in the docs.