Repository navigation
[#267] [#269] [#270] Read the type arrays of Nullable relationship properties - #271
Conversation
…atform#270] Read the type arrays of Nullable relationship properties OpenIdentityPlatform#267: ManagedObjectApiDescription.build read the type of a relationship property, and the type of the items of a relationship collection, as a single string. The type array ["relationship", "null"] of a Nullable relationship failed with "/type: Expecting a java.lang.String", and the whole managed object was left out of the API description. replaceRelationshipType also accepts a type array that lists "relationship". It replaces only that entry with "object", so ["relationship", "null"] becomes ["object", "null"]. ReverseRelationshipValidator reads the type of the reverse property from that description, and now accepts a type array as well. Otherwise a Nullable reverse property would still give NA, and the singleton and duplicate-assignment checks would be skipped. OpenIdentityPlatform#269: SchemaField.initializeType followed items only for the string "array". A relationship collection marked as Nullable, with the type ["array", "null"], became a core field: it got no relationship provider, and its value was stored with the managed object instead of as relationships. A type array of "array" and "null" with items is now read like "array". Without items it stays a core field, as before. A field that was marked as Nullable before is a relationship collection again after the upgrade; the upgrade note in OpenIdentityPlatform#269 says what that means for the values written to it meanwhile. OpenIdentityPlatform#270: the Admin UI finds relationship properties by comparing their type with "relationship" and "array", and JSONEditor has no editor for a "relationship" listed in a type array. The edit view did not read the value of a Nullable singleton relationship, the relationship widget left Nullable relationships out, and the linked view and the default value editor of a mapping to such a property threw "Unknown editor for schema". ResourceDelegate.normalizeRelationshipTypes reads ["relationship", "null"] as "relationship", and ["array", "null"] with relationship items as "array", in the schemas that ResourceDelegate.getSchema, with allSchemas, and AdminUtils.findPropertiesList return. Both work on a copy of managed.json, which keeps the type arrays. The OpenAPI transformer still has to describe the type arrays ["object", "null"] and ["array", "null"]. That is OpenIdentityPlatform/commons#382, which reaches OpenIDM with the commons version for OpenIdentityPlatform#266.
maximthomas
left a comment
There was a problem hiding this comment.
praise: The change sits exactly where the three type-array reads were, and it is careful about what it must not touch.
ResourceDelegate.normalizeRelationshipTypesrewrites only the result ofConfigDelegate.readEntity, which is a$.extend(true, {}, …)deep clone on both the fetch and the cache road (ConfigDelegate.js:59-84), so managed.json keeps its type arrays on the next save.SchemaField.isArraySchemakeeps a type array withoutitems, and one that lists another type next to"array", as a core field, andSchemaFieldTestpins both (testNullableArrayFieldWithoutItems,testArrayTypeUnionField).- The description's claim holds when measured: with BASE's
ResourceDelegate.jsandAdminUtils.jsput back, exactly the 3 new QUnit cases fail ("Actual: relationship,null / Expected: relationship").
issue (non-blocking): The new UI tests compare the normalized types with assert.equal, so a normalization that leaves a one-element array passes them.
openidm-ui/openidm-ui-common/src/test/qunit/tests/org/forgerock/openidm/ui/common/delegates/ResourceDelegateTest.js:38-40, :63-64, openidm-ui/openidm-ui-admin/src/test/qunit/org/forgerock/openidm/ui/admin/util/AdminUtilsTest.js:38
assert.equal is ==, and ["relationship"] == "relationship" is true. The views compare with === (RelationshipWidget.js:466, ResourceCollectionUtils.js:181), so such a result would still hide Nullable relationships — the #270 bug. Measured: with _.isArray(t) ? _.without(t, "null") : t in place of the three assignments at ResourceDelegate.js:52, :56, :58, all 3 new cases stay green. These are the only tests on #270.
// ResourceDelegateTest.js
assert.strictEqual(properties.manager.type, "relationship");
assert.strictEqual(properties.reports.type, "array");
assert.strictEqual(properties.roles.items.type, "relationship");
// ...
assert.strictEqual(schema.properties.manager.type, "relationship");
assert.strictEqual(_.findWhere(schema.allSchemas, { name: "role" }).schema.properties.members.type, "array",
"the schemas of the related managed objects are read the same way");
// AdminUtilsTest.js
assert.strictEqual(properties.manager.type, "relationship");Pin: with strictEqual, the one-element-array mutant above turns all 3 cases red.
issue (non-blocking): A ["array", "null"] property whose items has no type (or is a tuple list) loaded as a core field before, and now stops the whole managed service from activating.
openidm-core/src/main/java/org/forgerock/openidm/managed/SchemaField.java:248-256, :224, openidm-core/src/main/java/org/forgerock/openidm/managed/ManagedObjectService.java:241-247
isArraySchema only checks that items is not null, so initializeType(items) runs on items: {}, items: {"$ref": …} or items: [ … ] and throws "Schema field 'type' must be a String or List". At BASE these loaded as Nullable core fields (measured on all three shapes). Nothing catches the exception between new SchemaField and ManagedObjectService.activate, so every managed/* endpoint goes away after the upgrade, not just this object. Only hand-written configs are exposed: no shipped managed.json has a type array, and the schema editor always writes a typed items. Treating a type array without a typed items like one without items, which the PR already does, keeps BASE's behaviour. The type.isString() branch is left alone, because a plain "array" with typeless items already threw at BASE.
private static boolean isArraySchema(JsonValue schema) {
final JsonValue type = schema.get("type");
if (type.isString()) {
return type.asString().equals("array");
}
final JsonValue items = schema.get("items");
return type.isList()
&& items.isMap()
&& items.get("type").isNotNull()
&& type.asList().contains("array")
&& NULLABLE_ARRAY_TYPES.containsAll(type.asList());
}Pin, in SchemaFieldTest (and the Javadoc at :242 then reads "without a typed {@code items} schema"):
@Test
public void testNullableArrayFieldWithTypelessItems() throws Exception {
// a core field, as before: the items schema gives no type
final SchemaField schemaField = newSchemaField(json(object(
field("type", array("array", "null")),
field("items", object()))));
assertFalse(schemaField.isArray());
assertTrue(schemaField.isNullable());
}suggestion (non-blocking): The reverse-validator type-array rows always list the expected type first, so an order-dependent hasType survives.
openidm-core/src/test/java/org/forgerock/openidm/managed/RelationshipValidatorTest.java:362-368, :388-393, openidm-core/src/main/java/org/forgerock/openidm/managed/ReverseRelationshipValidator.java:85
type.asList().get(0).equals(expectedType) gives the same result as contains on array("object", "null") and array("array", "null"). replaceRelationshipType keeps the order of the entries, so a hand-written ["null", "relationship"] reaches hasType as ["null", "object"].
{ "object" },
// the type of a Nullable relationship
{ array("object", "null") },
{ array("null", "object") }
// ...
{ "array" },
{ array("array", "null") },
{ array("null", "array") }Pin: the get(0) mutant fails both new rows.
suggestion (non-blocking): No UI row combines a Nullable collection type with Nullable items, so swapping the two blocks of normalizeRelationshipTypes goes unnoticed.
openidm-ui/openidm-ui-common/src/main/js/org/forgerock/openidm/ui/common/delegates/ResourceDelegate.js:51-59, openidm-ui/openidm-ui-common/src/test/qunit/tests/org/forgerock/openidm/ui/common/delegates/ResourceDelegateTest.js:30-31
The property block depends on the items block having run first (items.type === "relationship" at :57). With the blocks swapped, {type: ["array", "null"], items: {type: ["relationship", "null"]}} stays ["array", "null"], and every existing row still passes. Only a hand-written managed.json has Nullable items, because SchemaUtils.setNullableProperties wraps only the top-level type.
teams: { type: ["array", "null"], items: { type: ["relationship", "null"] } },
// ...
assert.strictEqual(properties.teams.type, "array");
assert.strictEqual(properties.teams.items.type, "relationship");Pin: the block-swap mutant fails the first of the two assertions.
suggestion (non-blocking): No UI row has an array without items, so dropping the _.isObject(items) guard goes unnoticed.
openidm-ui/openidm-ui-common/src/main/js/org/forgerock/openidm/ui/common/delegates/ResourceDelegate.js:57, :75-79
Every "array" / ["array", "null"] row in both test files has an items object. Without the guard, one items-less array property throws a TypeError inside getSchema's loop over all managed objects, and getSchema then rejects for every object.
tags: { type: ["array", "null"] },
// ...
assert.deepEqual(properties.tags, { type: ["array", "null"] }, "an array without items is kept");Pin: the guard-dropped mutant throws in that case.
suggestion (non-blocking): Use "Part of #267" rather than "Fixes #267": in a running server the reverse validator still skips the singleton and duplicate checks for a Nullable reverse property.
openidm-core/src/main/java/org/forgerock/openidm/managed/ReverseRelationshipValidator.java:65-77, openidm-core/src/main/java/org/forgerock/openidm/managed/ManagedObjectService.java:157
parseReverseReferenceType reads the target object's API description. With commons 3.1.2 (the version this head resolves), OpenApiTransformer.execute still fails on a list-valued type ("/type: Expecting a java.lang.String", measured), so build returns null and the lookup ends in NA. The Dependency section says so. Merging with "Fixes" would still close #267 while its symptom remains until commons#382 reaches OpenIDM through #259. The transformer reads every property's type the same way, so any Nullable property on the target object has this effect, not just a Nullable relationship. That was already true before this PR, and commons#382 fixes it too.
suggestion (non-blocking): When the commons bump for #259 lands, add a build() row with type arrays. No test pins build()'s new isArray() selector or the isRelationship it now always sets on the collection branch.
openidm-core/src/main/java/org/forgerock/openidm/managed/ManagedObjectApiDescription.java:122-129
If :122 goes back to BASE's TYPE_RELATIONSHIP.equals(relationshipSchema.get(TYPE).asString()) with BASE's conditional collection body, ManagedObjectApiDescriptionTest stays 9/9 green (measured). Today that mutant is equivalent: string types take the same branch, and every type array makes build return null in the transformer. OpenApiTransformer.execute is a static call and openidm-core's Mockito has no MockedStatic, so no test can isolate the selector now. Once the transformer accepts type arrays, a row with ["relationship", "null"] and with ["array", "null"] + items "relationship" should assert ["object", "null"] and isRelationship: true.
…atform#270] Keep a Nullable array with typeless items a core field SchemaField.isArraySchema read a type array of "array" and "null" as an array whenever it had items. An items schema without a type, such as {} or {"$ref": ...}, or a tuple of items schemas then failed in initializeType with "Schema field 'type' must be a String or List". Nothing catches that before ManagedObjectService.activate, so the managed object service did not start and every managed object endpoint was gone. Before, such a field was a Nullable core field. It stays one now: only a typed items schema makes the type array an array. The string "array" with typeless items is rejected, as before. The tests now also catch these wrong implementations: - ResourceDelegateTest and AdminUtilsTest compare the normalized types with strictEqual. equal also passes a one-element array such as ["relationship"], which the views compare with === and leave out. - ResourceDelegateTest has a Nullable collection of Nullable relationships, which depends on the order of the two blocks of normalizeRelationshipTypes, and an array without items. - RelationshipValidatorTest also lists "null" first in a type array.
|
All seven points are addressed in 1bcce1c and in the description.
|
maximthomas
left a comment
There was a problem hiding this comment.
praise: Round 2 closes the config-loading edge case from round 1 where it starts, and the pins hold it.
SchemaField.isArraySchemanow asks for a typeditemsmap before it recurses (SchemaField.java:253-258), so{},$refand tupleitemsstay a core field again, and thetypelessItemsrows pin all three (SchemaFieldTest.java:279-299).- No Admin or End User UI action reaches the new 400 on a Nullable collection:
GenericEditResourceViewdrops a resource-collection key fromoldObject(GenericEditResourceView.js:478-479,:587-588), andRelationshipArrayViewremoves an item with a DELETE on the sub-resource (RelationshipArrayView.js:379).
issue (non-blocking): The typed-items guard checks only that items.type is not null, so ["array", "null"] with items {"type": 5} or {"type": "array"} (no inner items) now fails to load.
openidm-core/src/main/java/org/forgerock/openidm/managed/SchemaField.java:253-258, :224
isArraySchema lets such a schema into initializeType(items). That call hits the "Schema field 'type' must be a String or List" branch, for items.type directly or for items.items.type one level down. Measured with BASE and HEAD SchemaField.java compiled against openidm-core's classpath, on {"type":["array","null"]} with items set to {type:5}, {type:true}, {type:"array"} and {type:"array",items:{}}: BASE loads all four as a core, nullable field, and HEAD throws JsonValueException on all four. The exception is not caught in ManagedObjectSchema, ManagedObjectSet or ManagedObjectService.activate/modified, so one such hand-written property stops every managed object route. A plain "array" with the same items already throws at BASE, and that stays as it is. A fix that recurses:
private static boolean isArraySchema(JsonValue schema) {
final JsonValue type = schema.get("type");
if (type.isString()) {
return type.asString().equals("array");
}
return type.isList()
&& hasTypedItems(schema)
&& type.asList().contains("array")
&& NULLABLE_ARRAY_TYPES.containsAll(type.asList());
}
/**
* Returns whether the {@code items} schema of a schema has a type that {@link #initializeType} can read: a string,
* or a list of strings. An {@code items} schema of type {@code "array"} must have such an {@code items} schema too.
*
* @param schema a JSON object describing the schema field.
* @return true if the items schema is typed, false otherwise.
*/
private static boolean hasTypedItems(JsonValue schema) {
final JsonValue items = schema.get("items");
if (!items.isMap()) {
return false;
}
final JsonValue type = items.get("type");
if (type.isString()) {
return !type.asString().equals("array") || hasTypedItems(items);
}
return type.isList() && type.asList().stream().allMatch(String.class::isInstance);
}With this change compiled into HEAD's SchemaField.java, the four shapes above and items {type:["string",5]} load as a core, nullable field, as they did at BASE. Relationship and ["relationship","null"] items still give a relationship collection, and the plain-"array" controls throw as before. SchemaFieldTest was not run against the change. A broken relationship definition in items (reverseRelationship: true without reversePropertyName, :231) still throws, the same as in a plain "array".
Pin: add { object(field("type", 5)) } and { object(field("type", "array")) } to typelessItems (SchemaFieldTest.java:281-285). Both rows are red at HEAD, because the constructor throws, and the fix turns them green.
issue (non-blocking): getSchema now normalizes every managed object's schema, and normalizeRelationshipTypes reads property.items without checking whether property is null.
openidm-ui/openidm-ui-common/src/main/js/org/forgerock/openidm/ui/common/delegates/ResourceDelegate.js:48-49, :75-79
At BASE, getSchema for one object never walked the schemas of the other objects. At HEAD, a single "foo": null property in any object's schema throws a TypeError, and getSchema rejects for every managed object. GenericEditResourceView, ListResourceView, LinkedView, RelationshipWidget and EditAssignmentView then fail to load. The backend refuses such a schema, but only at startup. A managed.json change that fails ManagedObjectService.modified leaves the old routes up, while config/managed already returns the new raw config (SCR 2.1.20 road, read but not run).
_.each(_.filter(properties, _.isObject), function (property) {Pin: ResourceDelegate.normalizeRelationshipTypes({ foo: null, manager: { type: ["relationship", "null"] } }).manager.type is "relationship". At HEAD the call throws.
question (non-blocking): Should a relationship collection marked Nullable accept a PATCH remove that clears it, or is the 400 that a plain relationship collection gives intended?
openidm-core/src/main/java/org/forgerock/openidm/managed/ManagedObjectSet.java:1159-1161, SchemaField.java:210-215
A Nullable collection is now a relationship collection. PATCH [{"operation":"remove","field":"/coll"}], and move/copy into it, now get 400 "Cannot delete collection: /coll". At BASE, where the field was a core field, the same request got 200. isNullable() has no caller in openidm-core, so the Nullable mark does not change this road. replace with null is rejected at parse on both SHAs (CREST 3.1.2 PatchOperation), and no UI action sends any of these operations, so only REST clients and scripts see the change. If the 400 is intended, one sentence in the Upgrade note covers it, and this is a minor point. If Nullable is meant to allow clearing, :1160 must let a nullable collection through and clear its edges. That would be a change to this PR.
suggestion (non-blocking): replaceRelationshipType does not de-duplicate, so a type array that already lists "object" becomes ["object", "object"].
openidm-core/src/main/java/org/forgerock/openidm/managed/ManagedObjectApiDescription.java:244-254
setType is last-wins, so a hand-written ["object", "relationship"] is a relationship. Collections.replaceAll (:248) then writes ["object", "object"], and JSON Schema requires the entries of a type array to be unique. Today the transformer rejects every list type, so build() returns null at both SHAs and nothing changes. Once commons#382 lets list types through, the description carries the duplicate. Drop "relationship" when "object" is already listed, and add a testBuildReplacesRelationshipTypes row for it then.
suggestion (non-blocking): The Upgrade note covers only the upgrade. It does not describe a rollback to a release without this change.
openidm-core/src/main/java/org/forgerock/openidm/managed/SchemaField.java:210-215
After the upgrade, writes to a Nullable relationship collection store relationship edges. Back at the older release, the owner returns none of them, and its next update stores the value inline. A plain reverse property on the other object still lists the edges, so the two sides disagree. One sentence next to the upgrade direction would cover it.
suggestion (non-blocking): AdminUtils.findPropertiesList now calls ResourceDelegate.normalizeRelationshipTypes, and an older ui/admin/extension copy of the common ResourceDelegate.js does not have it.
openidm-ui/openidm-ui-admin/src/main/js/org/forgerock/openidm/ui/admin/util/AdminUtils.js:110
ResourceServlet serves ui/admin/extension first. On an installation that overrides org/forgerock/openidm/ui/common/delegates/ResourceDelegate.js there, the call throws a TypeError for every managed type. The mapping, self-service and resource-collection dialogs then get no properties. The re-apply procedure in the update guide (chap-update.adoc:920-946) avoids this. One line in the Upgrade note would point at it. No code change.
…atform#270] Keep a Nullable array with unreadable items a core field SchemaField.isArraySchema checked only that the items schema of a type array of "array" and "null" had a type. An items type that is not a string or a list of strings, such as 5 or ["string", 5], and an items schema of type "array" without a typed items schema of its own then made initializeType fail with "Schema field 'type' must be a String or List", so the managed object service did not start. An items type array that lists no type besides "null" made the field an array without a type. hasTypedItems now asks for a type that initializeType can read, and follows an items schema of type "array" to its own items schema. Such a field stays a Nullable core field, as before. ResourceDelegate.normalizeRelationshipTypes read the items of every property, and getSchema runs it on the schemas of all managed objects. A property that is not an object, such as null, made getSchema fail for every managed object. Such a property is skipped now. ManagedObjectApiDescription.replaceRelationshipType keeps the types of a type array unique, so ["object", "relationship"] becomes ["object"] instead of ["object", "object"].
|
All six points are addressed in 3207134 and in the description.
|
maximthomas
left a comment
There was a problem hiding this comment.
praise: The round closes the three code findings of round 2 where they sit.
SchemaField.hasTypedItems(SchemaField.java:267) letsinitializeTypesee only anitemstype it can read, so a hand-written{"type": 5}or an untyped array of arrays keeps a core field instead of failing the managed service.replaceRelationshipTypede-duplicates through aLinkedHashSet, pinned by the["object", "relationship"]row.normalizeRelationshipTypesskips properties that are not objects; reverting the filter turns thenormalizeRelationshipTypesQUnit case red (132/1, "Cannot read properties of null (reading 'items')").
Part of #267
Fixes #269
Fixes #270
Problem
A managed object property marked as "Nullable" gets a type array:
SchemaUtils.setNullablePropertiesin the Admin UI turns itstypeinto[type, "null"]. For a relationship this is["relationship", "null"], and for a relationship collection["array", "null"]. Three places did not read these type arrays:ManagedObjectApiDescription.buildread the type of a relationship property, and of the items of a relationship collection, withasString(). It failed with/type: Expecting a java.lang.String, and the managed object was left out of the API description.ReverseRelationshipValidatorthen fell back toNAand skipped the singleton and duplicate-assignment checks.SchemaField.initializeTypefolloweditemsonly for the string"array". A Nullable relationship collection became a core field: no relationship provider, its value stored inside the managed object, the reverse property not kept in sync."relationship"and"array". The edit view did not read the value of a Nullable singleton relationship, the relationship widget left Nullable relationships out, and the linked view and the default value editor of a mapping to such a property threwUnknown editor for schema.Change
ManagedObjectApiDescription.buildtells a singleton from a collection bySchemaField.isArray()of its relationship provider.replaceRelationshipTypereplaces"relationship"with"object"in a type string or a type array, so["relationship", "null"]becomes["object", "null"]. The types of a type array stay unique:["object", "relationship"]becomes["object"].ReverseRelationshipValidator.parseReverseReferenceTypeaccepts a type array that lists"array"or"object".SchemaField.initializeTypereads a type array of"array"and"null"with a typeditemsschema like"array". Without one it stays a core field, as before: noitems, anitemsschema withouttypesuch as{}or{"$ref": …}, a tuple ofitemsschemas, anitemstype that is not a string or a list of strings or that lists only"null", or anitemsschema of type"array"without a typeditemsschema of its own. A type array that lists another type next to"array"is not changed either.ResourceDelegate.normalizeRelationshipTypesreads["relationship", "null"]as"relationship", and["array", "null"]with relationship items as"array", in the schemas thatResourceDelegate.getSchema(withallSchemas) andAdminUtils.findPropertiesListreturn. Both work on the copy thatConfigDelegate.readEntityreturns, somanaged.jsonkeeps the type arrays. A property that is not an object, such asnull, is skipped, so it does not stopgetSchemafor every managed object.Upgrade note
A relationship collection marked as Nullable before this change is a relationship collection again after the upgrade. The values written to it while it was a core field are no longer returned, and the next update of the object does not keep them inside it. The relationships stored before it was marked as Nullable are returned again. See the upgrade note in #269 for how to keep the values.
As for any relationship collection, a PATCH
removeof the whole field without a value, and amoveorcopyonto it, now fail with400 Cannot delete collection. While the field was a core field they succeeded. Areplacewithnullis rejected by the PATCH parser, before and after this change.A rollback to a release without this change makes such a field a core field again. The relationships written to it after the upgrade are then no longer returned for the object that holds the field, while a reverse property on the other object that is not Nullable still lists them, and the next update of the object stores the value inside it again.
An older copy of
org/forgerock/openidm/ui/common/delegates/ResourceDelegate.jsinui/admin/extensionoverrides the new file and has nonormalizeRelationshipTypes.AdminUtils.findPropertiesListthen fails, and the mapping, assignment, self-service and resource-collection views get no properties. Re-apply such customizations to the new files, as "What You Should Do With Your UI After Updating" in the update guide describes.Dependency
The OpenAPI transformer still has to describe the type arrays
["object", "null"]and["array", "null"]. That is OpenIdentityPlatform/commons#382, which reaches OpenIDM through OpenDJ#1243, OpenICF#210 and #259. Until thenbuildstill drops the API description of a managed object with a Nullable relationship, now in the transformer, andReverseRelationshipValidatorstill falls back toNA. This pull request is therefore only part of #267, which #259 closes. TheSchemaFieldand Admin UI parts do not depend on it.Once OpenIDM resolves the fixed transformer,
testBuildReplacesRelationshipTypesalso gets rows with["relationship", "null"]and with["array", "null"]and relationship items, which expect["object", "null"]andisRelationship: true. Until then no test can tellbuild'sisArray()selector from the old type comparison: every type array makesbuildreturnnullin the transformer.Tests
openidm-core: 143 tests, 0 failures. With the oldReverseRelationshipValidatorandSchemaField, exactly the new type-array rows fail. With theitems.get("type").isNotNull()check of the previous round inSchemaField.isArraySchema, exactly the 6 new rows with an unreadableitemstype fail. Without the check for a type besides"null"inhasTypedItems, exactly the 2 rows[]and["null"]fail; without its recursion into anitemsschema of type"array", exactly the 2 array-of-arrays rows fail. Withget(0)instead ofcontainsinhasType, exactly the 2 rows that list"null"first fail. Without the de-duplication inreplaceRelationshipType, the["object", "relationship"]row fails.openidm-ui-admin(withopenidm-uiandopenidm-ui-common): eslint clean, 132 QUnit tests, 0 failures; with the oldResourceDelegate.jsandAdminUtils.js, exactly the 3 new tests fail. They compare withstrictEqual, so a normalization that leaves a one-element array such as["relationship"]fails all 3 of them as well. Swapping the two blocks ofnormalizeRelationshipTypes, dropping itsitemsguard in the collection branch, or dropping the filter of properties that are not objects, fails thenormalizeRelationshipTypestest.openidm-ui-enduser: 2 tests, 0 failures.