Repository navigation
Publish wiki How-To guides - #275
Conversation
maximthomas
left a comment
There was a problem hiding this comment.
praise: The guide is wired into the existing documentation pipeline end to end, and the claims hold against the build.
Check How-To documentation linksis green on the Linux/JDK 17 leg (Ran 8 tests … OK,Checked 5 How-To xrefs in 6 pages; 0 errors.), andopenidm-doc/check-how-to-links.py:24-26fails the step when no pages were generated instead of passing.- All 37 fragments in
wiki-how-to.patchresolve to explicit ids on their How-To page; the 8 ids with an intraword--render verbatim under Asciidoctor (id="…-configuring-active-directory--opendj-synchronization"). HowTo_Guides.pdfinsite-how-to.patchmatches doc-maven-plugin 3.2.2's naming of the doctitleHow-To Guides, and its AntoraMojo buildsmodules/how-to/nav.adocfrom the index'sinclude::./lines.
issue (non-blocking): The AD connector table gives principal as DOMAIN/Administrator, which AD does not accept as a bind name.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:49
AD simple bind takes a DN, a UPN or the down-level DOMAIN\Administrator. A reader who copies the example gets invalidCredentials on the AD connector, and nothing later in the tutorial works. The page's own screenshot 0-openidm-ad-connector.png shows DS\Administrator.
|principal |Active Directory account name |DOMAIN\Administratorquestion (non-blocking): Should check-how-to-links.py accept only explicit [[id]] / [#id] destinations?
openidm-doc/check-how-to-links.py:53, :55, :44
An xref to a section's auto-generated id, such as xref:../integrators-guide/chap-workflow.adoc#_openidmworkflowprocessdefinition[...] (that section has no explicit id), is reported as broken anchor and turns the Linux/JDK 17 leg red, although Asciidoctor resolves it. An Antora-style xref:integrators-guide:chap-workflow.adoc[...] is reported as missing page. Nothing fails today, because the 5 xrefs are page-only. If explicit ids are the rule, say so in the error. If not, also accept Asciidoctor's default _<title> ids.
if anchor not in ids:
errors.append(f"{article.name}: broken anchor (needs an explicit [[id]] or [#id]): {target}")Pin, if deliberate: self.check_link("_module", "== Module\n") exits 1 with broken anchor.
suggestion (non-blocking): The 8 regression tests leave most of the checker's rules unpinned.
openidm-doc/test_check_how_to_links.py:27
Every case goes through check_link, which always creates the target page and links to it with a fragment. These mutants of check-how-to-links.py pass all 8 tests:
- the
is_file()guard dropped (:45) - a same-page
xref:#idresolved against the directory (:44) - the
ROOT:index.adocskip dropped (:34) - the empty-articles guard dropped (
:24), which lets CI pass with no generated docs - the
[[id,reftext]]stop and the[#id.role]/[#id%opt]stops widened (:53,:55) - only anchored xrefs counted (
:37)
The cases below are green at the head, and each one kills one of these mutants (measured).
def run_checker(self, files):
with tempfile.TemporaryDirectory() as directory:
source = Path(directory)
(source / "how-to").mkdir()
for name, text in files.items():
path = source / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
return subprocess.run([sys.executable, str(CHECKER), str(source)],
capture_output=True, text=True)
def test_missing_page_is_reported(self):
result = self.run_checker({"how-to/page.adoc": "xref:../reference/absent.adoc[M]\n"})
self.assertEqual(1, result.returncode, result.stdout + result.stderr)
self.assertIn("missing page", result.stdout)
def test_same_page_fragment_resolves_on_the_article(self):
result = self.run_checker({"how-to/page.adoc": "[[local]]\nxref:#local[L]\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
def test_component_landing_page_is_skipped(self):
result = self.run_checker({"how-to/page.adoc": "xref:ROOT:index.adoc[Home]\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
def test_no_generated_pages_fails(self):
result = self.run_checker({})
self.assertEqual(1, result.returncode, result.stdout + result.stderr)
self.assertIn("No generated How-To pages", result.stdout)
def test_reftext_role_and_option_are_not_part_of_the_id(self):
for definition in ("[[module-ref,Module]]", "[#module-ref.role]", "[#module-ref%opt]"):
with self.subTest(definition=definition):
result = self.check_link("module-ref", definition + "\n")
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
def test_page_only_xref_is_counted(self):
result = self.run_checker({"how-to/page.adoc": "xref:../reference/module.adoc[M]\n",
"reference/module.adoc": "= Module\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
self.assertIn("Checked 1 How-To xrefs", result.stdout)suggestion (non-blocking): Nothing pins the 37 How-To section ids that wiki-how-to.patch links to.
openidm-doc/wiki-migration/wiki-how-to.patch, openidm-doc/src/main/asciidoc/how-to/*.adoc
The checker reads only the xrefs inside the How-To pages. Renaming an id such as …-active-directory-and-opendj-introduction therefore keeps CI green, and once the wiki patch is applied, that wiki section link lands at the top of the page. The test below is green at the head and fails on that rename (measured). It needs import re.
def test_wiki_patch_destinations_exist(self):
doc = Path(__file__).parent
patch = (doc / "wiki-migration/wiki-how-to.patch").read_text(encoding="utf-8")
links = set(re.findall(
r"https://doc\.openidentityplatform\.org/openidm/how-to/([a-z0-9-]+)#([\w.-]+)", patch))
self.assertEqual(37, len(links))
for page, anchor in sorted(links):
with self.subTest(page=page, anchor=anchor):
text = (doc / "src/main/asciidoc/how-to" / f"{page}.adoc").read_text(encoding="utf-8")
self.assertRegex(text, rf"(?m)^\[(\[|#){re.escape(anchor)}[\],.%]")suggestion (non-blocking): The page says to add the password snippet after whenCreated, which is the last property of account, and the snippet has no leading comma.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:75, :79
In samples/provisioners/provisioner.openicf-adldap.json, whenCreated is the last of the account properties. Pasting the snippet as shown gives } "password" : {, which is invalid JSON, so the connector configuration does not load.
Also, to synchronize passwords from OpenIDM, add a comma after the closing brace of the `whenCreated` field of the account object in the Active Directory connector configuration file `provisioner.openicf-adldap.json`, then add the `password` field:suggestion (non-blocking): The example account filters drop the sample's exclusion of disabled accounts.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:53, :54
(objectClass=user) replaces the sample's filter (openidm-zip/src/main/resources/samples/provisioners/provisioner.openicf-adldap.json:17-18). As a result, the disabled built-in accounts under CN=Users (Guest, krbtgt) show up in Data (account).
|accountSearchFilter |LDAP Filter for Retrieving Accounts |(&(!(userAccountControl:1.2.840.113556.1.4.803:=2))(!(objectClass=Computer)))
|accountSynchronizationFilter |An optional LDAP filter for the objects to synchronize. |(&(!(userAccountControl:1.2.840.113556.1.4.803:=2))(!(objectClass=Computer)))question (non-blocking): Is the rename of the wiki Home heading ### How-To intended?
openidm-doc/wiki-migration/wiki-how-to.patch:995
The new heading ### [How-To Guides](…) moves the GitHub anchor from Home#how-to to Home#how-to-guides, so an incoming Home#how-to link lands at the top of the page. openidm-doc/wiki-migration/README.md:22 says the patch keeps the original headings so that incoming section links keep landing. No link to Home#how-to was found in the wiki or in this repository. If the rename is intended, limit the README sentence to the tutorial pages. Otherwise, keep the heading and put the link below it.
### How-To
[How-To Guides](https://doc.openidentityplatform.org/openidm/how-to/)nitpick (non-blocking): The REST create example swaps sn and givenName.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:156
With the cn transformation at :122, the AD account gets cn "Doe John". Swap the two values in the request body and in the sample output.
--data-raw '{"mail":"jdoe@example.org","sn":"Doe","givenName":"John","telephoneNumber":"+7(999)123-45-67","userName":"idmUser"}' | json_ppnitpick (non-blocking): The AD page has a run-on sentence, a literal -> and two alt-text typos.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:37, :39, :41, :136
:37 is missing a period before "Change the configuration values". :39 renders -++>++ as a literal ->, while the rest of the page uses →. The alt texts read "OpenID LDAP Active Directory Connector.png" and "OpenIDM Email SMPT Settings".
In the administrator console select Configure → Connectors. In the list select AD LDAP Connector and set the settings according to the following table
image:images/how-to/openidm-ad/0-openidm-ad-connector.png[OpenIDM LDAP Active Directory Connector]
image:images/how-to/openidm-ad/5-openidm-email-settings.png[OpenIDM Email SMTP Settings]nitpick (non-blocking): wiki-migration/README.md does not list the How-To-Guides.md page that the patch creates.
openidm-doc/wiki-migration/wiki-how-to.patch:1011, openidm-doc/wiki-migration/README.md
Nothing in the patched wiki links to it either: not Home, _Sidebar.md or _Footer.md. Add it to the inventory that the README tells reviewers to apply from.
|
Addressed all ten review comments in 3e6e967907.
Validation: 17 tests pass; generated-source check reports 5 xrefs in 6 pages with 0 errors; JDK 17 preprocessing, Antora and How-To PDF generation pass with The wiki patch remains deferred until the documentation, companion site navigation and PDF have deployed and been verified. The full site integration build, full product tests and external directory/Docker integrations were not rerun locally for this review round; CI at the new head has not been validated here. |
maximthomas
left a comment
There was a problem hiding this comment.
praise: Round 1's feedback is addressed throughout, and the new checks hold up when tested.
- The How-To check runs in CI at this head: build-maven (ubuntu-latest, 17) prints
Ran 17 testsandChecked 5 How-To xrefs in 6 pages; 0 errors.(run 42352691878). test_wiki_patch_destinations_existnow pins all 37 wiki section destinations (openidm-doc/test_check_how_to_links.py:87-96).- The wiki patch keeps every wiki heading anchor. GitHub's live ids for the
→headings and for the duplicateSending Password by Email(-1) match the unchanged heading text.
issue (blocking): In the AD tutorial, deleting the managed user does not delete the AD account.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:131, :296-316
The page has the reader apply only "Default Actions", and that policy set maps SOURCE_MISSING to Exception (PolicyPatterns.json:57-58). When the linked user is deleted, notifyDelete sees a link and a valid target, so it assesses SOURCE_MISSING (SourceSyncOperation.java:203-206) and the Exception action throws. ManagedObjectSet.performSyncAction (:1763-1773) catches that exception. The DELETE therefore returns 200 while idmUser stays in CN=Users, and the ldapsearch at :308-312 prints 1, not 0. I traced this path at this head but did not run it against a live AD.
On the Behaviors tab, under Policies, select Default Actions, change the action for the SOURCE_MISSING situation (Source Missing) from Exception to Delete, and click the Save button.The set offers Delete for this situation (PolicyPatterns.json:59). I did not check the exact Admin UI label of the situation.
issue (blocking): In the reverse OpenLDAP mapping, the cn row replaces cn without its RDN value, so the OpenDJ → OpenLDAP step fails for user01 and user02.
openidm-doc/src/main/asciidoc/how-to/configuring-openidm-for-synchronization-between-openldap-and-opendj.adoc:247, :244, :288-298
The dn is cn=<userName> (:244), but cn is source.displayName || (givenName + ' ' + sn) (:247). user01 from openldap-data.ldif is linked, so Confirmed → Update computes cn "Jane Doe", which differs from "user01". The wiki's provisioner.openicf-openldap.json gives cn no NOT_UPDATEABLE flag, so the LDAP connector sends a REPLACE. slapd 2.6.10 (bitnamilegacy/openldap) rejects it with Naming violation (64), and a replace that keeps cn: user02 succeeds. bjensen is created rather than updated, so the check at :294-298 passes and hides the failures. I did not run the full reconcile.
Replace the :247 row with:
|userName |cn |Then cn always equals the RDN value that the :244 dn row builds. source.cn || … (as at :158) is not enough here. bjensen's managed cn comes from OpenDJ as "Barbara Jensen", so a later update would replace cn the same way. I reasoned this through but did not test it.
suggestion (non-blocking): test_wiki_patch_destinations_exist checks only fragment URLs, so the patch's 11 page-only How-To links are not tested.
openidm-doc/test_check_how_to_links.py:90-91
The regex requires #. how-to-persist-openidm-configuration-data-between-restarts is linked only without a fragment, and so are the five "This article has moved" banners, the guide index link and the five Home links. If that page is renamed, or a page-only URL becomes can-your-idm-play-chez / openidm/how-too/, the suite still prints Ran 17 tests ... OK.
def test_wiki_patch_page_links_exist(self):
doc = Path(__file__).parent
patch = (doc / "wiki-migration/wiki-how-to.patch").read_text(encoding="utf-8")
added = "\n".join(line for line in patch.splitlines() if line.startswith("+"))
paths = re.findall(r"https://doc\.openidentityplatform\.org/openidm/([^)\s#\]]*)", added)
self.assertEqual(48, len(paths))
for path in sorted(set(paths)):
with self.subTest(path=path):
self.assertTrue(path.startswith("how-to/"), path)
page = path[len("how-to/"):] or "index"
self.assertTrue((doc / "src/main/asciidoc/how-to" / f"{page}.adoc").is_file(), path)Pin: at 3e6e967 this finds 48 URLs and 0 misses. The chez / how-too mutant reports 3 misses. I measured this as a standalone probe, not inside the suite.
suggestion (non-blocking): The wiki-destination test accepts an id that sits inside a listing, literal or comment block.
openidm-doc/test_check_how_to_links.py:95-96
The test matches ^[[id against the raw page text. The checker strips verbatim blocks first (check-how-to-links.py:57-58), but this test does not. If [[can-your-idm-play-chess-introduction]] is wrapped in ----, the suite stays green, but Asciidoctor renders the id as listing text and the wiki link lands on the top of the page.
text = re.sub(r"^(/{4,}|-{4,}|\.{4,})[ \t]*\n.*?^\1[ \t]*$", "", text,
flags=re.MULTILINE | re.DOTALL)Pin: insert this before the assertRegex at :96. All 37 ids are still found at 3e6e967, and the wrapped id no longer is.
issue (non-blocking): A valid same-page xref:section-id[] is reported as missing page.
openidm-doc/check-how-to-links.py:38, :50-52
A target with no # and no extension is treated as a page path. Asciidoctor.js 3.0.4 renders xref:section-id[] as <a href="#section-id">, so a correct link fails the Linux/JDK 17 leg with a message about a page that does not exist. No page at this head uses this form.
page, _, anchor = target.partition("#")
if not anchor and not page.endswith(".adoc"):
page, anchor = "", page # xref:id[] and <<id>> are same-page idsissue (non-blocking): In the AD mapping, the cn row never reaches AD.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:124
The page has the reader copy samples/provisioners/provisioner.openicf-adldap.json, which flags account cn as NOT_CREATABLE/NOT_UPDATEABLE. ObjectClassInfoHelper drops such attributes silently, so AD cn stays the RDN (idmUser) from the dn row. Remove the row, or say that AD keeps cn equal to the userName.
suggestion (non-blocking): The checker does not scan the <<…>> xref shorthand.
openidm-doc/check-how-to-links.py:33
See <<nope,Nope>>. or <<../reference/m.adoc#nope,M>> gives Checked 0 How-To xrefs ... 0 errors. No page at this head uses the shorthand.
for match in re.finditer(r"xref:([^\s\[]+)\[|<<([^,>]+)", article.read_text(encoding="utf-8")):
target = match.group(1) or match.group(2)<<id>> also needs the same-page rule from the comment above.
suggestion (non-blocking): The verbatim-block strip only works when the block has a closing delimiter.
openidm-doc/check-how-to-links.py:57
For ----\n[[hid]]\ntext\n with no closing ----, xref:…#hid[] passes. Asciidoctor warns unterminated listing block and emits no id.
text = re.sub(r"^(/{4,}|-{4,}|\.{4,})[ \t]*\n.*?(?:^\1[ \t]*$|\Z)", "",
text, flags=re.MULTILINE | re.DOTALL)With the three checker changes above applied together, the 17 tests still pass. The unclosed-block case is then reported, and valid xref:sec[] / <<sec,S>> / <<../reference/m.adoc#ok,M>> links pass.
nitpick (non-blocking): The Antora page-ID diagnostic suggests a fix that cannot work for a cross-component xref.
openidm-doc/check-how-to-links.py:41-42
No relative path reaches xref:opendj:install-guide:index.adoc[]. Following the advice gives missing page, and only a link: URL passes. Suggested wording: (use a relative .adoc path, or a link: URL for another component).
nitpick (non-blocking): The heading "Active Directory Connector Configuration." ends with a period.
openidm-doc/src/main/asciidoc/how-to/active-directory-identity-management.adoc:35
|
Addressed all ten points from the second review in b780404f53.
Validation: all 26 regression tests pass; the generated-source checker reports 5 xrefs in 6 pages with 0 errors. doc-maven-plugin 3.2.2 preprocessing, Antora generation and the How-To PDF build pass on JDK 17 with The wiki/site patches are unchanged and remain deferred until the documentation, companion site navigation and PDF have deployed and been verified. The full product tests, full site integration build and live AD/OpenLDAP/Docker integrations were not rerun for this round. CI at the new head has not been validated here. |
maximthomas
left a comment
There was a problem hiding this comment.
praise: Both round-2 blocking issues are fixed in the text where they occur, and the checker's new rules are pinned by tests that CI runs.
active-directory-identity-management.adoc:132changes Source Missing (SOURCE_MISSING) from Exception to Delete, so deleting a managed user removes the linked AD account;configuring-openidm-for-synchronization-between-openldap-and-opendj.adoc:247mapsuserNametocn, and:303-321adds the repeated-sync check.check-how-to-links.py:45-47resolves#-less shorthand and dot-freexref:targets on the current page; the Linux/JDK 17 leg ran the suite ("Ran 26 tests ... OK", "Checked 5 How-To xrefs in 6 pages; 0 errors.").
issue (non-blocking): The <<…>> pattern does not follow Asciidoctor's xref rule: it matches inside verbatim blocks and on escaped or space-led <<, and it misses link text that wraps a line.
openidm-doc/check-how-to-links.py:33-35, :59-62
Line 33 scans the raw page text, and verbatim blocks are stripped only from the destination text (:61). A How-To page with cat <<EOF >> out.txt or 1 << 3 >> 1 in a ---- block, <<x>> in a .... or //// block, an escaped \<<x>>, or prose a << b >> c gets broken anchor ...: EOF (or 3, x, b). The "Check How-To documentation links" step then fails. Asciidoctor.js 3.0.4 renders none of these as a link: its InlineXrefMacroRx requires a word character, #, /, . or : first, and verbatim blocks get no macro substitution. The checker also misses the reverse case: <<absent,two / lines>> renders as a link to a missing anchor, but [^>\n]* stops at the newline, so the checker never checks it. No page contains << today, so this protects the next migrated page.
# openidm-doc/check-how-to-links.py, after the imports
# Comment, listing and literal blocks are verbatim: they neither link nor define anchors.
VERBATIM = re.compile(r"^(/{4,}|-{4,}|\.{4,})[ \t]*\n.*?(?:^\1[ \t]*$|\Z)", re.MULTILINE | re.DOTALL)
# Asciidoctor skips a backslash-escaped macro. A shorthand target starts with a word character,
# '#', '/', '.' or ':', and its text may wrap lines within the paragraph.
XREF = re.compile(r"(?<!\\)(?:xref:([^\s\[]+)\[|<<([\w#/.:](?:(?!\n[ \t]*\n).)*?)>>)", re.DOTALL)
# lines 33-35
for match in XREF.finditer(VERBATIM.sub("", article.read_text(encoding="utf-8"))):
target = match.group(1) or match.group(2).split(",", 1)[0]
# lines 59-62
text = VERBATIM.sub("", destination.read_text(encoding="utf-8"))
# openidm-doc/test_check_how_to_links.py
def test_verbatim_and_escaped_shorthand_is_not_a_link(self):
result = self.run_checker({"how-to/page.adoc":
"[source,bash]\n----\ncat <<EOF >> out.txt\n----\n"
"....\n<<literal>>\n....\n////\n<<comment>>\n////\n"
"\\<<escaped>> and 1 << 3 >> 1\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
self.assertIn("Checked 0 How-To xrefs", result.stdout)
def test_shorthand_text_may_wrap_lines(self):
result = self.run_checker({"how-to/page.adoc": "<<absent,two\nlines>>\n"})
self.assertEqual(1, result.returncode, result.stdout + result.stderr)
self.assertIn("broken anchor", result.stdout)Pin: both new tests fail against the head checker and pass with the change. With the change, the 26 existing tests pass and the head pages still give "Checked 5 How-To xrefs in 6 pages; 0 errors." The change still leaves these unchecked: shorthand targets that start with { (attribute references), ++++ passthrough blocks, indented literal paragraphs, and xrefs that follow an unclosed block in the article (see the next comment).
suggestion (non-blocking): An unclosed listing inside an example block or a table cell now hides every later anchor on the destination page.
openidm-doc/check-how-to-links.py:61-62
The new \Z alternative ends an unclosed block at end of file. Asciidoctor ends it at the parent's closing delimiter (====, |===), warns unterminated listing block, and still renders a later [[later]]. Take a destination page with ==== / unclosed ---- / ==== / [[later]]: the checker reports broken anchor ...: page.adoc#later and exits 1, while 3e6e967 passed the same page. The page is malformed, so the cost is an error that names the wrong cause. When \Z ends the strip, reporting the unclosed block itself (<page>: unclosed ---- block) would name the real cause. This review did not implement or run that variant.
b780404 to
fd5abc3
Compare
|
Addressed both points from the latest review in fd5abc3c2c.
Validation: all 30 Python tests pass; the checker reports 5 xrefs in 6 source How-To pages with 0 errors, and |
maximthomas
left a comment
There was a problem hiding this comment.
praise: The checker now handles what the round-3 findings named.
source_textstrips comment, listing and literal blocks from articles as well as from destinations. It caches each file by resolved path, so a page with an unclosed block is reported once, however many xrefs point at it.- An unclosed block is now reported as
<file>: unclosed <delimiter> blockinstead of a misleading broken anchor.test_unclosed_block_reports_its_location_instead_of_a_missing_anchorandtest_unclosed_block_in_article_is_reportedpin this. - The real pages give the same result as before: 5 How-To xrefs, 0 errors.
issue (non-blocking): The multi-line shorthand pattern pairs a << in one table cell, list item or block with a >> in the next.
openidm-doc/check-how-to-links.py:24-25, :58
This pattern is the one my round-3 review suggested, and that suggestion was wrong: it handles paragraph breaks only. Asciidoctor substitutes each paragraph, table cell, list item and delimited block separately. The pattern stops only at a blank line, and VERBATIM does not strip tables or lists, so three inputs now fail with a broken-anchor error even when [[a]] exists:
- a table with
| `cat <<EOF` | here-documenton one line and| `echo x >> f` | appendon the next; - the list
* <<a/* b>>; see <<a/====/b>>/====.
Asciidoctor.js 2.2.9 (Antora 3.1.9) and 3.0.4 render no link for any of them, and the previous head passed all three. The fix below keeps a target on one line and in one cell, and ends the link text at a blank line, a cell line or a delimiter line. Measured: the 30 existing tests and the new one pass, and the new test fails at this head. Each of five mutants of the fix fails it: | or \n put back into the target class, or the cell, delimiter or blank-line break dropped. <<absent,two\nlines>> is still reported, and the real pages are unchanged. The fix also covers cells on one line (| `cat <<EOF` | `echo x >> f`), which failed at the previous head too.
# Asciidoctor substitutes each paragraph, table cell and delimited block separately: a
# shorthand target stays on one line and in one cell, and its text stops at a blank line,
# a cell line or a delimiter line.
BLOCK_BREAK = r"\n[ \t]*(?:\n|\||(?:={4,}|\*{4,}|_{4,}|\+{4,}|-{2,}|/{4,}|\.{4,})[ \t]*(?:\n|\Z))"
XREF = re.compile(r"(?<!\\)(?:xref:([^\s\[]+)\[|"
r"<<([\w#/.:][^,|\n]*?(?:,(?:(?!" + BLOCK_BREAK + r").)*?)?)>>)", re.DOTALL) def test_shorthand_does_not_join_cells_items_or_blocks(self):
result = self.run_checker({"how-to/page.adoc":
"|===\n| `cat <<EOF` | `echo x >> f`\n| <<a\n| b>>\n| <<a,x\n| y>>\n|===\n\n"
"* <<a\n* b>>\n\nsee <<a,x\n====\ny>>\n====\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
self.assertIn("Checked 0 How-To xrefs", result.stdout)Some cases are still not handled. <<a\nb>> and <<a|b>> inside a paragraph are no longer counted; Asciidoctor renders each as a link to an id that no anchor can have. * <<a,x / * y>> is still counted, because a * line does not end a paragraph in Asciidoctor (see <<a,x\n* y>> is a link), and a regex cannot tell a list from a paragraph.
issue (non-blocking): The checker still scans line comments, ``` fences, [source]/`[listing]`/`[literal]` paragraphs and inline passthroughs. It also accepts an `[[id]]` inside a fence as a destination.
openidm-doc/check-how-to-links.py:22-23, :39-53, :85-87, openidm-doc/wiki-migration/README.md:34
Asciidoctor (2.2.9 and 3.0.4) renders no link for any of the following, but the checker fails on each:
// see xref:missing.adoc[old]fails withmissing page;// <<gone>>fails with a broken-anchor error;cat <<EOF >> out.txtinside a```bashfence or a[source,bash]paragraph fails with a broken-anchor error;+<<id>>+andpass:[<<x>>]fail with a broken-anchor error.
In the other direction, [[fenced]] inside a ``` fence passes as a destination, but Asciidoctor renders no id for it. README:34 says that text inside "comment, listing and literal blocks" is ignored, yet a fence and a [source] paragraph are listing blocks. The fix below removes a comment line together with its newline, so a paragraph stays one paragraph. Asciidoctor still links `<<a,x` / `// c` / `y>>`. `/// x` is not a comment in Asciidoctor, and the fix keeps it. Measured: the 30 existing tests and both new ones pass, both new tests fail at this head, and the real pages are unchanged. With all three fixes and the two pins below applied, all 36 tests pass.
VERBATIM = re.compile(r"^(/{4,}|-{4,}|\.{4,}|```)(?:[ \t]*|(?<=`)(?!`)[^\n]*)\n.*?"
r"(?:(?P<closed>^\1[ \t]*$)|\Z)", re.MULTILINE | re.DOTALL)
# Source, listing and literal paragraphs and line comments are not substituted either.
VERBATIM_PARAGRAPH = re.compile(r"^\[(?:source|listing|literal)\b[^\]\n]*\]\n(?:[ \t]*\S.*(?:\n|\Z))*",
re.MULTILINE)
LINE_COMMENT = re.compile(r"^//(?=[^/\n]|$)[^\n]*\n?", re.MULTILINE) text = VERBATIM_PARAGRAPH.sub("", VERBATIM.sub(strip_block, original))
sources[path] = LINE_COMMENT.sub("", text), unclosed def test_line_comments_fences_and_verbatim_paragraphs_are_not_links(self):
result = self.run_checker({"how-to/page.adoc":
"// see xref:missing.adoc[old] and <<gone>>\n\n"
"```bash\ncat <<EOF >> out.txt\n```\n\n"
"[source,bash]\ncat <<EOF >> out.txt\n\n[literal]\n<<literal>>\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
self.assertIn("Checked 0 How-To xrefs", result.stdout)
def test_anchor_in_fence_is_not_a_destination(self):
result = self.check_link("fenced", "```\n[[fenced]]\n```\n")
self.assertEqual(1, result.returncode, result.stdout + result.stderr)
self.assertIn("broken anchor", result.stdout)Leave inline passthroughs (+...+, pass:[...]) out of scope, and add them to the out-of-scope list in README:34. Stripping ++...++ would also strip the #++a--b++ fragments that xref: targets use. Also update the "runs 30 regression tests" count in README:36.
suggestion (non-blocking): The unclosed-block diagnostic names only the file, so unclosed blocks in two different index.adoc pages print the same line.
openidm-doc/check-how-to-links.py:48, :88
Take how-to/page.adoc linking xref:../a/index.adoc#x and xref:../b/index.adoc#y, with an unclosed ---- block in both destinations. The checker prints index.adoc: unclosed ---- block twice. An unclosed block in getting-started/index.adoc reads as if it were in the article how-to/index.adoc. Line 88 now suppresses the broken-anchor error, which named the target path, so this line is the only pointer to the broken file. This head has 6 index.adoc and 5 preface.adoc files. The fix prints the path relative to the source root; it resolves the root because path is resolved, and macOS temp directories sit behind /var → /private/var. The new test fails at this head. The existing assertIn("module.adoc: unclosed ...") and assertIn("page.adoc: unclosed ...") assertions still pass.
import os
...
errors.append(f"{os.path.relpath(path, source.resolve())}: "
f"unclosed {match.group(1)} block") def test_unclosed_block_names_the_destination_directory(self):
result = self.run_checker({
"how-to/page.adoc": "xref:../a/index.adoc#x[A] xref:../b/index.adoc#y[B]\n",
"a/index.adoc": "----\n", "b/index.adoc": "----\n"})
self.assertEqual(1, result.returncode, result.stdout + result.stderr)
self.assertIn("a/index.adoc: unclosed ---- block", result.stdout)
self.assertIn("b/index.adoc: unclosed ---- block", result.stdout)suggestion (non-blocking): No test pins the blank-line stop in the shorthand pattern.
openidm-doc/test_check_how_to_links.py:104, openidm-doc/check-how-to-links.py:24
The only multi-line shorthand test, <<absent,two\nlines>>, has no blank line in it. Mutant (?:(?!\n[ \t]*\n).)*? → (?:.)*? passes all 30 tests, including both wiki-patch tests. Under that mutant, link text that spans two paragraphs fails with a broken-anchor error, while Asciidoctor renders it as text.
Pin: the test below passes at this head and with the first fix above, and fails under the mutant.
def test_shorthand_text_stops_at_a_blank_line(self):
result = self.run_checker({"how-to/page.adoc": "<<absent,two\n\nparagraphs>>\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
self.assertIn("Checked 0 How-To xrefs", result.stdout)suggestion (non-blocking): No test pins the shorthand pattern's first-character class.
openidm-doc/check-how-to-links.py:24, openidm-doc/test_check_how_to_links.py:100
<<([\w#/.:] excludes ' and - as the first character of a target, as Asciidoctor does. The only test of the class is 1 << 3 >> 1, which starts with a space, and [^\s] rejects that too. Mutant <<([\w#/.:] → <<([^\s] passes all 30 tests. Under it, cat <<'EOF' >> out.txt and cat <<-EOF >> out.txt in prose fail with a broken-anchor error, while Asciidoctor 2.2.9 and 3.0.4 render neither as a link.
Pin: the test below passes at this head and with the first fix above, and fails under the mutant.
def test_shorthand_target_starts_with_a_word_character(self):
result = self.run_checker({"how-to/page.adoc":
"cat <<'EOF' >> out.txt\ncat <<-EOF >> out.txt\n"})
self.assertEqual(0, result.returncode, result.stdout + result.stderr)
self.assertIn("Checked 0 How-To xrefs", result.stdout)|
Addressed the latest review in 14830b8480.
Validation: all 42 Python tests pass. doc-maven-plugin 3.2.2 preprocessing passed on JDK 17 with a temporary standalone POM and cache; the checker found 5 xrefs across 6 generated How-To pages with 0 errors. |
maximthomas
left a comment
There was a problem hiding this comment.
praise: This round closes most of the boundary cases from round 4, and each fix has a test.
LINE_COMMENT's(?=[^/\n]|$)keeps///out of the comment rule.test_three_slashes_are_not_a_line_commentpins it.TABLE.subturns only unescaped cell pipes ((?<!\\)\|) into\x00, so<<absent,one \| two>>inside a cell is still checked (test_shorthand_links_inside_a_table_cell_are_checked).- The unclosed-block diagnostic now prints
os.path.relpath(path, source.resolve()), so two unclosedindex.adocpages produce different lines. On the Linux/JDK 17 leg, 42 tests pass and the checker reportsChecked 5 How-To xrefs in 6 pages; 0 errors.
Adds OpenIDM How-To Guides to the site landing page, OpenIDM navigation and PDF mapping. The companion OpenIDM change has been merged, and its generated `how-to` module and `HowTo_Guides.pdf` have been published. Generated pages and screenshots remain owned by the existing OpenIDM documentation deployment; this PR changes only site registration. Validation: before the rebase, a current-version Antora integration build with the companion generated module checked all 6 How-To pages, local article links and fragments, 28 images and all 37 wiki section destinations, with no How-To diagnostics. The navigation/PDF YAML mapping passed. Historical release tags were excluded from the integration playbook. After rebasing onto the published module, `git range-diff` confirmed that the commit content is unchanged, and `git diff --check` passed. The published guide and PDF URLs returned HTTP 200. Companion PR: OpenIdentityPlatform/OpenIDM#275. The deferred wiki migration patch is attached to that PR through its committed artifact and download link.
Publishes five standalone OpenIDM wiki tutorials as a searchable How-To guide, following OpenIdentityPlatform/OpenAM#1218. The guide covers Active Directory identity management, the chess workflow, AD/OpenDJ and OpenLDAP/OpenDJ synchronization, and persistent Docker data. It includes 28 local screenshots and preserves all 37 source section anchors with article prefixes.
Registers the guide in Maven and runs the How-To link checker and its 42 regression tests on the Linux/JDK 17 build. The checker validates relative AsciiDoc source paths and explicit destination anchors, including bare same-page IDs and
<<...>>shorthand. Tests validate all 48 documentation URLs in the deferred wiki patch and check its 37 section destinations through the same anchor validation. Linux build, deploy and release jobs use a 6 GB Maven heap for illustrated PDF generation; other build platforms use 4 GB.Repairs malformed JavaScript table cells, the AD
sAMAccountNametypo and AD-only attributes copied into the reverse OpenLDAP mapping. The AD tutorial usesDOMAIN\Administrator, retains the sample filters excluding disabled accounts and computers, explains the comma needed before adding the password property, and uses consistent given names and surnames throughout the REST examples. It explicitly sets Source Missing to Delete and removes the ignoredcnmapping. The reverse OpenLDAP mapping retains the RDN value withuserNametocnand includes checks for existing accounts after a repeated synchronization. Historical example output and external integration prerequisites remain documented; external integrations were not retested.Link-checker scope
Fragment destinations require explicit
[[id]]or[#id]anchors outside comment, listing and literal blocks or paragraphs. The checker ignores escaped links and line comments, handles fenced listings, accepts multiline shorthand link text, and identifies unclosed blocks by source-relative path. Shorthand references do not cross supported table-cell or block boundaries. Dottedxref:targets are document paths; shorthand targets without#, including dotted IDs, address the current page. Automatic section IDs and Antora page IDs remain outside the checker's scope and produce explanatory diagnostics. Attribute references in shorthand targets, passthrough blocks, inline passthroughs and indented literal paragraphs are also outside its scope. Shorthand targets containing a newline or pipe are not checked, and shorthand link text can still be mistakenly joined across list items. Cross-component links can uselink:URLs.ROOT:index.adocis skipped because Antora generates that landing page, and intraword double-hyphen fragments retain the passthrough requirement.Wiki migration patch
The patch is included in this PR for separate review and later application:
It retains the original wiki headings, spelling and duplicate-heading suffixes, linking each section to the matching published destination. Home keeps its
How-Toheading and links directly to the published guide below it, preservingHome#how-to; the two synchronization tutorials previously missing from Home are added. The patch does not create a separate wiki landing page. Existing wiki images and OpenLDAP downloads are retained.Do not apply the wiki patch until OpenIDM's generated documentation, the companion site navigation and the How-To PDF have deployed and their links/images have been verified. The product build does not apply either deferred patch automatically.
Validation at the current head
git diff --checkpasses.origin/master. The wiki and site patches are unchanged. The PDF, full site integration build, full product tests and live directory/Docker integrations were not rerun for this round. CI at the new head has not been validated here.Previous review-round validation
-Xmx6g. The PDF had 38 pages and was 2,864,405 bytes. Rendered HTML assertions, extracted PDF text and visual inspection covered the AD deletion policy, reverse OpenLDAP mapping and repeated-synchronization checks.Earlier migration validation
The previous round passed 17 tests, seven targeted checker mutations and a renamed wiki destination. The REST create request and all four account responses parsed as JSON with consistent names; following the password insertion instruction produced valid connector JSON. The regenerated wiki patch passed forward/reverse applicability checks at revision
c2b7bfa16d9e024771014c4f01b8ab00f376edfb; its Home post-image hash matched the produced file (52b25e9).The current-version site integration build checked all 6 How-To pages, local article links/fragments, 28 images, 37 wiki section destinations and repaired JavaScript expressions, with no How-To diagnostics. All screenshot copies were byte-identical to the wiki originals; the site patch passed forward/reverse applicability checks at its recorded source revision, and workflow YAML parsing passed.
Companion site navigation PR: OpenIdentityPlatform/doc.openidentityplatform.org#46.