Skip to content

[#165] Compile each runScriptOnConnector script once per engine and let evicted scripts unload - #196

Merged
maximthomas merged 8 commits into
OpenIdentityPlatform:masterfrom
maximthomas:issues/165-run-script-on-connector-cache
Oct 20, 2026
Merged

maximthomas merged 8 commits into
OpenIdentityPlatform:masterfrom
maximthomas:issues/165-run-script-on-connector-cache

Conversation

@maximthomas

@maximthomas maximthomas commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #165.

runScriptOnConnector no longer compiles a new class into the script engine's loader on every call. Each distinct script text is compiled once per engine and kept in a bounded LRU cache. Classes that leave the cache, and every cached class on release(), are unregistered so that they can be unloaded. With the default size an engine keeps at most 64 runScriptOnConnector scripts, each with its closures and declared classes, however many calls it serves.

  • One class per text, in its own loader. ScriptedConfiguration.getScriptOnConnectorClass compiles a text with a ScriptOnConnectorLoader, a child of the engine's loader, which keeps the new classes out of the engine's class cache. Before, every call went through getGroovyScriptEngine().getGroovyClassLoader().parseClass(...). Each class then stayed in the engine loader, and in GroovyScriptEngine's script cache, until release().
  • Script-root classes are the engine's. A class whose source is in the script roots, such as Helper, resolves through GroovyScriptEngine.loadScriptByName, as it does in the engine's own compilations. A text therefore uses the Helper that the operation scripts use. The engine compiles it when it is first needed or after its source changes, and a script-root class still shadows a class of the connector's class loader. Other classes resolve through the engine's loader.
  • A cached text runs against the classes it was compiled against. Its code links a class by name the first time each reference runs. ScriptOnConnectorLoader keeps every class the compiler resolved for the text, and the loaders of the script-root classes among them. Its loadClass returns a resolved class, or else a class that one of those loaders defined: a nested class, a closure or another class compiled from the same source. Without that, code that first ran after an operation script had recompiled an edited Helper linked the new classes. @CompileStatic code then failed with NoSuchMethodError where it was compiled against an old signature, and with a LinkageError (loader constraint violation) where it reached a class such as Helper.Greeting only through the return type of a Helper method.
  • The engine's configuration and compiler lock. A text compiles with the engine's compiler configuration, the one getRoots(CompilerConfiguration, GroovyClassLoader) receives, so with the default imports and with whatever a subclass adds there, a SecureASTCustomizer for example. It compiles under the lock that the engine's loader holds while it compiles an operation script or a script-root class, as texts compiled on master: the monitor of the loader's GroovyClassLoader.sourceCache. The engine therefore cannot recompile a script-root class while a text resolves the classes it uses, and the customizers of the configuration are not used by two compilations at once. The compiler resolves the class nodes of an ImportCustomizer in place. sourceCache is protected and the loader is a private class of GroovyScriptEngine, so the field is read by reflection; Groovy 2.2.1, 2.4.21, 4.0.22 and 5.0.3 have it and lock it in GroovyScriptEngine's loader. getImportCustomizer(ImportCustomizer) and getImports() are called once per engine, as on master.
  • One compilation at a time, no lock for a cached text. The configuration monitor is held only to read the engine and the cache, and the cache's monitor only around map operations. Texts compile one at a time, and not while the engine compiles, as on master. A cached text takes no lock and does not wait for a compilation. If two threads compile the same new text, the first one cached wins and the other's classes are unregistered.
  • What a text adds or compiles through its loader goes to the engine. A text's class loader delegates addURL, addClasspath and parseClass to the loader that compiled the text, now its ScriptOnConnectorLoader. That loader passes them on to the engine's loader, as on master. The operation scripts and the other texts load classes from a URL that a text adds, and see a class that a text compiles at run time. That compilation takes the engine's compiler lock, and the class resolves Helper as the operation scripts do. The text itself compiles through a separate method, into its own loader.
  • Bounded, least recently used first. An engine keeps 64 scripts by default. The system property org.identityconnectors.framework.groovy.scriptOnConnectorCacheSize changes the size. It is read when an engine is created. A value that is not a positive integer logs a WARN and falls back to 64.
  • Evicted and released classes can unload. Eviction and release() call InvokerHelper.removeClass on every class of the script's child loader, closures and declared classes included, outside the cache monitor. The cache is created with the engine once its customizer has succeeded. release() clears it last in its finally.
  • Run from Java. ScriptedConnectorBase.runScriptOnConnector now calls evaluateScriptOnConnector, which runs InvokerHelper.createScript(...).run() in Java. A Groovy call site caches the metaclass of the last script it ran, and so keeps that class alive. The arguments, the log binding and the catch blocks are unchanged.
  • Without a cache, as on master. While the customizer runs, the engine is not published and there is no cache. A subclass that overrides getGroovyScriptEngine() without calling super has no cache either. A text then compiles through the engine's loader, with the engine's configuration, as on master, and is not cached. The configuration of an engine that a subclass builds, a SecureASTCustomizer for example, therefore still applies to its texts. If GroovyClassLoader.sourceCache cannot be read, there is no cache either, and a WARN is logged when ScriptedConfiguration is loaded.
  • A broken script-root class still fails as a compilation error. The engine compiles it, and its CompilationFailedException reaches the caller, as on master. A ScriptException from loadScriptByName, which a GroovyScriptEngine subclass may throw, also fails the text's compilation with a CompilationFailedException, not a GroovyBugError.

Behaviour changes:

  • a script's class name is stable while its text stays cached;
  • static fields, metaclass changes and classes declared in a script survive between calls with the same text;
  • classes declared in one text are no longer visible to other texts or to the operation scripts, and a text can no longer replace a script-root class for everyone else;
  • a text's class loader answers getURLs, findResource, findResources, getResourceLoader, setResourceLoader, getLoadedClasses and clearCache for its ScriptOnConnectorLoader, not for the engine's loader. getResource and getResourceAsStream give the same answers as on master;
  • a cached script runs against the classes it was compiled against, script-root classes such as Helper and the classes compiled with them included, until it is evicted or the configuration is released. This holds even after their source changes on disk and the operation scripts move to the new classes. Master recompiled the text on every call and so picked up the change; a newly compiled text still does;
  • a class evicted while another thread runs it loses any metaclass changes it made. Its metaclass is rebuilt on its next dynamic call, after which the class is only softly held;
  • a compilation that finishes after release() goes into the orphaned cache, so its class is only softly held.

Not changed:

  • operation scripts (loadScript, evaluate) are still compiled and cached by the engine;
  • the customizer lifecycle and the exceptions runScriptOnConnector maps.

Test strength

All in ScriptOnConnectorCacheTest. The ed25f73, ce3a86f, 37b50d1 and 21f99b4 cells are runs of the new tests against the production code of those earlier heads of this PR.

arm case evidence
one class per text #testSameTextReusesTheCompiledClass compile-red at c328553
classes stay out of the engine loader #testScriptClassStaysOutOfTheEngineLoader compile-red at c328553; mutants "child loader's parent is getParentLoader(), not the engine's loader" and "the text compiles through the parseClass it passes on to the engine" killed
the text's configuration has the default imports #testScriptSeesTheDefaultImports mutant "text configuration without the import customizer" killed
script-root classes resolve #testScriptSeesTheClassesInTheScriptRoots kept behaviour
a text shares script-root classes with the operation scripts #testScriptSharesScriptRootClassesWithOperationScripts red at ed25f73; mutants "no engine resolver" and "default resolver first" killed
… also after a source edit, the text compiled first #testNewScriptSharesTheEditedScriptRootClassWithOperationScripts red at ed25f73; both resolver mutants killed
script roots shadow the connector's classes #testScriptRootClassShadowsAClassOfTheConnectorLoader red at ed25f73; both resolver mutants killed
a cached text keeps the class it was compiled against #testCachedScriptKeepsTheScriptRootClassItWasCompiledAgainst kept behaviour, green at ed25f73
… also where its code first runs after the engine recompiled the class, nested classes included #testCachedScriptLinksTheScriptRootClassesItWasCompiledAgainst red at ce3a86f (hi hi); mutant "no loadClass override" killed
… a class of a script-root source that the text names without that source's own class #testCachedScriptLinksTheEngineClassesItWasCompiledAgainst kept behaviour, green at 37b50d1; mutants "no lookup of the resolved classes" and "keep only the script-root classes" killed
… statically compiled code, with classes reached only through the methods of a script-root class #testCachedStaticScriptLinksTheClassesCompiledWithAScriptRootClass red at 37b50d1 (LinkageError, loader constraint violation); mutants "no lookup in the script-root loaders" and "nested classes only" killed
texts compiled together share a script-root class #testScriptsCompiledTogetherShareTheScriptRootClasses red at ed25f73; mutant "no engine compiler lock" killed (five rounds of four threads: a race, so a likely kill, not a certain one)
a text and an operation script that load a script-root class together share it #testTextAndOperationScriptThatLoadAScriptRootClassTogetherShareIt red at 37b50d1 (two copies of Helper); mutant "no engine compiler lock" killed
texts compile together on a new engine #testScriptsCompileTogetherOnANewEngine red at ed25f73 (unable to resolve class org$forgerock$openicf$misc$scriptedcommon$ICFObjectBuilder); mutant "no engine compiler lock" killed, with the engine's configuration shared
what a subclass adds to the engine's configuration in getRoots applies to texts #testScriptCompilesWithWhatASubclassAddsToTheEngineConfiguration red at 37b50d1 (unable to resolve class RootsList); mutant "a new configuration built as the engine's" killed
binding reaches the script #testScriptSeesItsBinding compile-red at c328553
a URL that a text adds reaches the operation scripts #testUrlThatATextAddsToItsLoaderReachesTheOperationScripts red at 37b50d1 (ClassNotFoundException: drv.Driver); mutant "no addURL override" killed
a class that a text compiles at run time reaches the operation scripts #testClassThatATextParsesReachesTheOperationScripts red at 21f99b4 (MissingPropertyException: No such property: Parsed); mutant "the parseClass override takes the engine's lock but compiles into the text's loader" killed
… and waits for the engine's compilation #testClassThatATextParsesWaitsForTheEngineCompilation red at 21f99b4 (parseClass returns while the engine compiles an operation script)
release() unregisters and recompiles #testReleaseUnregistersTheCachedClasses compile-red at c328553
… also when the release closure throws #testReleaseUnregistersTheCachedClassesWhenTheReleaseClosureThrows mutant "clear the cache after the try/finally" killed
customizer before publish #testCustomizerCanRunAScriptOnConnector mutant without the null-cache guard killed (NullPointerException)
… and its script is not cached #testScriptRunByTheCustomizerIsNotCached mutant "cache created before the customizer runs" killed
an engine built by a subclass compiles with its own configuration #testEngineBuiltByASubclassCompilesTheScriptWithItsOwnConfiguration red at ce3a86f (unable to resolve class EngineList); mutants "no cache: compile with the child loader anyway" and "no null-cache guard" killed
a hit does not wait for a compile #testCachedScriptDoesNotWaitForAnotherCompilation mutants synchronized getScriptOnConnectorClass and "cache lookup under the engine's compiler lock" killed (TimeoutException)
size property: name, default, 5, 7, 0, -3, many #testCacheSizeComesFromTheSystemProperty compile-red at f4b1286
LRU, not insertion order #testLeastRecentlyUsedScriptIsEvicted mutant accessOrder false killed
eviction unregisters closures too #testEvictedScriptClassesAreUnregistered mutants "unregister only the main class" and "no unregister in put" killed
an evicted class unloads #testEvictedScriptClassCanBeUnloaded same two mutants killed
the connector reuses the class #testConnectorReusesTheCompiledScript red at a40030c
arguments and log binding #testConnectorScriptSeesItsArgumentsAndTheLogger kept behaviour, green at a40030c; mutants "log is Log.getLog(Object.class)" and "evaluateScriptOnConnector rebinds log to the script's class" killed
a connector-run class unloads once evicted #testScriptRunByTheConnectorCanBeUnloadedOnceEvicted red at a40030c; mutant Groovy-side createScript(...).run() killed (the call site keeps the class)
broken script-root class → compile error #testBrokenScriptRootClassFailsAsACompilationError red at a872c5b (GroovyBugError); both resolver mutants killed
… also from the connector #testConnectorReportsABrokenScriptRootClassAsACompilationError red at a872c5b (GroovyBugError); both resolver mutants killed
a ScriptException from the engine → compile error #testScriptExceptionFromTheEngineFailsAsACompilationError red at 37b50d1 (GroovyBugError); mutants "ConnectorException.wrap" and "no cause" killed

The two unload tests run System.gc() for up to about 5 s and return as soon as the class is gone. They rely on the JVM's default soft-reference policy, which surefire leaves unchanged.

Not pinned by a test:

  • the first-wins rule when two threads compile the same new text;
  • the WARN for an invalid size, and reading the size once per engine;
  • unregistering closure classes on release() (eviction pins the shared unregister());
  • the fallback without a cache, and its WARN, when GroovyClassLoader.sourceCache cannot be read;
  • that loadClass takes from the loader of a script-root class only the classes that loader defined. A mutant that takes any class the loader returns passes: the two differ only for a class that the script-root class linked before the engine replaced it.

mvn -pl OpenICF-groovy-connector,OpenICF-ssh-connector verify on JDK 26:

  • groovy: Tests run: 179, Failures: 0, Errors: 0, Skipped: 42. The 42 skipped tests are the five *SampleTest classes, which need a private configuration;
  • ssh: Tests run: 135, Failures: 0, Errors: 0, Skipped: 120. The skipped sample tests need a live host.

Master is at 4648e38 now. #188, #191, #194 and #197 touch none of these files. The branch merges into it cleanly, and the groovy module then runs Tests run: 210, Failures: 0, Errors: 0, Skipped: 42.

@maximthomas maximthomas added bug Something isn't working java Pull requests that update java code tests Test additions or fixes connector:groovy Groovy connector concurrency Races, locking and thread-safety fixes resource-leak Memory, class-loader, thread or handle leaks performance Performance and scalability fixes labels Oct 15, 2026
…gine and compile each text with its own import customizer
…as compiled against, and compile as master did without a cache
@maximthomas
maximthomas marked this pull request as ready for review October 15, 2026 18:40
@maximthomas
maximthomas requested a review from vharseko October 15, 2026 18:40

@vharseko vharseko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the description and the test-strength table make this easy to follow. I reproduced points 1–4 with a test class that uses only API present on both master and this branch. It is at the end, under "Test used for points 1–4". All of its checks pass on master (1eeff90) and fail on 37b50d1. Point 5 comes from reading the Groovy 2.4.21 sources.

1. Under @CompileStatic, a type reached only through a signature of a pinned class is not pinned (ScriptedConfiguration.java:1112)

resolvedClasses holds only the names the compiler asked the resolver for. Some types never go through findClassNode: those a text reaches only through a member signature of a script-root class, such as the return type Helper.Greeting of Helper.make(). The static type checker takes such a type from the ClassNode of the resolved Helper. At run time the JVM still loads it through the text's loader. That lookup misses resolvedClasses and lands in the engine's current class cache.

Repro: Helper.groovy has static Greeting make() and a nested static class Greeting { String text() }. The text is @CompileStatic String use(boolean b) { b ? Helper.make().text() : 'not run' }. It is compiled and cached with b = false. Then Helper.groovy is edited and an operation script recompiles it. The same text then runs with b = true:

  • this branch: java.lang.LinkageError: loader constraint violation: loader groovy.lang.GroovyClassLoader$InnerLoader @2453f95d wants to load class Helper$Greeting. A different class with the same name was previously loaded by groovy.lang.GroovyClassLoader$InnerLoader @71e5f61d.
  • the same text without @CompileStatic: hello
  • master: hi

This is the case the description covers with "script-root classes such as Helper and their nested classes included". testCachedScriptLinksTheScriptRootClassesItWasCompiledAgainst names Helper.Greeting in the text, so the type goes through the resolver.

2. One compilation can mix two versions of a script-root class (ScriptedConfiguration.java:1092)

Helper resolves under scriptRootLock. Helper$Greeting has no source file of its own. It falls to super.findClassNode outside the lock and reads the engine's class cache as it is at that moment. If an operation script recompiles an edited Helper between the two lookups, the text keeps Helper v1 and Helper$Greeting v2 for as long as it stays cached.

Repro: I added a script root with a URL handler that holds the lookup of Trigger.groovy. This stops the text def a = Helper.greet(); if (false) { Trigger.x() }; def b = Helper.Greeting.text(); return a + ' ' + b after it has resolved Helper. While it waits, Helper.groovy is edited and the operation script runs.

  • this branch: the operation script returns hi hi while the text is still compiling, and the text returns hello hi.
  • master: the operation script waits for the text's compilation (the engine's compiler lock), and the text returns hello hello.

In production the window between the two lookups is short, but the mixed set lasts as long as the cache entry does. This is a different case from the first-load race listed under behaviour changes.

For both points, a name that misses resolvedClasses could be looked up through the loader of the script-root class it is nested in. Helper v1's InnerLoader returns its own Helper$Greeting from findLoadedClass. I have not tried this.

3. Changes a subclass makes to the engine's configuration in getRoots no longer reach texts (ScriptedConfiguration.java:835)

getRoots(CompilerConfiguration, GroovyClassLoader) is protected and receives the engine's configuration (line 928). Texts now compile with a fresh newCompilerConfiguration() instead. Repro: a subclass adds an ImportCustomizer with the alias RootsList and a SecureASTCustomizer with setMethodDefinitionAllowed(false) in getRoots.

  • this branch: return new RootsList() fails with unable to resolve class RootsList, and def f() { 1 }; return f() returns 1.
  • master: [] and SecurityException: Method definitions are not allowed.

Operation scripts still resolve RootsList on both branches. The description says a SecureASTCustomizer "still applies to its texts" for a subclass that builds its own engine, but this hook is a second way to configure the engine. If the change is intended, it belongs under behaviour changes. If not, the text configuration has to carry what the hook added.

4. A URL a text adds to its class loader no longer reaches the engine (ScriptedConfiguration.java:1082)

A text's InnerLoader delegates addURL, addClasspath and parseClass to the loader that compiled it. That used to be the engine's loader. Now it is the text's own ScriptOnConnectorLoader. Repro: one text runs this.class.classLoader.addURL(new URL('file:/…/lib/')), and then something else loads drv.Driver from that directory.

  • this branch: the URL is not in the engine's loader, and both an operation script and a second text fail with ClassNotFoundException: drv.Driver.
  • master: both load it and return drv.

This is probably rare, but it deserves a line under behaviour changes.

5. Nit: the ScriptException branch in loadScriptRootClass (ScriptedConfiguration.java:1134)

loadScriptByName declares ScriptException, so the branch has to stay. In Groovy 2.4.21, however, the method's body never throws it. If a GroovyScriptEngine subclass did throw it, the ConnectorException thrown from inside the resolver would reach CompilationUnit.applyToSourceUnits, which turns any Exception from the resolve phase into a GroovyBugError. That is the outcome testBrokenScriptRootClassFailsAsACompilationError guards against. Throwing a CompilationFailedException there would keep it a compilation error.

Test used for points 1–4 (same package as ScriptOnConnectorCacheTest; passes on master, four failures on this branch)
package org.forgerock.openicf.misc.scriptedcommon;

import static java.util.concurrent.TimeUnit.SECONDS;
import static org.fest.assertions.api.Assertions.assertThat;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLStreamHandler;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeoutException;

import org.codehaus.groovy.control.CompilationUnit;
import org.codehaus.groovy.control.CompilerConfiguration;
import org.codehaus.groovy.control.customizers.ImportCustomizer;
import org.codehaus.groovy.control.customizers.SecureASTCustomizer;
import org.forgerock.openicf.connectors.groovy.ScriptedPoolableConnector;
import org.identityconnectors.framework.common.objects.ScriptContext;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import groovy.lang.Binding;
import groovy.lang.GroovyClassLoader;

/** Temporary checks of the review findings on PR #196; uses only API that master has too. */
public class ReviewFindings196Test {

    private static volatile CountDownLatch triggerGate;

    private static volatile CountDownLatch triggerWaiting;

    private File scriptRoot;

    private ScriptedConfiguration configuration;

    private ExecutorService executor;

    @BeforeMethod
    public void setUp() throws Exception {
        scriptRoot = Files.createTempDirectory("review-196").toFile();
        configuration = withScriptRoot(new ScriptedConfiguration());
        executor = Executors.newCachedThreadPool();
    }

    @AfterMethod
    public void tearDown() {
        final CountDownLatch gate = triggerGate;
        triggerGate = null;
        if (null != gate) {
            gate.countDown();
        }
        executor.shutdownNow();
        configuration.release();
        delete(scriptRoot);
    }

    /** F1: a type reached only through a signature of a pinned class, under @CompileStatic. */
    @Test
    public void f1SignatureOnlyNestedTypeCompileStatic() throws Exception {
        f1(true);
    }

    /** F1 control: the same text without @CompileStatic. */
    @Test
    public void f1SignatureOnlyNestedTypeDynamic() throws Exception {
        f1(false);
    }

    private void f1(boolean compileStatic) throws Exception {
        writeHelper("class Helper { static Greeting make() { new Greeting() }\n"
                + "static class Greeting { String text() { 'hello' } } }\n");
        write("op.groovy", "return Helper.make().text()\n");
        final String text = (compileStatic ? "@groovy.transform.CompileStatic\n" : "")
                + "String use(boolean b) { b ? Helper.make().text() : 'not run' }\n"
                + "return use((boolean) useHelper)\n";
        assertThat(runOnConnector(text, useHelper(false))).isEqualTo("not run");

        editHelper("class Helper { static Greeting make() { new Greeting() }\n"
                + "static class Greeting { String text() { 'hi' } } }\n");
        assertThat(configuration.evaluate("op.groovy", new Binding(), null)).isEqualTo("hi");

        final Object result = outcome(() -> runOnConnector(text, useHelper(true)));
        System.out.println("REVIEW-F1 compileStatic=" + compileStatic + " result: " + result);
        assertThat(result).isIn("hello", "hi");
    }

    /** F2: one compilation of a text resolves Helper and Helper$Greeting from different versions. */
    @Test(timeOut = 120000)
    public void f2OneCompilationMixesTwoVersions() throws Exception {
        configuration.release();
        configuration = withScriptRoot(new ScriptedConfiguration() {
            @Override
            protected URL[] getRoots(CompilerConfiguration compilerConfiguration, GroovyClassLoader loader) {
                final URL[] roots = super.getRoots(compilerConfiguration, loader);
                final URL[] result = Arrays.copyOf(roots, roots.length + 1);
                result[roots.length] = blockingRoot();
                return result;
            }
        });
        writeHelper("class Helper { static String greet() { 'hello' }\n"
                + "static class Greeting { static String text() { 'hello' } } }\n");
        write("op.groovy", "return Helper.greet() + ' ' + Helper.Greeting.text()\n");
        assertThat(configuration.evaluate("op.groovy", new Binding(), null)).isEqualTo("hello hello");

        final String text = "def a = Helper.greet()\n"
                + "if (false) { Trigger.x() }\n"
                + "def b = Helper.Greeting.text()\n"
                + "return a + ' ' + b\n";
        triggerWaiting = new CountDownLatch(1);
        triggerGate = new CountDownLatch(1);
        final Future<Object> textRun = executor.submit(
                () -> runOnConnector(text, Collections.<String, Object>emptyMap()));
        assertThat(triggerWaiting.await(30, SECONDS)).isTrue();

        // The text has resolved Helper and is now looking up Trigger.
        editHelper("class Helper { static String greet() { 'hi' }\n"
                + "static class Greeting { static String text() { 'hi' } } }\n");
        final Future<Object> op = executor.submit(() -> configuration.evaluate("op.groovy", new Binding(), null));
        Object opResult;
        boolean opRanWhileTextCompiled;
        try {
            opResult = op.get(5, SECONDS);
            opRanWhileTextCompiled = true;
        } catch (TimeoutException e) {
            opResult = null;
            opRanWhileTextCompiled = false;
        }
        triggerGate.countDown();
        if (!opRanWhileTextCompiled) {
            opResult = op.get(30, SECONDS);
        }
        final Object result = outcome(() -> textRun.get(30, SECONDS));
        System.out.println("REVIEW-F2 op ran while text compiled: " + opRanWhileTextCompiled
                + ", op: " + opResult + ", text: " + result);
        assertThat(result).isIn("hello hello", "hi hi");
    }

    /** F3: what a subclass adds to the engine's configuration in getRoots. */
    @Test
    public void f3GetRootsConfigurationReachesTexts() throws Exception {
        configuration.release();
        configuration = withScriptRoot(new ScriptedConfiguration() {
            @Override
            protected URL[] getRoots(CompilerConfiguration compilerConfiguration, GroovyClassLoader loader) {
                compilerConfiguration.addCompilationCustomizers(
                        new ImportCustomizer().addImport("RootsList", ArrayList.class.getName()));
                final SecureASTCustomizer secure = new SecureASTCustomizer();
                secure.setMethodDefinitionAllowed(false);
                compilerConfiguration.addCompilationCustomizers(secure);
                return super.getRoots(compilerConfiguration, loader);
            }
        });
        write("op.groovy", "return new RootsList()\n");
        final Object op = outcome(() -> configuration.evaluate("op.groovy", new Binding(), null));
        final Object imports = outcome(() -> runOnConnector("return new RootsList()",
                Collections.<String, Object>emptyMap()));
        final Object secured = outcome(() -> runOnConnector("def f() { 1 }\nreturn f()",
                Collections.<String, Object>emptyMap()));
        System.out.println("REVIEW-F3 op: " + op + " | text import: " + imports + " | text method def: " + secured);
        assertThat(imports).isInstanceOf(ArrayList.class);
        assertThat(String.valueOf(secured)).startsWith("THREW");
    }

    /** F4: a URL a text adds to its class loader. */
    @Test
    public void f4UrlAddedByATextReachesTheEngine() throws Exception {
        final File lib = Files.createTempDirectory("review-196-lib").toFile();
        try {
            final CompilerConfiguration libConfig = new CompilerConfiguration();
            libConfig.setTargetDirectory(lib);
            final CompilationUnit unit = new CompilationUnit(libConfig);
            unit.addSource("Driver.groovy", "package drv\nclass Driver { static String id() { 'drv' } }\n");
            unit.compile();
            final URL libUrl = lib.toURI().toURL();

            runOnConnector("this.class.classLoader.addURL(new URL('" + libUrl + "')); return null",
                    Collections.<String, Object>emptyMap());
            final boolean inEngine = Arrays.asList(
                    configuration.getGroovyScriptEngine().getGroovyClassLoader().getURLs()).contains(libUrl);
            write("op.groovy", "return this.class.classLoader.loadClass('drv.Driver').id()\n");
            final Object op = outcome(() -> configuration.evaluate("op.groovy", new Binding(), null));
            final Object otherText = outcome(() -> runOnConnector(
                    "return this.class.classLoader.loadClass('drv.Driver').id()",
                    Collections.<String, Object>emptyMap()));
            System.out.println("REVIEW-F4 URL in engine loader: " + inEngine + " | op: " + op
                    + " | other text: " + otherText);
            assertThat(inEngine).isTrue();
        } finally {
            delete(lib);
        }
    }

    private static URL blockingRoot() {
        try {
            return new URL(null, "blocking:/", new URLStreamHandler() {
                @Override
                protected URLConnection openConnection(final URL u) {
                    return new URLConnection(u) {
                        @Override
                        public void connect() throws IOException {
                            throw new FileNotFoundException(u.toString());
                        }

                        @Override
                        public InputStream getInputStream() throws IOException {
                            final CountDownLatch gate = triggerGate;
                            if ("/Trigger.groovy".equals(u.getPath()) && null != gate) {
                                triggerWaiting.countDown();
                                try {
                                    gate.await();
                                } catch (InterruptedException e) {
                                    Thread.currentThread().interrupt();
                                }
                            }
                            throw new FileNotFoundException(u.toString());
                        }
                    };
                }
            });
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }
    }

    private static Object outcome(Callable<Object> call) {
        try {
            return call.call();
        } catch (Throwable t) {
            Throwable e = t instanceof ExecutionException && null != t.getCause() ? t.getCause() : t;
            final StringBuilder chain = new StringBuilder("THREW ");
            for (int i = 0; null != e && i < 4; e = e.getCause(), i++) {
                chain.append(i > 0 ? " <- " : "").append(e.getClass().getName()).append(": ")
                        .append(String.valueOf(e.getMessage()).replace('\n', ' '));
            }
            return chain.toString();
        }
    }

    private Object runOnConnector(String text, Map<String, Object> arguments) {
        final ScriptedPoolableConnector connector = new ScriptedPoolableConnector();
        connector.init(configuration);
        return connector.runScriptOnConnector(new ScriptContext("Groovy", text, arguments), null);
    }

    private static Map<String, Object> useHelper(boolean useHelper) {
        return Collections.<String, Object>singletonMap("useHelper", useHelper);
    }

    private File write(String fileName, String text) throws IOException {
        final File file = new File(scriptRoot, fileName);
        file.getParentFile().mkdirs();
        Files.write(file.toPath(), text.getBytes(StandardCharsets.UTF_8));
        return file;
    }

    private void writeHelper(String text) throws IOException {
        final File helper = write("Helper.groovy", text);
        assertThat(helper.setLastModified(System.currentTimeMillis() - 60000L)).isTrue();
    }

    private void editHelper(String text) throws Exception {
        Thread.sleep(200);
        write("Helper.groovy", text);
        Thread.sleep(200);
    }

    private ScriptedConfiguration withScriptRoot(ScriptedConfiguration result) {
        result.setScriptRoots(new String[] { scriptRoot.getAbsolutePath() });
        return result;
    }

    private static void delete(File file) {
        final File[] children = file.listFiles();
        if (null != children) {
            for (File child : children) {
                delete(child);
            }
        }
        file.delete();
    }
}

…he engine's compiler lock with its configuration, and link the classes compiled with a script-root class
@maximthomas

Copy link
Copy Markdown
Contributor Author

Your test class reproduced all four checks on 37b50d1 and passes on master, and point 5 matches the 2.4.21 sources. All five are fixed in 21f99b4. Your class, unchanged, now passes 5/5 on this branch.

1. Types reached only through a signature. ScriptOnConnectorLoader now also keeps the loaders of the script-root classes the compiler resolved. For a name it has not resolved, loadClass returns the class that one of those loaders defined: a nested class, a closure, or another class compiled from the same source. Helper$Greeting, reached only through Helper.make(), therefore links to the copy that Helper v1 was compiled with. Your F1 returns hello with and without @CompileStatic. testCachedStaticScriptLinksTheClassesCompiledWithAScriptRootClass reaches a nested class and a second top-level class of Helper.groovy, both only through return types. At 37b50d1 it fails with your LinkageError. A mutant that looks only at nested names ($) fails on the top-level class.

2. Two versions in one compilation. A text now compiles under the engine's compiler lock, as it did on master through the engine's loader. That lock is the monitor of GroovyClassLoader.sourceCache of the engine's loader, which GroovyScriptEngine's loader holds for every compilation. The field is protected and the loader is a private class of GroovyScriptEngine, so it is read by reflection. 2.2.1, 2.4.21, 4.0.22 and 5.0.3 all have the field and lock it in ScriptClassLoader.parseClass. If it cannot be read, texts are not cached and compile through the engine's loader, as on master. Your F2 now gives the master result: the operation script waits, and the text returns hello hello. The nested lookup is also served by Helper v1's loader since the change for point 1, so F2 is covered twice.

The lock also removes scriptRootLock, and with it the first-load race from the behaviour changes. testTextAndOperationScriptThatLoadAScriptRootClassTogetherShareIt is red at 37b50d1 with two copies of Helper. Texts compile one at a time again, as on master. A cached text still takes no lock.

3. What getRoots adds to the configuration. Now that all compilations are serialised, texts compile with the engine's own configuration again. The ImportCustomizer the engine and the texts share is safe for the same reason as on master. Your F3 gives [] and the SecurityException. getImportCustomizer and getImports are again called once per engine, and their Javadoc changes are reverted. Pinned by testScriptCompilesWithWhatASubclassAddsToTheEngineConfiguration.

4. addURL. ScriptOnConnectorLoader.addURL passes the URL to the engine's loader, and addClasspath goes through it. In F4 the URL is now in the engine's loader, and the operation script and the second text both return drv. Pinned by testUrlThatATextAddsToItsLoaderReachesTheOperationScripts.

5. The ScriptException branch. It now throws a CompilationFailedException with the unit's current phase and the ScriptException as its cause. testScriptExceptionFromTheEngineFailsAsACompilationError uses a subclass whose engine throws it. At 37b50d1 it fails with BUG! exception in phase 'semantic analysis'.

The description is updated: the bullets on the configuration and the lock, the behaviour changes, the test-strength rows and the counts.

@maximthomas
maximthomas requested a review from vharseko October 20, 2026 10:39

@vharseko vharseko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I re-ran my test class on 21f99b4, and all five fixes hold. F1 returns hello with and without @CompileStatic. In F2 the operation script waits for the text's compilation, and the text returns hello hello. F3 gives [] and the SecurityException. In F4 the URL is in the engine's loader, and both scripts return drv. mvn package -pl OpenICF-groovy-connector -am on JDK 21 gives Tests run: 182, Failures: 0, Errors: 0, Skipped: 42: the module's 177 tests plus my five. In the Groovy 2.4.21 sources, ScriptClassLoader.parseClass holds sourceCache for every engine compilation, and loadScriptByName gets there through groovyLoader.parseClass, so the lock you take is the one the engine uses.

One leftover from point 4. Point 4 said that a text's InnerLoader delegates addURL, addClasspath and parseClass to the loader that compiled it. addURL and addClasspath now reach the engine. parseClass still ends at ScriptOnConnectorLoader (ScriptedConfiguration.java:1082), which inherits GroovyClassLoader.parseClass. So a class that a text compiles at run time with this.class.classLoader.parseClass(...) is compiled:

  • without the engine's compiler lock, so it can run at the same time as an engine compilation that uses the same CompilerConfiguration. The lock is now there to prevent exactly that.
  • into the text's own class cache, so the operation scripts do not see it. This is close to the behaviour change you already list for classes declared in a text.

Repro (same package, using only API that both branches have): a script root has a URL handler that holds the lookup of Trigger.groovy. A text returns this.class.classLoader. While the compilation of op.groovy (if (false) { Trigger.x() }) is held, another thread calls parseClass("class Parsed { static String id() { 'parsed' } }") on that loader. Then op2.groovy runs return Parsed.id().

  • this branch: parseClass returns while the engine is still compiling, and op2 fails with MissingPropertyException: No such property: Parsed for class: op2.
  • master (4648e38): parseClass waits for the engine's compilation, and op2 returns parsed.

I have not reproduced a compilation failure caused by the missing lock. I don't expect the import race from testScriptsCompileTogetherOnANewEngine to come back this way. The text that calls parseClass was itself compiled with the shared configuration, and that compilation already resolved the class nodes of the import customizer. The remaining risk is a customizer that a subclass adds in getRoots and that keeps state between calls. Still, "all compilations are serialised" does not hold for this path.

One way to close it: override parseClass(GroovyCodeSource, boolean) in ScriptOnConnectorLoader so that it passes the call to the engine's loader, as master did. getScriptOnConnectorClass would then compile the text through a separate method that calls super.parseClass under the lock. Classes that a text compiles this way would then live in the engine, as on master. If you would rather keep them in the text's loader, it is enough to take the engine's lock in the override, plus a line under behaviour changes. I'd like this closed before merge, and either option works for me. Apart from this, I have nothing else.

… text parses at run time with the engine's loader

A text's InnerLoader delegates parseClass to the loader that compiled the
text. ScriptOnConnectorLoader now passes it on to the engine's loader, as
master did, so such a class compiles under the engine's compiler lock and
reaches the operation scripts. The text itself compiles through
compileText().
@maximthomas

Copy link
Copy Markdown
Contributor Author

Your repro holds on 21f99b4. I took your first option in 7b06bd4. ScriptOnConnectorLoader now overrides parseClass(GroovyCodeSource, boolean) and passes the call to the engine's loader, as master did. getScriptOnConnectorClass compiles the text through a private compileText, which calls super.parseClass under the engine's lock. The other parseClass methods of GroovyClassLoader all end in the overridden one. A class that a text parses at run time therefore compiles under the engine's compiler lock and goes into the engine's class cache. It resolves Helper as the operation scripts do, not as the cached text does.

Your repro is now two tests in ScriptOnConnectorCacheTest:

  • testClassThatATextParsesReachesTheOperationScripts: on 21f99b4, op.groovy fails with MissingPropertyException: No such property: Parsed for class: op. It now returns parsed.
  • testClassThatATextParsesWaitsForTheEngineCompilation: on 21f99b4, parseClass returns while the compilation of an operation script is held. It now waits for that compilation.

A mutant of your second option, where the override takes the engine's lock but compiles into the text's loader, fails the first test. A mutant that compiles the text itself through the overridden parseClass fails nine tests, testScriptClassStaysOutOfTheEngineLoader among them.

I also went through the rest of what GroovyClassLoader.InnerLoader passes to its parent in 2.4.21. loadClass stays with the text's loader by design. addURL, addClasspath and parseClass now reach the engine. getResource and getResourceAsStream give the same answers as on master, because the text's loader has no URLs of its own and asks the engine's loader first. getURLs, findResource, findResources, getResourceLoader, setResourceLoader, getLoadedClasses and clearCache still answer for the text's own loader. I have kept them that way: unregister() relies on the text loader's own getLoadedClasses, and on master a text's setResourceLoader or clearCache changed the engine's loader for every script. They are now listed under behaviour changes.

mvn -pl OpenICF-groovy-connector,OpenICF-ssh-connector verify on JDK 26 gives groovy Tests run: 179, Failures: 0, Errors: 0, Skipped: 42 (+2) and ssh Tests run: 135, Failures: 0, Errors: 0, Skipped: 120. Merged into master 4648e38, the groovy module runs Tests run: 210, Failures: 0, Errors: 0, Skipped: 42.

The description changes in four places: the bullet on what a text adds or compiles through its loader, a new line under behaviour changes, two new rows in the test-strength table, and the test counts.

@maximthomas
maximthomas requested a review from vharseko October 20, 2026 11:20

@vharseko vharseko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, 7b06bd4 closes it. My follow-up test now gives the master result on this branch: parseClass through the text's loader waits for the engine's compilation, and op2 returns parsed. The five checks from the first round still hold. mvn package -pl OpenICF-groovy-connector -am on JDK 21 gives Tests run: 185, Failures: 0, Errors: 0, Skipped: 42: the module's 179 tests plus my six.

In Groovy 2.4.21 the other parseClass overloads, and recompile, all end in the overridden parseClass(GroovyCodeSource, boolean). compileText takes the text loader's own sourceCache inside the engine's lock, and the run-time path through recompile takes them the other way round. The two cannot meet: compileText runs only before cache.put publishes the loader, so no other thread can hold its sourceCache at that point. Approving.

@maximthomas
maximthomas merged commit 95c8375 into OpenIdentityPlatform:master Oct 20, 2026
15 checks passed
@maximthomas
maximthomas deleted the issues/165-run-script-on-connector-cache branch October 20, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working concurrency Races, locking and thread-safety fixes connector:groovy Groovy connector java Pull requests that update java code performance Performance and scalability fixes resource-leak Memory, class-loader, thread or handle leaks tests Test additions or fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Groovy connectors: runScriptOnConnector recompiles the script on every call and keeps every compiled class until the configuration is released

2 participants