feat(regex): execute embedded callbacks in vendored Joni - #965
Merged
Conversation
Import the official Joni 2.2.7 source snapshot from tag joni-2.2.7 at 57fd57b4f977813a7b4b35e0179943b1f06f51d7. Retain its source packages, tests, MIT license, copyright headers, and authorship notices unchanged except for the generic callout extension maintained here. Replace the binary Joni dependency with the vendored source and retain JCodings 1.0.64 as the upstream binary dependency. Its license was copied from tag jcodings-1.0.64 at 996ae0f72c5cc6bb28ef92f29bbd5ef0f63d5250. Add matcher-local callout handlers, provisional capture views, a callout opcode, and matcher-stack unwind tokens. Cleanup now covers backtracking, success, failure, handler exceptions, interruption, and timeout in reverse execution order. Matcher-local handlers keep shared compiled regexes safe for runtime-local callback state. Relocate Joni and JCodings into org.perlonjava.internal in the standalone JAR, package all upstream license and modification notices, and record the vendored component and JCodings relationship in the combined SBOM. Add the history-free design contract in docs/design/joni-callout-fork.md, the imported upstream Joni suite, focused callout tests, and an automated packaging verifier. Verification: full make passes, including the upstream Joni suite, focused callout tests, five PerlOnJava unit shards, standalone JAR relocation and notice checks, and SBOM validation. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
Implement phases 36.2 through 36.4 of the vendored Joni callout design.
Phase 36.2 represents literal (?{...}) bodies and callback conditions as
parser-created lexical closures. Structured regex templates preserve those
closures and callback tables across interpolation, qr embedding, JVM codegen,
and interpreter bytecode without exposing internal callout syntax to runtime
strings.
Phase 36.3 makes every structured executable callback select Joni, including
constant-bodied callbacks. The runtime bridge publishes provisional captures,
executes closures in scalar context, maintains $^R, checkpoints regex and
dynamic-local state, and restores state exactly once on backtracking, failure,
exceptions, or successful completion.
Phase 36.4 adds callback-selected conditional branches and callback-table ID
remapping for embedded regex values. Preserve literal CALL-like pattern text,
retain the compatibility no-op for eval groups introduced only as runtime
text, and keep dynamic (??{...}) patterns gated until nested alternatives can
participate in outer backtracking.
Stabilize literal match-target scalar identity for /g call sites and fix
undefining a typeglob scalar slot that aliases a read-only callback value while
preserving immediate DESTROY behavior for mutable referents.
Track runtime regex interpolation explicitly so compile-time transforms such as
\Q...\E keep eager qr validation. Preserve a lone interpolated value's runtime
type for qr overloading and already-compiled callback regexes.
Keep Perl source policy and Java-only rewrites in PerlOnJava rather than moving
the whole RegexPreprocessor into Joni. Move only matcher semantics incrementally
and use the missing-regex feature matrix as the Joni routing roadmap. Original
Joni copyright, authorship, license, and modification notices remain intact.
Verification:
- full make passes, including Joni tests, five unit shards, packaging, notices,
relocation, and SBOM checks
- executable_callbacks.t passes 18/18 under system Perl, the JVM backend, and
the interpreter backend
- perl5_t/t/re completes 80 files with zero timeouts and 50458/94827 passing
- versus test_20260815_080000_958.log: 17 files improve, 63 are unchanged,
none regress, and the net gain is 185 passing assertions
Generated with [Codex](https://openai.com/codex)
Co-Authored-By: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
org.perlonjava.internalin the standalone JAR(?{ ... })closures and callback conditions inside Joni with provisional captures,$^R, and exact backtracking unwindqr//values on both JVM and interpreter backends(??{ ... })folding and defer runtime-dependent dynamic patterns until nested alternatives can join outer backtrackingDesign decisions
The history-free architecture and acceptance contract are in
docs/design/joni-callout-fork.md.The complete
RegexPreprocessoris not moved into the fork. Perl source policy, security, diagnostics, and backend selection remain in PerlOnJava; matcher semantics move into Joni incrementally. The missing regex features indocs/reference/feature-matrix.mdare the Joni routing roadmap.Original Joni copyright, authorship, MIT license, tests, and modification notices are retained. Packaging checks verify namespace isolation, notices, and SBOM metadata.
Verification
make: passes after rebase ontoorigin/masterexecutable_callbacks.t: 18/18 under system Perl, JVM backend, and interpreter backendperl dev/tools/perl_test_runner.pl perl5_t/t/re/: 80 files, 94,827 assertions, zero outer timeouts, 50,458 passing in the full runCompared file-by-file with
../PerlOnJava/logs/test_20260815_080000_958.log:speed_thr.t: 26/59 on both this branch and cleanorigin/master; the full-run 25/59 observation was its internal watchdog under shared CPU contentionpat.tandpat_thr.t+32 each; advanced variants +5 each;rxcode.t+39; substitution variants +2 eachDeferred
(??{ ... })nested-pattern execution