Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ dependencies {
implementation libs.jing // RELAX NG validation for XML::LibXML
implementation libs.snakeyaml.engine // YAML processing
implementation libs.tomlj // TOML processing
implementation libs.zxing.core // QR encoding for Text::QRCode
implementation libs.commons.csv // CSV processing
implementation libs.commonmark // CommonMark rendering
implementation libs.commonmark.autolink // GFM autolinks
Expand Down
29 changes: 26 additions & 3 deletions dev/design/jcpan-compiler-tooling-followup.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ The following are not treated as PerlOnJava regressions because their current di

## Progress Tracking

### Current Status: implementation complete; PR ready for review
### Current Status: unified into PR #964; combined CI passing

### Completed Phases

Expand All @@ -59,11 +59,34 @@ The following are not treated as PerlOnJava regressions because their current di
- Opened PR #962 from `fix/jcpan-compiler-tooling-followup`.
- GitHub Actions passed on Ubuntu and Windows; the Windows run specifically
confirmed the platform-default newline fix in `pipe_jperl_shebang.t`.
- [x] Phase 7: post-merge warning cleanup (2026-08-15)
- Replaced deprecated `Zstd.decompressedSize` calls with
`Zstd.getFrameContentSize` and reject unknown or invalid frame sizes before
allocating decoder buffers.
- [x] Phase 8: core-suite regression audit (2026-08-15)
- Compared the reported core files against isolated current-master and
historical baseline builds instead of treating aggregate TAP counts as
deterministic.
- Fixed PVLV filehandle handling so `-t` follows the glob's IO slot without
stringification and `close` warnings retain ASCII and Unicode glob names.
- Restored `op/gv.t` from 253/304 to 255/304 and `uni/gv.t` from 176/206 to
178/206. `re/pat_advanced.t`, `re/pat_advanced_thr.t`, and
`test_pl/examples.t` reproduce their higher reported counts. The remaining
`japh/abigail.t` 109/130 result matches both current master and the
documented historical baseline, so it is not a PR #963 regression.
- Added a four-case system-Perl oracle and verified it with both PerlOnJava
backends; the full `make` suite passes.
- PR #963 CI passed on Ubuntu and Windows at commit `b48f504d8`.
- [x] Phase 9: unified review branch (2026-08-15)
- Merged all five PR #963 commits into PR #964 so the compiler, runtime,
CPAN-tooling, and Java-module changes can be tested and approved together.
- The combined full `make` suite passed before the unified branch was pushed.
- Unified PR #964 CI passed on Ubuntu in 14m35s and Windows in 17m27s.

### Next Steps

1. Review PR #962.
2. Merge after approval.
1. Mark PR #963 as superseded by PR #964.
2. Hand PR #964 back for user testing and review.

### Open Questions

Expand Down
66 changes: 66 additions & 0 deletions dev/design/jcpan-compiler-tooling-modules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# jcpan compiler and tooling modules

## Goal

Fix reusable compiler and CPAN-tooling blockers exposed by Pegex::JSON,
Music::Factory, Bio::Data::Plasmid::CloningVector,
Catalyst::Engine::HTTP::POE::YieldCC, App::Chained, Queue, and their
dependencies. Prefer shared runtime/tooling fixes and existing Java libraries
over distribution preferences.

## Progress Tracking

### Current Status: PR #963 commits unified into PR #964; combined CI passing

### Completed Phases

- [x] Phase 1: baseline and system-Perl classification (2026-08-15)
- Captured bounded `jcpan -t` logs for all six requested modules.
- Identified malformed Bio-MCPrimers packaging and Catalyst's omitted
Restarter::Watcher as upstream distribution failures.
- [x] Phase 2: shared root-cause implementation (2026-08-15)
- Restored YAML::PP's standard object `dump` API.
- Serialized scalar references through their referents for boolean.pm parity.
- Made gzip EOF status compatible with CPAN single-file extraction.
- Extended generic missing-prerequisite discovery to TAP diagnostics.
- Routed Object::Pad's core syntax to PerlOnJava's native class compiler.
- [x] Phase 3: cross-runtime regression validation (2026-08-15)
- Validated YAML, gzip, native-class, and CPAN-tooling regressions with
system Perl where applicable.
- Passed the focused regressions on JVM and interpreter backends.
- [x] Phase 4: requested module verification (2026-08-15)
- Pegex::JSON: 4 files, 21 assertions, PASS.
- Music::Factory: 5 files, 20 assertions, PASS.
- App::Chained: 2 files, 9 assertions, PASS after generic dependency retry.
- Queue: single-file distribution built and tested successfully; upstream
ships no test directory.
- Bio::Data::Plasmid::CloningVector excluded because Bio-MCPrimers has no
Makefile.PL and fails system-Perl configuration.
- Catalyst::Engine::HTTP::POE::YieldCC excluded because its distribution
requires but does not ship or declare Restarter::Watcher; system Perl
reproduces the missing-module failure.
- [x] Phase 5: full verification (2026-08-15)
- Full `make` passed all unit shards.
- [x] Phase 6: pull request and CI (2026-08-15)
- Opened [PR #964](https://github.com/fglock/PerlOnJava/pull/964).
- Ubuntu CI passed in 11m20s.
- Windows CI passed in 18m10s.
- [x] Phase 7: unified review branch (2026-08-15)
- Merged all five commits from PR #963 into PR #964 for joint testing and
approval.
- The combined full `make` suite passed before push.
- Unified PR #964 CI passed on Ubuntu in 14m35s and Windows in 17m27s.

### Next Steps

1. Mark PR #963 as superseded by PR #964.
2. Await joint testing and review of PR #964.

### Open Questions

- Object::Pad-specific MOP and extension APIs remain outside the native class
compatibility pragma; the requested Music::Factory surface uses core syntax.

## References

- Skills: `debug-perlonjava`, `port-cpan-module`
5 changes: 5 additions & 0 deletions docs/about/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ Release history of PerlOnJava. See [Roadmap](roadmap.md) for future plans.

## Work in progress

- CPAN/compiler tooling: restore YAML::PP's object `dump` API, recognize
TAP-indented missing prerequisites, report gzip stream completion for CPAN
single-file distributions, and route Object::Pad's core syntax through the
native class compiler. This unblocks Pegex::JSON, Music::Factory,
App::Chained, and Queue without distribution preferences.
- CPAN/compiler tooling: add transitive prerequisites to the bundled-provider
manifest, provide a JAXP-backed `XML::LibXSLT`, and preserve descriptors for
anonymous handles stored in container lvalues. This unblocks
Expand Down
1 change: 1 addition & 0 deletions docs/reference/bundled-modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,7 @@ These are loaded automatically or via `use`:

| Module | Implementation | Notes |
|--------|---------------|-------|
| `Object::Pad` | Perl/compiler | Core `class`, `field`, `method`, `:param`, and `:isa` syntax uses PerlOnJava's native class compiler; Object::Pad MOP extensions are not included |
| `Scalar::Util` | Java | |
| `Sub::Name` | Java | |
| `Sub::Util` | Java | |
Expand Down
3 changes: 3 additions & 0 deletions docs/reference/feature-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -769,6 +769,9 @@ The `:encoding()` layer supports all encodings provided by Java's `Charset.forNa
- ❌ **Safe** module.

### Non-core modules
- 🟡 **Object::Pad**: core class, field, method, parameter, and inheritance
syntax is handled by PerlOnJava's native class compiler; Object::Pad-specific
MOP extensions are not implemented.
- ✅ **JSON::DWIW**: relaxed JSON conversion implemented over the bundled
pure-Perl `JSON::PP` backend.
- ✅ **Taint::Runtime**: Java XS replacement for runtime taint toggling and
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/xs-compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ These modules have optimized Java implementations built into PerlOnJava:
| Exporter::Lexical | ExporterLexical.java | 0.02 | Installs lexical subs into the enclosing compile-time scope |
| HTML::Content::Extractor | HTMLContentExtractor.java | 0.17 | Uses jsoup's HTML5 parser with a small legacy tree-compatibility layer |
| Crypt::Twofish2 | CryptTwofish2.java | 1.03 | Uses BouncyCastle Twofish; ECB, stateful zero-IV CBC, and CFB1 |
| Math::Cephes | MathCephes.java | 0.5308 | Implements the normal and chi-square distribution functions used by CPAN statistics modules |
| Text::QRCode | TextQRCode.java | 0.05 | Uses ZXing in forced byte mode with libqrencode-compatible mask selection |

## Modules with PP Fallbacks or Shims

Expand Down
2 changes: 2 additions & 0 deletions gradle/libs.versions.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ snappy-java = "1.1.10.8"
sqlite-jdbc = "3.53.2.1"
tomlj = "1.1.1"
zstd-jni = "1.5.7-8"
zxing = "3.5.4"

[libraries]
asm = { module = "org.ow2.asm:asm", version.ref = "asm" }
Expand All @@ -37,6 +38,7 @@ snappy-java = { module = "org.xerial.snappy:snappy-java", version.ref = "snappy-
sqlite-jdbc = { module = "org.xerial:sqlite-jdbc", version.ref = "sqlite-jdbc" }
tomlj = { module = "org.tomlj:tomlj", version.ref = "tomlj" }
zstd-jni = { module = "com.github.luben:zstd-jni", version.ref = "zstd-jni" }
zxing-core = { module = "com.google.zxing:core", version.ref = "zxing" }

[plugins]
cyclonedx = "org.cyclonedx.bom:2.3.0"
Expand Down
2 changes: 1 addition & 1 deletion jcpan
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ export JPERL_ORPHAN_EXIT=1
# CPAN build tools may install copies of their own implementation modules into
# the user library. Keep PerlOnJava's narrow compatibility overlays ahead of
# those copies while jcpan and its child build processes run.
export PERLONJAVA_PREFER_BUNDLED_MODULES="Module/Build/Base.pm${PERLONJAVA_PREFER_BUNDLED_MODULES:+,$PERLONJAVA_PREFER_BUNDLED_MODULES}"
export PERLONJAVA_PREFER_BUNDLED_MODULES="Module/Build/Base.pm,Object/Pad.pm${PERLONJAVA_PREFER_BUNDLED_MODULES:+,$PERLONJAVA_PREFER_BUNDLED_MODULES}"

# CPAN test suites should run with deterministic semantics. User-interface
# color preferences such as NO_COLOR can change module behavior under test
Expand Down
8 changes: 7 additions & 1 deletion src/main/java/com/booking/sereal/Decoder.java
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,13 @@ private void uncompressZstd() throws SerealException {
int len = (int) read_varint();

byte[] compressedData = Arrays.copyOfRange(originalData.array, position, position + len);
long decompressedSize = Zstd.decompressedSize(compressedData);
long decompressedSize = Zstd.getFrameContentSize(compressedData);
if (Zstd.isError(decompressedSize)) {
String message = decompressedSize == -1
? "Zstd frame content size is unknown"
: Zstd.getErrorName(decompressedSize);
throw new SerealException(message);
}

if (decompressedSize > this.maxSize) {
throw new SerealException("The expected uncompressed size is larger than the allowed maximum size");
Expand Down
8 changes: 7 additions & 1 deletion src/main/java/com/booking/sereal/TokenDecoder.java
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,13 @@ private void uncompressZstd() throws SerealException {
int len = (int) readVarint();

byte[] compressedData = Arrays.copyOfRange(originalData.array, position, position + len);
long decompressedSize = Zstd.decompressedSize(compressedData);
long decompressedSize = Zstd.getFrameContentSize(compressedData);
if (Zstd.isError(decompressedSize)) {
String message = decompressedSize == -1
? "Zstd frame content size is unknown"
: Zstd.getErrorName(decompressedSize);
throw new SerealException(message);
}
if (decompressedSize > Integer.MAX_VALUE) {
throw new SerealException("Decompressed size exceeds integer MAX_VALUE: " + decompressedSize);
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
package com.google.zxing.qrcode.encoder;

import com.google.zxing.WriterException;
import com.google.zxing.common.BitArray;
import com.google.zxing.qrcode.decoder.ErrorCorrectionLevel;
import com.google.zxing.qrcode.decoder.Mode;
import com.google.zxing.qrcode.decoder.Version;

import java.nio.charset.Charset;

/** Accesses ZXing's package-level encoder primitives to force QR byte mode. */
public final class PerlOnJavaByteModeEncoder {
private PerlOnJavaByteModeEncoder() {
}

public static QRCode encode(String text, Charset charset, ErrorCorrectionLevel level,
int requestedVersion, int mask) throws WriterException {
BitArray data = new BitArray();
Encoder.append8BitBytes(text, data, charset);

Version version = requestedVersion > 0
? Version.getVersionForNumber(requestedVersion)
: smallestVersion(data, level);
BitArray headerAndData = new BitArray();
Encoder.appendModeInfo(Mode.BYTE, headerAndData);
Encoder.appendLengthInfo(data.getSizeInBytes(), version, Mode.BYTE, headerAndData);
headerAndData.appendBitArray(data);

Version.ECBlocks ecBlocks = version.getECBlocksForLevel(level);
int totalCodewords = version.getTotalCodewords();
int dataCodewords = totalCodewords - ecBlocks.getTotalECCodewords();
if (!Encoder.willFit(headerAndData.getSize(), version, level)) {
throw new WriterException("Data too big for requested version");
}
Encoder.terminateBits(dataCodewords, headerAndData);
BitArray finalBits = Encoder.interleaveWithECBytes(
headerAndData, totalCodewords, dataCodewords, ecBlocks.getNumBlocks());

ByteMatrix matrix = new ByteMatrix(
version.getDimensionForVersion(), version.getDimensionForVersion());
MatrixUtil.buildMatrix(finalBits, level, version, mask, matrix);
QRCode result = new QRCode();
result.setMode(Mode.BYTE);
result.setECLevel(level);
result.setVersion(version);
result.setMaskPattern(mask);
result.setMatrix(matrix);
return result;
}

private static Version smallestVersion(BitArray data, ErrorCorrectionLevel level)
throws WriterException {
for (int number = 1; number <= 40; number++) {
Version version = Version.getVersionForNumber(number);
int bits = 4 + Mode.BYTE.getCharacterCountBits(version) + data.getSize();
if (Encoder.willFit(bits, version, level)) return version;
}
throw new WriterException("Data too big");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,13 @@ public static Node parseStatement(Parser parser, String label) {
if (peek(parser).text.equals("(")) {
// Parse the signature properly to generate parameter declarations
// Pass true for isMethod flag to account for implicit $self in error messages
signatureAST = SignatureParser.parseSignature(parser, methodName, true);
boolean wasInMethod = parser.isInMethod;
parser.isInMethod = true;
try {
signatureAST = SignatureParser.parseSignature(parser, methodName, true);
} finally {
parser.isInMethod = wasInMethod;
}
// Note: SignatureParser consumes the closing )
}

Expand Down Expand Up @@ -617,7 +623,13 @@ && nextNonWhitespaceTokenIs(parser, parser.tokenIndex + 1, "sub")) {
ListNode signatureAST = null;
if (peek(parser).text.equals("(")) {
// Pass true for isMethod flag to account for implicit $self in error messages
signatureAST = SignatureParser.parseSignature(parser, methodName, true);
boolean wasInMethod = parser.isInMethod;
parser.isInMethod = true;
try {
signatureAST = SignatureParser.parseSignature(parser, methodName, true);
} finally {
parser.isInMethod = wasInMethod;
}
}

try {
Expand Down
45 changes: 45 additions & 0 deletions src/main/java/org/perlonjava/frontend/parser/Variable.java
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,7 @@ static Node parseArrayHashAccessInBraces(Parser parser, Node operand, boolean is
}
case "{" -> {
// Hash access
preprocessBackslashQuotesInInterpolatedHashAccess(parser, parser.tokenIndex);
operand = ParseInfix.parseInfixOperation(parser, operand, 0);
if (operand == null) {
throw new PerlCompilerException(parser.tokenIndex, "syntax error: Missing closing brace", parser.ctx.errorUtil);
Expand All @@ -534,6 +535,10 @@ static Node parseArrayHashAccessInBraces(Parser parser, Node operand, boolean is
switch (text) {
case "[", "{", "@*", "$*", "%*", "&*", "$#", "@", "%" -> {
// Dereference followed by access: $var->[0] or $var->{key}
if (text.equals("{")) {
preprocessBackslashQuotesInInterpolatedHashAccess(
parser, parser.tokenIndex);
}
parser.tokenIndex = previousIndex; // Re-parse "->"
operand = ParseInfix.parseInfixOperation(parser, operand, 0);
if (operand == null) {
Expand Down Expand Up @@ -565,6 +570,41 @@ static Node parseArrayHashAccessInBraces(Parser parser, Node operand, boolean is
return operand;
}

/**
* Inside a double-quoted string, quotes delimiting an interpolated hash key
* are escaped for the outer string: {@code "$ref->{\"key\"}"}. The string
* lexer preserves those backslashes, but the embedded expression parser must
* see an ordinary quoted key.
*/
private static void preprocessBackslashQuotesInInterpolatedHashAccess(
Parser parser, int openingBraceIndex) {
if (!parser.preprocessBracedBackslashQuotesInInterpolation) {
return;
}

int scan = openingBraceIndex;
int braceLevel = 0;
while (scan < parser.tokens.size()) {
String text = parser.tokens.get(scan).text;
if ("{".equals(text)) {
braceLevel++;
scan++;
} else if ("}".equals(text)) {
braceLevel--;
if (braceLevel == 0) {
return;
}
scan++;
} else if ("\\".equals(text)
&& scan + 1 < parser.tokens.size()
&& "\"".equals(parser.tokens.get(scan + 1).text)) {
parser.tokens.remove(scan);
} else {
scan++;
}
}
}

/**
* Parses array and hash access operations following a variable.
*
Expand Down Expand Up @@ -640,6 +680,7 @@ static Node parseArrayHashAccess(Parser parser, Node operand, boolean isRegex) {
}
case "{" -> {
// Hash access
preprocessBackslashQuotesInInterpolatedHashAccess(parser, parser.tokenIndex);
int savedIndex = parser.tokenIndex;
Node result = null;
try {
Expand All @@ -664,6 +705,10 @@ static Node parseArrayHashAccess(Parser parser, Node operand, boolean isRegex) {
switch (text) {
case "[", "{", "@*", "$*", "%*", "&*", "$#", "@", "%" -> {
// Dereference followed by access: $var->[0] or $var->{key}
if (text.equals("{")) {
preprocessBackslashQuotesInInterpolatedHashAccess(
parser, parser.tokenIndex);
}
parser.tokenIndex = previousIndex; // Re-parse "->"
Node result = ParseInfix.parseInfixOperation(parser, operand, 0);
if (result == null) {
Expand Down
Loading
Loading