From ffbfad176a3429d4d9657332c969389d90dca8dc Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:02:35 +0000 Subject: [PATCH 1/6] Stop packaging hadoop-client and avro-ipc in plugin jars Image scanning of gcr.io/cdapio/cdap reports jline 3.9.0 and jetty-http 9.4.51.v20230217 inside database-plugins-*.jar. dependency:tree shows both come from hadoop-client 3.3.6 (compile scope) -> hadoop-yarn-client 3.3.6 (jline 3.9.0, websocket-client -> jetty-client -> jetty-http/-util/-io 9.4.51). The main code only uses org.apache.hadoop.{conf,io,util,mapreduce,mapred.lib.db} classes, which come from hadoop-common / hadoop-mapreduce-client-core that CDAP provides at runtime, so hadoop-client is now provided. This also stops packaging the rest of the Hadoop client tree (httpclient, commons-io 2.8.0, commons-compress 1.21, reload4j, kerby, curator, jersey-client, jackson jaxrs 2.12.7, ...) in the database-plugins jar. The root-managed avro-mapred 1.11.4 pulled avro-ipc and avro-ipc-jetty, which brought velocity-engine-core 2.3 (with commons-io 2.8.0) into the format-avro jar (and through it format-parquet / google-cloud). No plugin uses Avro IPC, so both are excluded. Versions: - hadoop-client 3.3.6: compile -> provided (database-plugins/pom.xml) - avro-ipc, avro-ipc-jetty 1.11.4: excluded from avro-mapred (root dependencyManagement) Bugs: Plugins side done for jline b/578444907, b/578489244, b/578492108, b/578644778, b/578663138 and jetty-http b/578515513. Risk: low. No main-code change; Hadoop classes are supplied by CDAP. Downstream projects that relied on database-plugins pulling hadoop-client transitively at compile scope must declare it themselves (none in this repo). Testing: JDK 8, Checkstyle on (0 violations). mvn -DcloudBuild clean verify -pl database-plugins,format-avro,format-parquet -am: BUILD SUCCESS; hydrator-common, format-common, format-avro, format-parquet and database-plugins unit tests pass (database-plugins: 35 run, 1 skipped: DBConnectorTest assumes live-DB system properties). dependency:tree -Dincludes=org.jline,org.eclipse.jetty,org.apache.avro:avro-ipc*,org.apache.velocity shows jline/jetty only at provided/test scope and no avro-ipc or velocity. unzip -l database-plugins jar: lib/ now only has hydrator-common, guava, opentracing-* and common-0.1.4 (no jline, jetty, hadoop-*); format-avro jar has no avro-ipc / velocity. DB source/sink against real MySQL / Postgres not run (needs a database). --- database-plugins/pom.xml | 2 ++ pom.xml | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/database-plugins/pom.xml b/database-plugins/pom.xml index 90524b353..d4b9fcf89 100644 --- a/database-plugins/pom.xml +++ b/database-plugins/pom.xml @@ -35,6 +35,8 @@ org.apache.hadoop hadoop-client + + provided io.cdap.cdap diff --git a/pom.xml b/pom.xml index 594766c6c..12a3bfc27 100644 --- a/pom.xml +++ b/pom.xml @@ -624,6 +624,15 @@ avro-mapred ${avro.version} + + + org.apache.avro + avro-ipc + + + org.apache.avro + avro-ipc-jetty + org.mortbay.jetty jetty From bd562808683b9076f41857e48be726ce12878b62 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:23:06 +0000 Subject: [PATCH 2/6] Bump log4j2, bcprov, commons-io, json-smart and iq80 snappy Image scanning of gcr.io/cdapio/cdap reports CVEs in jars packaged by hydrator-plugins: log4j-core 2.20.0 / log4j-api 2.21.1 (format-xls, core-plugins via poi), bcprov-jdk15to18 1.84 (core-plugins), commons-io 2.5 (format-delimited), json-smart 2.2.1 (transform-plugins via json-path 2.2.0) and iq80 snappy 0.2 (format-orc via orc-core 1.1.0). The root POM now imports log4j-bom and manages commons-io, json-smart and iq80 snappy, so modules no longer pin their own versions. json-path stays on 2.2.0 (2.9 would pull json-smart 2.5.x). Versions: - log4j2 (log4j-core, log4j-api) 2.20.0 / 2.21.1 -> 2.25.5 (root log4j2.version, log4j-bom import; format-xls log4j-core.version removed) - bcprov-jdk15to18 1.84 -> 1.85 (core-plugins) - commons-io 2.5 / 2.15.0 -> 2.16.1 (root commons-io.version + managed entry; format-delimited property and core-plugins / format-xls pins removed) - json-smart 2.2.1 -> 2.4.10 (root json-smart.version, managed) - iq80 snappy 0.2 -> 0.5 (root iq80-snappy.version, managed) Bugs: Plugins side done for log4j-core b/578431300, b/578491647, b/578514361, b/578514875. iq80 snappy is rows only (b/578567520 stays open through the Platform leveldb exception). Risk: low-medium. Same-major bumps. log4j 2.25.x and commons-io 2.16.1 still run on Java 8. commons-compress (1.26.0 in core-plugins / format-xls) is not touched in this PR. Testing: JDK 8, Checkstyle on (0 violations). mvn -DcloudBuild clean verify -pl core-plugins,format-xls,format-delimited,transform-plugins,format-orc -am: BUILD SUCCESS; core-plugins 328 tests (2 skipped), transform-plugins 100 (1 skipped), format-xls / format-delimited / format-orc and dependencies all pass. dependency:tree -Dincludes=org.apache.logging.log4j,org.bouncycastle,commons-io,net.minidev:json-smart,org.iq80.snappy shows only log4j 2.25.5, bcprov-jdk15to18 1.85, commons-io 2.16.1, json-smart 2.4.10 and snappy 0.5 at compile scope (smbj's bcprov-jdk15on 1.70 is runtime scope and is not embedded). unzip -l of the built jars: core-plugins lib/ has bcprov-jdk15to18-1.85, commons-io-2.16.1, log4j-api-2.25.5; format-xls has log4j-core/-api 2.25.5, commons-io-2.16.1; format-delimited commons-io-2.16.1; transform-plugins json-smart-2.4.10; format-orc snappy-0.5. Excel / ORC / SFTP pipelines in a sandbox not run. --- core-plugins/pom.xml | 3 +-- format-delimited/pom.xml | 2 -- format-xls/pom.xml | 3 --- pom.xml | 28 ++++++++++++++++++++++++++++ 4 files changed, 29 insertions(+), 7 deletions(-) diff --git a/core-plugins/pom.xml b/core-plugins/pom.xml index 74e7d5c49..8ffed57ca 100644 --- a/core-plugins/pom.xml +++ b/core-plugins/pom.xml @@ -136,7 +136,7 @@ org.bouncycastle bcprov-jdk15to18 - 1.84 + 1.85 @@ -196,7 +196,6 @@ commons-io commons-io - 2.15.0 org.apache.commons diff --git a/format-delimited/pom.xml b/format-delimited/pom.xml index c17c45571..b35a556b8 100644 --- a/format-delimited/pom.xml +++ b/format-delimited/pom.xml @@ -30,7 +30,6 @@ jar - 2.5 3.1 @@ -78,7 +77,6 @@ commons-io commons-io - ${commons-io.version} diff --git a/format-xls/pom.xml b/format-xls/pom.xml index c3ef74b6f..8d40cf018 100644 --- a/format-xls/pom.xml +++ b/format-xls/pom.xml @@ -27,7 +27,6 @@ jar 5.2.5 - 2.20.0 @@ -45,7 +44,6 @@ org.apache.logging.log4j log4j-core compile - ${log4j-core.version} io.cdap.cdap @@ -71,7 +69,6 @@ commons-io commons-io - 2.15.0 org.apache.commons diff --git a/pom.xml b/pom.xml index 12a3bfc27..f7dc69b85 100644 --- a/pom.xml +++ b/pom.xml @@ -88,6 +88,7 @@ 2.0.2.2 1.10 3.1 + 2.16.1 2.6 3.5 3.2.2 @@ -101,6 +102,7 @@ 13.0.1 3.3.6 2.2.4 + 0.5 1.4.1 4.13.1 2.24.0 @@ -111,6 +113,8 @@ 3.4.5 2.2.0 20160212 + 2.4.10 + 2.25.5 4.1.75.Final 1.3.0 3.3.2 @@ -133,6 +137,30 @@ + + org.apache.logging.log4j + log4j-bom + ${log4j2.version} + pom + import + + + commons-io + commons-io + ${commons-io.version} + + + + net.minidev + json-smart + ${json-smart.version} + + + + org.iq80.snappy + snappy + ${iq80-snappy.version} + org.slf4j slf4j-api From f3d6233a2a68f52978dbf8cceba26119f56b861d Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:25:05 +0000 Subject: [PATCH 3/6] Bump org.json to 20260719 Image scanning of gcr.io/cdapio/cdap reports org.json:json 20160212 in transform-plugins-*.jar (3 CVEs). transform-plugins is the only module that uses ${json.version}; only XMLToJSON uses org.json (JSONObject, XML.toJSONObject). Kept separate from the other minor bumps because newer org.json changes behavior: numbers are parsed as BigDecimal / BigInteger where needed, and XML.toJSONObject keeps strings with leading zeros and some other values that the old version coerced. Versions: - org.json:json 20160212 -> 20260719 (root json.version) Bugs: none closed alone in hydrator-plugins (the 3 org.json bugs close once WR-3b and GC-3b also land). Risk: medium. The XMLToJSON output can differ for numeric-looking values (for example leading zeros and very large / high-precision numbers). Testing: JDK 8, Checkstyle on (0 violations). mvn -DcloudBuild clean verify -pl transform-plugins -am: BUILD SUCCESS; transform-plugins 100 tests pass (1 skipped, ValueMapperTest, as on the HP-2a run), including XMLToJSONConverterTest (4), XMLParserTest (8), XMLMultiParserTest (2), JSONParserTest (8), JSONFormatterTest (2). unzip -l transform-plugins jar: lib/json-20260719.jar (no json-20160212). A sandbox pipeline with XMLToJSON was not run. --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index f7dc69b85..1dbb14f79 100644 --- a/pom.xml +++ b/pom.xml @@ -112,7 +112,7 @@ 4.0.3 3.4.5 2.2.0 - 20160212 + 20260719 2.4.10 2.25.5 4.1.75.Final From d6115dec0cc2ff2c6f71a9d5a5b3c71290d03ab5 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:44:23 +0000 Subject: [PATCH 4/6] Import jackson-bom 2.18.12 Image scanning of gcr.io/cdapio/cdap reports jackson-databind 2.13.4.2 and jackson-core 2.13.4 / 2.14.3 in format-avro, format-parquet (also nested in google-cloud through format-avro) and spark-plugins jars. The root POM now imports com.fasterxml.jackson:jackson-bom so every jackson module (core, databind, annotations, module-scala for the Spark tests) resolves to one version; jackson.databind.version now points at the new jackson.version property, so existing module references keep working. Versions: - jackson (core, databind, annotations, modules) 2.13.4.2 / 2.14.3 -> 2.18.12 (root jackson.version + jackson-bom import) Bugs: none closed alone; the jackson rows still have a copy shaded in parquet-jackson 1.12.0 (HP-4) and in Platform / other plugins. Risk: medium. 2.15 added stream read limits (max string length 20M, nesting depth 1000, number length 1000) and 2.18 changed some property introspection. 2.18.x still runs on Java 8. The repo has no maven-shade-plugin usage, so the shade >= 3.5 rule does not apply. Testing: JDK 8, Checkstyle on (0 violations). mvn -DcloudBuild clean verify (whole reactor: 17 modules): BUILD SUCCESS; all unit tests pass (core-plugins 328 / 2 skipped, transform-plugins 100 / 1 skipped, database-plugins 35 / 1 skipped, spark-plugins, http-plugins, solrsearch-plugins, format-* modules). dependency:tree -Dincludes=com.fasterxml.jackson.core,com.fasterxml.jackson.module,com.fasterxml.jackson.dataformat shows only 2.18.12 at compile scope. unzip -l: format-avro, format-parquet and spark-plugins lib/ contain jackson-*-2.18.12 only (format-parquet still has parquet-jackson-1.12.0, fixed in HP-4). The submodules (google-cloud, kafka, azure, ...) are separate repos and are not built here (-DcloudBuild). --- pom.xml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 1dbb14f79..165459a08 100644 --- a/pom.xml +++ b/pom.xml @@ -118,7 +118,8 @@ 4.1.75.Final 1.3.0 3.3.2 - 2.13.4.2 + 2.18.12 + ${jackson.version} ${project.basedir}/src/test/java/ @@ -137,6 +138,13 @@ + + com.fasterxml.jackson + jackson-bom + ${jackson.version} + pom + import + org.apache.logging.log4j log4j-bom From bcc5ca22a85fbb2be4a82acc65779311e3c68a68 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 07:07:01 +0000 Subject: [PATCH 5/6] Bump parquet to 1.15.2, snappy-java and aircompressor Image scanning of gcr.io/cdapio/cdap reports parquet-avro 1.12.0 (1 Critical, 1 High) and the jackson-core 2.11.4 / jackson-databind 2.11.4 copy shaded inside parquet-jackson 1.12.0 in format-parquet-*.jar. parquet-avro and the other parquet modules move to 1.15.2 and parquet-jackson to 1.17.1 (it only holds the shaded jackson; 1.17.1 has a fixed copy). snappy-java (transform-plugins, root snappy.version) and aircompressor (parquet-hadoop) are aligned to the cross-track targets. Code: format-parquet ships its own copy of org.apache.parquet.avro.AvroSchemaConverter (to read INT96 as bytes). It was based on an old parquet release and shadows the 1.15.2 class, so AvroParquetWriter failed with NoSuchMethodError (AvroSchemaConverter(ParquetConfiguration)). The copy is re-based on the parquet-avro 1.15.2 source with the same INT96 change (INT96 -> Avro bytes unless parquet.avro.readInt96AsFixed is set). AvroIndexedRecordConverter / NoOpConverter are unchanged (1.12 -> 1.15 upstream changes to that class are formatting only). parquet-hadoop 1.15.2 declares parquet-jackson at runtime scope, which the bundle plugin does not embed (Embed-Dependency scope=compile), so format-parquet declares parquet-jackson explicitly to keep it in the plugin jar. Versions: - parquet-avro, -hadoop, -column, -common, -encoding, -format-structures 1.12.0 -> 1.15.2 (root parquet.version, managed) - parquet-jackson 1.12.0 -> 1.17.1 (root parquet-jackson.version, managed; explicit dependency in format-parquet) - snappy-java 1.1.2 -> 1.1.10.12 (root snappy.version) - aircompressor: new compile dependency of parquet-hadoop 1.15.2 (2.0.2) -> pinned to 2.0.3 (root aircompressor.version, managed; also replaces 0.21 from Spark's orc-core on the test classpath) Bugs: closes b/578421211 (Critical) and b/578599613 (High) alone; Plugins side done for the jackson bugs whose last hydrator copy was parquet-jackson 1.12.0. Risk: medium. New aircompressor 2.0.3, zstd-jni 1.5.6-6 and commons-pool 1.6 are now packaged (parquet-hadoop 1.15 compile deps). The managed parquet version also applies to the Spark 3.3.2 test classpath (parquet 1.12.x in Spark); all Spark-based unit tests pass. The plugin classloader loading parquet from the plugin jar on Spark 3.5.8 still needs the P2 sandbox check. All new jars are Java 8 bytecode (52). Testing: JDK 8, Checkstyle on (0 violations). mvn -DcloudBuild clean verify (whole reactor) passed for every module except format-parquet (NoSuchMethodError above, fixed in this commit); after the fix: mvn clean verify -pl format-parquet,core-plugins -am BUILD SUCCESS (format-parquet ParquetInputFormatProviderTest 3/3, core-plugins 328 / 2 skipped) and mvn clean verify -pl format-parquet -am BUILD SUCCESS with the explicit parquet-jackson. transform-plugins (100 / 1 skipped, CompressorTest / DecompressorTest with snappy), database, http, solr, spark plugins passed in the reactor run. dependency:tree -Dincludes=org.apache.parquet,io.airlift,org.xerial.snappy shows only parquet 1.15.2, parquet-jackson 1.17.1, aircompressor 2.0.3 and snappy-java 1.1.10.12 at compile scope. unzip -l format-parquet jar: lib/ has parquet-*-1.15.2, parquet-jackson-1.17.1, aircompressor-2.0.3, no parquet-*-1.12.0. --- format-parquet/pom.xml | 9 +- .../parquet/avro/AvroSchemaConverter.java | 556 +++++++++++------- pom.xml | 46 +- 3 files changed, 395 insertions(+), 216 deletions(-) diff --git a/format-parquet/pom.xml b/format-parquet/pom.xml index 3a1a08484..002e42361 100644 --- a/format-parquet/pom.xml +++ b/format-parquet/pom.xml @@ -79,7 +79,14 @@ org.apache.parquet parquet-avro - 1.12.0 + + + + org.apache.parquet + parquet-jackson com.google.guava diff --git a/format-parquet/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java b/format-parquet/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java index 62a3687af..105ce8b14 100644 --- a/format-parquet/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java +++ b/format-parquet/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java @@ -18,32 +18,16 @@ */ package org.apache.parquet.avro; -import org.apache.avro.LogicalType; -import org.apache.avro.LogicalTypes; -import org.apache.avro.Schema; - -import org.apache.hadoop.conf.Configuration; -import org.apache.parquet.schema.ConversionPatterns; -import org.apache.parquet.schema.GroupType; -import org.apache.parquet.schema.LogicalTypeAnnotation; -import org.apache.parquet.schema.MessageType; -import org.apache.parquet.schema.PrimitiveType; -import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName; -import org.apache.parquet.schema.Type; -import org.apache.parquet.schema.Types; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Optional; - import static java.util.Optional.empty; import static java.util.Optional.of; -import static org.apache.avro.JsonProperties.NULL_VALUE; +import static org.apache.parquet.avro.AvroReadSupport.READ_INT96_AS_FIXED; +import static org.apache.parquet.avro.AvroReadSupport.READ_INT96_AS_FIXED_DEFAULT; +import static org.apache.parquet.avro.AvroRecordConverter.getRuntimeAvroVersion; +import static org.apache.parquet.avro.AvroWriteSupport.WRITE_FIXED_AS_INT96; import static org.apache.parquet.avro.AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE; import static org.apache.parquet.avro.AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE_DEFAULT; +import static org.apache.parquet.avro.AvroWriteSupport.WRITE_PARQUET_UUID; +import static org.apache.parquet.avro.AvroWriteSupport.WRITE_PARQUET_UUID_DEFAULT; import static org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit.MICROS; import static org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit.MILLIS; import static org.apache.parquet.schema.LogicalTypeAnnotation.dateType; @@ -52,29 +36,64 @@ import static org.apache.parquet.schema.LogicalTypeAnnotation.stringType; import static org.apache.parquet.schema.LogicalTypeAnnotation.timeType; import static org.apache.parquet.schema.LogicalTypeAnnotation.timestampType; -import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*; +import static org.apache.parquet.schema.LogicalTypeAnnotation.uuidType; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BOOLEAN; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.DOUBLE; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.FLOAT; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT32; +import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64; import static org.apache.parquet.schema.Type.Repetition.REPEATED; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import org.apache.avro.LogicalType; +import org.apache.avro.LogicalTypes; +import org.apache.avro.Schema; +import org.apache.avro.SchemaBuilder; +import org.apache.hadoop.conf.Configuration; +import org.apache.parquet.conf.HadoopParquetConfiguration; +import org.apache.parquet.conf.ParquetConfiguration; +import org.apache.parquet.schema.ConversionPatterns; +import org.apache.parquet.schema.GroupType; +import org.apache.parquet.schema.LogicalTypeAnnotation; +import org.apache.parquet.schema.LogicalTypeAnnotation.UUIDLogicalTypeAnnotation; +import org.apache.parquet.schema.MessageType; +import org.apache.parquet.schema.PrimitiveType; +import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName; +import org.apache.parquet.schema.Type; +import org.apache.parquet.schema.Types; + /** *

* Converts an Avro schema into a Parquet schema, or vice versa. See package * documentation for details of the mapping. *

* - * A copy of the AvroSchemaConverter, except with handling for INT96. + * A copy of parquet-avro 1.15.2's AvroSchemaConverter, except with handling for INT96: INT96 columns + * are converted to Avro bytes instead of failing when READ_INT96_AS_FIXED is not set. */ public class AvroSchemaConverter { - public static final String ADD_LIST_ELEMENT_RECORDS = - "parquet.avro.add-list-element-records"; + public static final String ADD_LIST_ELEMENT_RECORDS = "parquet.avro.add-list-element-records"; private static final boolean ADD_LIST_ELEMENT_RECORDS_DEFAULT = true; private final boolean assumeRepeatedIsListElement; private final boolean writeOldListStructure; + private final boolean writeParquetUUID; + private final boolean readInt96AsFixed; + private final Set pathsToInt96; public AvroSchemaConverter() { - this.assumeRepeatedIsListElement = ADD_LIST_ELEMENT_RECORDS_DEFAULT; - this.writeOldListStructure = WRITE_OLD_LIST_STRUCTURE_DEFAULT; + this(ADD_LIST_ELEMENT_RECORDS_DEFAULT); } /** @@ -86,13 +105,21 @@ public AvroSchemaConverter() { AvroSchemaConverter(boolean assumeRepeatedIsListElement) { this.assumeRepeatedIsListElement = assumeRepeatedIsListElement; this.writeOldListStructure = WRITE_OLD_LIST_STRUCTURE_DEFAULT; + this.writeParquetUUID = WRITE_PARQUET_UUID_DEFAULT; + this.readInt96AsFixed = READ_INT96_AS_FIXED_DEFAULT; + this.pathsToInt96 = Collections.emptySet(); } public AvroSchemaConverter(Configuration conf) { - this.assumeRepeatedIsListElement = conf.getBoolean( - ADD_LIST_ELEMENT_RECORDS, ADD_LIST_ELEMENT_RECORDS_DEFAULT); - this.writeOldListStructure = conf.getBoolean( - WRITE_OLD_LIST_STRUCTURE, WRITE_OLD_LIST_STRUCTURE_DEFAULT); + this(new HadoopParquetConfiguration(conf)); + } + + public AvroSchemaConverter(ParquetConfiguration conf) { + this.assumeRepeatedIsListElement = conf.getBoolean(ADD_LIST_ELEMENT_RECORDS, ADD_LIST_ELEMENT_RECORDS_DEFAULT); + this.writeOldListStructure = conf.getBoolean(WRITE_OLD_LIST_STRUCTURE, WRITE_OLD_LIST_STRUCTURE_DEFAULT); + this.writeParquetUUID = conf.getBoolean(WRITE_PARQUET_UUID, WRITE_PARQUET_UUID_DEFAULT); + this.readInt96AsFixed = conf.getBoolean(READ_INT96_AS_FIXED, READ_INT96_AS_FIXED_DEFAULT); + this.pathsToInt96 = new HashSet<>(Arrays.asList(conf.getStrings(WRITE_FIXED_AS_INT96, new String[0]))); } /** @@ -125,28 +152,29 @@ public MessageType convert(Schema avroSchema) { if (!avroSchema.getType().equals(Schema.Type.RECORD)) { throw new IllegalArgumentException("Avro schema must be a record."); } - return new MessageType(avroSchema.getFullName(), convertFields(avroSchema.getFields())); + return new MessageType(avroSchema.getFullName(), convertFields(avroSchema.getFields(), "")); } - private List convertFields(List fields) { + private List convertFields(List fields, String schemaPath) { List types = new ArrayList(); for (Schema.Field field : fields) { if (field.schema().getType().equals(Schema.Type.NULL)) { continue; // Avro nulls are not encoded, unless they are null unions } - types.add(convertField(field)); + types.add(convertField(field, appendPath(schemaPath, field.name()))); } return types; } - private Type convertField(String fieldName, Schema schema) { - return convertField(fieldName, schema, Type.Repetition.REQUIRED); + private Type convertField(String fieldName, Schema schema, String schemaPath) { + return convertField(fieldName, schema, Type.Repetition.REQUIRED, schemaPath); } @SuppressWarnings("deprecation") - private Type convertField(String fieldName, Schema schema, Type.Repetition repetition) { + private Type convertField(String fieldName, Schema schema, Type.Repetition repetition, String schemaPath) { Types.PrimitiveBuilder builder; Schema.Type type = schema.getType(); + LogicalType logicalType = schema.getLogicalType(); if (type.equals(Schema.Type.BOOLEAN)) { builder = Types.primitive(BOOLEAN, repetition); } else if (type.equals(Schema.Type.INT)) { @@ -160,35 +188,50 @@ private Type convertField(String fieldName, Schema schema, Type.Repetition repet } else if (type.equals(Schema.Type.BYTES)) { builder = Types.primitive(BINARY, repetition); } else if (type.equals(Schema.Type.STRING)) { - builder = Types.primitive(BINARY, repetition).as(stringType()); + if (logicalType != null + && logicalType.getName().equals(LogicalTypes.uuid().getName()) + && writeParquetUUID) { + builder = Types.primitive(FIXED_LEN_BYTE_ARRAY, repetition) + .length(LogicalTypeAnnotation.UUIDLogicalTypeAnnotation.BYTES); + } else { + builder = Types.primitive(BINARY, repetition).as(stringType()); + } } else if (type.equals(Schema.Type.RECORD)) { - return new GroupType(repetition, fieldName, convertFields(schema.getFields())); + return new GroupType(repetition, fieldName, convertFields(schema.getFields(), schemaPath)); } else if (type.equals(Schema.Type.ENUM)) { builder = Types.primitive(BINARY, repetition).as(enumType()); } else if (type.equals(Schema.Type.ARRAY)) { if (writeOldListStructure) { - return ConversionPatterns.listType(repetition, fieldName, - convertField("array", schema.getElementType(), REPEATED)); + return ConversionPatterns.listType( + repetition, fieldName, convertField("array", schema.getElementType(), REPEATED, schemaPath)); } else { - return ConversionPatterns.listOfElements(repetition, fieldName, - convertField(AvroWriteSupport.LIST_ELEMENT_NAME, schema.getElementType())); + return ConversionPatterns.listOfElements( + repetition, + fieldName, + convertField(AvroWriteSupport.LIST_ELEMENT_NAME, schema.getElementType(), schemaPath)); } } else if (type.equals(Schema.Type.MAP)) { - Type valType = convertField("value", schema.getValueType()); + Type valType = convertField("value", schema.getValueType(), schemaPath); // avro map key type is always string return ConversionPatterns.stringKeyMapType(repetition, fieldName, valType); } else if (type.equals(Schema.Type.FIXED)) { - builder = Types.primitive(FIXED_LEN_BYTE_ARRAY, repetition) - .length(schema.getFixedSize()); + if (pathsToInt96.contains(schemaPath)) { + if (schema.getFixedSize() != 12) { + throw new IllegalArgumentException("The size of the fixed type field " + schemaPath + + " must be 12 bytes for INT96 conversion"); + } + builder = Types.primitive(PrimitiveTypeName.INT96, repetition); + } else { + builder = Types.primitive(FIXED_LEN_BYTE_ARRAY, repetition).length(schema.getFixedSize()); + } } else if (type.equals(Schema.Type.UNION)) { - return convertUnion(fieldName, schema, repetition); + return convertUnion(fieldName, schema, repetition, schemaPath); } else { throw new UnsupportedOperationException("Cannot convert Avro type " + type); } // schema translation can only be done for known logical types because this // creates an equivalence - LogicalType logicalType = schema.getLogicalType(); if (logicalType != null) { if (logicalType instanceof LogicalTypes.Decimal) { LogicalTypes.Decimal decimal = (LogicalTypes.Decimal) logicalType; @@ -204,7 +247,7 @@ private Type convertField(String fieldName, Schema schema, Type.Repetition repet return builder.named(fieldName); } - private Type convertUnion(String fieldName, Schema schema, Type.Repetition repetition) { + private Type convertUnion(String fieldName, Schema schema, Type.Repetition repetition, String schemaPath) { List nonNullSchemas = new ArrayList(schema.getTypes().size()); // Found any schemas in the union? Required for the edge case, where the union contains only a single type. boolean foundNullSchema = false; @@ -225,25 +268,27 @@ private Type convertUnion(String fieldName, Schema schema, Type.Repetition repet throw new UnsupportedOperationException("Cannot convert Avro union of only nulls"); case 1: - return foundNullSchema ? convertField(fieldName, nonNullSchemas.get(0), repetition) : - convertUnionToGroupType(fieldName, repetition, nonNullSchemas); + return foundNullSchema + ? convertField(fieldName, nonNullSchemas.get(0), repetition, schemaPath) + : convertUnionToGroupType(fieldName, repetition, nonNullSchemas, schemaPath); default: // complex union type - return convertUnionToGroupType(fieldName, repetition, nonNullSchemas); + return convertUnionToGroupType(fieldName, repetition, nonNullSchemas, schemaPath); } } - private Type convertUnionToGroupType(String fieldName, Type.Repetition repetition, List nonNullSchemas) { + private Type convertUnionToGroupType( + String fieldName, Type.Repetition repetition, List nonNullSchemas, String schemaPath) { List unionTypes = new ArrayList(nonNullSchemas.size()); int index = 0; for (Schema childSchema : nonNullSchemas) { - unionTypes.add( convertField("member" + index++, childSchema, Type.Repetition.OPTIONAL)); + unionTypes.add(convertField("member" + index++, childSchema, Type.Repetition.OPTIONAL, schemaPath)); } return new GroupType(repetition, fieldName, unionTypes); } - private Type convertField(Schema.Field field) { - return convertField(field.name(), field.schema()); + private Type convertField(Schema.Field field, String schemaPath) { + return convertField(field.name(), field.schema(), schemaPath); } public Schema convert(MessageType parquetSchema) { @@ -255,150 +300,177 @@ Schema convert(GroupType parquetSchema) { } private Schema convertFields(String name, List parquetFields, Map names) { - List fields = new ArrayList(); - Integer nameCount = names.merge(name, 1, (oldValue, value) -> oldValue + 1); + SchemaBuilder.FieldAssembler builder = + SchemaBuilder.builder(namespace(name, names)).record(name).fields(); for (Type parquetType : parquetFields) { Schema fieldSchema = convertField(parquetType, names); - if (parquetType.isRepetition(REPEATED)) { - throw new UnsupportedOperationException("REPEATED not supported outside LIST or MAP. Type: " + parquetType); + if (parquetType.isRepetition(REPEATED)) { // If a repeated field is ungrouped, treat as REQUIRED per spec + builder.name(parquetType.getName()) + .type() + .array() + .items() + .type(fieldSchema) + .arrayDefault(new ArrayList<>()); } else if (parquetType.isRepetition(Type.Repetition.OPTIONAL)) { - fields.add(new Schema.Field( - parquetType.getName(), optional(fieldSchema), null, NULL_VALUE)); + builder.name(parquetType.getName()).type().optional().type(fieldSchema); } else { // REQUIRED - fields.add(new Schema.Field( - parquetType.getName(), fieldSchema, null, (Object) null)); + builder.name(parquetType.getName()).type(fieldSchema).noDefault(); } } - Schema schema = Schema.createRecord(name, null, nameCount > 1 ? name + nameCount : null, false); - schema.setFields(fields); - return schema; + return builder.endRecord(); } private Schema convertField(final Type parquetType, Map names) { if (parquetType.isPrimitive()) { final PrimitiveType asPrimitive = parquetType.asPrimitiveType(); - final PrimitiveTypeName parquetPrimitiveTypeName = - asPrimitive.getPrimitiveTypeName(); + final PrimitiveTypeName parquetPrimitiveTypeName = asPrimitive.getPrimitiveTypeName(); final LogicalTypeAnnotation annotation = parquetType.getLogicalTypeAnnotation(); Schema schema = parquetPrimitiveTypeName.convert( - new PrimitiveType.PrimitiveTypeNameConverter() { - @Override - public Schema convertBOOLEAN(PrimitiveTypeName primitiveTypeName) { - return Schema.create(Schema.Type.BOOLEAN); - } - @Override - public Schema convertINT32(PrimitiveTypeName primitiveTypeName) { - return Schema.create(Schema.Type.INT); - } - @Override - public Schema convertINT64(PrimitiveTypeName primitiveTypeName) { - return Schema.create(Schema.Type.LONG); - } - @Override - public Schema convertINT96(PrimitiveTypeName primitiveTypeName) { - // this is the modified part - return Schema.create(Schema.Type.BYTES); - } - @Override - public Schema convertFLOAT(PrimitiveTypeName primitiveTypeName) { - return Schema.create(Schema.Type.FLOAT); - } - @Override - public Schema convertDOUBLE(PrimitiveTypeName primitiveTypeName) { - return Schema.create(Schema.Type.DOUBLE); - } - @Override - public Schema convertFIXED_LEN_BYTE_ARRAY(PrimitiveTypeName primitiveTypeName) { - int size = parquetType.asPrimitiveType().getTypeLength(); - return Schema.createFixed(parquetType.getName(), null, null, size); - } - @Override - public Schema convertBINARY(PrimitiveTypeName primitiveTypeName) { - if (annotation instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation || - annotation instanceof LogicalTypeAnnotation.EnumLogicalTypeAnnotation) { - return Schema.create(Schema.Type.STRING); - } else { + new PrimitiveType.PrimitiveTypeNameConverter() { + @Override + public Schema convertBOOLEAN(PrimitiveTypeName primitiveTypeName) { + return Schema.create(Schema.Type.BOOLEAN); + } + + @Override + public Schema convertINT32(PrimitiveTypeName primitiveTypeName) { + return Schema.create(Schema.Type.INT); + } + + @Override + public Schema convertINT64(PrimitiveTypeName primitiveTypeName) { + return Schema.create(Schema.Type.LONG); + } + + @Override + public Schema convertINT96(PrimitiveTypeName primitiveTypeName) { + if (readInt96AsFixed) { + return Schema.createFixed("INT96", "INT96 represented as byte[12]", null, 12); + } + // this is the modified part: CDAP reads INT96 as bytes (see AvroIndexedRecordConverter) return Schema.create(Schema.Type.BYTES); } - } - }); + + @Override + public Schema convertFLOAT(PrimitiveTypeName primitiveTypeName) { + return Schema.create(Schema.Type.FLOAT); + } + + @Override + public Schema convertDOUBLE(PrimitiveTypeName primitiveTypeName) { + return Schema.create(Schema.Type.DOUBLE); + } + + @Override + public Schema convertFIXED_LEN_BYTE_ARRAY(PrimitiveTypeName primitiveTypeName) { + if (annotation instanceof LogicalTypeAnnotation.UUIDLogicalTypeAnnotation) { + return Schema.create(Schema.Type.STRING); + } else { + int size = parquetType.asPrimitiveType().getTypeLength(); + String name = parquetType.getName(); + String ns = namespace(name, names); + return Schema.createFixed(name, null, ns, size); + } + } + + @Override + public Schema convertBINARY(PrimitiveTypeName primitiveTypeName) { + if (annotation instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation + || annotation instanceof LogicalTypeAnnotation.EnumLogicalTypeAnnotation) { + return Schema.create(Schema.Type.STRING); + } else { + return Schema.create(Schema.Type.BYTES); + } + } + }); LogicalType logicalType = convertLogicalType(annotation); - if (logicalType != null && (!(annotation instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) || - parquetPrimitiveTypeName == BINARY || - parquetPrimitiveTypeName == FIXED_LEN_BYTE_ARRAY)) { + if (logicalType != null + && (!(annotation instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) + || parquetPrimitiveTypeName == BINARY + || parquetPrimitiveTypeName == FIXED_LEN_BYTE_ARRAY)) { schema = logicalType.addToSchema(schema); } return schema; - } else { GroupType parquetGroupType = parquetType.asGroupType(); LogicalTypeAnnotation logicalTypeAnnotation = parquetGroupType.getLogicalTypeAnnotation(); if (logicalTypeAnnotation != null) { - return logicalTypeAnnotation.accept(new LogicalTypeAnnotation.LogicalTypeAnnotationVisitor() { - @Override - public Optional visit(LogicalTypeAnnotation.ListLogicalTypeAnnotation listLogicalType) { - if (parquetGroupType.getFieldCount()!= 1) { - throw new UnsupportedOperationException("Invalid list type " + parquetGroupType); - } - Type repeatedType = parquetGroupType.getType(0); - if (!repeatedType.isRepetition(REPEATED)) { - throw new UnsupportedOperationException("Invalid list type " + parquetGroupType); - } - if (isElementType(repeatedType, parquetGroupType.getName())) { - // repeated element types are always required - return of(Schema.createArray(convertField(repeatedType, names))); - } else { - Type elementType = repeatedType.asGroupType().getType(0); - if (elementType.isRepetition(Type.Repetition.OPTIONAL)) { - return of(Schema.createArray(optional(convertField(elementType, names)))); - } else { - return of(Schema.createArray(convertField(elementType, names))); + return logicalTypeAnnotation + .accept(new LogicalTypeAnnotation.LogicalTypeAnnotationVisitor() { + @Override + public Optional visit( + LogicalTypeAnnotation.ListLogicalTypeAnnotation listLogicalType) { + if (parquetGroupType.getFieldCount() != 1) { + throw new UnsupportedOperationException("Invalid list type " + parquetGroupType); + } + Type repeatedType = parquetGroupType.getType(0); + if (!repeatedType.isRepetition(REPEATED)) { + throw new UnsupportedOperationException("Invalid list type " + parquetGroupType); + } + if (isElementType(repeatedType, parquetGroupType.getName())) { + // repeated element types are always required + return of(Schema.createArray(convertField(repeatedType, names))); + } else { + Type elementType = + repeatedType.asGroupType().getType(0); + if (elementType.isRepetition(Type.Repetition.OPTIONAL)) { + return of(Schema.createArray(optional(convertField(elementType, names)))); + } else { + return of(Schema.createArray(convertField(elementType, names))); + } + } } - } - } - @Override - // for backward-compatibility - public Optional visit(LogicalTypeAnnotation.MapKeyValueTypeAnnotation mapKeyValueLogicalType) { - return visitMapOrMapKeyValue(); - } + @Override + // for backward-compatibility + public Optional visit( + LogicalTypeAnnotation.MapKeyValueTypeAnnotation mapKeyValueLogicalType) { + return visitMapOrMapKeyValue(); + } - @Override - public Optional visit(LogicalTypeAnnotation.MapLogicalTypeAnnotation mapLogicalType) { - return visitMapOrMapKeyValue(); - } + @Override + public Optional visit( + LogicalTypeAnnotation.MapLogicalTypeAnnotation mapLogicalType) { + return visitMapOrMapKeyValue(); + } - private Optional visitMapOrMapKeyValue() { - if (parquetGroupType.getFieldCount() != 1 || parquetGroupType.getType(0).isPrimitive()) { - throw new UnsupportedOperationException("Invalid map type " + parquetGroupType); - } - GroupType mapKeyValType = parquetGroupType.getType(0).asGroupType(); - if (!mapKeyValType.isRepetition(REPEATED) || - mapKeyValType.getFieldCount()!=2) { - throw new UnsupportedOperationException("Invalid map type " + parquetGroupType); - } - Type keyType = mapKeyValType.getType(0); - if (!keyType.isPrimitive() || - !keyType.asPrimitiveType().getPrimitiveTypeName().equals(PrimitiveTypeName.BINARY) || - !keyType.getLogicalTypeAnnotation().equals(stringType())) { - throw new IllegalArgumentException("Map key type must be binary (UTF8): " - + keyType); - } - Type valueType = mapKeyValType.getType(1); - if (valueType.isRepetition(Type.Repetition.OPTIONAL)) { - return of(Schema.createMap(optional(convertField(valueType, names)))); - } else { - return of(Schema.createMap(convertField(valueType, names))); - } - } + private Optional visitMapOrMapKeyValue() { + if (parquetGroupType.getFieldCount() != 1 + || parquetGroupType.getType(0).isPrimitive()) { + throw new UnsupportedOperationException("Invalid map type " + parquetGroupType); + } + GroupType mapKeyValType = + parquetGroupType.getType(0).asGroupType(); + if (!mapKeyValType.isRepetition(REPEATED) || mapKeyValType.getFieldCount() != 2) { + throw new UnsupportedOperationException("Invalid map type " + parquetGroupType); + } + Type keyType = mapKeyValType.getType(0); + if (!keyType.isPrimitive() + || !keyType.asPrimitiveType() + .getPrimitiveTypeName() + .equals(PrimitiveTypeName.BINARY) + || !keyType.getLogicalTypeAnnotation().equals(stringType())) { + throw new IllegalArgumentException( + "Map key type must be binary (UTF8): " + keyType); + } + Type valueType = mapKeyValType.getType(1); + if (valueType.isRepetition(Type.Repetition.OPTIONAL)) { + return of(Schema.createMap(optional(convertField(valueType, names)))); + } else { + return of(Schema.createMap(convertField(valueType, names))); + } + } - @Override - public Optional visit(LogicalTypeAnnotation.EnumLogicalTypeAnnotation enumLogicalType) { - return of(Schema.create(Schema.Type.STRING)); - } - }).orElseThrow(() -> new UnsupportedOperationException("Cannot convert Parquet type " + parquetType)); + @Override + public Optional visit( + LogicalTypeAnnotation.EnumLogicalTypeAnnotation enumLogicalType) { + return of(Schema.create(Schema.Type.STRING)); + } + }) + .orElseThrow( + () -> new UnsupportedOperationException("Cannot convert Parquet type " + parquetType)); } else { // if no original type then it's a record return convertFields(parquetGroupType.getName(), parquetGroupType.getFields(), names); @@ -422,7 +494,18 @@ private LogicalTypeAnnotation convertLogicalType(LogicalType logicalType) { return timestampType(true, MILLIS); } else if (logicalType instanceof LogicalTypes.TimestampMicros) { return timestampType(true, MICROS); + } else if (logicalType.getName().equals(LogicalTypes.uuid().getName()) && writeParquetUUID) { + return uuidType(); } + + if (avroVersionSupportsLocalTimestampTypes()) { + if (logicalType instanceof LogicalTypes.LocalTimestampMillis) { + return timestampType(false, MILLIS); + } else if (logicalType instanceof LogicalTypes.LocalTimestampMicros) { + return timestampType(false, MICROS); + } + } + return null; } @@ -430,41 +513,65 @@ private LogicalType convertLogicalType(LogicalTypeAnnotation annotation) { if (annotation == null) { return null; } - return annotation.accept(new LogicalTypeAnnotation.LogicalTypeAnnotationVisitor() { - @Override - public Optional visit(LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalLogicalType) { - return of(LogicalTypes.decimal(decimalLogicalType.getPrecision(), decimalLogicalType.getScale())); - } + return annotation + .accept(new LogicalTypeAnnotation.LogicalTypeAnnotationVisitor() { + @Override + public Optional visit( + LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalLogicalType) { + return of( + LogicalTypes.decimal(decimalLogicalType.getPrecision(), decimalLogicalType.getScale())); + } - @Override - public Optional visit(LogicalTypeAnnotation.DateLogicalTypeAnnotation dateLogicalType) { - return of(LogicalTypes.date()); - } + @Override + public Optional visit( + LogicalTypeAnnotation.DateLogicalTypeAnnotation dateLogicalType) { + return of(LogicalTypes.date()); + } - @Override - public Optional visit(LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeLogicalType) { - LogicalTypeAnnotation.TimeUnit unit = timeLogicalType.getUnit(); - switch (unit) { - case MILLIS: - return of(LogicalTypes.timeMillis()); - case MICROS: - return of(LogicalTypes.timeMicros()); - } - return empty(); - } + @Override + public Optional visit( + LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeLogicalType) { + LogicalTypeAnnotation.TimeUnit unit = timeLogicalType.getUnit(); + switch (unit) { + case MILLIS: + return of(LogicalTypes.timeMillis()); + case MICROS: + return of(LogicalTypes.timeMicros()); + } + return empty(); + } - @Override - public Optional visit(LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestampLogicalType) { - LogicalTypeAnnotation.TimeUnit unit = timestampLogicalType.getUnit(); - switch (unit) { - case MILLIS: - return of(LogicalTypes.timestampMillis()); - case MICROS: - return of(LogicalTypes.timestampMicros()); - } - return empty(); - } - }).orElse(null); + @Override + public Optional visit( + LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestampLogicalType) { + LogicalTypeAnnotation.TimeUnit unit = timestampLogicalType.getUnit(); + boolean isAdjustedToUTC = timestampLogicalType.isAdjustedToUTC(); + + if (isAdjustedToUTC || !avroVersionSupportsLocalTimestampTypes()) { + switch (unit) { + case MILLIS: + return of(LogicalTypes.timestampMillis()); + case MICROS: + return of(LogicalTypes.timestampMicros()); + } + return empty(); + } else { + switch (unit) { + case MILLIS: + return of(LogicalTypes.localTimestampMillis()); + case MICROS: + return of(LogicalTypes.localTimestampMicros()); + } + return empty(); + } + } + + @Override + public Optional visit(UUIDLogicalTypeAnnotation uuidLogicalType) { + return of(LogicalTypes.uuid()); + } + }) + .orElse(null); } /** @@ -478,22 +585,43 @@ public Optional visit(LogicalTypeAnnotation.TimestampLogicalTypeAnn */ private boolean isElementType(Type repeatedType, String parentName) { return ( - // can't be a synthetic layer because it would be invalid - repeatedType.isPrimitive() || - repeatedType.asGroupType().getFieldCount() > 1 || - repeatedType.asGroupType().getType(0).isRepetition(REPEATED) || + // can't be a synthetic layer because it would be invalid + repeatedType.isPrimitive() + || repeatedType.asGroupType().getFieldCount() > 1 + || repeatedType.asGroupType().getType(0).isRepetition(REPEATED) + || // known patterns without the synthetic layer - repeatedType.getName().equals("array") || - repeatedType.getName().equals(parentName + "_tuple") || + repeatedType.getName().equals("array") + || repeatedType.getName().equals(parentName + "_tuple") + || // default assumption - assumeRepeatedIsListElement - ); + assumeRepeatedIsListElement); } private static Schema optional(Schema original) { // null is first in the union because Parquet's default is always null - return Schema.createUnion(Arrays.asList( - Schema.create(Schema.Type.NULL), - original)); + return Schema.createUnion(Arrays.asList(Schema.create(Schema.Type.NULL), original)); + } + + private static String appendPath(String path, String fieldName) { + if (path == null || path.isEmpty()) { + return fieldName; + } + return path + '.' + fieldName; + } + + private static String namespace(String name, Map names) { + Integer nameCount = names.merge(name, 1, (oldValue, value) -> oldValue + 1); + return nameCount > 1 ? name + nameCount : null; + } + + /* Avro <= 1.9 does not support conversions to LocalTimestamp{Micros, Millis} classes */ + private static boolean avroVersionSupportsLocalTimestampTypes() { + final String avroVersion = getRuntimeAvroVersion(); + + return avroVersion == null + || !(avroVersion.startsWith("1.7.") + || avroVersion.startsWith("1.8.") + || avroVersion.startsWith("1.9.")); } } diff --git a/pom.xml b/pom.xml index 165459a08..29d2bfd65 100644 --- a/pom.xml +++ b/pom.xml @@ -82,6 +82,7 @@ UTF-8 1.2 + 2.0.3 7.1 1.11.4 3.11.12 @@ -107,7 +108,7 @@ 4.13.1 2.24.0 2.6 - 1.1.2 + 1.1.10.12 1.7.15 4.0.3 3.4.5 @@ -117,6 +118,9 @@ 2.25.5 4.1.75.Final 1.3.0 + 1.15.2 + + 1.17.1 3.3.2 2.18.12 ${jackson.version} @@ -531,6 +535,46 @@ snappy-java ${snappy.version}
+ + org.apache.parquet + parquet-avro + ${parquet.version} + + + org.apache.parquet + parquet-hadoop + ${parquet.version} + + + org.apache.parquet + parquet-column + ${parquet.version} + + + org.apache.parquet + parquet-common + ${parquet.version} + + + org.apache.parquet + parquet-encoding + ${parquet.version} + + + org.apache.parquet + parquet-format-structures + ${parquet.version} + + + org.apache.parquet + parquet-jackson + ${parquet-jackson.version} + + + io.airlift + aircompressor + ${aircompressor.version} + org.cassandraunit cassandra-unit From 1973c07ac264cb610a7b17dbcd1076593b0fc4d2 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 07:26:33 +0000 Subject: [PATCH 6/6] [deferred] Upgrade format-orc to ORC 1.9.8 and drop protobuf 2.5.0 Image scanning of gcr.io/cdapio/cdap reports protobuf-java 2.5.0 in format-orc-*.jar (3 CVEs), packaged as a direct dependency for orc-mapreduce 1.1.0. ORC 1.9.8 (the last Java 8 line; ORC 2.x needs Java 17) is generated with protobuf 3, so protobuf-java moves to 3.25.5. Deferred in the plan: it closes no bug on its own, because hbase-shaded-mapreduce and the Hadoop 3.3.6 tarball keep protobuf 2.5.0 (b/578423011, b/578459565, b/578641178 stay under the protobuf 2.5.0 exception). The published orc-core 1.9.8 pom does not declare hive-storage-api (orc-core needs org.apache.hadoop.hive.ql.exec.vector.*), and its commons-lang3 3.12.0 loses to 3.5 from hydrator-common, so format-orc declares hive-storage-api 2.8.1 (the version ORC 1.9.8 builds with) and commons-lang3 3.18.0 directly so they are packaged. No Java code change was needed: the plugin only uses TypeDescription, CompressionKind, OrcStruct and OrcOutputFormat, which are unchanged. Versions: - orc-mapreduce / orc-core 1.1.0 -> 1.9.8 (format-orc orc.version); adds orc-shims 1.9.8, threeten-extra 1.7.1, annotations 17.0.0 - protobuf-java 2.5.0 -> 3.25.5 (format-orc protobuf.version) - hive-storage-api: new direct dependency 2.8.1 (format-orc) - commons-lang3 3.5 -> 3.18.0 in the format-orc jar - iq80 snappy 0.5 (HP-2a) is no longer used: orc-core 1.9 uses aircompressor (2.0.3, managed in HP-4) instead Bugs: none closed alone (see above). Removes the protobuf 2.5.0 copy from format-orc; AZ-3 (azure adls) is the matching change in the azure repo. Risk: high. ORC 1.1 -> 1.9 changes the writer (file version 0.12, new defaults such as zstd support, dictionary and stripe settings); files are still readable by older readers that support ORC 0.12. ORC 1.9.8 was built against aircompressor 0.27; here it runs with the managed 2.0.3. Testing: JDK 8, Checkstyle on (0 violations). mvn -DcloudBuild clean verify -pl format-orc,core-plugins -am: BUILD SUCCESS; core-plugins 328 tests (2 skipped), including ETLTPFSTestRun.testOrc (writes ORC through the TPFSOrc sink and reads it back) and the parquet / avro TPFS tests. format-orc has no unit tests of its own. dependency:tree -pl format-orc shows orc 1.9.8, protobuf-java 3.25.5, hive-storage-api 2.8.1, commons-lang3 3.18.0, aircompressor 2.0.3 at compile scope; no protobuf 2.5.0 or iq80 snappy. unzip -l format-orc jar: lib/ has orc-core/-mapreduce/-shims-1.9.8, protobuf-java-3.25.5, hive-storage-api-2.8.1, commons-lang3-3.18.0, aircompressor-2.0.3 (no protobuf-java-2.5.0, no snappy-0.2/0.5). ORC read/write in a 6.11 sandbox (P2) not run. --- format-orc/pom.xml | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/format-orc/pom.xml b/format-orc/pom.xml index cc99170cc..dde22aa14 100644 --- a/format-orc/pom.xml +++ b/format-orc/pom.xml @@ -29,6 +29,14 @@ ORC format plugins jar + + + 1.9.8 + 3.25.5 + 2.8.1 + 3.18.0 + + io.cdap.cdap @@ -60,7 +68,7 @@ org.apache.orc orc-mapreduce - 1.1.0 + ${orc.version} org.apache.hadoop @@ -69,13 +77,30 @@ + + org.apache.hive + hive-storage-api + ${hive-storage-api.version} + + + * + * + + + + + org.apache.commons + commons-lang3 + ${commons-lang3.version} + com.google.protobuf protobuf-java - 2.5.0 + ${protobuf.version} junit