Skip to content

[INLONG-12189][SDK] Support Map and JsonObject key access via [...] in ArrayParser - #12190

Open
luchunliang wants to merge 2 commits into
apache:masterfrom
luchunliang:INLONG-12189
Open

[INLONG-12189][SDK] Support Map and JsonObject key access via [...] in ArrayParser#12190
luchunliang wants to merge 2 commits into
apache:masterfrom
luchunliang:INLONG-12189

Conversation

@luchunliang

Copy link
Copy Markdown
Contributor

Fixes #12189

Motivation

  1. KV map lookup after STR_TO_MAP — decode a URL-encoded query string and pluck a specific parameter in one line:

    SELECT
      STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] AS HY50
    FROM source
    WHERE STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] = 'welfare_milestone_operations'
      AND event_code = 'OnPageEnter';
  2. JSON key access — read fields off a raw JsonObject produced by JSON helper functions using the intuitive obj['field'] syntax, without additional casting.

  3. Consistency — align bracket-access semantics with common SQL / JSONPath expectations, reduce boilerplate, and remove edge cases where a valid-looking expression silently returned null.

Modifications

ArrayParser#parse now handles four shapes; the two new branches (Map, JsonObject) are highlighted:

@Override
public Object parse(SourceData sourceData, int rowIndex, Context context) {
    Object leftValue  = this.left.parse(sourceData, rowIndex, context);
    Object rightValue = this.right.parse(sourceData, rowIndex, context);

    // 1) List indexed by Number (existing)
    if (leftValue instanceof List<?> && rightValue instanceof Number) {
        return ((List<?>) leftValue).get(((Number) rightValue).intValue());
    }

    // 2) Map indexed by any key type (NEW)
    if (leftValue instanceof Map<?, ?>) {
        return ((Map<?, ?>) leftValue).get(rightValue);
    }

    // 3) JsonArray indexed by Number (existing) — primitives unwrapped
    if (leftValue instanceof JsonArray && rightValue instanceof Number) {
        JsonElement result = ((JsonArray) leftValue).get(((Number) rightValue).intValue());
        return unwrapJsonElement(result);
    }

    // 4) JsonObject indexed by String key (NEW) — primitives unwrapped
    if (leftValue instanceof JsonObject && rightValue instanceof String) {
        JsonElement result = ((JsonObject) leftValue).get((String) rightValue);
        return unwrapJsonElement(result);
    }

    return null;
}

Where unwrapJsonElement:

  • returns null for JsonNull;
  • returns the natural Java value for JsonPrimitive (String / Boolean / Number, fallback to toString());
  • returns the original JsonArray / JsonObject for structural elements, so chained bracket access keeps working.

Behavior matrix

leftValue type rightValue type Result
List<?> Number list.get(index.intValue())
Map<?, ?> any map.get(rightValue) (new)
JsonArray Number element at index, primitives unwrapped
JsonObject String value at key, primitives unwrapped (new)
Any other combination null

Example

Before this change, the following expression silently returned null and the WHERE clause could never match:

SELECT
  STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] AS HY50
FROM source
WHERE STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] = 'welfare_milestone_operations'
  AND event_code = 'OnPageEnter';

After this change the map lookup works as intended, so both projection and filter behave correctly.

Tests

  • Added TestCsv2RowDataProcessor (peer of TestCsv2KvProcessor) which drives the CSV → RowData pipeline with a SQL that uses STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] in both SELECT and WHERE, exercising the new Map branch end-to-end.
  • Existing List / JsonArray bracket-access tests continue to pass unchanged.

Backward compatibility

  • Fully backward compatible.
  • The two existing branches (List[Number], JsonArray[Number]) keep identical semantics.
  • The fallback for unrecognized type combinations still returns null, so previously-null expressions can only turn into a real value — never into an unexpected error.
  • No public API / method signature change.

Risks / Notes

  • Map<?, ?> uses map.get(rightValue) directly; when the SQL parser resolves the bracket key as a String while the map key is a boxed number (or vice versa), the lookup will return null due to .equals type mismatch. This matches Java Map semantics and is consistent with how other Transform SDK operators treat mixed key types.
  • For JsonObject, missing keys resolve to null (Gson returns null from get); no exception is thrown.
  • JsonPrimitive unwrapping intentionally mirrors what JsonSourceData already does elsewhere, so downstream comparison/arithmetic operators receive the same Java type regardless of whether the value originated from the source or from a JSON helper function.

Files changed

  • inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/ArrayParser.java
  • inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/processor/TestCsv2RowDataProcessor.java (new; end-to-end coverage for Map[key])

Checklist

  • Backward-compatible with existing List[Number] / JsonArray[Number] usage
  • New unit / integration test added and passing locally
  • No changes to public APIs / method signatures
  • Behavior of primitive-unwrapping for JSON branches aligned with JsonSourceData

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)

  • This change added tests and can be verified as follows:

    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][SDK] Support Map and JsonObject key access via [...] in ArrayParser

2 participants