Add seatunnel integration doc - #464
Conversation
Bumped default release version to 1.7.0 and updated default GPG user in the release workflow. Added installation step for subversion on Ubuntu. Appended WARP.md to .gitignore.
Deleted validate-release-in-local.sh and enhanced validate-release.sh to support both SVN and local directory validation. Added color-coded output, improved argument handling, and included Java version checks for better usability and error reporting.
Major rewrite of validate-release.sh for Apache HugeGraph, adding modular structure, improved logging, error/warning collection, colorized output, and comprehensive validation steps for source and binary packages. New features include dependency checks, GPG key management, license compliance, file size and binary checks, version consistency, and automated server/toolchain testing. Usage instructions and help output are expanded for clarity.
…s V3 Updated CI workflow to support additional OS and architectures (arm64, macOS 14). Improved documentation and script usage instructions. The license header check now covers more file types and excludes generated/vendor files. Maven build commands in docs and scripts now use '-DskipTests' and '-Dcheckstyle.skip=true' for consistency. Added a detailed README for the release validation script.
Adds contextual error and warning reporting with step and package information, enhances license category and header checks, improves version consistency logic, and refines summary output with execution time and clearer formatting. These changes make validation results more actionable and easier to interpret, especially for multi-package and multi-step validations.
The JSON license was added to the CATEGORY_X regex in the binary package validation step to ensure packages with this license are properly flagged during release validation.
[pull] master from apache:master
* chore: update release workflow and .gitignore entries Bumped default release version to 1.7.0 and updated default GPG user in the release workflow. Added installation step for subversion on Ubuntu. Appended WARP.md to .gitignore. * refactor: unify release validation script and add local path support Deleted validate-release-in-local.sh and enhanced validate-release.sh to support both SVN and local directory validation. Added color-coded output, improved argument handling, and included Java version checks for better usability and error reporting. * refactor: revamp release validation script with enhanced checks V2 Major rewrite of validate-release.sh for Apache HugeGraph, adding modular structure, improved logging, error/warning collection, colorized output, and comprehensive validation steps for source and binary packages. New features include dependency checks, GPG key management, license compliance, file size and binary checks, version consistency, and automated server/toolchain testing. Usage instructions and help output are expanded for clarity. * refactor: enhance release validation for multi-arch and license checks V3 Updated CI workflow to support additional OS and architectures (arm64, macOS 14). Improved documentation and script usage instructions. The license header check now covers more file types and excludes generated/vendor files. Maven build commands in docs and scripts now use '-DskipTests' and '-Dcheckstyle.skip=true' for consistency. Added a detailed README for the release validation script. * refactor: improve validation script error context and reporting V4 Adds contextual error and warning reporting with step and package information, enhances license category and header checks, improves version consistency logic, and refines summary output with execution time and clearer formatting. These changes make validation results more actionable and easier to interpret, especially for multi-package and multi-step validations. * fix: add JSON to CATEGORY_X license validation The JSON license was added to the CATEGORY_X regex in the binary package validation step to ensure packages with this license are properly flagged during release validation. * introduce new version of validation * Update GPG username and regex in workflow --------- Co-authored-by: imbajin <jin@apache.org>
…#415) * fixed mvn version to 1.7.0 added graphspace part for docs of client changed client examples to NEWER version fixed parameters in loader docs
[pull] master from apache:master
[pull] master from apache:master
dosubot
Bot
added
size:L
docs: add Apache SeaTunnel connector integration guide (CN only) Add HugeGraph-SeaTunnel Connector documentation as an ecosystem entry point for users who want to sync data between HugeGraph and external systems via Apache SeaTunnel. Changes: - Add CN SeaTunnel connector doc in quickstart/toolchain/ - Add SeaTunnel to toolchain table and architecture diagram in CN introduction page - Follow existing Spark-connector documentation style and structure Co-Authored-By: Claude <noreply@anthropic.com> @
cf93179 to
4ae78ff
Compare
dosubot
Bot
added
size:M
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The PR adds an unrelated release workflow that uses obsolete artifact rules, and its SeaTunnel guide contains unsupported claims and non-runnable configuration. Evidence: actionlint /tmp/validate-release-new-464.yml; exact-head patch; and comparison with the parent release workflow.
| @@ -0,0 +1,805 @@ | |||
| name: "Validate Apache Release (New)" | |||
There was a problem hiding this comment.
release-* push/pull_request triggers. Please remove it from this PR and review it separately; otherwise unrelated CI behavior is merged without focused validation.
| RELEASE_VERSION: ${{ inputs.release_version || '1.7.0' }} | ||
| GPG_USER: ${{ inputs.gpg_user || 'pengjunzhi' }} | ||
| JAVA_VERSION: ${{ inputs.java_version || matrix.java_version || '11' }} | ||
| SVN_URL_PREFIX: https://dist.apache.org/repos/dist/dev/incubator/hugegraph |
There was a problem hiding this comment.
dist/dev/incubator/hugegraph and downloads.apache.org/incubator/hugegraph/KEYS). The parent workflow uses the post-graduation /hugegraph/ paths, so this workflow cannot fetch current release material. Replace both URLs with the TLP paths.
| pushd "$PACKAGE_DIR" | ||
|
|
||
| # 5.1: Check incubating name | ||
| if [[ ! "$src_pkg" =~ "incubating" ]]; then |
There was a problem hiding this comment.
incubating in source and binary names and checks incubator-shaped directories (also at line 631), while the parent workflow rejects that naming for current TLP artifacts. Valid packages will therefore be rejected or not found. Port the post-graduation naming/glob rules and remove the mandatory DISCLAIMER checks.
| - 'release-*' | ||
|
|
||
| jobs: | ||
| validate: |
There was a problem hiding this comment.
permissions: contents: read and persist-credentials: false so candidate code cannot inherit broader repository access.
| echo "Step [2/9]: Prepare Release Files" | ||
| echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" | ||
|
|
||
| DIST_DIR="dist/${{ env.RELEASE_VERSION }}" |
There was a problem hiding this comment.
${{ env.RELEASE_VERSION }} and GPG_USER usages below). A value containing shell syntax can execute before the script's quoted variables are applied. Pass values through the process environment, quote "$RELEASE_VERSION"/"$GPG_USER", and validate them against strict allowlists.
|
|
||
| #### 4.1 安装 SeaTunnel | ||
|
|
||
| 请参考 [Apache SeaTunnel 安装指南](https://seatunnel.apache.org/docs/start-v2/) 完成 SeaTunnel 的部署。 |
There was a problem hiding this comment.
/docs/start-v2/, which is not the current deployment guide, and the page says the connector can be referenced directly although current SeaTunnel deployments require connector-plugin installation. Link the current deployment guide and document the required plugin installation step.
| ```hocon | ||
| source { | ||
| HugeGraph { | ||
| url = "http://127.0.0.1:8080" |
There was a problem hiding this comment.
url and graph, which are not the documented connection fields, and omits required connection parameters such as host, port, and graph_name. Provide a versioned, runnable Source example only if that Source implementation is released; otherwise remove it.
| ```hocon | ||
| sink { | ||
| HugeGraph { | ||
| url = "http://127.0.0.1:8080" |
There was a problem hiding this comment.
url/graph fields and omits required host, port, graph_name, and schema_config, so it cannot run. Replace it with a complete versioned example using the official HugeGraph Sink options and state that the HugeGraph schema must already exist.
| | [Loader](/cn/docs/quickstart/toolchain/hugegraph-loader) | 数据导入工具：支持本地文件、HDFS、MySQL 等多数据源，TXT/CSV/JSON 等格式 | | ||
| | [Client](/cn/docs/quickstart/client/hugegraph-client) | 多语言 SDK：Java / Python / Go | | ||
| | [Spark-connector](/cn/docs/quickstart/toolchain/hugegraph-spark-connector) | Spark 集成：支持通过 Spark 批量读写图数据，适合大数据离线处理场景 | | ||
| | [SeaTunnel-connector](/cn/docs/quickstart/toolchain/hugegraph-seatunnel-connector) | SeaTunnel 集成：支持通过 SeaTunnel 在 HugeGraph 与外部数据系统之间同步数据 | |
There was a problem hiding this comment.
content/cn and content/en documentation. Add the corresponding English page and content/en/docs/introduction/_index.md entry so the integration is discoverable on both sites.
| ├──────────────────┴────────────────────┴──────────────────────┤ | ||
| │ HugeGraph Toolchain │ | ||
| │ Hubble | Loader | Client(Java/Go/Py) | Spark | Tools │ | ||
| │ Hubble | Loader | Client(Java/Go/Py) | Spark | SeaTunnel | Tools │ |
There was a problem hiding this comment.
🧹 minor — The new SeaTunnel row is wider than the ASCII diagram's top and bottom borders, so the right edge no longer aligns. Extend the frame or shorten the row and adjust the padding.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The new release workflow still fails valid artifacts or validates them incompletely, and the SeaTunnel documentation has unsupported compatibility and reference claims. Evidence: exact-head workflow/source review, actionlint, macOS grep reproduction, and current Apache documentation checks at 4ae78ff.
| MAX_FILE_SIZE: 800k | ||
| SERVER_START_DELAY: 3 | ||
| # License Patterns (ASF Category X - Prohibited) | ||
| CATEGORY_X: '\bGPL|\bLGPL|Sleepycat License|BSD-4-Clause|\bBCL\b|JSR-275|Amazon Software License|\bRSAL\b|\bQPL\b|\bSSPL|\bCPOL|\bNPL1|Creative Commons Non-Commercial|JSON\.org' |
There was a problem hiding this comment.
\b, but the workflow runs on macos-latest where BSD grep does not treat \b as a word boundary (printf 'GPL\n' | grep -E '\\bGPL' returns no match). Prohibited licenses can therefore pass on macOS while the Linux job rejects them. Use portable POSIX boundaries or a portable matcher and add a macOS regression check.
|
|
||
| # 5.3: Check license categories (Category X - Prohibited) | ||
| CAT_X_MATCHES=$(grep -r -E "${{ env.CATEGORY_X }}" LICENSE NOTICE 2>/dev/null || true) | ||
| CAT_X_COUNT=$(echo "$CAT_X_MATCHES" | grep -v '^$' | wc -l | tr -d ' ') |
There was a problem hiding this comment.
CAT_X_MATCHES is empty, grep -v '^$' exits 1; under GitHub Actions' default bash -eo pipefail, this assignment exits the step before the package check. I reproduced the non-zero status with bash -eo pipefail; the same pattern appears again at line 665. Count matches with an error-safe tool such as awk and keep read errors fail-closed.
| echo " [${CURRENT}/${PACKAGE_COUNT}] $pkg" | ||
|
|
||
| # Check SHA512 | ||
| if shasum -a 512 --check "${pkg}.sha512"; then |
There was a problem hiding this comment.
shasum --check "${pkg}.sha512" accepts the filenames embedded in the sidecar and never asserts that the sidecar contains exactly $pkg. A sidecar entry for another existing file can make this package pass without its own digest being checked. Parse one exact digest for $pkg and reject missing, duplicate, malformed, or extra records.
| # Extract package | ||
| PACKAGE_DIR=$(basename "$src_pkg" .tar.gz) | ||
| rm -rf "$PACKAGE_DIR" | ||
| tar -xzf "$src_pkg" |
There was a problem hiding this comment.
tar -xzf before compiling and executing their contents. It does not reject absolute, .., symlink, or hardlink members, so a crafted archive can escape the staging directory or redirect later validation commands. Validate members first and extract into a fresh isolated directory with resolved-path checks. Apply the same fix to the binary extraction at line 621.
| echo " ✓ No Category X licenses found" | ||
|
|
||
| # 5.4: Check license categories (Category B - Warning) | ||
| CAT_B_COUNT=$(grep -r -E "${{ env.CATEGORY_B }}" LICENSE NOTICE 2>/dev/null | wc -l | tr -d ' ' || echo "0") |
There was a problem hiding this comment.
| fi | ||
| cd .. | ||
| else | ||
| if mvn clean package -DskipTests -Dcheckstyle.skip=true -ntp -e; then |
There was a problem hiding this comment.
mvn clean package -DskipTests -Dcheckstyle.skip=true without the apache-release profile used by the repository's release workflow. Profile-specific packaging and release checks are consequently never exercised. Build with the release profile or explicitly run the omitted checks before declaring validation passed.
| if grep -q "<revision>" "$pom_file" 2>/dev/null; then | ||
| REVISION_VALUE=$(grep "<revision>" "$pom_file" | head -1 | sed 's/.*<revision>\(.*\)<\/revision>.*/\1/') | ||
| ROOT_POM="$pom_file" | ||
| break |
There was a problem hiding this comment.
<revision> and skips validation entirely if none is found. A child module can therefore carry a mismatched version while the workflow reports success. Validate all relevant parent/child POM versions against the release version and fail when the expected version metadata is absent.
|
|
||
| jobs: | ||
| validate: | ||
| name: "Validate Release On ${{ matrix.os }} (java-${{ matrix.java_version }})" |
There was a problem hiding this comment.
🧹 minor — java_version accepts 11 or 17, but the job name always uses matrix.java_version, whose matrix contains only 11. A manual Java 17 run will execute Java 17 while being labeled java-11. Use the effective input in the job name or remove the misleading override.
| } | ||
| ``` | ||
|
|
||
| > 完整配置参数和详细用法请参考 [Apache SeaTunnel Connector-V2 文档](https://seatunnel.apache.org/docs/connector-v2/)。 |
There was a problem hiding this comment.
/docs/connector-v2/ index rather than the concrete HugeGraph connector reference, so users cannot find the options used by this page. Link to the versioned/current HugeGraph Sink Connector documentation and qualify any Source claim separately.
|
|
||
| ### 3 环境要求 | ||
|
|
||
| - Java 8+ |
There was a problem hiding this comment.
Java 8+ and HugeGraph Server 1.0.0+, but the released SeaTunnel 2.3.13 connector depends on HugeGraph client/common 1.5.0, and HugeGraph 1.5.0+ requires Java 11. Publish a verified SeaTunnel/HugeGraph version matrix and do not advertise an unsupported Java 8 or Server 1.0.x combination.
6 participants
Purpose of the PR
Changes
content/cn/docs/quickstart/toolchain/hugegraph-seatunnel-connector.md— Chinese SeaTunnel integration doccontent/en/docs/quickstart/toolchain/hugegraph-seatunnel-connector.md— English SeaTunnel integration doccontent/cn/docs/introduction/_index.md— Add SeaTunnel to toolchain table & architecture diagramcontent/en/docs/introduction/_index.md— Add SeaTunnel to toolchain table & architecture diagram (EN)Background
The HugeGraph community has contributed HugeGraph Connector-V2 to Apache SeaTunnel, which is now in the merge pipeline. This PR adds the ecosystem entry in the official HugeGraph documentation so users know HugeGraph supports Apache SeaTunnel for data synchronization.
Doc Preview
New SeaTunnel Doc (CN)
![seatunnel-cn-doc]
Introduction Page — Toolchain Table (CN)
Hubble | Loader | Client | Spark-connector | ToolsHubble | Loader | Client | Spark-connector | SeaTunnel-connector | ToolsNew SeaTunnel Doc (EN)
![seatunnel-en-doc]
Introduction Page — Toolchain Table (EN)
Hubble | Loader | Client | Spark-connector | ToolsHubble | Loader | Client | Spark-connector | SeaTunnel-connector | Tools