Skip to content
View tangyf07's full-sized avatar

Achievements

Achievement: YOLO

Achievements

Achievement: YOLO

Highlights

  • Pro

Block or report tangyf07

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
tangyf07/README.md

Yuanfei Tang · tangyf07

哈工大 · 数据科学

Data Engineering · Data Warehouse · Stream Processing

Building reproducible data systems around correctness, observability and safe consumption.

Systems

Focus: RetailDW → GameStream → SQLGuard → ChangeLake

Repo What it is Run
RetailDW Offline retail warehouse on PySpark (ODS→DWD→DWS→ADS): quality gates, refund semantics, dt-partition idempotent reruns, cross-layer GMV reconcile bash scripts/run_local.sh
GameStream Realtime game analytics: Kafka → Flink → Doris; event-time, dedup, checkpoint recovery; Golden Path docker compose up -d && bash scripts/demo_golden_path.sh
SQLGuard Deterministic SQL write/execution gate (AST policy): make seal prints ALLOW/BLOCK + rule_id + evidence make seal
ChangeLake Reproducible CDC lakehouse: MySQL → Flink CDC → Paimon on MinIO; schema evolution, recovery, backfill, time travel, reconcile, compaction make demo

Other work

Other work: DataPilot · DocPilot · AgentOpsLite · RucBaseLab

Correctness & observability

  • RetailDW — 7-day repurchase window + net GMV; quality gate fail-fast; same-DT rerun reconcile
  • GameStream — metrics keyed by metric_id / ADS tables; Golden Path six beats (incl. late/dup/kill-TM)
  • SQLGuard — unsupported SQL fail-closed; seal cases are deterministic and evidence-bearing
  • ChangeLake — G9 source↔lake Decimal reconcile; G10 compaction fingerprint identity; Golden Path fail-fast (DEMO_EXIT=0)

Pinned Loading

  1. SQLGuard SQLGuard Public

    SQLGuard: deterministic SQL write/execution gate for agents (AST policy) — freshness, schema, and PII checks; ALLOW/BLOCK with rule_id + evidence; read-only passthrough.

    Python 1

  2. RetailDW RetailDW Public

    RetailDW: offline retail data warehouse on PySpark (ODS→DWD→DWS→ADS) with quality gates, refund semantics, dt-partition idempotent reruns, and cross-layer GMV reconcile.

    Python 1

  3. GameStream GameStream Public

    Realtime game analytics with Kafka → Flink → Doris: event-time, event_id dedup, checkpoint recovery and idempotent ADS upserts.

    Python 1

  4. ChangeLake ChangeLake Public

    Reproducible CDC lakehouse: MySQL → Flink CDC → Apache Paimon on MinIO, with schema evolution, recovery, backfill, time travel, reconciliation and compaction.

    Shell 1