🐍 data-engineering-portfolio

A portfolio of Python data engineering pipelines and learning projects. Projects are designed to map existing enterprise skills onto programmatic equivalents — pandas ↔ Power Query, SQLite/DuckDB ↔ SQL Server, dbt/ETL structure ↔ SSIS packages.

---

Projects

🕹️ rawg_pipeline

An automated Medallion Architecture (Bronze/Silver/Gold) ETL pipeline that extracts rich video game metadata from the RAWG.io API, processes transformations via dbt and DuckDB, and serves insights through an interactive Streamlit dashboard.

Skills demonstrated: REST API extraction (requests), JSON flattening, pipeline orchestration, analytics engineering with dbt, local data warehouse optimisation with DuckDB, interactive data presentation.
BI equivalent: Power Query API Connection ➔ SQL Server Staging ➔ dbt Core Transformations ➔ Power BI / Tableau Dashboard

---

🐱 cat_shelter_pipeline

An end-to-end Medallion Architecture (Bronze/Silver/Gold) ETL pipeline that extracts real-world cat adoption data from the RescueGroups.org v5 API, transforms it through pandas and Parquet intermediate layers, and serves it via an interactive Streamlit dashboard.

Skills demonstrated: REST API integration, medallion architecture, pandas transformation, config-driven field selection, SQLAlchemy upserts, Streamlit dashboard with pipeline bootstrap and staleness guard.
BI equivalent: SSIS package ➔ SQL Server staging ➔ Power BI Dashboard

---

📘 python_intermediate_d2i

Exercises and worked examples based on the Data to Insight ERN intermediate Python sessions — a structured course covering core Python and pandas concepts for data practitioners.

Topics covered: pandas Series & DataFrames, filtering & aggregation, merging datasets, handling nulls, data visualization.
BI equivalent: Moving from Excel/SQL-based analysis to programmatic pandas-based data wrangling.

---

Tech Stack

Tool / Framework	Purpose
Python 3.x	Core language for ingestion, scripting, and application serving
pandas / PyArrow	High-performance data manipulation and evaluation
PySpark	Distributed transformation layer for larger-scale processing
dbt-duckdb	Modern Data Stack modeling, transformation logic, and testing
DuckDB	In-process analytical database for lightning-fast local queries
Streamlit / Plotly	Interactive visualization and user-facing presentation layer
GitHub Actions	CI/CD for linting and pipeline automation
pip-tools	Explicit dependency management (requirements.in / requirements.txt compilation)

---

Setup & Project Isolation

This repository uses a monorepo structure where each independent project contains its own isolated virtual environment (.venv) and pinned dependencies via pip-tools. This ensures individual pipelines never suffer from package version conflicts.

1. Clone the repository

git clone https://github.com/data-fox-coder/data-engineering-portfolio.git
cd data-engineering-portfolio

2. Install Ruff

Install ruff in an isolated global sandbox (or inside your root environment) to run repository-wide linting:

pipx install ruff

3. Code Quality Commands

Run these commands from the workspace root to check and format your code before pushing:

# Check all projects against repository linting rules
ruff check .

# Automatically fix all auto-fixable issues (imports, formatting, preview rules)
ruff check --fix .

4. Git Pre-Commit Hook (Optional)

To automatically run Ruff checks and auto-fix formatting issues before every git commit, set up a local Git hook:

# Create the pre-commit hook script
cat << 'EOF' > .git/hooks/pre-commit
#!/bin/sh
echo "🔍 Running Ruff linting check..."

ruff check --fix .
status=$?

if [ $status -ne 0 ]; then
    echo "❌ Ruff linting failed. Please fix remaining errors before committing."
    exit 1
fi

echo "✅ Ruff linting passed!"
EOF

# Make the hook executable
chmod +x .git/hooks/pre-commit

---

Background

I am a data professional with a deep background in SQL Server, local government data analysis, and Excel automation (Power Query M / DAX), expanding into programmatic data engineering. This repository documents that journey, each project is carefully engineered to handle messy, real-world API data rather than generic toy datasets.

I have the Databricks Certified Data Analyst Associate, Microsoft Fabric Analytics Engineer Associate and SnowPro Associate: Platform certifications.

---