From 41114dae7f8fe801a7f7e5abfc3896a377e03b8f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 21:19:15 +0000 Subject: [PATCH 1/9] Improve discoverability: package metadata, usage guidance and database pages - pyproject: add keywords, Topic/Audience/Status classifiers and supported Python versions; describe the XSD-driven table generation in the summary. - README and docs home: add a worked example (XML to tables) and a "When to use xml2db" comparison with common alternatives. - New docs pages per backend (PostgreSQL, SQL Server, MySQL/MariaDB, DuckDB) covering connection, bulk loading, names and type advice. - New page on copying loaded tables from a DuckDB staging file to SQLite, Parquet, DuckLake and Delta Lake, with full and incremental copies (tested with DuckDB 2.0). - New "XSD to SQL tables" page on generating DDL from an XSD. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- README.md | 21 ++++- docs/databases/duckdb.md | 64 +++++++++++++ docs/databases/mysql.md | 82 +++++++++++++++++ docs/databases/other_targets.md | 154 ++++++++++++++++++++++++++++++++ docs/databases/postgresql.md | 100 +++++++++++++++++++++ docs/databases/sql_server.md | 108 ++++++++++++++++++++++ docs/how_it_works.md | 3 + docs/index.md | 86 +++++++++++++++++- docs/xsd_to_sql.md | 107 ++++++++++++++++++++++ mkdocs.yml | 9 ++ pyproject.toml | 20 ++++- 11 files changed, 749 insertions(+), 5 deletions(-) create mode 100644 docs/databases/duckdb.md create mode 100644 docs/databases/mysql.md create mode 100644 docs/databases/other_targets.md create mode 100644 docs/databases/postgresql.md create mode 100644 docs/databases/sql_server.md create mode 100644 docs/xsd_to_sql.md diff --git a/README.md b/README.md index e9a6743..e7c6c76 100644 --- a/README.md +++ b/README.md @@ -15,10 +15,29 @@ database, and getting them back from the database into XML format if needed. This package uses `sqlalchemy` to interact with the database, so it should work with different database backends. Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for -MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). +MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). Through DuckDB, the loaded tables can also be +copied to SQLite, Parquet files or Delta Lake. **Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!** +## When to use xml2db + +| Situation | Usually enough | What xml2db adds | +|---|---|---| +| A few fields from flat, record-like XML | `pandas.read_xml`, a short `lxml` script, or the XML functions of your database | Little: these are simpler | +| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables instead of nested columns to explode | +| Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | +| Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | +| All the data must be kept, or written back to XML | Hand-written mappings, with the risk of dropping fields | Every element and attribute kept, with a round trip back to XML | + +`xml2db` needs an XSD schema. See the documentation for an [example](https://cre-dev.github.io/xml2db/#from-xml-to-tables) +and for database-specific guides: [PostgreSQL](https://cre-dev.github.io/xml2db/databases/postgresql/), +[SQL Server](https://cre-dev.github.io/xml2db/databases/sql_server/), +[MySQL and MariaDB](https://cre-dev.github.io/xml2db/databases/mysql/), +[DuckDB](https://cre-dev.github.io/xml2db/databases/duckdb/), +[SQLite, Parquet and Delta Lake](https://cre-dev.github.io/xml2db/databases/other_targets/). To only generate SQL +tables from an XSD, see [XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd_to_sql/). + ## Installation The package can be installed, preferably in a virtual environment, using `pip`: diff --git a/docs/databases/duckdb.md b/docs/databases/duckdb.md new file mode 100644 index 0000000..857a6e8 --- /dev/null +++ b/docs/databases/duckdb.md @@ -0,0 +1,64 @@ +--- +title: "Load XML into DuckDB" +description: "Load complex XML files into DuckDB tables generated from an XSD schema with xml2db, query them locally without a server, and use DuckDB as a staging database for other targets." +--- + +# Load XML into DuckDB + +DuckDB runs inside the Python process and stores a whole database in a single file. With `xml2db`, it gives a +relational copy of XML files without any server to set up, which suits local analysis, trying a data model before +deploying it, and tests. + +## Connecting + +``` bash +pip install xml2db duckdb-engine + +xml2db import invoices.xml invoices.xsd --connection-string "duckdb:///invoices.duckdb" +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="invoices.xsd", + connection_string="duckdb:///invoices.duckdb", +) +for xml_file in ["invoices_1.xml", "invoices_2.xml"]: + data_model.parse_xml(xml_file).insert_into_target_tables() +``` + +Use `duckdb:///:memory:` for a database that only lives during the Python process. + +Then query the file with the DuckDB command line or from Python: + +``` py +import duckdb + +con = duckdb.connect("invoices.duckdb", read_only=True) +con.sql("SELECT id, customer_name, CAST(date AS DATE) AS date FROM invoice").show() +``` + +XSD `date` and `time` values are stored as text, since they may carry a time zone: cast them in queries or views. + +## Bulk loading + +Batches of 100 rows or more are written to a temporary CSV file and loaded with DuckDB's `read_csv`, every column being +read as text and then cast to its target type. Smaller batches use `executemany`. + +## One writer at a time + +A DuckDB file accepts a single writing process. When loading files from several processes, parse in parallel and +load one file at a time, as shown in [Multiprocessing](../how_it_works.md#multiprocessing). Readers using +`read_only=True` cannot open the file while a process writes to it either. + +## Keys and names + +Primary keys are filled from DuckDB sequences, named `pk_sequ_`. Table and column names longer than 63 +characters are shortened with a hash suffix, as for PostgreSQL. This limit comes from the SQLAlchemy integration, not +from DuckDB. + +## Writing to other targets + +DuckDB extensions can copy the loaded tables to SQLite, Parquet files, Delta Lake, DuckLake and other targets that +`xml2db` does not support directly. See [Other targets through DuckDB](other_targets.md). diff --git a/docs/databases/mysql.md b/docs/databases/mysql.md new file mode 100644 index 0000000..a204bd5 --- /dev/null +++ b/docs/databases/mysql.md @@ -0,0 +1,82 @@ +--- +title: "Load XML into MySQL and MariaDB" +description: "Load complex XML files into MySQL or MariaDB tables generated from an XSD schema with xml2db: drivers, LOAD DATA LOCAL INFILE bulk loading, identifier limits and string sizes." +--- + +# Load XML into MySQL and MariaDB + +`xml2db` creates MySQL tables from an XSD schema and loads XML files into them, using `LOAD DATA LOCAL INFILE` for +bulk loading. MariaDB uses the same code path; integration tests run against MySQL only. + +## Connecting + +Install `pymysql` or `mysqlclient`, then pass a SQLAlchemy connection string: + +``` bash +pip install xml2db pymysql + +xml2db import file.xml schema.xsd \ + --connection-string "mysql+pymysql://user:pw@host:3306/db" \ + --db-schema xml_data +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="schema.xsd", + connection_string="mysql+pymysql://user:pw@host:3306/db", + db_schema="xml_data", +) +data_model.parse_xml("file.xml").insert_into_target_tables() +``` + +In MySQL, a schema is a database: `--db-schema xml_data` creates the `xml_data` database if it does not exist. Use +`mariadb+pymysql://` for MariaDB. + +## Bulk loading + +With `pymysql` or `mysqlclient`, batches of 100 rows or more are written to a temporary tab-separated file and sent +with `LOAD DATA LOCAL INFILE`. Two settings are needed: + +* on the client, `local_infile=True` in the connection arguments. `DataModel` adds it when it creates the engine from a + connection string. If you pass your own engine, add it yourself: + `sqlalchemy.create_engine(url, connect_args={"local_infile": True})`; +* on the server, `local_infile=ON`, for instance by starting it with `--local-infile=1`. Check with + `SHOW VARIABLES LIKE 'local_infile'`. + +When the server refuses `LOAD DATA LOCAL INFILE`, `xml2db` falls back to `executemany` and keeps doing so for the +following batches of the same `DataModel`. +`insert_into_target_tables(bulk_load=True)` raises an error instead, which helps to check the setup. + +## String sizes + +Strings without a `maxLength` in the XSD become `VARCHAR(255)` on MySQL, instead of `VARCHAR(1000)` on other +backends. Longer values are rejected in strict SQL mode, which is the default. Override the type of free-text fields: + +``` yaml title="model_config.yml" +tables: + line: + fields: + description: + type: Text +``` + +MySQL also limits the total size of a row to 65,535 bytes, counting each `VARCHAR` column at its maximum size. Tables +with many string columns can reach this limit, especially with `utf8mb4` (4 bytes per character). `Text` columns only +count for a few bytes each, so the same override solves the "Row size too large" error. + +Repeated simple values joined in one column use `VARCHAR(4000)`. + +## Table and column names + +MySQL accepts identifiers up to 64 characters. `xml2db` shortens names longer than 56 characters, keeping room for the +suffixes SQLAlchemy adds to constraint names. Shortened names keep their first characters and get a 7-character hash +suffix, so they stay unique and stable between runs. Use `rename` in the configuration to choose readable names, and +`xml2db render schema.xsd --format ddl --db-type mysql` to see the result. + +## Concurrent loads + +Each `DataModel` instance uses its own staging tables (named with a random `temp_prefix`), so several processes can +fill staging tables at the same time. The merge into target tables should run one process at a time: see +[Multiprocessing](../how_it_works.md#multiprocessing) for an example. diff --git a/docs/databases/other_targets.md b/docs/databases/other_targets.md new file mode 100644 index 0000000..bc9135a --- /dev/null +++ b/docs/databases/other_targets.md @@ -0,0 +1,154 @@ +--- +title: "Load XML into SQLite, Parquet or Delta Lake" +description: "Use xml2db with DuckDB as a staging database to load complex XML files into SQLite, Parquet files, Delta Lake, DuckLake or Iceberg tables, with full or incremental copies." +--- + +# Load XML into SQLite, Parquet or Delta Lake + +`xml2db` merges each file into existing tables: it matches records by hash, generates keys and links rows through +foreign keys. This needs a database with sequences or identity columns, unique constraints and `UPDATE` statements. Files +and table formats such as Parquet or Delta Lake do not offer this, and SQLite is not supported. + +[DuckDB](duckdb.md) fills the gap. Load the XML files into a DuckDB file, which keeps the state needed for +deduplication, then copy its tables to the target with a DuckDB extension: + +| Target | DuckDB extension | Copy method | +|---|---|---| +| SQLite | `sqlite` | `ATTACH ... (TYPE sqlite)`, then `CREATE TABLE ... AS` | +| Parquet files | built in | `COPY ... TO ... (FORMAT parquet)` | +| DuckLake | `ducklake` | `ATTACH 'ducklake:...'`, then `CREATE TABLE ... AS` | +| Delta Lake | `delta` | `ATTACH ... (TYPE delta)`, then `CREATE TABLE` and `INSERT` | +| Iceberg | `iceberg` | Through an Iceberg REST catalog, see the [DuckDB documentation](https://duckdb.org/docs/current/core_extensions/iceberg/overview) | + +The examples below were tested with DuckDB 2.0 and local files. Extensions are downloaded on first use. + +!!! note + Pointing `xml2db` directly at a database attached in DuckDB does not work: DuckDB cannot create the sequences + used for primary keys in attached PostgreSQL or SQLite databases. + +## Load into a staging DuckDB file + +``` py title="load.py" linenums="1" +from xml2db import DataModel + +data_model = DataModel(xsd_file="schema.xsd", connection_string="duckdb:///staging.duckdb") +for xml_file in xml_files: + data_model.parse_xml(xml_file).insert_into_target_tables() +data_model.engine.dispose() +``` + +Keep `staging.duckdb` between runs: it holds the hashes used to store each record once. Starting from an empty file +restarts deduplication and key numbering. + +## Copy all tables + +The function below lists the tables of the staging file and copies them to an attached database, replacing previous +copies. Run it once loading is finished, from a single process. + +``` py title="copy_all.py" linenums="1" +import duckdb + + +def staging_tables(con): + """List the tables xml2db created in the staging DuckDB file.""" + return [ + row[0] + for row in con.execute( + "SELECT table_name FROM duckdb_tables() " + "WHERE database_name = current_database() AND schema_name = 'main'" + ).fetchall() + ] + + +def copy_all(con, target): + """Copy every table to an attached database, replacing previous copies.""" + for table in staging_tables(con): + con.execute(f'CREATE OR REPLACE TABLE {target}."{table}" AS FROM "{table}"') + + +con = duckdb.connect("staging.duckdb") +con.execute("ATTACH 'xml_data.sqlite' AS target (TYPE sqlite)") +copy_all(con, "target") +``` + +The same function writes to DuckLake, with `ATTACH 'ducklake:xml_data.ducklake' AS target (DATA_PATH 'files/')`. + +For Parquet, write one file per table: + +``` py +for table in staging_tables(con): + con.execute(f"COPY \"{table}\" TO 'parquet/{table}.parquet' (FORMAT parquet)") +``` + +Copies made with `CREATE TABLE ... AS` keep columns and their types, as far as the target supports them, but not +primary keys, foreign keys or unique constraints. + +## Copy new rows only + +`xml2db` never updates or deletes rows of the target tables: loading a file only appends rows. Keys come from +sequences, so new rows have higher primary keys than existing ones. Relationship tables have no primary key, but a new +row always points to a new parent row, and the parent key is their first column. + +This gives a simple incremental copy: for each table, append the rows whose first column is greater than its highest +value in the target. + +``` py title="copy_new_rows.py" linenums="1" +def copy_new_rows(con, target): + """Append the rows added since the previous copy to an attached database.""" + existing = { + row[0] + for row in con.execute( + "SELECT table_name FROM duckdb_tables() WHERE database_name = ?", [target] + ).fetchall() + } + for table in staging_tables(con): + if table not in existing: + con.execute(f'CREATE TABLE {target}."{table}" AS FROM "{table}"') + continue + # first column: the primary key, or the parent key of a relationship table + key = con.execute( + "SELECT column_name FROM duckdb_columns() WHERE database_name = current_database() " + "AND table_name = ? ORDER BY column_index LIMIT 1", + [table], + ).fetchone()[0] + con.execute( + f'INSERT INTO {target}."{table}" FROM "{table}" ' + f'WHERE "{key}" > (SELECT coalesce(max("{key}"), 0) FROM {target}."{table}")' + ) +``` + +## Delta Lake + +A Delta table is attached on its own, and DuckDB does not support `CREATE TABLE ... AS` for Delta. Create each table +from the staging column types, then append new rows as above: + +``` py title="copy_to_delta.py" linenums="1" +def copy_new_rows_to_delta(con, root): + """Append new rows to one Delta table per staging table, under the root path.""" + for table in staging_tables(con): + con.execute(f"ATTACH '{root}/{table}' AS delta_target (TYPE delta)") + columns = con.execute( + "SELECT column_name, data_type FROM duckdb_columns() " + "WHERE database_name = current_database() AND table_name = ? ORDER BY column_index", + [table], + ).fetchall() + if not con.execute( + "SELECT count(*) FROM duckdb_tables() WHERE database_name = 'delta_target'" + ).fetchone()[0]: + definition = ", ".join(f'"{name}" {data_type}' for name, data_type in columns) + con.execute(f"CREATE TABLE delta_target.delta_target ({definition})") + key = columns[0][0] + con.execute( + f'INSERT INTO delta_target FROM "{table}" ' + f'WHERE "{key}" > (SELECT coalesce(max("{key}"), 0) FROM delta_target)' + ) + con.execute("DETACH delta_target") + + +con = duckdb.connect("staging.duckdb") +copy_new_rows_to_delta(con, "s3://bucket/xml_data") # or a local directory +``` + +Delta write support depends on the DuckDB version and platform: check the +[delta extension documentation](https://duckdb.org/docs/current/core_extensions/delta) for current limitations. +Writing to object storage also needs credentials, set up with DuckDB secrets. diff --git a/docs/databases/postgresql.md b/docs/databases/postgresql.md new file mode 100644 index 0000000..e729481 --- /dev/null +++ b/docs/databases/postgresql.md @@ -0,0 +1,100 @@ +--- +title: "Load XML into PostgreSQL" +description: "Load complex XML files into PostgreSQL tables generated from an XSD schema with xml2db: drivers, COPY bulk loading, identifier limits, type overrides and concurrent loads." +--- + +# Load XML into PostgreSQL + +`xml2db` creates PostgreSQL tables from an XSD schema and loads XML files into them, using `COPY` for bulk loading. + +## Connecting + +Install a driver alongside `xml2db`: + +``` bash +pip install xml2db psycopg2-binary # or: pip install xml2db "psycopg[binary]" +``` + +Then pass a SQLAlchemy connection string: + +``` bash +xml2db import file.xml schema.xsd \ + --connection-string "postgresql+psycopg2://user:pw@host:5432/db" \ + --db-schema xml_data +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="schema.xsd", + connection_string="postgresql+psycopg://user:pw@host:5432/db", + db_schema="xml_data", +) +data_model.parse_xml("file.xml").insert_into_target_tables() +``` + +The schema given with `--db-schema` or `db_schema` is created if it does not exist. Without it, tables go to the +default schema of the connection, usually `public`. + +## Bulk loading + +With `psycopg2` or `psycopg` (version 3), staging tables are filled with `COPY ... FROM STDIN`: rows are written to an +in-memory CSV buffer and streamed to the server, with no temporary file. `COPY` is used for every batch, whatever its +size. Other drivers, such as `pg8000`, fall back to standard `executemany` inserts. + +`insert_into_target_tables(bulk_load=True)` raises an error instead of falling back when the driver does not support +`COPY`. See [Bulk loading](../how_it_works.md#bulk-loading) for the other options. + +## Table and column names + +PostgreSQL truncates identifiers longer than 63 characters. `xml2db` shortens such names itself, keeping the first +characters and adding a 7-character hash suffix, so names stay unique and stable between runs. Deep XSD hierarchies +often produce long names: rename them in the configuration to keep them readable. + +``` yaml title="model_config.yml" +tables: + TimeSeries: + fields: + production_RegisteredResource_pSRType_powerSystemResources_nominalP: + rename: nominal_power +``` + +`xml2db render schema.xsd --format ddl --db-type postgresql` shows the physical names before anything is created. + +## Column types + +The default mapping favours loading every file without errors over precise types: + +| XSD type | PostgreSQL type | +|---|---| +| `string` without `maxLength` | `VARCHAR(1000)` | +| `string` with `maxLength` | `VARCHAR(n)` | +| `decimal`, `float`, `double` | `DOUBLE PRECISION` | +| `integer`, `int` and derived types | `INTEGER` | +| `long` | `BIGINT` | +| `boolean` | `BOOLEAN` | +| `dateTime` | `TIMESTAMP WITH TIME ZONE` | +| `date`, `time` | `VARCHAR` (XSD values may carry a time zone) | +| Repeated simple values joined in one column | `VARCHAR(8000)` | +| Other types, such as `anyURI` | `VARCHAR(1000)`, with a warning | + +The `xml2db_record_hash` column used for deduplication is `BYTEA`. + +Override a type when the default does not fit, for instance to keep exact decimal values for amounts: + +``` yaml title="model_config.yml" +tables: + line: + fields: + price: + type: Numeric(12, 2) + description: + type: Text +``` + +## Concurrent loads + +Each `DataModel` instance uses its own staging tables (named with a random `temp_prefix`), so several processes can +fill staging tables at the same time. The merge into target tables should run one process at a time: see +[Multiprocessing](../how_it_works.md#multiprocessing) for an example. diff --git a/docs/databases/sql_server.md b/docs/databases/sql_server.md new file mode 100644 index 0000000..177923d --- /dev/null +++ b/docs/databases/sql_server.md @@ -0,0 +1,108 @@ +--- +title: "Load XML into SQL Server" +description: "Load complex XML files into Microsoft SQL Server tables generated from an XSD schema with xml2db: pyodbc connection, bcp bulk loading, columnstore indexes and type mapping." +--- + +# Load XML into SQL Server + +`xml2db` creates SQL Server tables from an XSD schema and loads XML files into them, using the `bcp` utility for bulk +loading when it is installed. + +## Connecting + +Install `pyodbc` and a Microsoft ODBC driver (for example ODBC Driver 18 for SQL Server), then pass a SQLAlchemy +connection string: + +``` bash +pip install xml2db pyodbc + +xml2db import file.xml schema.xsd \ + --connection-string "mssql+pyodbc://user:pw@host:1433/db?driver=ODBC+Driver+18+for+SQL+Server&TrustServerCertificate=yes" \ + --db-schema xml_data +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="schema.xsd", + connection_string="mssql+pyodbc://user:pw@host:1433/db?driver=ODBC+Driver+18+for+SQL+Server", + db_schema="xml_data", +) +data_model.parse_xml("file.xml").insert_into_target_tables() +``` + +When `DataModel` creates the engine from a connection string, it enables `fast_executemany` and uses the +`SERIALIZABLE` isolation level. If you pass your own engine with `db_engine`, set these options yourself: + +``` py +import sqlalchemy + +engine = sqlalchemy.create_engine( + connection_string, fast_executemany=True, isolation_level="SERIALIZABLE" +) +data_model = DataModel(xsd_file="schema.xsd", db_engine=engine) +``` + +## Bulk loading + +Batches of 100 rows or more are loaded into staging tables with `bcp` when these conditions are met: + +* the `bcp` executable is on the `PATH` (package `mssql-tools18` on Linux and macOS, SQL Server Command Line + Utilities on Windows); +* the connection uses SQL Server authentication (user and password in the connection string) or a trusted connection + (`Trusted_Connection=yes`). + +`TrustServerCertificate=yes` in the connection string is passed on to `bcp`. Smaller batches, or any batch when these +conditions are not met, use `fast_executemany`. + +`bcp` runs as a separate process, outside the SQLAlchemy transaction. This only concerns staging tables: the merge +into target tables still runs in a transaction. Tab characters inside values are replaced by spaces on this path; pass +`bulk_load=False` to `insert_into_target_tables` if they must be kept. + +## Columnstore indexes + +For large analytical tables, `xml2db` can create a clustered columnstore index on selected tables. The primary key +of these tables then becomes nonclustered. + +``` yaml title="model_config.yml" +tables: + TimeSeries: + as_columnstore: true +``` + +Relationship tables between deduplicated tables have no primary key: they get a clustered index on their two foreign +keys. + +## Column types + +| XSD type | SQL Server type | +|---|---| +| `string` without `maxLength` | `VARCHAR(1000)` | +| `string` with `maxLength` | `VARCHAR(n)`, or `CHAR(n)` when the length is fixed | +| `decimal`, `float`, `double` | `DOUBLE PRECISION` (`FLOAT(53)`) | +| `dateTime` | `DATETIMEOFFSET` | +| `date`, `time` | `VARCHAR` (XSD values may carry a time zone) | +| Repeated simple values joined in one column | `VARCHAR(8000)` | +| Other types, such as `anyURI` | `VARCHAR(1000)`, with a warning | + +The `xml2db_record_hash` column used for deduplication is `BINARY(20)` with the default hash function. + +Text columns are `VARCHAR`. To store characters outside the code page of your database collation, use a UTF-8 +collation (SQL Server 2019 and later), or override column types with `NVARCHAR` in a Python configuration: + +``` py +from sqlalchemy.dialects.mssql import NVARCHAR + +model_config = { + "tables": {"invoice": {"fields": {"customer_name": {"type": NVARCHAR(200)}}}} +} +``` + +SQL Server accepts identifiers up to 128 characters, so table and column names are not shortened. + +## Concurrent loads + +Each `DataModel` instance uses its own staging tables (named with a random `temp_prefix`), so several processes can +fill staging tables at the same time. The merge into target tables should run one process at a time: see +[Multiprocessing](../how_it_works.md#multiprocessing) for an example. diff --git a/docs/how_it_works.md b/docs/how_it_works.md index 0019f2b..fb7419b 100644 --- a/docs/how_it_works.md +++ b/docs/how_it_works.md @@ -197,6 +197,9 @@ missing tool, or server configuration), it falls back silently to SQLAlchemy's s | MS SQL Server | `bcp` utility | `bcp` on PATH; SQL or Windows/Kerberos auth | 100 rows | | DuckDB | `read_csv()` | built-in | 100 rows | +The pages for [PostgreSQL](databases/postgresql.md#bulk-loading), [SQL Server](databases/sql_server.md#bulk-loading), +[MySQL](databases/mysql.md#bulk-loading) and [DuckDB](databases/duckdb.md#bulk-loading) give the setup details. + The threshold column means that batches smaller than that number always use `executemany` (avoiding temp-file overhead for small inserts). PostgreSQL's `COPY` is in-protocol and has no file overhead, so there is no threshold. diff --git a/docs/index.md b/docs/index.md index 3a72127..24821d4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -20,12 +20,92 @@ more user-friendly tables. See the [Getting started](getting_started.md) guide for installation, CLI usage, and the Python API. +## From XML to tables + +Take this XSD-described file, where each invoice has a customer and repeated lines: + +``` xml title="invoices.xml" + + + AliceFR + X129.90 + Y7124.00 + + + BobDE + X129.90 + + +``` + +A single command, with no mapping to write: + +``` bash +xml2db import invoices.xml invoices.xsd --connection-string "duckdb:///invoices.duckdb" +``` + +gives an `invoice` table, a `line` table and an `invoice_line` table linking them, plus tables for the `invoices` root +element. The `customer` element has a single occurrence, so its fields become columns of `invoice`. The `X1` line appears twice in the file but is stored once. + +=== "invoice" + + | pk_invoice | id | date | customer_name | customer_country | + |-----------:|---------|------------|---------------|------------------| + | 1 | INV-001 | 2026-10-01 | Alice | FR | + | 2 | INV-002 | 2026-10-02 | Bob | DE | + +=== "line" + + | pk_line | sku | quantity | price | + |--------:|-----|---------:|------:| + | 1 | X1 | 2 | 9.9 | + | 2 | Y7 | 1 | 24.0 | + +=== "invoice_line" + + | fk_invoice | fk_line | + |-----------:|--------:| + | 1 | 1 | + | 1 | 2 | + | 2 | 1 | + +The tables are then queried with plain SQL joins: + +``` sql +SELECT i.id, i.customer_name, l.sku, l.quantity, l.price +FROM invoice i +JOIN invoice_line il ON il.fk_invoice = i.pk_invoice +JOIN line l ON l.pk_line = il.fk_line; +``` + +## When to use xml2db + +Many XML files do not need `xml2db`. The table below compares it with common alternatives. + +| Situation | Usually enough | What xml2db adds | +|---|---|---| +| A few fields from flat, record-like XML | `pandas.read_xml`, a short `lxml` script, or the XML functions of your database | Little: these are simpler | +| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables instead of nested columns to explode | +| Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | +| Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | +| All the data must be kept, or written back to XML | Hand-written mappings, with the risk of dropping fields | Every element and attribute kept, with a round trip back to XML | + +`xml2db` needs an XSD schema. Files are parsed in memory: files around 500 MB work in practice. Recursive schemas, +wildcards and mixed content are only partly supported, see the [caveats](how_it_works.md#caveats). + ## Supported backends Built on `sqlalchemy`, `xml2db` supports multiple database backends. Integration tests cover PostgreSQL, MySQL, -MS SQL Server, and DuckDB. You may need to install a connector package (e.g. `psycopg2` or `psycopg` for PostgreSQL, -`pymysql` or `mysqlclient` for MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). See -[How it works](how_it_works.md#bulk-loading) for which drivers enable native bulk loading. +MS SQL Server, and DuckDB. Each backend has its own page with connection strings, bulk loading and configuration +advice: + +* [PostgreSQL](databases/postgresql.md) +* [SQL Server](databases/sql_server.md) +* [MySQL and MariaDB](databases/mysql.md) +* [DuckDB](databases/duckdb.md) +* [SQLite, Parquet, Delta Lake and other targets through DuckDB](databases/other_targets.md) + +To only generate `CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd_to_sql.md). ## Visualising data models diff --git a/docs/xsd_to_sql.md b/docs/xsd_to_sql.md new file mode 100644 index 0000000..4722915 --- /dev/null +++ b/docs/xsd_to_sql.md @@ -0,0 +1,107 @@ +--- +title: "XSD to SQL tables" +description: "Generate SQL CREATE TABLE statements from an XSD schema with xml2db, for PostgreSQL, SQL Server or MySQL, and adjust the resulting relational schema with a configuration file." +--- + +# XSD to SQL tables + +`xml2db` derives a relational schema from an XSD file. You can generate the `CREATE TABLE` statements without +connecting to a database or loading any XML file, to review the schema, share it, or create the tables with your own +deployment tools. + +## From the command line + +``` bash +xml2db render invoices.xsd --format ddl --db-type postgresql --output invoices.sql +``` + +`--db-type` accepts `postgresql`, `mssql`, `mysql` and `mariadb`. Without it, the statements use generic SQL. + +For the invoices schema of the [introduction](index.md#from-xml-to-tables), this gives, among other tables: + +``` sql +CREATE TABLE line ( + pk_line SERIAL NOT NULL, + sku VARCHAR(1000), + quantity INTEGER, + price DOUBLE PRECISION, + xml2db_record_hash BYTEA, + CONSTRAINT cx_pk_line PRIMARY KEY (pk_line), + CONSTRAINT line_xml2db_record_hash UNIQUE (xml2db_record_hash) +) + +CREATE TABLE invoice ( + pk_invoice SERIAL NOT NULL, + id VARCHAR(1000), + date VARCHAR(16), + customer_name VARCHAR(1000), + customer_country VARCHAR(1000), + xml2db_record_hash BYTEA, + CONSTRAINT cx_pk_invoice PRIMARY KEY (pk_invoice), + CONSTRAINT invoice_xml2db_record_hash UNIQUE (xml2db_record_hash) +) + +CREATE TABLE invoice_line ( + fk_invoice INTEGER NOT NULL, + fk_line INTEGER NOT NULL, + FOREIGN KEY(fk_invoice) REFERENCES invoice (pk_invoice), + FOREIGN KEY(fk_line) REFERENCES line (pk_line) +) +``` + +The same output is shown in the DDL tab of `xml2db serve`, next to the diagram, and updates as you edit the +configuration. + +## How the schema is built + +* Each complex type becomes a table, with an integer primary key named `pk_
`. +* Simple elements and attributes become columns, with types derived from the XSD types and their restrictions + (`maxLength`, for instance). +* A child element that occurs at most once is merged into its parent as prefixed columns, such as `customer_name`. +* A repeated child element gets its own table. By default, identical records are stored once, identified by the + `xml2db_record_hash` column, and a relationship table such as `invoice_line` links parents and children. + +[How it works](how_it_works.md#building-a-data-model) describes these rules in detail. + +## Adjusting the schema + +The configuration file changes names, types and structure. For instance, this configuration renames a column, sets an +exact decimal type, and stores lines without deduplication, which replaces the `invoice_line` table with a foreign key +to the parent invoice: + +``` yaml title="model_config.yml" +tables: + invoice: + fields: + customer_name: + rename: customer + line: + reuse: false + fields: + price: + type: Numeric(12, 2) +``` + +``` bash +xml2db render invoices.xsd --format ddl --db-type postgresql --config model_config.yml +``` + +See [Configuring your data model](configuring.md) for all options, and the database pages for backend-specific +advice: [PostgreSQL](databases/postgresql.md), [SQL Server](databases/sql_server.md), +[MySQL and MariaDB](databases/mysql.md), [DuckDB](databases/duckdb.md). + +## From Python + +``` py title="Write the DDL to a file" linenums="1" +from sqlalchemy.dialects import postgresql +from xml2db import DataModel + +data_model = DataModel(xsd_file="invoices.xsd", db_type="postgresql") +with open("invoices.sql", "w") as f: + for statement in data_model.get_all_create_table_statements(): + f.write(f"{statement.compile(dialect=postgresql.dialect())};\n") + for statement in data_model.get_all_create_index_statements(): + f.write(f"{statement.compile(dialect=postgresql.dialect())};\n") +``` + +When `xml2db` loads data, it creates missing tables itself: generating the DDL is not a required step. diff --git a/mkdocs.yml b/mkdocs.yml index 1850517..4133389 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,6 +9,13 @@ site_description: xml2db is a Python package to load XML files into a relational nav: - Introduction: "index.md" - Getting started: "getting_started.md" + - Databases: + - PostgreSQL: "databases/postgresql.md" + - SQL Server: "databases/sql_server.md" + - MySQL and MariaDB: "databases/mysql.md" + - DuckDB: "databases/duckdb.md" + - SQLite, Parquet, Delta Lake: "databases/other_targets.md" + - XSD to SQL tables: "xsd_to_sql.md" - Configuring: "configuring.md" - How it works: "how_it_works.md" - CLI usage: "cli.md" @@ -54,6 +61,8 @@ markdown_extensions: pygments_lang_class: true - pymdownx.inlinehilite - pymdownx.snippets + - pymdownx.tabbed: + alternate_style: true - pymdownx.superfences: custom_fences: - name: mermaid diff --git a/pyproject.toml b/pyproject.toml index 48f5a3c..4423cc9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,13 +8,31 @@ version = "0.15.0" authors = [ { name="Commission de régulation de l'énergie", email="opensource@cre.fr" }, ] -description = "Import complex XML files to a relational database" +description = "Load complex XML files into a relational database, with tables generated from the XSD schema" readme = "README.md" requires-python = ">=3.11" +keywords = [ + "xml", "xsd", "xml-schema", "etl", "elt", "data-engineering", "sql", "ddl", + "sqlalchemy", "postgresql", "mysql", "mariadb", "sql-server", "duckdb", +] classifiers = [ + "Development Status :: 4 - Beta", + "Environment :: Console", + "Intended Audience :: Developers", + "Intended Audience :: Information Technology", + "Intended Audience :: Science/Research", "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", + "Programming Language :: SQL", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", + "Topic :: Database", + "Topic :: Scientific/Engineering :: Information Analysis", + "Topic :: Text Processing :: Markup :: XML", ] dependencies = [ "sqlalchemy>1.4", From dbceb05a8ad55cfd5553a733ac0169ba03ae5a70 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 20:19:17 +0000 Subject: [PATCH 2/9] Docs: emphasize ELT, add dbt page, transform before exporting - Introduction: add a "Load first, transform later" section, show the invoices XSD with the example, and use the same schema for the ERD. - New "Transform with dbt" page: raw schema as dbt sources, staging and flattening models, key-based incremental models, grain choice with deduplication, corrected records (tested with dbt-duckdb and dbt-postgres). - Other targets page: load into DuckDB, transform with SQL, then export the result; incremental Delta export filtered on the grain key; raw table copy kept as a secondary option. - README and DuckDB page: same ELT framing and links. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- README.md | 15 ++- docs/databases/duckdb.md | 7 +- docs/databases/other_targets.md | 200 ++++++++++++++----------------- docs/dbt.md | 203 ++++++++++++++++++++++++++++++++ docs/index.md | 141 +++++++++++----------- mkdocs.yml | 1 + 6 files changed, 380 insertions(+), 187 deletions(-) create mode 100644 docs/dbt.md diff --git a/README.md b/README.md index e7c6c76..d9d90a4 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,10 @@ XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules. It fits naturally into an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) pipeline: it -loads XML files into a relational data model that stays close to the source data while remaining easy to query as -flat database tables. The raw data can then be transformed using [DBT](https://www.getdbt.com/), SQL views, or stored procedures to produce -more user-friendly tables. +loads all the content of XML files into a relational data model that stays close to the source data while remaining +easy to query as flat database tables. Parsing XML is slow compared to SQL, so you do not need to decide upfront what +to extract: explore the loaded data, then build more user-friendly tables with [dbt](https://www.getdbt.com/), SQL +views, or stored procedures, and change them later without parsing the files again. Starting from an XSD schema which represents a given XML structure, `xml2db` builds a data model, i.e. a set of database tables linked to each other by foreign keys relationships. Then, it allows parsing and loading XML files into the @@ -16,7 +17,7 @@ This package uses `sqlalchemy` to interact with the database, so it should work Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). Through DuckDB, the loaded tables can also be -copied to SQLite, Parquet files or Delta Lake. +transformed and exported to SQLite, Parquet files or Delta Lake. **Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!** @@ -28,6 +29,7 @@ copied to SQLite, Parquet files or Delta Lake. | One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables instead of nested columns to explode | | Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | | Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | +| Needs not settled yet: which fields, which aggregations | Extracting the fields you expect to need, and parsing the files again when needs change | Everything loaded once: new questions are answered with SQL | | All the data must be kept, or written back to XML | Hand-written mappings, with the risk of dropping fields | Every element and attribute kept, with a round trip back to XML | `xml2db` needs an XSD schema. See the documentation for an [example](https://cre-dev.github.io/xml2db/#from-xml-to-tables) @@ -35,8 +37,9 @@ and for database-specific guides: [PostgreSQL](https://cre-dev.github.io/xml2db/ [SQL Server](https://cre-dev.github.io/xml2db/databases/sql_server/), [MySQL and MariaDB](https://cre-dev.github.io/xml2db/databases/mysql/), [DuckDB](https://cre-dev.github.io/xml2db/databases/duckdb/), -[SQLite, Parquet and Delta Lake](https://cre-dev.github.io/xml2db/databases/other_targets/). To only generate SQL -tables from an XSD, see [XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd_to_sql/). +[SQLite, Parquet and Delta Lake](https://cre-dev.github.io/xml2db/databases/other_targets/). To transform the loaded +data, see [Transform with dbt](https://cre-dev.github.io/xml2db/dbt/). To only generate SQL tables from an XSD, see +[XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd_to_sql/). ## Installation diff --git a/docs/databases/duckdb.md b/docs/databases/duckdb.md index 857a6e8..87877cd 100644 --- a/docs/databases/duckdb.md +++ b/docs/databases/duckdb.md @@ -58,7 +58,8 @@ Primary keys are filled from DuckDB sequences, named `pk_sequ_
`. Table an characters are shortened with a hash suffix, as for PostgreSQL. This limit comes from the SQLAlchemy integration, not from DuckDB. -## Writing to other targets +## Transforming and exporting -DuckDB extensions can copy the loaded tables to SQLite, Parquet files, Delta Lake, DuckLake and other targets that -`xml2db` does not support directly. See [Other targets through DuckDB](other_targets.md). +Keep the DuckDB file as the raw copy of your XML data, and build the tables you need with SQL views or +[dbt](../dbt.md). DuckDB extensions then export them to Parquet files, SQLite, Delta Lake, DuckLake and other targets +that `xml2db` does not support directly: see [SQLite, Parquet or Delta Lake](other_targets.md). diff --git a/docs/databases/other_targets.md b/docs/databases/other_targets.md index bc9135a..8fb4a2f 100644 --- a/docs/databases/other_targets.md +++ b/docs/databases/other_targets.md @@ -1,154 +1,130 @@ --- title: "Load XML into SQLite, Parquet or Delta Lake" -description: "Use xml2db with DuckDB as a staging database to load complex XML files into SQLite, Parquet files, Delta Lake, DuckLake or Iceberg tables, with full or incremental copies." +description: "Use xml2db with DuckDB to keep all the data of complex XML files, transform it with SQL, and export the resulting tables to Parquet files, SQLite, DuckLake, Delta Lake or Iceberg." --- # Load XML into SQLite, Parquet or Delta Lake `xml2db` merges each file into existing tables: it matches records by hash, generates keys and links rows through -foreign keys. This needs a database with sequences or identity columns, unique constraints and `UPDATE` statements. Files -and table formats such as Parquet or Delta Lake do not offer this, and SQLite is not supported. +foreign keys. This needs a database with sequences or identity columns, unique constraints and `UPDATE` statements. +Files and table formats such as Parquet or Delta Lake do not offer this, and SQLite is not supported. -[DuckDB](duckdb.md) fills the gap. Load the XML files into a DuckDB file, which keeps the state needed for -deduplication, then copy its tables to the target with a DuckDB extension: +[DuckDB](duckdb.md) fills the gap, and fits the [ELT approach](../index.md#load-first-transform-later): -| Target | DuckDB extension | Copy method | -|---|---|---| -| SQLite | `sqlite` | `ATTACH ... (TYPE sqlite)`, then `CREATE TABLE ... AS` | -| Parquet files | built in | `COPY ... TO ... (FORMAT parquet)` | -| DuckLake | `ducklake` | `ATTACH 'ducklake:...'`, then `CREATE TABLE ... AS` | -| Delta Lake | `delta` | `ATTACH ... (TYPE delta)`, then `CREATE TABLE` and `INSERT` | -| Iceberg | `iceberg` | Through an Iceberg REST catalog, see the [DuckDB documentation](https://duckdb.org/docs/current/core_extensions/iceberg/overview) | +1. load all the XML data into a DuckDB file, which keeps every element and the state needed for deduplication; +2. explore it, then join, filter and flatten it with SQL into the tables you need; +3. export these tables to the target with a DuckDB extension. -The examples below were tested with DuckDB 2.0 and local files. Extensions are downloaded on first use. +When the transformation changes, rerun steps 2 and 3 from the DuckDB file, without parsing the XML files again. -!!! note - Pointing `xml2db` directly at a database attached in DuckDB does not work: DuckDB cannot create the sequences - used for primary keys in attached PostgreSQL or SQLite databases. +The examples below use the invoices schema of the [introduction](../index.md#from-xml-to-tables). They were tested +with DuckDB 2.0 and local files. Extensions are downloaded on first use. -## Load into a staging DuckDB file +## Load everything into DuckDB ``` py title="load.py" linenums="1" from xml2db import DataModel -data_model = DataModel(xsd_file="schema.xsd", connection_string="duckdb:///staging.duckdb") +data_model = DataModel(xsd_file="invoices.xsd", connection_string="duckdb:///invoices.duckdb") for xml_file in xml_files: data_model.parse_xml(xml_file).insert_into_target_tables() data_model.engine.dispose() ``` -Keep `staging.duckdb` between runs: it holds the hashes used to store each record once. Starting from an empty file -restarts deduplication and key numbering. +Keep `invoices.duckdb` between runs: it holds all the source data, and the hashes used to store each record once. -## Copy all tables +## Transform with SQL -The function below lists the tables of the staging file and copies them to an attached database, replacing previous -copies. Run it once loading is finished, from a single process. +Build the tables your users need from the xml2db tables, for instance one row per invoice line, with proper types: -``` py title="copy_all.py" linenums="1" +``` py title="transform.py" linenums="1" import duckdb +con = duckdb.connect("invoices.duckdb") +con.execute(""" + CREATE OR REPLACE VIEW invoice_lines AS + SELECT + i.pk_invoice, + i.id AS invoice_id, + CAST(i.date AS DATE) AS invoice_date, + i.customer_name, + i.customer_country, + l.sku, + l.quantity, + CAST(l.price AS DECIMAL(12, 2)) AS price + FROM invoice AS i + JOIN invoice_line AS il ON il.fk_invoice = i.pk_invoice + JOIN line AS l ON l.pk_line = il.fk_line +""") +``` -def staging_tables(con): - """List the tables xml2db created in the staging DuckDB file.""" - return [ - row[0] - for row in con.execute( - "SELECT table_name FROM duckdb_tables() " - "WHERE database_name = current_database() AND schema_name = 'main'" - ).fetchall() - ] +For more than a few views, [dbt](../dbt.md) organizes these transformations, tests them and makes them incremental. +## Export -def copy_all(con, target): - """Copy every table to an attached database, replacing previous copies.""" - for table in staging_tables(con): - con.execute(f'CREATE OR REPLACE TABLE {target}."{table}" AS FROM "{table}"') +| Target | DuckDB extension | Export | +|---|---|---| +| Parquet files | built in | `COPY invoice_lines TO 'invoice_lines.parquet' (FORMAT parquet)` | +| SQLite | `sqlite` | `ATTACH 'analytics.sqlite' AS lite (TYPE sqlite)`, then `CREATE OR REPLACE TABLE lite.invoice_lines AS FROM invoice_lines` | +| DuckLake | `ducklake` | `ATTACH 'ducklake:analytics.ducklake' AS lake (DATA_PATH 'files/')`, then `CREATE OR REPLACE TABLE lake.invoice_lines AS FROM invoice_lines` | +| Delta Lake | `delta` | `ATTACH ... (TYPE delta)`, then `CREATE TABLE` and `INSERT`, see below | +| Iceberg | `iceberg` | Through an Iceberg REST catalog, see the [DuckDB documentation](https://duckdb.org/docs/current/core_extensions/iceberg/overview) | +SQLite has no date or decimal types: DuckDB stores these columns as text there. -con = duckdb.connect("staging.duckdb") -con.execute("ATTACH 'xml_data.sqlite' AS target (TYPE sqlite)") -copy_all(con, "target") -``` +### Delta Lake and incremental exports -The same function writes to DuckLake, with `ATTACH 'ducklake:xml_data.ducklake' AS target (DATA_PATH 'files/')`. +DuckDB does not support `CREATE TABLE ... AS` for Delta tables: create the table, then insert into it. -For Parquet, write one file per table: +`xml2db` only appends rows to its tables, and new rows get higher keys. Filtering on the key of the table that defines +the grain of the export, `pk_invoice` here, appends new invoices only: -``` py -for table in staging_tables(con): - con.execute(f"COPY \"{table}\" TO 'parquet/{table}.parquet' (FORMAT parquet)") +``` py title="export_delta.py" linenums="1" +con.execute("ATTACH 'delta/invoice_lines' AS delta_lines (TYPE delta)") +if not con.execute( + "SELECT count(*) FROM duckdb_tables() WHERE database_name = 'delta_lines'" +).fetchone()[0]: + con.execute(""" + CREATE TABLE delta_lines.delta_lines ( + pk_invoice INTEGER, invoice_id VARCHAR, invoice_date DATE, customer_name VARCHAR, + customer_country VARCHAR, sku VARCHAR, quantity INTEGER, price DECIMAL(12, 2) + ) + """) +con.execute(""" + INSERT INTO delta_lines + FROM invoice_lines + WHERE pk_invoice > (SELECT coalesce(max(pk_invoice), 0) FROM delta_lines) +""") ``` -Copies made with `CREATE TABLE ... AS` keep columns and their types, as far as the target supports them, but not -primary keys, foreign keys or unique constraints. +The same filter works for DuckLake, SQLite or any attached database with `INSERT INTO`. The +[dbt page](../dbt.md#incremental-models) explains how to choose the key. -## Copy new rows only - -`xml2db` never updates or deletes rows of the target tables: loading a file only appends rows. Keys come from -sequences, so new rows have higher primary keys than existing ones. Relationship tables have no primary key, but a new -row always points to a new parent row, and the parent key is their first column. - -This gives a simple incremental copy: for each table, append the rows whose first column is greater than its highest -value in the target. - -``` py title="copy_new_rows.py" linenums="1" -def copy_new_rows(con, target): - """Append the rows added since the previous copy to an attached database.""" - existing = { - row[0] - for row in con.execute( - "SELECT table_name FROM duckdb_tables() WHERE database_name = ?", [target] - ).fetchall() - } - for table in staging_tables(con): - if table not in existing: - con.execute(f'CREATE TABLE {target}."{table}" AS FROM "{table}"') - continue - # first column: the primary key, or the parent key of a relationship table - key = con.execute( - "SELECT column_name FROM duckdb_columns() WHERE database_name = current_database() " - "AND table_name = ? ORDER BY column_index LIMIT 1", - [table], - ).fetchone()[0] - con.execute( - f'INSERT INTO {target}."{table}" FROM "{table}" ' - f'WHERE "{key}" > (SELECT coalesce(max("{key}"), 0) FROM {target}."{table}")' - ) -``` +Delta write support depends on the DuckDB version and platform: check the +[delta extension documentation](https://duckdb.org/docs/current/core_extensions/delta) for current limitations. +Writing to object storage also needs credentials, set up with DuckDB secrets. -## Delta Lake - -A Delta table is attached on its own, and DuckDB does not support `CREATE TABLE ... AS` for Delta. Create each table -from the staging column types, then append new rows as above: - -``` py title="copy_to_delta.py" linenums="1" -def copy_new_rows_to_delta(con, root): - """Append new rows to one Delta table per staging table, under the root path.""" - for table in staging_tables(con): - con.execute(f"ATTACH '{root}/{table}' AS delta_target (TYPE delta)") - columns = con.execute( - "SELECT column_name, data_type FROM duckdb_columns() " - "WHERE database_name = current_database() AND table_name = ? ORDER BY column_index", - [table], - ).fetchall() - if not con.execute( - "SELECT count(*) FROM duckdb_tables() WHERE database_name = 'delta_target'" - ).fetchone()[0]: - definition = ", ".join(f'"{name}" {data_type}' for name, data_type in columns) - con.execute(f"CREATE TABLE delta_target.delta_target ({definition})") - key = columns[0][0] - con.execute( - f'INSERT INTO delta_target FROM "{table}" ' - f'WHERE "{key}" > (SELECT coalesce(max("{key}"), 0) FROM delta_target)' - ) - con.execute("DETACH delta_target") +## Copying the xml2db tables as they are + +To give another tool the full relational copy, copy every table. Run it once loading is finished, from a single +process: +``` py title="copy_all.py" linenums="1" +import duckdb -con = duckdb.connect("staging.duckdb") -copy_new_rows_to_delta(con, "s3://bucket/xml_data") # or a local directory +con = duckdb.connect("invoices.duckdb") +con.execute("ATTACH 'invoices.sqlite' AS target (TYPE sqlite)") +tables = con.execute( + "SELECT table_name FROM duckdb_tables() " + "WHERE database_name = current_database() AND schema_name = 'main'" +).fetchall() +for (table,) in tables: + con.execute(f'CREATE OR REPLACE TABLE target."{table}" AS FROM "{table}"') ``` -Delta write support depends on the DuckDB version and platform: check the -[delta extension documentation](https://duckdb.org/docs/current/core_extensions/delta) for current limitations. -Writing to object storage also needs credentials, set up with DuckDB secrets. +Copies made with `CREATE TABLE ... AS` keep columns and their types, as far as the target supports them, but not +primary keys, foreign keys or unique constraints. + +!!! note + Pointing `xml2db` directly at a database attached in DuckDB does not work: DuckDB cannot create the sequences + used for primary keys in attached PostgreSQL or SQLite databases. diff --git a/docs/dbt.md b/docs/dbt.md new file mode 100644 index 0000000..5f8f5ac --- /dev/null +++ b/docs/dbt.md @@ -0,0 +1,203 @@ +--- +title: "Transform XML data with dbt" +description: "Use dbt on top of xml2db tables: keep all the XML data as a raw layer, flatten it with staging and mart models, and build incremental models with key-based filters." +--- + +# Transform XML data with dbt + +`xml2db` loads XML files into tables that stay close to the XSD. These tables are a raw layer: they keep all the +source data, and [dbt](https://www.getdbt.com/) builds the tables people query on top of them, with SQL models that +are versioned, tested and documented. + +Parsing XML is the slow part of the pipeline. Once the files are loaded, changing a model and rebuilding it only runs +SQL in the database, so you can start with a few models and add others as questions come, without parsing the files +again. + +The examples below use the invoices schema of the [introduction](index.md#from-xml-to-tables). They were tested with +dbt 1.12 on DuckDB (`dbt-duckdb`) and PostgreSQL (`dbt-postgres`). Adapters also exist for SQL Server and other +databases. + +## Loading into a raw schema + +Load the files into a dedicated schema, and record the file name in a [metadata column](configuring.md#model-configuration): + +``` py title="load.py" linenums="1" +import sqlalchemy +from xml2db import DataModel + +data_model = DataModel( + xsd_file="invoices.xsd", + connection_string="duckdb:///warehouse.duckdb", + db_schema="xml_raw", + model_config={ + "metadata_columns": [{"name": "input_file_path", "type": sqlalchemy.String(256)}] + }, +) +for xml_file in xml_files: + data_model.parse_xml(xml_file, metadata={"input_file_path": xml_file}).insert_into_target_tables() +``` + +Declare the xml2db tables as dbt sources, and write dbt models to another schema: + +``` yaml title="models/staging/sources.yml" +sources: + - name: xml2db + schema: xml_raw + tables: + - name: invoices + - name: invoices_invoice + - name: invoice + - name: invoice_line + - name: line +``` + +``` yaml title="profiles.yml" +invoices: + target: dev + outputs: + dev: + type: duckdb + path: warehouse.duckdb + schema: analytics +``` + +## Staging models + +Staging models rename columns and fix types. `xml2db` stores XSD `date` values as text, since they may carry a time +zone, and decimals as floating point numbers by default: + +``` sql title="models/staging/stg_invoice.sql" +select + pk_invoice, + id as invoice_id, + cast(date as date) as invoice_date, + customer_name, + customer_country +from {{ source('xml2db', 'invoice') }} +``` + +``` sql title="models/staging/stg_line.sql" +select + pk_line, + sku, + quantity, + cast(price as decimal(12, 2)) as price +from {{ source('xml2db', 'line') }} +``` + +## Flattening + +Repeated elements are linked through relationship tables such as `invoice_line`. Joining through them gives one row +per invoice line: + +``` sql title="models/marts/fct_invoice_lines.sql" +{{ config(materialized='incremental') }} + +select + i.pk_invoice, + i.invoice_id, + i.invoice_date, + i.customer_name, + l.sku, + l.quantity, + l.price, + l.quantity * l.price as amount +from {{ ref('stg_invoice') }} as i +join {{ source('xml2db', 'invoice_line') }} as il on il.fk_invoice = i.pk_invoice +join {{ ref('stg_line') }} as l on l.pk_line = il.fk_line +{% if is_incremental() %} +where i.pk_invoice > (select coalesce(max(pk_invoice), 0) from {{ this }}) +{% endif %} +``` + +## Incremental models + +Three properties of `xml2db` tables make incremental models simple: + +* rows are only appended: `xml2db` never updates or deletes rows in its tables; +* keys increase: a new row gets a higher primary key than all existing rows of its table; +* a new row in a relationship table always belongs to a new parent row, inserted in the same load. + +So an incremental model filters on the primary key of the table that defines its grain, and keeps the rows above the +highest key already in the model, as in `fct_invoice_lines` above. + +Choose that table with deduplication in mind. An invoice found in several files is stored once, so a model driven by +`pk_invoice` holds each invoice once. To keep one row per file and invoice, as received, drive the model from the root +table, which gets one row per loaded file with new content: + +``` sql title="models/marts/fct_received_invoices.sql" +{{ config(materialized='incremental') }} + +select + f.pk_invoices as pk_file, + f.input_file_path, + i.invoice_id, + i.invoice_date +from {{ source('xml2db', 'invoices') }} as f +join {{ source('xml2db', 'invoices_invoice') }} as fi on fi.fk_invoices = f.pk_invoices +join {{ ref('stg_invoice') }} as i on i.pk_invoice = fi.fk_invoice +{% if is_incremental() %} +where f.pk_invoices > (select coalesce(max(pk_file), 0) from {{ this }}) +{% endif %} +``` + +After loading `invoices_1.xml`, with invoices `INV-001` and `INV-002`, then `invoices_2.xml`, with `INV-002` and +`INV-003`: + +=== "fct_invoice_lines" + + | pk_invoice | invoice_id | sku | quantity | amount | + |-----------:|------------|-----|---------:|-------:| + | 1 | INV-001 | X1 | 2 | 19.80 | + | 1 | INV-001 | Y7 | 1 | 24.00 | + | 2 | INV-002 | X1 | 2 | 19.80 | + | 3 | INV-003 | Z9 | 5 | 17.50 | + +=== "fct_received_invoices" + + | pk_file | input_file_path | invoice_id | + |--------:|-----------------|------------| + | 1 | invoices_1.xml | INV-001 | + | 1 | invoices_1.xml | INV-002 | + | 2 | invoices_2.xml | INV-002 | + | 2 | invoices_2.xml | INV-003 | + +Metadata columns are not part of the record hash: a file whose content was already loaded adds no row to the root +table, even under another name, and only the first file name is kept. + +Do not filter on the key of a child table that can be shared: the line `X1` of `INV-002` reuses the row created for +`INV-001`, so its key is not new. + +The filter relies on each load being complete when dbt runs. Run dbt after loads, merge files one at a time as +recommended in [Multiprocessing](how_it_works.md#multiprocessing), and keep the default single transaction of +`insert_into_target_tables`. `dbt run --full-refresh` rebuilds a model from all the raw data at any time. + +## Corrected records + +Deduplication compares whole records, not business keys. When a later file contains a corrected version of an +invoice, the new version is stored as a new row, with a higher key. A model can keep the latest version of each +invoice: + +``` sql title="models/marts/dim_invoice_latest.sql" +select invoice_id, invoice_date, customer_name, customer_country, pk_invoice +from ( + select + *, + row_number() over (partition by invoice_id order by pk_invoice desc) as version_rank + from {{ ref('stg_invoice') }} +) as versions +where version_rank = 1 +``` + +The order of the keys follows the order of the loads: load files in the order they were produced. + +## Exporting from dbt + +With `dbt-duckdb`, the `external` materialization writes a model to a Parquet file: + +``` sql title="models/marts/invoice_lines_parquet.sql" +{{ config(materialized='external', location='invoice_lines.parquet') }} +select * from {{ ref('fct_invoice_lines') }} +``` + +See [SQLite, Parquet or Delta Lake](databases/other_targets.md) for other targets. diff --git a/docs/index.md b/docs/index.md index 24821d4..daeb7d6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -11,18 +11,54 @@ description: "xml2db is a Python package that automatically maps an XSD schema t * it can handle complex XML files which cannot be denormalized into flat tables * it works out of the box, without any custom mapping rules, while still configurable. -`xml2db` fits naturally into an ETL or [ELT (Extract, Load, Transform)](https://docs.getdbt.com/terms/elt) pipeline. It loads -XML files into a relational model that stays close to the source data while remaining easy to query as flat database -tables. - -The raw data can then be transformed using [DBT](https://www.getdbt.com/), SQL views, or stored procedures to produce -more user-friendly tables. +`xml2db` fits naturally into an [ELT (Extract, Load, Transform)](https://docs.getdbt.com/terms/elt) pipeline: it +loads all the content of XML files into tables that stay close to the source, and the transformations happen later, +in the database. See the [Getting started](getting_started.md) guide for installation, CLI usage, and the Python API. ## From XML to tables -Take this XSD-described file, where each invoice has a customer and repeated lines: +Take this file, where each invoice has a customer and repeated lines, described by an XSD schema: + +??? example "invoices.xsd" + + ``` xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ``` ``` xml title="invoices.xml" @@ -78,6 +114,19 @@ JOIN invoice_line il ON il.fk_invoice = i.pk_invoice JOIN line l ON l.pk_line = il.fk_line; ``` +## Load first, transform later + +Parsing XML is slow compared to running SQL on tables already in a database. Since `xml2db` loads every element and +attribute, you do not need to decide upfront which data to extract and how to reshape it: + +* explore the loaded tables with SQL before choosing transformations; +* when a new question comes, or the transformation logic changes, rerun SQL instead of parsing the files again; +* keep the `xml2db` tables as a raw layer, and build the tables people query on top of them with + [dbt](dbt.md), SQL views or stored procedures. + +The same applies when the final tables live elsewhere: load into [DuckDB](databases/duckdb.md), transform with SQL, +then [export the result](databases/other_targets.md) to Parquet files, Delta Lake or SQLite. + ## When to use xml2db Many XML files do not need `xml2db`. The table below compares it with common alternatives. @@ -88,6 +137,7 @@ Many XML files do not need `xml2db`. The table below compares it with common alt | One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables instead of nested columns to explode | | Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | | Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | +| Needs not settled yet: which fields, which aggregations | Extracting the fields you expect to need, and parsing the files again when needs change | Everything loaded once: new questions are answered with SQL | | All the data must be kept, or written back to XML | Hand-written mappings, with the risk of dropping fields | Every element and attribute kept, with a round trip back to XML | `xml2db` needs an XSD schema. Files are parsed in memory: files around 500 MB work in practice. Recursive schemas, @@ -105,7 +155,8 @@ advice: * [DuckDB](databases/duckdb.md) * [SQLite, Parquet, Delta Lake and other targets through DuckDB](databases/other_targets.md) -To only generate `CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd_to_sql.md). +To build transformed tables on top of the loaded data, see [Transform with dbt](dbt.md). To only generate +`CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd_to_sql.md). ## Visualising data models @@ -113,72 +164,30 @@ To only generate `CREATE TABLE` statements from an XSD, see [XSD to SQL tables]( [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) to represent tables and their relationships. You can see this representation and others through the interactive configuration panel in your browser (`xml2db serve`). -It looks like this: +For the invoices schema above, it looks like this: ```mermaid erDiagram - Unavailability_MarketDocument ||--o{ TimeSeries : "TimeSeries*" - Unavailability_MarketDocument ||--|{ Reason : "Reason*" - Unavailability_MarketDocument { - string mRID - string revisionNumber - NMTOKEN type - NMTOKEN process_processType - dateTime createdDateTime - string sender_MarketParticipant_mRID - NMTOKEN sender_MarketParticipant_marketRole_type - string receiver_MarketParticipant_mRID - NMTOKEN receiver_MarketParticipant_marketRole_type - string unavailability_Time_Period_timeInterval_start - string unavailability_Time_Period_timeInterval_end - NMTOKEN docStatus_value + invoices ||--|{ invoice : "invoice*" + invoices { } - TimeSeries ||--o{ Available_Period : "Available_Period*" - TimeSeries ||--o{ Available_Period : "WindPowerFeedin_Period*" - TimeSeries ||--o{ Asset_RegisteredResource : "Asset_RegisteredResource*" - TimeSeries ||--o{ Reason : "Reason*" - TimeSeries { - string mRID - NMTOKEN businessType - string biddingZone_Domain_mRID - string in_Domain_mRID - string out_Domain_mRID - date start_DateAndOrTime_date - time start_DateAndOrTime_time - date end_DateAndOrTime_date - time end_DateAndOrTime_time - NMTOKEN quantity_Measure_Unit_name - NMTOKEN curveType - string production_RegisteredResource_mRID - string production_RegisteredResource_name - string production_RegisteredResource_location_name - NMTOKEN production_RegisteredResource_pSRType_psrType - string production_RegisteredResource_pSRType_powerSystemResources_mRID - string production_RegisteredResource_pSRType_powerSystemResources_name - float production_RegisteredResource_pSRType_powerSystemResources_nominalP + invoice ||--|{ line : "line*" + invoice { + string id + date date + string customer_name + string customer_country } - Available_Period ||--|{ Point : "Point*" - Available_Period { - string timeInterval_start - string timeInterval_end - duration resolution - } - Point { - integer position - decimal quantity - } - Asset_RegisteredResource { - string mRID - string name - NMTOKEN asset_PSRType_psrType - string location_name - } - Reason { - NMTOKEN code - string text + line { + string sku + integer quantity + decimal price } ``` +Each box is a table with its columns and XSD types. The asterisk on `line*` means that a relationship table, +`invoice_line`, links the two tables, since a line can be shared by several invoices. + ## How to contribute to this project `xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex diff --git a/mkdocs.yml b/mkdocs.yml index 4133389..265d158 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -15,6 +15,7 @@ nav: - MySQL and MariaDB: "databases/mysql.md" - DuckDB: "databases/duckdb.md" - SQLite, Parquet, Delta Lake: "databases/other_targets.md" + - Transform with dbt: "dbt.md" - XSD to SQL tables: "xsd_to_sql.md" - Configuring: "configuring.md" - How it works: "how_it_works.md" From c7a5b232c8fb1b4e165722c81605f6a680a1378e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 20:37:09 +0000 Subject: [PATCH 3/9] Raise a clear error when reuse is disabled on the root table Setting "reuse": False on the root table failed with an AttributeError while building the model. The root table is always deduplicated so that loads are idempotent, so raise a DataModelConfigError explaining it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- docs/configuring.md | 3 +++ src/xml2db/model.py | 5 +++++ tests/test_config.py | 7 +++++++ 3 files changed, 15 insertions(+) diff --git a/docs/configuring.md b/docs/configuring.md index 54c4e51..f5d29b2 100644 --- a/docs/configuring.md +++ b/docs/configuring.md @@ -391,6 +391,9 @@ records. Configuration: `"reuse":` `True` (default) or `False` (disable) +The root table is always deduplicated, so that loading the same file twice stores it once: setting `"reuse": False` on +it raises an error. + !!! example Disabling deduplication for a given table: ``` python diff --git a/src/xml2db/model.py b/src/xml2db/model.py index 920db4b..717b117 100644 --- a/src/xml2db/model.py +++ b/src/xml2db/model.py @@ -197,6 +197,11 @@ def _create_table_model( table_config = self.tables_config.get(table_name, {}) if table_config is None: table_config = {} + if is_root_table and not table_config.get("reuse", True): + raise DataModelConfigError( + f"'reuse: false' is not supported for the root table '{table_name}': the root table is always " + f"deduplicated, so that loading the same file twice stores it once" + ) if table_config.get("reuse", True): return DataModelTableReused( table_name, diff --git a/tests/test_config.py b/tests/test_config.py index d6333e5..8fa74d7 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -168,6 +168,7 @@ def test_invalid_transform_value_raises(self): with pytest.raises(DataModelConfigError, match="Invalid 'transform'"): self._make_model({"transform": "none"}) + def test_transform_false_per_field_auto_overrides_global(self): # Global false but explicit per-field "auto" should restore default elevation m_off = self._make_model({"transform": False}) @@ -185,6 +186,12 @@ def test_transform_false_per_field_auto_overrides_global(self): ) +def test_root_table_without_reuse_raises(): + # the root table is always deduplicated, so that loads are idempotent + with pytest.raises(DataModelConfigError, match="root table 'orders'"): + DataModel(xsd_file=ORDERS_XSD, model_config={"tables": {"orders": {"reuse": False}}}) + + # --------------------------------------------------------------------------- # Field-level transform: "auto" and choice_transform: "auto" # --------------------------------------------------------------------------- From a154720afe84dad97e82828fd4ca363f71acff0f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 20:55:04 +0000 Subject: [PATCH 4/9] Docs: focus export targets on Parquet, lakehouse tables and servers - Export page: drop SQLite; cover Parquet (with partitioning and object storage), loading into cloud warehouses from Parquet, Delta Lake, Iceberg, DuckLake, publishing transformed tables to a server database, DataFrames, and CSV/JSON/Excel extracts. Untested paths are worded as such. - Remove SQLite mentions from README, home page and DuckDB page. - Rephrase "X, not Y" style contrasts, and add a writing style rule against them in CLAUDE.md. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- CLAUDE.md | 1 + README.md | 6 +- docs/databases/duckdb.md | 8 +- docs/databases/mysql.md | 4 +- docs/databases/other_targets.md | 139 ++++++++++++++++++++++---------- docs/databases/postgresql.md | 2 +- docs/dbt.md | 8 +- docs/index.md | 8 +- mkdocs.yml | 2 +- 9 files changed, 118 insertions(+), 60 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e42d2d0..4a08371 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,6 +81,7 @@ Check that the bug reproduces before the fix and is gone after it. - After any code change, check whether docstrings, inline docs, or `docs/` pages need updating and update them as part of the same task. - After updating `docs/`, verify it builds without errors: `python -m mkdocs build --strict`. In strict mode warnings are also errors, matching CI. - Write docstrings and documentation concisely and directly. No em-dashes or en-dashes; use commas, colons, or plain sentences instead. Avoid AI-sounding phrasing such as bold lead-ins ("**Why this matters:**"), filler constructions ("it is worth noting", "this allows", "this ensures"), and verbose qualifiers. +- Avoid contrast constructions used for emphasis, such as "X, not Y", "not X but Y", "X rather than Y" or "X instead of Y". State what is true directly; keep a comparison only when the reader needs both sides. ### Key configuration options (`model_config`) diff --git a/README.md b/README.md index d9d90a4..d6087f9 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ This package uses `sqlalchemy` to interact with the database, so it should work Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). Through DuckDB, the loaded tables can also be -transformed and exported to SQLite, Parquet files or Delta Lake. +transformed and exported to Parquet files, Delta Lake or Iceberg. **Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!** @@ -26,7 +26,7 @@ transformed and exported to SQLite, Parquet files or Delta Lake. | Situation | Usually enough | What xml2db adds | |---|---|---| | A few fields from flat, record-like XML | `pandas.read_xml`, a short `lxml` script, or the XML functions of your database | Little: these are simpler | -| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables instead of nested columns to explode | +| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables, with no nested columns to explode | | Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | | Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | | Needs not settled yet: which fields, which aggregations | Extracting the fields you expect to need, and parsing the files again when needs change | Everything loaded once: new questions are answered with SQL | @@ -37,7 +37,7 @@ and for database-specific guides: [PostgreSQL](https://cre-dev.github.io/xml2db/ [SQL Server](https://cre-dev.github.io/xml2db/databases/sql_server/), [MySQL and MariaDB](https://cre-dev.github.io/xml2db/databases/mysql/), [DuckDB](https://cre-dev.github.io/xml2db/databases/duckdb/), -[SQLite, Parquet and Delta Lake](https://cre-dev.github.io/xml2db/databases/other_targets/). To transform the loaded +[Parquet, Delta Lake and Iceberg](https://cre-dev.github.io/xml2db/databases/other_targets/). To transform the loaded data, see [Transform with dbt](https://cre-dev.github.io/xml2db/dbt/). To only generate SQL tables from an XSD, see [XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd_to_sql/). diff --git a/docs/databases/duckdb.md b/docs/databases/duckdb.md index 87877cd..8046179 100644 --- a/docs/databases/duckdb.md +++ b/docs/databases/duckdb.md @@ -55,11 +55,11 @@ load one file at a time, as shown in [Multiprocessing](../how_it_works.md#multip ## Keys and names Primary keys are filled from DuckDB sequences, named `pk_sequ_
`. Table and column names longer than 63 -characters are shortened with a hash suffix, as for PostgreSQL. This limit comes from the SQLAlchemy integration, not -from DuckDB. +characters are shortened with a hash suffix, as for PostgreSQL. This limit comes from the SQLAlchemy integration: DuckDB +itself accepts longer names. ## Transforming and exporting Keep the DuckDB file as the raw copy of your XML data, and build the tables you need with SQL views or -[dbt](../dbt.md). DuckDB extensions then export them to Parquet files, SQLite, Delta Lake, DuckLake and other targets -that `xml2db` does not support directly: see [SQLite, Parquet or Delta Lake](other_targets.md). +[dbt](../dbt.md). DuckDB then exports them to Parquet files, Delta Lake, Iceberg and other targets that `xml2db` does +not support directly: see [Parquet, Delta Lake or Iceberg](other_targets.md). diff --git a/docs/databases/mysql.md b/docs/databases/mysql.md index a204bd5..2449078 100644 --- a/docs/databases/mysql.md +++ b/docs/databases/mysql.md @@ -51,8 +51,8 @@ following batches of the same `DataModel`. ## String sizes -Strings without a `maxLength` in the XSD become `VARCHAR(255)` on MySQL, instead of `VARCHAR(1000)` on other -backends. Longer values are rejected in strict SQL mode, which is the default. Override the type of free-text fields: +Strings without a `maxLength` in the XSD become `VARCHAR(255)` on MySQL, while other backends use +`VARCHAR(1000)`. Longer values are rejected in strict SQL mode, which is the default. Override the type of free-text fields: ``` yaml title="model_config.yml" tables: diff --git a/docs/databases/other_targets.md b/docs/databases/other_targets.md index 8fb4a2f..faa7ccd 100644 --- a/docs/databases/other_targets.md +++ b/docs/databases/other_targets.md @@ -1,24 +1,26 @@ --- -title: "Load XML into SQLite, Parquet or Delta Lake" -description: "Use xml2db with DuckDB to keep all the data of complex XML files, transform it with SQL, and export the resulting tables to Parquet files, SQLite, DuckLake, Delta Lake or Iceberg." +title: "Load XML into Parquet, Delta Lake or Iceberg" +description: "Use xml2db with DuckDB to keep all the data of complex XML files, transform it with SQL, and export the resulting tables to Parquet files, object storage, Delta Lake, Iceberg, a server database or DataFrames." --- -# Load XML into SQLite, Parquet or Delta Lake +# Load XML into Parquet, Delta Lake or Iceberg `xml2db` merges each file into existing tables: it matches records by hash, generates keys and links rows through -foreign keys. This needs a database with sequences or identity columns, unique constraints and `UPDATE` statements. -Files and table formats such as Parquet or Delta Lake do not offer this, and SQLite is not supported. +foreign keys. This needs a database with sequences or identity columns, unique constraints and `UPDATE` statements, +which files and table formats such as Parquet or Delta Lake do not offer. [DuckDB](duckdb.md) fills the gap, and fits the [ELT approach](../index.md#load-first-transform-later): 1. load all the XML data into a DuckDB file, which keeps every element and the state needed for deduplication; 2. explore it, then join, filter and flatten it with SQL into the tables you need; -3. export these tables to the target with a DuckDB extension. +3. export these tables with DuckDB. -When the transformation changes, rerun steps 2 and 3 from the DuckDB file, without parsing the XML files again. +When the transformation changes, rerun steps 2 and 3 from the DuckDB file: the XML files are parsed once. -The examples below use the invoices schema of the [introduction](../index.md#from-xml-to-tables). They were tested -with DuckDB 2.0 and local files. Extensions are downloaded on first use. +The examples below use the invoices schema of the [introduction](../index.md#from-xml-to-tables). The single-file +Parquet, Delta Lake, DuckLake and PostgreSQL examples were tested with DuckDB 2.0 and local files. The other examples +follow the DuckDB documentation but were not tested with `xml2db`: they should work, check the linked pages for +details. ## Load everything into DuckDB @@ -35,7 +37,7 @@ Keep `invoices.duckdb` between runs: it holds all the source data, and the hashe ## Transform with SQL -Build the tables your users need from the xml2db tables, for instance one row per invoice line, with proper types: +Build the tables your users need from the `xml2db` tables, for instance one row per invoice line, with proper types: ``` py title="transform.py" linenums="1" import duckdb @@ -60,19 +62,49 @@ con.execute(""" For more than a few views, [dbt](../dbt.md) organizes these transformations, tests them and makes them incremental. -## Export +## Parquet files -| Target | DuckDB extension | Export | -|---|---|---| -| Parquet files | built in | `COPY invoice_lines TO 'invoice_lines.parquet' (FORMAT parquet)` | -| SQLite | `sqlite` | `ATTACH 'analytics.sqlite' AS lite (TYPE sqlite)`, then `CREATE OR REPLACE TABLE lite.invoice_lines AS FROM invoice_lines` | -| DuckLake | `ducklake` | `ATTACH 'ducklake:analytics.ducklake' AS lake (DATA_PATH 'files/')`, then `CREATE OR REPLACE TABLE lake.invoice_lines AS FROM invoice_lines` | -| Delta Lake | `delta` | `ATTACH ... (TYPE delta)`, then `CREATE TABLE` and `INSERT`, see below | -| Iceberg | `iceberg` | Through an Iceberg REST catalog, see the [DuckDB documentation](https://duckdb.org/docs/current/core_extensions/iceberg/overview) | +Parquet is read by most data tools: Spark, pandas, Polars, and the loaders of cloud data warehouses. -SQLite has no date or decimal types: DuckDB stores these columns as text there. +``` py +con.execute("COPY invoice_lines TO 'invoice_lines.parquet' (FORMAT parquet)") +``` + +Large tables can be split into one directory per value of a column, which should let tools reading the files skip +partitions: + +``` py +con.execute(""" + COPY (SELECT *, year(invoice_date) AS invoice_year FROM invoice_lines) + TO 'invoice_lines' (FORMAT parquet, PARTITION_BY (invoice_year)) +""") +``` + +See [Parquet export](https://duckdb.org/docs/current/data/parquet/overview) and +[partitioned writes](https://duckdb.org/docs/current/data/partitioning/partitioned_writes) in the DuckDB +documentation. + +### Object storage + +With the `httpfs` extension and a secret holding the credentials, the same `COPY` statement should write directly to +S3 or S3-compatible storage: + +``` py +con.execute("CREATE SECRET (TYPE s3, PROVIDER credential_chain)") +con.execute("COPY invoice_lines TO 's3://my-bucket/invoice_lines.parquet' (FORMAT parquet)") +``` + +The [S3 API page](https://duckdb.org/docs/current/core_extensions/httpfs/s3api) of the DuckDB documentation +describes the supported providers. For Azure Blob Storage or Google Cloud Storage, check the `azure` and `httpfs` +extension pages for write support. + +### Cloud data warehouses + +Cloud data warehouses load Parquet files from object storage with their own tools, such as `COPY INTO` statements or +external tables. Exporting Parquet files to a bucket, then loading them with these tools, should be the most reliable +path to a warehouse. -### Delta Lake and incremental exports +## Delta Lake DuckDB does not support `CREATE TABLE ... AS` for Delta tables: create the table, then insert into it. @@ -97,34 +129,59 @@ con.execute(""" """) ``` -The same filter works for DuckLake, SQLite or any attached database with `INSERT INTO`. The -[dbt page](../dbt.md#incremental-models) explains how to choose the key. +The [dbt page](../dbt.md#incremental-models) explains how to choose the key. Delta write support depends on the +DuckDB version and platform: check the [delta extension documentation](https://duckdb.org/docs/current/core_extensions/delta) +for current limitations. -Delta write support depends on the DuckDB version and platform: check the -[delta extension documentation](https://duckdb.org/docs/current/core_extensions/delta) for current limitations. -Writing to object storage also needs credentials, set up with DuckDB secrets. +## Iceberg -## Copying the xml2db tables as they are +DuckDB writes Iceberg tables through an Iceberg REST catalog. Once the catalog is attached as described in the +[DuckDB documentation](https://duckdb.org/docs/current/core_extensions/iceberg/overview), `CREATE TABLE` and +`INSERT INTO` statements on the attached catalog should allow the same full or incremental exports as for Delta Lake. -To give another tool the full relational copy, copy every table. Run it once loading is finished, from a single -process: +## DuckLake -``` py title="copy_all.py" linenums="1" -import duckdb +If your tools already rely on DuckDB, a [DuckLake](https://duckdb.org/docs/current/core_extensions/ducklake) catalog +gives a table format with snapshots on top of Parquet files: -con = duckdb.connect("invoices.duckdb") -con.execute("ATTACH 'invoices.sqlite' AS target (TYPE sqlite)") -tables = con.execute( - "SELECT table_name FROM duckdb_tables() " - "WHERE database_name = current_database() AND schema_name = 'main'" -).fetchall() -for (table,) in tables: - con.execute(f'CREATE OR REPLACE TABLE target."{table}" AS FROM "{table}"') +``` py +con.execute("ATTACH 'ducklake:analytics.ducklake' AS lake (DATA_PATH 'files/')") +con.execute("CREATE OR REPLACE TABLE lake.invoice_lines AS FROM invoice_lines") +``` + +## Publishing to a server database + +`xml2db` loads [PostgreSQL](postgresql.md) and [MySQL](mysql.md) directly. Another option is to keep the +`xml2db` tables in a local DuckDB file, and publish only the transformed tables to a shared server. The server +holds only the tables people query, and the parsing work stays on the machine running `xml2db`. + +``` py +con.execute("ATTACH 'host=db.example.com dbname=analytics user=etl' AS pg (TYPE postgres)") +con.execute("CREATE OR REPLACE TABLE pg.public.invoice_lines AS FROM invoice_lines") +``` + +The `mysql` extension should allow the same with `ATTACH ... (TYPE mysql)`. Tables created this way have no primary +key, foreign keys or indexes: add them on the server if needed. + +## DataFrames + +To continue the analysis in Python, DuckDB results should convert directly to pandas or Polars DataFrames: + +``` py +invoice_lines = con.sql("FROM invoice_lines").df() # pandas; .pl() for Polars ``` -Copies made with `CREATE TABLE ... AS` keep columns and their types, as far as the target supports them, but not -primary keys, foreign keys or unique constraints. +## Other formats + +DuckDB also writes CSV and JSON files with `COPY`, and the `excel` extension should allow writing `.xlsx` files for +small extracts (Excel sheets hold about one million rows): + +``` py +con.execute("INSTALL excel") +con.execute("LOAD excel") +con.execute("COPY invoice_lines TO 'invoice_lines.xlsx' (FORMAT xlsx, HEADER true)") +``` !!! note Pointing `xml2db` directly at a database attached in DuckDB does not work: DuckDB cannot create the sequences - used for primary keys in attached PostgreSQL or SQLite databases. + used for primary keys in attached databases such as PostgreSQL. diff --git a/docs/databases/postgresql.md b/docs/databases/postgresql.md index e729481..ce9646b 100644 --- a/docs/databases/postgresql.md +++ b/docs/databases/postgresql.md @@ -43,7 +43,7 @@ With `psycopg2` or `psycopg` (version 3), staging tables are filled with `COPY . in-memory CSV buffer and streamed to the server, with no temporary file. `COPY` is used for every batch, whatever its size. Other drivers, such as `pg8000`, fall back to standard `executemany` inserts. -`insert_into_target_tables(bulk_load=True)` raises an error instead of falling back when the driver does not support +`insert_into_target_tables(bulk_load=True)` raises an error when the driver does not support `COPY`. See [Bulk loading](../how_it_works.md#bulk-loading) for the other options. ## Table and column names diff --git a/docs/dbt.md b/docs/dbt.md index 5f8f5ac..2a94aa1 100644 --- a/docs/dbt.md +++ b/docs/dbt.md @@ -174,9 +174,9 @@ recommended in [Multiprocessing](how_it_works.md#multiprocessing), and keep the ## Corrected records -Deduplication compares whole records, not business keys. When a later file contains a corrected version of an -invoice, the new version is stored as a new row, with a higher key. A model can keep the latest version of each -invoice: +Deduplication compares whole records, so two versions of an invoice with the same identifier are two different +records. When a later file contains a corrected version of an invoice, it is stored as a new row, with a higher key. +A model can keep the latest version of each invoice: ``` sql title="models/marts/dim_invoice_latest.sql" select invoice_id, invoice_date, customer_name, customer_country, pk_invoice @@ -200,4 +200,4 @@ With `dbt-duckdb`, the `external` materialization writes a model to a Parquet fi select * from {{ ref('fct_invoice_lines') }} ``` -See [SQLite, Parquet or Delta Lake](databases/other_targets.md) for other targets. +See [Parquet, Delta Lake or Iceberg](databases/other_targets.md) for other targets. diff --git a/docs/index.md b/docs/index.md index daeb7d6..8b70436 100644 --- a/docs/index.md +++ b/docs/index.md @@ -120,12 +120,12 @@ Parsing XML is slow compared to running SQL on tables already in a database. Sin attribute, you do not need to decide upfront which data to extract and how to reshape it: * explore the loaded tables with SQL before choosing transformations; -* when a new question comes, or the transformation logic changes, rerun SQL instead of parsing the files again; +* when a new question comes, or the transformation logic changes, rerun the SQL on the loaded tables; * keep the `xml2db` tables as a raw layer, and build the tables people query on top of them with [dbt](dbt.md), SQL views or stored procedures. The same applies when the final tables live elsewhere: load into [DuckDB](databases/duckdb.md), transform with SQL, -then [export the result](databases/other_targets.md) to Parquet files, Delta Lake or SQLite. +then [export the result](databases/other_targets.md) to Parquet files, Delta Lake or Iceberg. ## When to use xml2db @@ -134,7 +134,7 @@ Many XML files do not need `xml2db`. The table below compares it with common alt | Situation | Usually enough | What xml2db adds | |---|---|---| | A few fields from flat, record-like XML | `pandas.read_xml`, a short `lxml` script, or the XML functions of your database | Little: these are simpler | -| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables instead of nested columns to explode | +| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables, with no nested columns to explode | | Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | | Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | | Needs not settled yet: which fields, which aggregations | Extracting the fields you expect to need, and parsing the files again when needs change | Everything loaded once: new questions are answered with SQL | @@ -153,7 +153,7 @@ advice: * [SQL Server](databases/sql_server.md) * [MySQL and MariaDB](databases/mysql.md) * [DuckDB](databases/duckdb.md) -* [SQLite, Parquet, Delta Lake and other targets through DuckDB](databases/other_targets.md) +* [Parquet, Delta Lake, Iceberg and other targets through DuckDB](databases/other_targets.md) To build transformed tables on top of the loaded data, see [Transform with dbt](dbt.md). To only generate `CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd_to_sql.md). diff --git a/mkdocs.yml b/mkdocs.yml index 265d158..1537e68 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,7 +14,7 @@ nav: - SQL Server: "databases/sql_server.md" - MySQL and MariaDB: "databases/mysql.md" - DuckDB: "databases/duckdb.md" - - SQLite, Parquet, Delta Lake: "databases/other_targets.md" + - Parquet, Delta Lake, Iceberg: "databases/other_targets.md" - Transform with dbt: "dbt.md" - XSD to SQL tables: "xsd_to_sql.md" - Configuring: "configuring.md" From 953ab408c4724dbb92d1685e4f9315253f2f5cf2 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 21:00:01 +0000 Subject: [PATCH 5/9] Docs: mark DataFrame, CSV, JSON and Excel exports as tested Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- docs/databases/other_targets.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/databases/other_targets.md b/docs/databases/other_targets.md index faa7ccd..881cef2 100644 --- a/docs/databases/other_targets.md +++ b/docs/databases/other_targets.md @@ -17,10 +17,10 @@ which files and table formats such as Parquet or Delta Lake do not offer. When the transformation changes, rerun steps 2 and 3 from the DuckDB file: the XML files are parsed once. -The examples below use the invoices schema of the [introduction](../index.md#from-xml-to-tables). The single-file -Parquet, Delta Lake, DuckLake and PostgreSQL examples were tested with DuckDB 2.0 and local files. The other examples -follow the DuckDB documentation but were not tested with `xml2db`: they should work, check the linked pages for -details. +The examples below use the invoices schema of the [introduction](../index.md#from-xml-to-tables). The examples +for single Parquet files, Delta Lake, DuckLake, PostgreSQL, DataFrames, CSV, JSON and Excel were tested with DuckDB 2.0 +and local files. The examples for partitioned Parquet files, object storage, Iceberg and MySQL follow the DuckDB +documentation but were not tested with `xml2db`: they should work, check the linked pages for details. ## Load everything into DuckDB @@ -165,18 +165,24 @@ key, foreign keys or indexes: add them on the server if needed. ## DataFrames -To continue the analysis in Python, DuckDB results should convert directly to pandas or Polars DataFrames: +To continue the analysis in Python, convert DuckDB results to pandas or Polars DataFrames: ``` py -invoice_lines = con.sql("FROM invoice_lines").df() # pandas; .pl() for Polars +invoice_lines = con.sql("FROM invoice_lines").df() # pandas +invoice_lines = con.sql("FROM invoice_lines").pl() # Polars ``` +pandas converts `DECIMAL` columns to floating point numbers, while Polars keeps them as decimals. + ## Other formats -DuckDB also writes CSV and JSON files with `COPY`, and the `excel` extension should allow writing `.xlsx` files for -small extracts (Excel sheets hold about one million rows): +DuckDB also writes CSV and JSON files, and `.xlsx` files with the `excel` extension, for small extracts (Excel sheets +hold about one million rows): ``` py +con.execute("COPY invoice_lines TO 'invoice_lines.csv'") +con.execute("COPY invoice_lines TO 'invoice_lines.json'") # one JSON object per line + con.execute("INSTALL excel") con.execute("LOAD excel") con.execute("COPY invoice_lines TO 'invoice_lines.xlsx' (FORMAT xlsx, HEADER true)") From ed1bd79ece7178ec39241953a4cb79832b0c0c57 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 25 Oct 2026 19:59:27 +0000 Subject: [PATCH 6/9] README: document local testing with DuckDB Show the install command with duckdb_engine and pytz, the full test run against an in-memory DuckDB database, why TZ must be Europe/Paris, how to target another database, and the databases covered by CI. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- README.md | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index d6087f9..e171ecb 100644 --- a/README.md +++ b/README.md @@ -101,24 +101,35 @@ document.insert_into_target_tables() ## Testing -Running the tests requires installing additional development dependencies, after cloning the repo, with: +After cloning the repo, install the package with its development dependencies, plus DuckDB and `pytz` to run the +database tests locally: ```bash -pip install -e .[tests,docs] +pip install -e .[tests,docs] duckdb_engine pytz ``` -Run all tests with the following command: +Run all tests, including database integration tests, against an in-memory DuckDB database: ```bash -python -m pytest +TZ="Europe/Paris" DB_STRING="duckdb:///:memory:" python -m pytest ``` -Integration tests require write access to a PostgreSQL or MS SQL Server database; the connection string is provided as an -environment variable `DB_STRING`. If you want to run only conversion tests that do not require a database you can run: +Round-trip tests compare datetimes read back from the database in the `Europe/Paris` time zone, so `TZ` must be set to +it. + +Integration tests use the database given by the `DB_STRING` connection string. To run them against another database, +for instance PostgreSQL: + +```bash +TZ="Europe/Paris" DB_STRING="postgresql+psycopg2://user:pw@localhost:5432/testdb" python -m pytest +``` + +Continuous integration runs them against DuckDB, PostgreSQL, MySQL and MS SQL Server. To run only the tests that do +not need a database: ```bash -pytest -m "not dbtest" -````` +TZ="Europe/Paris" python -m pytest -m "not dbtest" +``` ## Contributing From 791d687ad760354c3b1e33b27f83801bb8f8eb22 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 25 Oct 2026 20:19:43 +0000 Subject: [PATCH 7/9] Docs: retitle the DuckDB export page to match its content Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- README.md | 2 +- docs/databases/duckdb.md | 2 +- docs/databases/other_targets.md | 6 +++--- docs/dbt.md | 2 +- docs/index.md | 4 ++-- mkdocs.yml | 2 +- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index e171ecb..49b786e 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ and for database-specific guides: [PostgreSQL](https://cre-dev.github.io/xml2db/ [SQL Server](https://cre-dev.github.io/xml2db/databases/sql_server/), [MySQL and MariaDB](https://cre-dev.github.io/xml2db/databases/mysql/), [DuckDB](https://cre-dev.github.io/xml2db/databases/duckdb/), -[Parquet, Delta Lake and Iceberg](https://cre-dev.github.io/xml2db/databases/other_targets/). To transform the loaded +[export to files, lakehouses and other databases](https://cre-dev.github.io/xml2db/databases/other_targets/). To transform the loaded data, see [Transform with dbt](https://cre-dev.github.io/xml2db/dbt/). To only generate SQL tables from an XSD, see [XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd_to_sql/). diff --git a/docs/databases/duckdb.md b/docs/databases/duckdb.md index 8046179..bad6d54 100644 --- a/docs/databases/duckdb.md +++ b/docs/databases/duckdb.md @@ -62,4 +62,4 @@ itself accepts longer names. Keep the DuckDB file as the raw copy of your XML data, and build the tables you need with SQL views or [dbt](../dbt.md). DuckDB then exports them to Parquet files, Delta Lake, Iceberg and other targets that `xml2db` does -not support directly: see [Parquet, Delta Lake or Iceberg](other_targets.md). +not support directly: see [Export to files, lakehouses and other databases](other_targets.md). diff --git a/docs/databases/other_targets.md b/docs/databases/other_targets.md index 881cef2..c03b1da 100644 --- a/docs/databases/other_targets.md +++ b/docs/databases/other_targets.md @@ -1,9 +1,9 @@ --- -title: "Load XML into Parquet, Delta Lake or Iceberg" -description: "Use xml2db with DuckDB to keep all the data of complex XML files, transform it with SQL, and export the resulting tables to Parquet files, object storage, Delta Lake, Iceberg, a server database or DataFrames." +title: "Export to files, lakehouses and other databases" +description: "Use xml2db with DuckDB to keep all the data of complex XML files, transform it with SQL, and export the resulting tables to Parquet, CSV or Excel files, object storage, cloud data warehouses, Delta Lake, Iceberg, DuckLake, server databases or DataFrames." --- -# Load XML into Parquet, Delta Lake or Iceberg +# Export to files, lakehouses and other databases `xml2db` merges each file into existing tables: it matches records by hash, generates keys and links rows through foreign keys. This needs a database with sequences or identity columns, unique constraints and `UPDATE` statements, diff --git a/docs/dbt.md b/docs/dbt.md index 2a94aa1..9e90413 100644 --- a/docs/dbt.md +++ b/docs/dbt.md @@ -200,4 +200,4 @@ With `dbt-duckdb`, the `external` materialization writes a model to a Parquet fi select * from {{ ref('fct_invoice_lines') }} ``` -See [Parquet, Delta Lake or Iceberg](databases/other_targets.md) for other targets. +See [Export to files, lakehouses and other databases](databases/other_targets.md) for other targets. diff --git a/docs/index.md b/docs/index.md index 8b70436..e8c8a76 100644 --- a/docs/index.md +++ b/docs/index.md @@ -125,7 +125,7 @@ attribute, you do not need to decide upfront which data to extract and how to re [dbt](dbt.md), SQL views or stored procedures. The same applies when the final tables live elsewhere: load into [DuckDB](databases/duckdb.md), transform with SQL, -then [export the result](databases/other_targets.md) to Parquet files, Delta Lake or Iceberg. +then [export the result](databases/other_targets.md) to files, lakehouse tables or other databases. ## When to use xml2db @@ -153,7 +153,7 @@ advice: * [SQL Server](databases/sql_server.md) * [MySQL and MariaDB](databases/mysql.md) * [DuckDB](databases/duckdb.md) -* [Parquet, Delta Lake, Iceberg and other targets through DuckDB](databases/other_targets.md) +* [Files, lakehouses and other databases, through DuckDB](databases/other_targets.md) To build transformed tables on top of the loaded data, see [Transform with dbt](dbt.md). To only generate `CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd_to_sql.md). diff --git a/mkdocs.yml b/mkdocs.yml index 1537e68..f9df328 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,7 +14,7 @@ nav: - SQL Server: "databases/sql_server.md" - MySQL and MariaDB: "databases/mysql.md" - DuckDB: "databases/duckdb.md" - - Parquet, Delta Lake, Iceberg: "databases/other_targets.md" + - Export through DuckDB: "databases/other_targets.md" - Transform with dbt: "dbt.md" - XSD to SQL tables: "xsd_to_sql.md" - Configuring: "configuring.md" From 26b857d4222d1cb4b08ec624b9b236de38831b2e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 25 Oct 2026 20:32:55 +0000 Subject: [PATCH 8/9] Docs: SEO improvements for titles, URLs, previews and internal links - Rename new pages to hyphenated, descriptive URLs before their first release: databases/sql-server, databases/export, xsd-to-sql. - Align page titles with search queries (home, getting started, CLI, how it works, XSD to SQL) and keep descriptions under 155 characters. - DuckDB page: add a section on reading XML with DuckDB extensions and when xml2db fits better. - Add Open Graph and Twitter tags for link previews. - Add internal links to the CLI, XSD to SQL, database and dbt pages. - Move the theme override out of docs/ so it is no longer published. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012tCdFJo9K4E39QdqU3Mxvv --- README.md | 8 +++--- docs/api/data_model.md | 2 +- docs/api/overview.md | 2 +- docs/cli.md | 6 ++-- docs/configuring.md | 2 +- docs/databases/duckdb.md | 14 ++++++++-- .../databases/{other_targets.md => export.md} | 2 +- docs/databases/mysql.md | 2 +- docs/databases/postgresql.md | 2 +- .../{sql_server.md => sql-server.md} | 2 +- docs/dbt.md | 4 +-- docs/getting_started.md | 16 +++++++++-- docs/how_it_works.md | 6 ++-- docs/index.md | 17 +++++------ docs/overrides/main.html | 5 ---- docs/{xsd_to_sql.md => xsd-to-sql.md} | 8 +++--- mkdocs.yml | 12 ++++---- overrides/main.html | 28 +++++++++++++++++++ 18 files changed, 92 insertions(+), 46 deletions(-) rename docs/databases/{other_targets.md => export.md} (96%) rename docs/databases/{sql_server.md => sql-server.md} (95%) delete mode 100644 docs/overrides/main.html rename docs/{xsd_to_sql.md => xsd-to-sql.md} (93%) create mode 100644 overrides/main.html diff --git a/README.md b/README.md index 49b786e..8a03f7a 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Loading XML files into a relational database +# Load XML files into a SQL database with Python `xml2db` is a Python package that parses and loads XML files into a relational database. It handles complex XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules. @@ -34,12 +34,12 @@ transformed and exported to Parquet files, Delta Lake or Iceberg. `xml2db` needs an XSD schema. See the documentation for an [example](https://cre-dev.github.io/xml2db/#from-xml-to-tables) and for database-specific guides: [PostgreSQL](https://cre-dev.github.io/xml2db/databases/postgresql/), -[SQL Server](https://cre-dev.github.io/xml2db/databases/sql_server/), +[SQL Server](https://cre-dev.github.io/xml2db/databases/sql-server/), [MySQL and MariaDB](https://cre-dev.github.io/xml2db/databases/mysql/), [DuckDB](https://cre-dev.github.io/xml2db/databases/duckdb/), -[export to files, lakehouses and other databases](https://cre-dev.github.io/xml2db/databases/other_targets/). To transform the loaded +[export to files, lakehouses and other databases](https://cre-dev.github.io/xml2db/databases/export/). To transform the loaded data, see [Transform with dbt](https://cre-dev.github.io/xml2db/dbt/). To only generate SQL tables from an XSD, see -[XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd_to_sql/). +[XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd-to-sql/). ## Installation diff --git a/docs/api/data_model.md b/docs/api/data_model.md index 2e03956..0cc24a6 100644 --- a/docs/api/data_model.md +++ b/docs/api/data_model.md @@ -1,6 +1,6 @@ --- title: "DataModel" -description: "API reference for the xml2db DataModel class: constructor parameters, methods for parsing XML files, visualizing the schema as ERD or tree, and generating DDL statements." +description: "API reference for the xml2db DataModel class: constructor parameters, parsing XML files, ERD and tree views of the schema, and DDL generation." --- # DataModel diff --git a/docs/api/overview.md b/docs/api/overview.md index e1d9a29..f2c2f66 100644 --- a/docs/api/overview.md +++ b/docs/api/overview.md @@ -1,6 +1,6 @@ --- title: "API Overview" -description: "Overview of the xml2db Python API: building DataModel objects, parsing and loading XML files into a database, and extracting data back to XML, including multiprocessing patterns." +description: "Overview of the xml2db Python API: building a DataModel, loading XML files into a database, extracting data back to XML, and multiprocessing." --- # API Overview diff --git a/docs/cli.md b/docs/cli.md index 9cd91a0..fba774c 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,6 +1,6 @@ --- -title: "CLI usage" -description: "Reference for the xml2db command-line interface: import XML files, render ERDs and DDL, and launch the interactive browser explorer." +title: "Command line: import XML into a database, render DDL" +description: "Reference for the xml2db command line: import XML files into a database, render diagrams and DDL from an XSD, and explore a schema in the browser." --- # CLI usage @@ -81,6 +81,8 @@ xml2db render schema.xsd --format ddl --db-type postgresql xml2db render schema.xsd --format erd --output diagram.md ``` +[XSD to SQL](xsd-to-sql.md) shows the generated DDL and how to adjust it. + ## xml2db serve Launch an interactive schema explorer in the browser. diff --git a/docs/configuring.md b/docs/configuring.md index f5d29b2..b3e02b0 100644 --- a/docs/configuring.md +++ b/docs/configuring.md @@ -1,6 +1,6 @@ --- title: "Configuring your data model" -description: "Configure xml2db's data model via model_config: override column types, control field elevation, adjust deduplication, simplify XSD choice groups, and add custom indexes." +description: "Configure the tables xml2db derives from an XSD: column types and names, flattening of child elements, deduplication, choice groups and indexes." --- # Configuring your data model diff --git a/docs/databases/duckdb.md b/docs/databases/duckdb.md index bad6d54..26e6762 100644 --- a/docs/databases/duckdb.md +++ b/docs/databases/duckdb.md @@ -1,6 +1,6 @@ --- title: "Load XML into DuckDB" -description: "Load complex XML files into DuckDB tables generated from an XSD schema with xml2db, query them locally without a server, and use DuckDB as a staging database for other targets." +description: "Load complex XML files into DuckDB tables generated from an XSD with xml2db, query them without a server, or read simple XML with DuckDB extensions." --- # Load XML into DuckDB @@ -9,6 +9,16 @@ DuckDB runs inside the Python process and stores a whole database in a single fi relational copy of XML files without any server to set up, which suits local analysis, trying a data model before deploying it, and tests. +## Reading XML in DuckDB + +DuckDB community extensions, such as [webbed](https://duckdb.org/community_extensions/extensions/webbed), add a +`read_xml` function that reads XML files directly in SQL. It turns a repeated element into rows, with nested elements +as `STRUCT` and `LIST` columns, and needs no XSD. It fits quick queries on record-like files. + +`xml2db` fits files described by an XSD, with deep nesting or repeated elements at several levels: it loads them into +normalized tables with keys, stores identical records once, and gives the same tables for every file of a feed. Both +work in the same DuckDB database, so each can be used where it fits. + ## Connecting ``` bash @@ -62,4 +72,4 @@ itself accepts longer names. Keep the DuckDB file as the raw copy of your XML data, and build the tables you need with SQL views or [dbt](../dbt.md). DuckDB then exports them to Parquet files, Delta Lake, Iceberg and other targets that `xml2db` does -not support directly: see [Export to files, lakehouses and other databases](other_targets.md). +not support directly: see [Export to files, lakehouses and other databases](export.md). diff --git a/docs/databases/other_targets.md b/docs/databases/export.md similarity index 96% rename from docs/databases/other_targets.md rename to docs/databases/export.md index c03b1da..548b4c1 100644 --- a/docs/databases/other_targets.md +++ b/docs/databases/export.md @@ -1,6 +1,6 @@ --- title: "Export to files, lakehouses and other databases" -description: "Use xml2db with DuckDB to keep all the data of complex XML files, transform it with SQL, and export the resulting tables to Parquet, CSV or Excel files, object storage, cloud data warehouses, Delta Lake, Iceberg, DuckLake, server databases or DataFrames." +description: "Load XML with xml2db into DuckDB, transform it with SQL, then export to Parquet, CSV or Excel, Delta Lake, Iceberg, warehouses or server databases." --- # Export to files, lakehouses and other databases diff --git a/docs/databases/mysql.md b/docs/databases/mysql.md index 2449078..f5bb765 100644 --- a/docs/databases/mysql.md +++ b/docs/databases/mysql.md @@ -1,6 +1,6 @@ --- title: "Load XML into MySQL and MariaDB" -description: "Load complex XML files into MySQL or MariaDB tables generated from an XSD schema with xml2db: drivers, LOAD DATA LOCAL INFILE bulk loading, identifier limits and string sizes." +description: "Load complex XML files into MySQL or MariaDB tables generated from an XSD with xml2db: drivers, LOAD DATA bulk loading, names and string sizes." --- # Load XML into MySQL and MariaDB diff --git a/docs/databases/postgresql.md b/docs/databases/postgresql.md index ce9646b..c0915fe 100644 --- a/docs/databases/postgresql.md +++ b/docs/databases/postgresql.md @@ -1,6 +1,6 @@ --- title: "Load XML into PostgreSQL" -description: "Load complex XML files into PostgreSQL tables generated from an XSD schema with xml2db: drivers, COPY bulk loading, identifier limits, type overrides and concurrent loads." +description: "Load complex XML files into PostgreSQL tables generated from an XSD with xml2db: drivers, COPY bulk loading, names, types and concurrent loads." --- # Load XML into PostgreSQL diff --git a/docs/databases/sql_server.md b/docs/databases/sql-server.md similarity index 95% rename from docs/databases/sql_server.md rename to docs/databases/sql-server.md index 177923d..a5deb7f 100644 --- a/docs/databases/sql_server.md +++ b/docs/databases/sql-server.md @@ -1,6 +1,6 @@ --- title: "Load XML into SQL Server" -description: "Load complex XML files into Microsoft SQL Server tables generated from an XSD schema with xml2db: pyodbc connection, bcp bulk loading, columnstore indexes and type mapping." +description: "Load complex XML files into SQL Server tables generated from an XSD with xml2db: pyodbc connection, bcp bulk loading, columnstore indexes, types." --- # Load XML into SQL Server diff --git a/docs/dbt.md b/docs/dbt.md index 9e90413..62963fb 100644 --- a/docs/dbt.md +++ b/docs/dbt.md @@ -1,6 +1,6 @@ --- title: "Transform XML data with dbt" -description: "Use dbt on top of xml2db tables: keep all the XML data as a raw layer, flatten it with staging and mart models, and build incremental models with key-based filters." +description: "Use dbt on top of xml2db tables: keep all the XML data as a raw layer, flatten it with SQL models, and build incremental models filtered on keys." --- # Transform XML data with dbt @@ -200,4 +200,4 @@ With `dbt-duckdb`, the `external` materialization writes a model to a Parquet fi select * from {{ ref('fct_invoice_lines') }} ``` -See [Export to files, lakehouses and other databases](databases/other_targets.md) for other targets. +See [Export to files, lakehouses and other databases](databases/export.md) for other targets. diff --git a/docs/getting_started.md b/docs/getting_started.md index 829211e..7fc0f89 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,6 +1,6 @@ --- -title: "Getting started" -description: "Step-by-step guide to installing xml2db, exploring an XSD schema with the CLI, importing XML into a relational database, and visualizing the resulting data model." +title: "Getting started: load XML files into a database" +description: "Install xml2db, explore an XSD schema in the browser, and load XML files into a database from the command line or from Python." --- # Getting started @@ -15,7 +15,8 @@ Install the package, preferably in a virtual environment: pip install xml2db ``` -You will also need a database driver for your backend (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for SQL Server, `duckdb-engine` for DuckDB). See [How it works](how_it_works.md#bulk-loading) for which drivers enable native bulk loading. +You will also need a database driver for your backend (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for SQL Server, `duckdb-engine` for DuckDB). The database pages give connection strings and bulk loading setup for each backend: [PostgreSQL](databases/postgresql.md), +[SQL Server](databases/sql-server.md), [MySQL and MariaDB](databases/mysql.md) and [DuckDB](databases/duckdb.md). !!! note To contribute to `xml2db` development, clone the repository and install in editable mode: @@ -51,6 +52,8 @@ xml2db render schema.xsd --format source-tree xml2db render schema.xsd --format ddl --db-type postgresql ``` +To generate `CREATE TABLE` statements for review or deployment, see [XSD to SQL](xsd-to-sql.md). + !!! note "Documents mixing several schemas" `DataModel` reads a single XSD file, but that file can pull in others with `xs:import` (for @@ -79,6 +82,8 @@ Key options: - `--validate`: validate the XML against the schema before importing - `--recover`: attempt to parse malformed XML +See the [command line reference](cli.md) for all options. + ## Using the Python API The same operations are available programmatically. Create a [`DataModel`](api/data_model.md) from an XSD file: @@ -161,6 +166,11 @@ By default, XML files are not validated against the schema. Enable validation if document.insert_into_target_tables() ``` +## Transforming the loaded data + +The loaded tables keep all the content of the XML files. Build the tables people query on top of them with SQL +views or [dbt](dbt.md), or [export them](databases/export.md) to Parquet files, Delta Lake and other targets. + ## Getting data back to XML Data can be extracted from the database back to XML, primarily for round-trip testing: diff --git a/docs/how_it_works.md b/docs/how_it_works.md index fb7419b..d73ab84 100644 --- a/docs/how_it_works.md +++ b/docs/how_it_works.md @@ -1,6 +1,6 @@ --- -title: "How it works" -description: "Deep dive into how xml2db builds a relational data model from an XSD schema, performs hash-based deduplication, models 1-1/1-n/n-n relationships, and loads XML data into a database." +title: "How it works: from XSD schema to SQL tables" +description: "How xml2db builds SQL tables from an XSD schema, deduplicates records with hashes, models relationships, and loads XML files into the database." --- # How it works @@ -197,7 +197,7 @@ missing tool, or server configuration), it falls back silently to SQLAlchemy's s | MS SQL Server | `bcp` utility | `bcp` on PATH; SQL or Windows/Kerberos auth | 100 rows | | DuckDB | `read_csv()` | built-in | 100 rows | -The pages for [PostgreSQL](databases/postgresql.md#bulk-loading), [SQL Server](databases/sql_server.md#bulk-loading), +The pages for [PostgreSQL](databases/postgresql.md#bulk-loading), [SQL Server](databases/sql-server.md#bulk-loading), [MySQL](databases/mysql.md#bulk-loading) and [DuckDB](databases/duckdb.md#bulk-loading) give the setup details. The threshold column means that batches smaller than that number always use `executemany` (avoiding temp-file overhead diff --git a/docs/index.md b/docs/index.md index e8c8a76..45223e9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,9 +1,9 @@ --- -title: "Loading XML into a relational database" -description: "xml2db is a Python package that automatically maps an XSD schema to relational database tables and loads XML files into them, with no custom mapping rules required." +title: "Load XML into a SQL database with Python" +description: "xml2db is a Python package that turns an XSD schema into SQL tables and loads XML files into them, with no mapping rules to write." --- -# Loading XML into a relational database +# Load XML into a SQL database with Python `xml2db` is a Python package that parses and loads XML files into a relational database: @@ -15,7 +15,8 @@ description: "xml2db is a Python package that automatically maps an XSD schema t loads all the content of XML files into tables that stay close to the source, and the transformations happen later, in the database. -See the [Getting started](getting_started.md) guide for installation, CLI usage, and the Python API. +See the [Getting started](getting_started.md) guide for installation and the Python API, and the +[command line reference](cli.md) for all commands. ## From XML to tables @@ -125,7 +126,7 @@ attribute, you do not need to decide upfront which data to extract and how to re [dbt](dbt.md), SQL views or stored procedures. The same applies when the final tables live elsewhere: load into [DuckDB](databases/duckdb.md), transform with SQL, -then [export the result](databases/other_targets.md) to files, lakehouse tables or other databases. +then [export the result](databases/export.md) to files, lakehouse tables or other databases. ## When to use xml2db @@ -150,13 +151,13 @@ MS SQL Server, and DuckDB. Each backend has its own page with connection strings advice: * [PostgreSQL](databases/postgresql.md) -* [SQL Server](databases/sql_server.md) +* [SQL Server](databases/sql-server.md) * [MySQL and MariaDB](databases/mysql.md) * [DuckDB](databases/duckdb.md) -* [Files, lakehouses and other databases, through DuckDB](databases/other_targets.md) +* [Files, lakehouses and other databases, through DuckDB](databases/export.md) To build transformed tables on top of the loaded data, see [Transform with dbt](dbt.md). To only generate -`CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd_to_sql.md). +`CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd-to-sql.md). ## Visualising data models diff --git a/docs/overrides/main.html b/docs/overrides/main.html deleted file mode 100644 index 59ac178..0000000 --- a/docs/overrides/main.html +++ /dev/null @@ -1,5 +0,0 @@ -{% extends "base.html" %} - -{% block extrahead %} - -{% endblock %} \ No newline at end of file diff --git a/docs/xsd_to_sql.md b/docs/xsd-to-sql.md similarity index 93% rename from docs/xsd_to_sql.md rename to docs/xsd-to-sql.md index 4722915..1618a11 100644 --- a/docs/xsd_to_sql.md +++ b/docs/xsd-to-sql.md @@ -1,9 +1,9 @@ --- -title: "XSD to SQL tables" -description: "Generate SQL CREATE TABLE statements from an XSD schema with xml2db, for PostgreSQL, SQL Server or MySQL, and adjust the resulting relational schema with a configuration file." +title: "XSD to SQL: generate database tables from an XML schema" +description: "Generate SQL CREATE TABLE statements from an XSD schema for PostgreSQL, SQL Server or MySQL with xml2db, and shape the tables with a config file." --- -# XSD to SQL tables +# XSD to SQL: generate database tables from an XML schema `xml2db` derives a relational schema from an XSD file. You can generate the `CREATE TABLE` statements without connecting to a database or loading any XML file, to review the schema, share it, or create the tables with your own @@ -87,7 +87,7 @@ xml2db render invoices.xsd --format ddl --db-type postgresql --config model_conf ``` See [Configuring your data model](configuring.md) for all options, and the database pages for backend-specific -advice: [PostgreSQL](databases/postgresql.md), [SQL Server](databases/sql_server.md), +advice: [PostgreSQL](databases/postgresql.md), [SQL Server](databases/sql-server.md), [MySQL and MariaDB](databases/mysql.md), [DuckDB](databases/duckdb.md). ## From Python diff --git a/mkdocs.yml b/mkdocs.yml index f9df328..dab3361 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -3,20 +3,20 @@ site_url: https://cre-dev.github.io/xml2db repo_url: https://github.com/cre-dev/xml2db repo_name: cre-dev/xml2db copyright: Copyright © 2026 Commission de régulation de l'énergie -site_description: xml2db is a Python package to load XML files into a relational database. It handles complex XML and - works out of the box, without any custom mapping rules. +site_description: xml2db is a Python package that turns an XSD schema into SQL tables and loads XML files into them, + with no mapping rules to write. nav: - Introduction: "index.md" - Getting started: "getting_started.md" - Databases: - PostgreSQL: "databases/postgresql.md" - - SQL Server: "databases/sql_server.md" + - SQL Server: "databases/sql-server.md" - MySQL and MariaDB: "databases/mysql.md" - DuckDB: "databases/duckdb.md" - - Export through DuckDB: "databases/other_targets.md" + - Export through DuckDB: "databases/export.md" - Transform with dbt: "dbt.md" - - XSD to SQL tables: "xsd_to_sql.md" + - XSD to SQL tables: "xsd-to-sql.md" - Configuring: "configuring.md" - How it works: "how_it_works.md" - CLI usage: "cli.md" @@ -28,7 +28,7 @@ nav: theme: name: material - custom_dir: docs/overrides + custom_dir: overrides features: - navigation.instant - navigation.instant.prefetch diff --git a/overrides/main.html b/overrides/main.html new file mode 100644 index 0000000..ef35ffa --- /dev/null +++ b/overrides/main.html @@ -0,0 +1,28 @@ +{% extends "base.html" %} + +{% block extrahead %} + + {#- Open Graph and Twitter tags for link previews on social networks and chat apps -#} + {%- if page and page.meta and page.meta.title -%} + {%- set preview_title = page.meta.title -%} + {%- elif page and page.title -%} + {%- set preview_title = page.title -%} + {%- else -%} + {%- set preview_title = config.site_name -%} + {%- endif -%} + {%- if page and page.meta and page.meta.description -%} + {%- set preview_description = page.meta.description -%} + {%- else -%} + {%- set preview_description = config.site_description -%} + {%- endif %} + + + + + {%- if page and page.canonical_url %} + + {%- endif %} + + + +{% endblock %} From 9df7d85a433b9d193379a31e0fb1d8bf2f35c182 Mon Sep 17 00:00:00 2001 From: Martin Vergier Date: Sun, 25 Oct 2026 21:44:36 +0100 Subject: [PATCH 9/9] Bump version from 0.15.0 to 0.15.1 --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 4423cc9..fa08fd5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "xml2db" -version = "0.15.0" +version = "0.15.1" authors = [ { name="Commission de régulation de l'énergie", email="opensource@cre.fr" }, ]