diff --git a/CLAUDE.md b/CLAUDE.md index e42d2d0..4a08371 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,6 +81,7 @@ Check that the bug reproduces before the fix and is gone after it. - After any code change, check whether docstrings, inline docs, or `docs/` pages need updating and update them as part of the same task. - After updating `docs/`, verify it builds without errors: `python -m mkdocs build --strict`. In strict mode warnings are also errors, matching CI. - Write docstrings and documentation concisely and directly. No em-dashes or en-dashes; use commas, colons, or plain sentences instead. Avoid AI-sounding phrasing such as bold lead-ins ("**Why this matters:**"), filler constructions ("it is worth noting", "this allows", "this ensures"), and verbose qualifiers. +- Avoid contrast constructions used for emphasis, such as "X, not Y", "not X but Y", "X rather than Y" or "X instead of Y". State what is true directly; keep a comparison only when the reader needs both sides. ### Key configuration options (`model_config`) diff --git a/README.md b/README.md index e9a6743..8a03f7a 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,13 @@ -# Loading XML files into a relational database +# Load XML files into a SQL database with Python `xml2db` is a Python package that parses and loads XML files into a relational database. It handles complex XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules. It fits naturally into an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) pipeline: it -loads XML files into a relational data model that stays close to the source data while remaining easy to query as -flat database tables. The raw data can then be transformed using [DBT](https://www.getdbt.com/), SQL views, or stored procedures to produce -more user-friendly tables. +loads all the content of XML files into a relational data model that stays close to the source data while remaining +easy to query as flat database tables. Parsing XML is slow compared to SQL, so you do not need to decide upfront what +to extract: explore the loaded data, then build more user-friendly tables with [dbt](https://www.getdbt.com/), SQL +views, or stored procedures, and change them later without parsing the files again. Starting from an XSD schema which represents a given XML structure, `xml2db` builds a data model, i.e. a set of database tables linked to each other by foreign keys relationships. Then, it allows parsing and loading XML files into the @@ -15,10 +16,31 @@ database, and getting them back from the database into XML format if needed. This package uses `sqlalchemy` to interact with the database, so it should work with different database backends. Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for -MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). +MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). Through DuckDB, the loaded tables can also be +transformed and exported to Parquet files, Delta Lake or Iceberg. **Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!** +## When to use xml2db + +| Situation | Usually enough | What xml2db adds | +|---|---|---| +| A few fields from flat, record-like XML | `pandas.read_xml`, a short `lxml` script, or the XML functions of your database | Little: these are simpler | +| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables, with no nested columns to explode | +| Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | +| Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | +| Needs not settled yet: which fields, which aggregations | Extracting the fields you expect to need, and parsing the files again when needs change | Everything loaded once: new questions are answered with SQL | +| All the data must be kept, or written back to XML | Hand-written mappings, with the risk of dropping fields | Every element and attribute kept, with a round trip back to XML | + +`xml2db` needs an XSD schema. See the documentation for an [example](https://cre-dev.github.io/xml2db/#from-xml-to-tables) +and for database-specific guides: [PostgreSQL](https://cre-dev.github.io/xml2db/databases/postgresql/), +[SQL Server](https://cre-dev.github.io/xml2db/databases/sql-server/), +[MySQL and MariaDB](https://cre-dev.github.io/xml2db/databases/mysql/), +[DuckDB](https://cre-dev.github.io/xml2db/databases/duckdb/), +[export to files, lakehouses and other databases](https://cre-dev.github.io/xml2db/databases/export/). To transform the loaded +data, see [Transform with dbt](https://cre-dev.github.io/xml2db/dbt/). To only generate SQL tables from an XSD, see +[XSD to SQL tables](https://cre-dev.github.io/xml2db/xsd-to-sql/). + ## Installation The package can be installed, preferably in a virtual environment, using `pip`: @@ -79,24 +101,35 @@ document.insert_into_target_tables() ## Testing -Running the tests requires installing additional development dependencies, after cloning the repo, with: +After cloning the repo, install the package with its development dependencies, plus DuckDB and `pytz` to run the +database tests locally: ```bash -pip install -e .[tests,docs] +pip install -e .[tests,docs] duckdb_engine pytz ``` -Run all tests with the following command: +Run all tests, including database integration tests, against an in-memory DuckDB database: ```bash -python -m pytest +TZ="Europe/Paris" DB_STRING="duckdb:///:memory:" python -m pytest ``` -Integration tests require write access to a PostgreSQL or MS SQL Server database; the connection string is provided as an -environment variable `DB_STRING`. If you want to run only conversion tests that do not require a database you can run: +Round-trip tests compare datetimes read back from the database in the `Europe/Paris` time zone, so `TZ` must be set to +it. + +Integration tests use the database given by the `DB_STRING` connection string. To run them against another database, +for instance PostgreSQL: ```bash -pytest -m "not dbtest" -````` +TZ="Europe/Paris" DB_STRING="postgresql+psycopg2://user:pw@localhost:5432/testdb" python -m pytest +``` + +Continuous integration runs them against DuckDB, PostgreSQL, MySQL and MS SQL Server. To run only the tests that do +not need a database: + +```bash +TZ="Europe/Paris" python -m pytest -m "not dbtest" +``` ## Contributing diff --git a/docs/api/data_model.md b/docs/api/data_model.md index 2e03956..0cc24a6 100644 --- a/docs/api/data_model.md +++ b/docs/api/data_model.md @@ -1,6 +1,6 @@ --- title: "DataModel" -description: "API reference for the xml2db DataModel class: constructor parameters, methods for parsing XML files, visualizing the schema as ERD or tree, and generating DDL statements." +description: "API reference for the xml2db DataModel class: constructor parameters, parsing XML files, ERD and tree views of the schema, and DDL generation." --- # DataModel diff --git a/docs/api/overview.md b/docs/api/overview.md index e1d9a29..f2c2f66 100644 --- a/docs/api/overview.md +++ b/docs/api/overview.md @@ -1,6 +1,6 @@ --- title: "API Overview" -description: "Overview of the xml2db Python API: building DataModel objects, parsing and loading XML files into a database, and extracting data back to XML, including multiprocessing patterns." +description: "Overview of the xml2db Python API: building a DataModel, loading XML files into a database, extracting data back to XML, and multiprocessing." --- # API Overview diff --git a/docs/cli.md b/docs/cli.md index 9cd91a0..fba774c 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,6 +1,6 @@ --- -title: "CLI usage" -description: "Reference for the xml2db command-line interface: import XML files, render ERDs and DDL, and launch the interactive browser explorer." +title: "Command line: import XML into a database, render DDL" +description: "Reference for the xml2db command line: import XML files into a database, render diagrams and DDL from an XSD, and explore a schema in the browser." --- # CLI usage @@ -81,6 +81,8 @@ xml2db render schema.xsd --format ddl --db-type postgresql xml2db render schema.xsd --format erd --output diagram.md ``` +[XSD to SQL](xsd-to-sql.md) shows the generated DDL and how to adjust it. + ## xml2db serve Launch an interactive schema explorer in the browser. diff --git a/docs/configuring.md b/docs/configuring.md index 54c4e51..b3e02b0 100644 --- a/docs/configuring.md +++ b/docs/configuring.md @@ -1,6 +1,6 @@ --- title: "Configuring your data model" -description: "Configure xml2db's data model via model_config: override column types, control field elevation, adjust deduplication, simplify XSD choice groups, and add custom indexes." +description: "Configure the tables xml2db derives from an XSD: column types and names, flattening of child elements, deduplication, choice groups and indexes." --- # Configuring your data model @@ -391,6 +391,9 @@ records. Configuration: `"reuse":` `True` (default) or `False` (disable) +The root table is always deduplicated, so that loading the same file twice stores it once: setting `"reuse": False` on +it raises an error. + !!! example Disabling deduplication for a given table: ``` python diff --git a/docs/databases/duckdb.md b/docs/databases/duckdb.md new file mode 100644 index 0000000..26e6762 --- /dev/null +++ b/docs/databases/duckdb.md @@ -0,0 +1,75 @@ +--- +title: "Load XML into DuckDB" +description: "Load complex XML files into DuckDB tables generated from an XSD with xml2db, query them without a server, or read simple XML with DuckDB extensions." +--- + +# Load XML into DuckDB + +DuckDB runs inside the Python process and stores a whole database in a single file. With `xml2db`, it gives a +relational copy of XML files without any server to set up, which suits local analysis, trying a data model before +deploying it, and tests. + +## Reading XML in DuckDB + +DuckDB community extensions, such as [webbed](https://duckdb.org/community_extensions/extensions/webbed), add a +`read_xml` function that reads XML files directly in SQL. It turns a repeated element into rows, with nested elements +as `STRUCT` and `LIST` columns, and needs no XSD. It fits quick queries on record-like files. + +`xml2db` fits files described by an XSD, with deep nesting or repeated elements at several levels: it loads them into +normalized tables with keys, stores identical records once, and gives the same tables for every file of a feed. Both +work in the same DuckDB database, so each can be used where it fits. + +## Connecting + +``` bash +pip install xml2db duckdb-engine + +xml2db import invoices.xml invoices.xsd --connection-string "duckdb:///invoices.duckdb" +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="invoices.xsd", + connection_string="duckdb:///invoices.duckdb", +) +for xml_file in ["invoices_1.xml", "invoices_2.xml"]: + data_model.parse_xml(xml_file).insert_into_target_tables() +``` + +Use `duckdb:///:memory:` for a database that only lives during the Python process. + +Then query the file with the DuckDB command line or from Python: + +``` py +import duckdb + +con = duckdb.connect("invoices.duckdb", read_only=True) +con.sql("SELECT id, customer_name, CAST(date AS DATE) AS date FROM invoice").show() +``` + +XSD `date` and `time` values are stored as text, since they may carry a time zone: cast them in queries or views. + +## Bulk loading + +Batches of 100 rows or more are written to a temporary CSV file and loaded with DuckDB's `read_csv`, every column being +read as text and then cast to its target type. Smaller batches use `executemany`. + +## One writer at a time + +A DuckDB file accepts a single writing process. When loading files from several processes, parse in parallel and +load one file at a time, as shown in [Multiprocessing](../how_it_works.md#multiprocessing). Readers using +`read_only=True` cannot open the file while a process writes to it either. + +## Keys and names + +Primary keys are filled from DuckDB sequences, named `pk_sequ_`. Table and column names longer than 63 +characters are shortened with a hash suffix, as for PostgreSQL. This limit comes from the SQLAlchemy integration: DuckDB +itself accepts longer names. + +## Transforming and exporting + +Keep the DuckDB file as the raw copy of your XML data, and build the tables you need with SQL views or +[dbt](../dbt.md). DuckDB then exports them to Parquet files, Delta Lake, Iceberg and other targets that `xml2db` does +not support directly: see [Export to files, lakehouses and other databases](export.md). diff --git a/docs/databases/export.md b/docs/databases/export.md new file mode 100644 index 0000000..548b4c1 --- /dev/null +++ b/docs/databases/export.md @@ -0,0 +1,193 @@ +--- +title: "Export to files, lakehouses and other databases" +description: "Load XML with xml2db into DuckDB, transform it with SQL, then export to Parquet, CSV or Excel, Delta Lake, Iceberg, warehouses or server databases." +--- + +# Export to files, lakehouses and other databases + +`xml2db` merges each file into existing tables: it matches records by hash, generates keys and links rows through +foreign keys. This needs a database with sequences or identity columns, unique constraints and `UPDATE` statements, +which files and table formats such as Parquet or Delta Lake do not offer. + +[DuckDB](duckdb.md) fills the gap, and fits the [ELT approach](../index.md#load-first-transform-later): + +1. load all the XML data into a DuckDB file, which keeps every element and the state needed for deduplication; +2. explore it, then join, filter and flatten it with SQL into the tables you need; +3. export these tables with DuckDB. + +When the transformation changes, rerun steps 2 and 3 from the DuckDB file: the XML files are parsed once. + +The examples below use the invoices schema of the [introduction](../index.md#from-xml-to-tables). The examples +for single Parquet files, Delta Lake, DuckLake, PostgreSQL, DataFrames, CSV, JSON and Excel were tested with DuckDB 2.0 +and local files. The examples for partitioned Parquet files, object storage, Iceberg and MySQL follow the DuckDB +documentation but were not tested with `xml2db`: they should work, check the linked pages for details. + +## Load everything into DuckDB + +``` py title="load.py" linenums="1" +from xml2db import DataModel + +data_model = DataModel(xsd_file="invoices.xsd", connection_string="duckdb:///invoices.duckdb") +for xml_file in xml_files: + data_model.parse_xml(xml_file).insert_into_target_tables() +data_model.engine.dispose() +``` + +Keep `invoices.duckdb` between runs: it holds all the source data, and the hashes used to store each record once. + +## Transform with SQL + +Build the tables your users need from the `xml2db` tables, for instance one row per invoice line, with proper types: + +``` py title="transform.py" linenums="1" +import duckdb + +con = duckdb.connect("invoices.duckdb") +con.execute(""" + CREATE OR REPLACE VIEW invoice_lines AS + SELECT + i.pk_invoice, + i.id AS invoice_id, + CAST(i.date AS DATE) AS invoice_date, + i.customer_name, + i.customer_country, + l.sku, + l.quantity, + CAST(l.price AS DECIMAL(12, 2)) AS price + FROM invoice AS i + JOIN invoice_line AS il ON il.fk_invoice = i.pk_invoice + JOIN line AS l ON l.pk_line = il.fk_line +""") +``` + +For more than a few views, [dbt](../dbt.md) organizes these transformations, tests them and makes them incremental. + +## Parquet files + +Parquet is read by most data tools: Spark, pandas, Polars, and the loaders of cloud data warehouses. + +``` py +con.execute("COPY invoice_lines TO 'invoice_lines.parquet' (FORMAT parquet)") +``` + +Large tables can be split into one directory per value of a column, which should let tools reading the files skip +partitions: + +``` py +con.execute(""" + COPY (SELECT *, year(invoice_date) AS invoice_year FROM invoice_lines) + TO 'invoice_lines' (FORMAT parquet, PARTITION_BY (invoice_year)) +""") +``` + +See [Parquet export](https://duckdb.org/docs/current/data/parquet/overview) and +[partitioned writes](https://duckdb.org/docs/current/data/partitioning/partitioned_writes) in the DuckDB +documentation. + +### Object storage + +With the `httpfs` extension and a secret holding the credentials, the same `COPY` statement should write directly to +S3 or S3-compatible storage: + +``` py +con.execute("CREATE SECRET (TYPE s3, PROVIDER credential_chain)") +con.execute("COPY invoice_lines TO 's3://my-bucket/invoice_lines.parquet' (FORMAT parquet)") +``` + +The [S3 API page](https://duckdb.org/docs/current/core_extensions/httpfs/s3api) of the DuckDB documentation +describes the supported providers. For Azure Blob Storage or Google Cloud Storage, check the `azure` and `httpfs` +extension pages for write support. + +### Cloud data warehouses + +Cloud data warehouses load Parquet files from object storage with their own tools, such as `COPY INTO` statements or +external tables. Exporting Parquet files to a bucket, then loading them with these tools, should be the most reliable +path to a warehouse. + +## Delta Lake + +DuckDB does not support `CREATE TABLE ... AS` for Delta tables: create the table, then insert into it. + +`xml2db` only appends rows to its tables, and new rows get higher keys. Filtering on the key of the table that defines +the grain of the export, `pk_invoice` here, appends new invoices only: + +``` py title="export_delta.py" linenums="1" +con.execute("ATTACH 'delta/invoice_lines' AS delta_lines (TYPE delta)") +if not con.execute( + "SELECT count(*) FROM duckdb_tables() WHERE database_name = 'delta_lines'" +).fetchone()[0]: + con.execute(""" + CREATE TABLE delta_lines.delta_lines ( + pk_invoice INTEGER, invoice_id VARCHAR, invoice_date DATE, customer_name VARCHAR, + customer_country VARCHAR, sku VARCHAR, quantity INTEGER, price DECIMAL(12, 2) + ) + """) +con.execute(""" + INSERT INTO delta_lines + FROM invoice_lines + WHERE pk_invoice > (SELECT coalesce(max(pk_invoice), 0) FROM delta_lines) +""") +``` + +The [dbt page](../dbt.md#incremental-models) explains how to choose the key. Delta write support depends on the +DuckDB version and platform: check the [delta extension documentation](https://duckdb.org/docs/current/core_extensions/delta) +for current limitations. + +## Iceberg + +DuckDB writes Iceberg tables through an Iceberg REST catalog. Once the catalog is attached as described in the +[DuckDB documentation](https://duckdb.org/docs/current/core_extensions/iceberg/overview), `CREATE TABLE` and +`INSERT INTO` statements on the attached catalog should allow the same full or incremental exports as for Delta Lake. + +## DuckLake + +If your tools already rely on DuckDB, a [DuckLake](https://duckdb.org/docs/current/core_extensions/ducklake) catalog +gives a table format with snapshots on top of Parquet files: + +``` py +con.execute("ATTACH 'ducklake:analytics.ducklake' AS lake (DATA_PATH 'files/')") +con.execute("CREATE OR REPLACE TABLE lake.invoice_lines AS FROM invoice_lines") +``` + +## Publishing to a server database + +`xml2db` loads [PostgreSQL](postgresql.md) and [MySQL](mysql.md) directly. Another option is to keep the +`xml2db` tables in a local DuckDB file, and publish only the transformed tables to a shared server. The server +holds only the tables people query, and the parsing work stays on the machine running `xml2db`. + +``` py +con.execute("ATTACH 'host=db.example.com dbname=analytics user=etl' AS pg (TYPE postgres)") +con.execute("CREATE OR REPLACE TABLE pg.public.invoice_lines AS FROM invoice_lines") +``` + +The `mysql` extension should allow the same with `ATTACH ... (TYPE mysql)`. Tables created this way have no primary +key, foreign keys or indexes: add them on the server if needed. + +## DataFrames + +To continue the analysis in Python, convert DuckDB results to pandas or Polars DataFrames: + +``` py +invoice_lines = con.sql("FROM invoice_lines").df() # pandas +invoice_lines = con.sql("FROM invoice_lines").pl() # Polars +``` + +pandas converts `DECIMAL` columns to floating point numbers, while Polars keeps them as decimals. + +## Other formats + +DuckDB also writes CSV and JSON files, and `.xlsx` files with the `excel` extension, for small extracts (Excel sheets +hold about one million rows): + +``` py +con.execute("COPY invoice_lines TO 'invoice_lines.csv'") +con.execute("COPY invoice_lines TO 'invoice_lines.json'") # one JSON object per line + +con.execute("INSTALL excel") +con.execute("LOAD excel") +con.execute("COPY invoice_lines TO 'invoice_lines.xlsx' (FORMAT xlsx, HEADER true)") +``` + +!!! note + Pointing `xml2db` directly at a database attached in DuckDB does not work: DuckDB cannot create the sequences + used for primary keys in attached databases such as PostgreSQL. diff --git a/docs/databases/mysql.md b/docs/databases/mysql.md new file mode 100644 index 0000000..f5bb765 --- /dev/null +++ b/docs/databases/mysql.md @@ -0,0 +1,82 @@ +--- +title: "Load XML into MySQL and MariaDB" +description: "Load complex XML files into MySQL or MariaDB tables generated from an XSD with xml2db: drivers, LOAD DATA bulk loading, names and string sizes." +--- + +# Load XML into MySQL and MariaDB + +`xml2db` creates MySQL tables from an XSD schema and loads XML files into them, using `LOAD DATA LOCAL INFILE` for +bulk loading. MariaDB uses the same code path; integration tests run against MySQL only. + +## Connecting + +Install `pymysql` or `mysqlclient`, then pass a SQLAlchemy connection string: + +``` bash +pip install xml2db pymysql + +xml2db import file.xml schema.xsd \ + --connection-string "mysql+pymysql://user:pw@host:3306/db" \ + --db-schema xml_data +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="schema.xsd", + connection_string="mysql+pymysql://user:pw@host:3306/db", + db_schema="xml_data", +) +data_model.parse_xml("file.xml").insert_into_target_tables() +``` + +In MySQL, a schema is a database: `--db-schema xml_data` creates the `xml_data` database if it does not exist. Use +`mariadb+pymysql://` for MariaDB. + +## Bulk loading + +With `pymysql` or `mysqlclient`, batches of 100 rows or more are written to a temporary tab-separated file and sent +with `LOAD DATA LOCAL INFILE`. Two settings are needed: + +* on the client, `local_infile=True` in the connection arguments. `DataModel` adds it when it creates the engine from a + connection string. If you pass your own engine, add it yourself: + `sqlalchemy.create_engine(url, connect_args={"local_infile": True})`; +* on the server, `local_infile=ON`, for instance by starting it with `--local-infile=1`. Check with + `SHOW VARIABLES LIKE 'local_infile'`. + +When the server refuses `LOAD DATA LOCAL INFILE`, `xml2db` falls back to `executemany` and keeps doing so for the +following batches of the same `DataModel`. +`insert_into_target_tables(bulk_load=True)` raises an error instead, which helps to check the setup. + +## String sizes + +Strings without a `maxLength` in the XSD become `VARCHAR(255)` on MySQL, while other backends use +`VARCHAR(1000)`. Longer values are rejected in strict SQL mode, which is the default. Override the type of free-text fields: + +``` yaml title="model_config.yml" +tables: + line: + fields: + description: + type: Text +``` + +MySQL also limits the total size of a row to 65,535 bytes, counting each `VARCHAR` column at its maximum size. Tables +with many string columns can reach this limit, especially with `utf8mb4` (4 bytes per character). `Text` columns only +count for a few bytes each, so the same override solves the "Row size too large" error. + +Repeated simple values joined in one column use `VARCHAR(4000)`. + +## Table and column names + +MySQL accepts identifiers up to 64 characters. `xml2db` shortens names longer than 56 characters, keeping room for the +suffixes SQLAlchemy adds to constraint names. Shortened names keep their first characters and get a 7-character hash +suffix, so they stay unique and stable between runs. Use `rename` in the configuration to choose readable names, and +`xml2db render schema.xsd --format ddl --db-type mysql` to see the result. + +## Concurrent loads + +Each `DataModel` instance uses its own staging tables (named with a random `temp_prefix`), so several processes can +fill staging tables at the same time. The merge into target tables should run one process at a time: see +[Multiprocessing](../how_it_works.md#multiprocessing) for an example. diff --git a/docs/databases/postgresql.md b/docs/databases/postgresql.md new file mode 100644 index 0000000..c0915fe --- /dev/null +++ b/docs/databases/postgresql.md @@ -0,0 +1,100 @@ +--- +title: "Load XML into PostgreSQL" +description: "Load complex XML files into PostgreSQL tables generated from an XSD with xml2db: drivers, COPY bulk loading, names, types and concurrent loads." +--- + +# Load XML into PostgreSQL + +`xml2db` creates PostgreSQL tables from an XSD schema and loads XML files into them, using `COPY` for bulk loading. + +## Connecting + +Install a driver alongside `xml2db`: + +``` bash +pip install xml2db psycopg2-binary # or: pip install xml2db "psycopg[binary]" +``` + +Then pass a SQLAlchemy connection string: + +``` bash +xml2db import file.xml schema.xsd \ + --connection-string "postgresql+psycopg2://user:pw@host:5432/db" \ + --db-schema xml_data +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="schema.xsd", + connection_string="postgresql+psycopg://user:pw@host:5432/db", + db_schema="xml_data", +) +data_model.parse_xml("file.xml").insert_into_target_tables() +``` + +The schema given with `--db-schema` or `db_schema` is created if it does not exist. Without it, tables go to the +default schema of the connection, usually `public`. + +## Bulk loading + +With `psycopg2` or `psycopg` (version 3), staging tables are filled with `COPY ... FROM STDIN`: rows are written to an +in-memory CSV buffer and streamed to the server, with no temporary file. `COPY` is used for every batch, whatever its +size. Other drivers, such as `pg8000`, fall back to standard `executemany` inserts. + +`insert_into_target_tables(bulk_load=True)` raises an error when the driver does not support +`COPY`. See [Bulk loading](../how_it_works.md#bulk-loading) for the other options. + +## Table and column names + +PostgreSQL truncates identifiers longer than 63 characters. `xml2db` shortens such names itself, keeping the first +characters and adding a 7-character hash suffix, so names stay unique and stable between runs. Deep XSD hierarchies +often produce long names: rename them in the configuration to keep them readable. + +``` yaml title="model_config.yml" +tables: + TimeSeries: + fields: + production_RegisteredResource_pSRType_powerSystemResources_nominalP: + rename: nominal_power +``` + +`xml2db render schema.xsd --format ddl --db-type postgresql` shows the physical names before anything is created. + +## Column types + +The default mapping favours loading every file without errors over precise types: + +| XSD type | PostgreSQL type | +|---|---| +| `string` without `maxLength` | `VARCHAR(1000)` | +| `string` with `maxLength` | `VARCHAR(n)` | +| `decimal`, `float`, `double` | `DOUBLE PRECISION` | +| `integer`, `int` and derived types | `INTEGER` | +| `long` | `BIGINT` | +| `boolean` | `BOOLEAN` | +| `dateTime` | `TIMESTAMP WITH TIME ZONE` | +| `date`, `time` | `VARCHAR` (XSD values may carry a time zone) | +| Repeated simple values joined in one column | `VARCHAR(8000)` | +| Other types, such as `anyURI` | `VARCHAR(1000)`, with a warning | + +The `xml2db_record_hash` column used for deduplication is `BYTEA`. + +Override a type when the default does not fit, for instance to keep exact decimal values for amounts: + +``` yaml title="model_config.yml" +tables: + line: + fields: + price: + type: Numeric(12, 2) + description: + type: Text +``` + +## Concurrent loads + +Each `DataModel` instance uses its own staging tables (named with a random `temp_prefix`), so several processes can +fill staging tables at the same time. The merge into target tables should run one process at a time: see +[Multiprocessing](../how_it_works.md#multiprocessing) for an example. diff --git a/docs/databases/sql-server.md b/docs/databases/sql-server.md new file mode 100644 index 0000000..a5deb7f --- /dev/null +++ b/docs/databases/sql-server.md @@ -0,0 +1,108 @@ +--- +title: "Load XML into SQL Server" +description: "Load complex XML files into SQL Server tables generated from an XSD with xml2db: pyodbc connection, bcp bulk loading, columnstore indexes, types." +--- + +# Load XML into SQL Server + +`xml2db` creates SQL Server tables from an XSD schema and loads XML files into them, using the `bcp` utility for bulk +loading when it is installed. + +## Connecting + +Install `pyodbc` and a Microsoft ODBC driver (for example ODBC Driver 18 for SQL Server), then pass a SQLAlchemy +connection string: + +``` bash +pip install xml2db pyodbc + +xml2db import file.xml schema.xsd \ + --connection-string "mssql+pyodbc://user:pw@host:1433/db?driver=ODBC+Driver+18+for+SQL+Server&TrustServerCertificate=yes" \ + --db-schema xml_data +``` + +``` py title="Python API" linenums="1" +from xml2db import DataModel + +data_model = DataModel( + xsd_file="schema.xsd", + connection_string="mssql+pyodbc://user:pw@host:1433/db?driver=ODBC+Driver+18+for+SQL+Server", + db_schema="xml_data", +) +data_model.parse_xml("file.xml").insert_into_target_tables() +``` + +When `DataModel` creates the engine from a connection string, it enables `fast_executemany` and uses the +`SERIALIZABLE` isolation level. If you pass your own engine with `db_engine`, set these options yourself: + +``` py +import sqlalchemy + +engine = sqlalchemy.create_engine( + connection_string, fast_executemany=True, isolation_level="SERIALIZABLE" +) +data_model = DataModel(xsd_file="schema.xsd", db_engine=engine) +``` + +## Bulk loading + +Batches of 100 rows or more are loaded into staging tables with `bcp` when these conditions are met: + +* the `bcp` executable is on the `PATH` (package `mssql-tools18` on Linux and macOS, SQL Server Command Line + Utilities on Windows); +* the connection uses SQL Server authentication (user and password in the connection string) or a trusted connection + (`Trusted_Connection=yes`). + +`TrustServerCertificate=yes` in the connection string is passed on to `bcp`. Smaller batches, or any batch when these +conditions are not met, use `fast_executemany`. + +`bcp` runs as a separate process, outside the SQLAlchemy transaction. This only concerns staging tables: the merge +into target tables still runs in a transaction. Tab characters inside values are replaced by spaces on this path; pass +`bulk_load=False` to `insert_into_target_tables` if they must be kept. + +## Columnstore indexes + +For large analytical tables, `xml2db` can create a clustered columnstore index on selected tables. The primary key +of these tables then becomes nonclustered. + +``` yaml title="model_config.yml" +tables: + TimeSeries: + as_columnstore: true +``` + +Relationship tables between deduplicated tables have no primary key: they get a clustered index on their two foreign +keys. + +## Column types + +| XSD type | SQL Server type | +|---|---| +| `string` without `maxLength` | `VARCHAR(1000)` | +| `string` with `maxLength` | `VARCHAR(n)`, or `CHAR(n)` when the length is fixed | +| `decimal`, `float`, `double` | `DOUBLE PRECISION` (`FLOAT(53)`) | +| `dateTime` | `DATETIMEOFFSET` | +| `date`, `time` | `VARCHAR` (XSD values may carry a time zone) | +| Repeated simple values joined in one column | `VARCHAR(8000)` | +| Other types, such as `anyURI` | `VARCHAR(1000)`, with a warning | + +The `xml2db_record_hash` column used for deduplication is `BINARY(20)` with the default hash function. + +Text columns are `VARCHAR`. To store characters outside the code page of your database collation, use a UTF-8 +collation (SQL Server 2019 and later), or override column types with `NVARCHAR` in a Python configuration: + +``` py +from sqlalchemy.dialects.mssql import NVARCHAR + +model_config = { + "tables": {"invoice": {"fields": {"customer_name": {"type": NVARCHAR(200)}}}} +} +``` + +SQL Server accepts identifiers up to 128 characters, so table and column names are not shortened. + +## Concurrent loads + +Each `DataModel` instance uses its own staging tables (named with a random `temp_prefix`), so several processes can +fill staging tables at the same time. The merge into target tables should run one process at a time: see +[Multiprocessing](../how_it_works.md#multiprocessing) for an example. diff --git a/docs/dbt.md b/docs/dbt.md new file mode 100644 index 0000000..62963fb --- /dev/null +++ b/docs/dbt.md @@ -0,0 +1,203 @@ +--- +title: "Transform XML data with dbt" +description: "Use dbt on top of xml2db tables: keep all the XML data as a raw layer, flatten it with SQL models, and build incremental models filtered on keys." +--- + +# Transform XML data with dbt + +`xml2db` loads XML files into tables that stay close to the XSD. These tables are a raw layer: they keep all the +source data, and [dbt](https://www.getdbt.com/) builds the tables people query on top of them, with SQL models that +are versioned, tested and documented. + +Parsing XML is the slow part of the pipeline. Once the files are loaded, changing a model and rebuilding it only runs +SQL in the database, so you can start with a few models and add others as questions come, without parsing the files +again. + +The examples below use the invoices schema of the [introduction](index.md#from-xml-to-tables). They were tested with +dbt 1.12 on DuckDB (`dbt-duckdb`) and PostgreSQL (`dbt-postgres`). Adapters also exist for SQL Server and other +databases. + +## Loading into a raw schema + +Load the files into a dedicated schema, and record the file name in a [metadata column](configuring.md#model-configuration): + +``` py title="load.py" linenums="1" +import sqlalchemy +from xml2db import DataModel + +data_model = DataModel( + xsd_file="invoices.xsd", + connection_string="duckdb:///warehouse.duckdb", + db_schema="xml_raw", + model_config={ + "metadata_columns": [{"name": "input_file_path", "type": sqlalchemy.String(256)}] + }, +) +for xml_file in xml_files: + data_model.parse_xml(xml_file, metadata={"input_file_path": xml_file}).insert_into_target_tables() +``` + +Declare the xml2db tables as dbt sources, and write dbt models to another schema: + +``` yaml title="models/staging/sources.yml" +sources: + - name: xml2db + schema: xml_raw + tables: + - name: invoices + - name: invoices_invoice + - name: invoice + - name: invoice_line + - name: line +``` + +``` yaml title="profiles.yml" +invoices: + target: dev + outputs: + dev: + type: duckdb + path: warehouse.duckdb + schema: analytics +``` + +## Staging models + +Staging models rename columns and fix types. `xml2db` stores XSD `date` values as text, since they may carry a time +zone, and decimals as floating point numbers by default: + +``` sql title="models/staging/stg_invoice.sql" +select + pk_invoice, + id as invoice_id, + cast(date as date) as invoice_date, + customer_name, + customer_country +from {{ source('xml2db', 'invoice') }} +``` + +``` sql title="models/staging/stg_line.sql" +select + pk_line, + sku, + quantity, + cast(price as decimal(12, 2)) as price +from {{ source('xml2db', 'line') }} +``` + +## Flattening + +Repeated elements are linked through relationship tables such as `invoice_line`. Joining through them gives one row +per invoice line: + +``` sql title="models/marts/fct_invoice_lines.sql" +{{ config(materialized='incremental') }} + +select + i.pk_invoice, + i.invoice_id, + i.invoice_date, + i.customer_name, + l.sku, + l.quantity, + l.price, + l.quantity * l.price as amount +from {{ ref('stg_invoice') }} as i +join {{ source('xml2db', 'invoice_line') }} as il on il.fk_invoice = i.pk_invoice +join {{ ref('stg_line') }} as l on l.pk_line = il.fk_line +{% if is_incremental() %} +where i.pk_invoice > (select coalesce(max(pk_invoice), 0) from {{ this }}) +{% endif %} +``` + +## Incremental models + +Three properties of `xml2db` tables make incremental models simple: + +* rows are only appended: `xml2db` never updates or deletes rows in its tables; +* keys increase: a new row gets a higher primary key than all existing rows of its table; +* a new row in a relationship table always belongs to a new parent row, inserted in the same load. + +So an incremental model filters on the primary key of the table that defines its grain, and keeps the rows above the +highest key already in the model, as in `fct_invoice_lines` above. + +Choose that table with deduplication in mind. An invoice found in several files is stored once, so a model driven by +`pk_invoice` holds each invoice once. To keep one row per file and invoice, as received, drive the model from the root +table, which gets one row per loaded file with new content: + +``` sql title="models/marts/fct_received_invoices.sql" +{{ config(materialized='incremental') }} + +select + f.pk_invoices as pk_file, + f.input_file_path, + i.invoice_id, + i.invoice_date +from {{ source('xml2db', 'invoices') }} as f +join {{ source('xml2db', 'invoices_invoice') }} as fi on fi.fk_invoices = f.pk_invoices +join {{ ref('stg_invoice') }} as i on i.pk_invoice = fi.fk_invoice +{% if is_incremental() %} +where f.pk_invoices > (select coalesce(max(pk_file), 0) from {{ this }}) +{% endif %} +``` + +After loading `invoices_1.xml`, with invoices `INV-001` and `INV-002`, then `invoices_2.xml`, with `INV-002` and +`INV-003`: + +=== "fct_invoice_lines" + + | pk_invoice | invoice_id | sku | quantity | amount | + |-----------:|------------|-----|---------:|-------:| + | 1 | INV-001 | X1 | 2 | 19.80 | + | 1 | INV-001 | Y7 | 1 | 24.00 | + | 2 | INV-002 | X1 | 2 | 19.80 | + | 3 | INV-003 | Z9 | 5 | 17.50 | + +=== "fct_received_invoices" + + | pk_file | input_file_path | invoice_id | + |--------:|-----------------|------------| + | 1 | invoices_1.xml | INV-001 | + | 1 | invoices_1.xml | INV-002 | + | 2 | invoices_2.xml | INV-002 | + | 2 | invoices_2.xml | INV-003 | + +Metadata columns are not part of the record hash: a file whose content was already loaded adds no row to the root +table, even under another name, and only the first file name is kept. + +Do not filter on the key of a child table that can be shared: the line `X1` of `INV-002` reuses the row created for +`INV-001`, so its key is not new. + +The filter relies on each load being complete when dbt runs. Run dbt after loads, merge files one at a time as +recommended in [Multiprocessing](how_it_works.md#multiprocessing), and keep the default single transaction of +`insert_into_target_tables`. `dbt run --full-refresh` rebuilds a model from all the raw data at any time. + +## Corrected records + +Deduplication compares whole records, so two versions of an invoice with the same identifier are two different +records. When a later file contains a corrected version of an invoice, it is stored as a new row, with a higher key. +A model can keep the latest version of each invoice: + +``` sql title="models/marts/dim_invoice_latest.sql" +select invoice_id, invoice_date, customer_name, customer_country, pk_invoice +from ( + select + *, + row_number() over (partition by invoice_id order by pk_invoice desc) as version_rank + from {{ ref('stg_invoice') }} +) as versions +where version_rank = 1 +``` + +The order of the keys follows the order of the loads: load files in the order they were produced. + +## Exporting from dbt + +With `dbt-duckdb`, the `external` materialization writes a model to a Parquet file: + +``` sql title="models/marts/invoice_lines_parquet.sql" +{{ config(materialized='external', location='invoice_lines.parquet') }} +select * from {{ ref('fct_invoice_lines') }} +``` + +See [Export to files, lakehouses and other databases](databases/export.md) for other targets. diff --git a/docs/getting_started.md b/docs/getting_started.md index 829211e..7fc0f89 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,6 +1,6 @@ --- -title: "Getting started" -description: "Step-by-step guide to installing xml2db, exploring an XSD schema with the CLI, importing XML into a relational database, and visualizing the resulting data model." +title: "Getting started: load XML files into a database" +description: "Install xml2db, explore an XSD schema in the browser, and load XML files into a database from the command line or from Python." --- # Getting started @@ -15,7 +15,8 @@ Install the package, preferably in a virtual environment: pip install xml2db ``` -You will also need a database driver for your backend (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for SQL Server, `duckdb-engine` for DuckDB). See [How it works](how_it_works.md#bulk-loading) for which drivers enable native bulk loading. +You will also need a database driver for your backend (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for SQL Server, `duckdb-engine` for DuckDB). The database pages give connection strings and bulk loading setup for each backend: [PostgreSQL](databases/postgresql.md), +[SQL Server](databases/sql-server.md), [MySQL and MariaDB](databases/mysql.md) and [DuckDB](databases/duckdb.md). !!! note To contribute to `xml2db` development, clone the repository and install in editable mode: @@ -51,6 +52,8 @@ xml2db render schema.xsd --format source-tree xml2db render schema.xsd --format ddl --db-type postgresql ``` +To generate `CREATE TABLE` statements for review or deployment, see [XSD to SQL](xsd-to-sql.md). + !!! note "Documents mixing several schemas" `DataModel` reads a single XSD file, but that file can pull in others with `xs:import` (for @@ -79,6 +82,8 @@ Key options: - `--validate`: validate the XML against the schema before importing - `--recover`: attempt to parse malformed XML +See the [command line reference](cli.md) for all options. + ## Using the Python API The same operations are available programmatically. Create a [`DataModel`](api/data_model.md) from an XSD file: @@ -161,6 +166,11 @@ By default, XML files are not validated against the schema. Enable validation if document.insert_into_target_tables() ``` +## Transforming the loaded data + +The loaded tables keep all the content of the XML files. Build the tables people query on top of them with SQL +views or [dbt](dbt.md), or [export them](databases/export.md) to Parquet files, Delta Lake and other targets. + ## Getting data back to XML Data can be extracted from the database back to XML, primarily for round-trip testing: diff --git a/docs/how_it_works.md b/docs/how_it_works.md index 0019f2b..d73ab84 100644 --- a/docs/how_it_works.md +++ b/docs/how_it_works.md @@ -1,6 +1,6 @@ --- -title: "How it works" -description: "Deep dive into how xml2db builds a relational data model from an XSD schema, performs hash-based deduplication, models 1-1/1-n/n-n relationships, and loads XML data into a database." +title: "How it works: from XSD schema to SQL tables" +description: "How xml2db builds SQL tables from an XSD schema, deduplicates records with hashes, models relationships, and loads XML files into the database." --- # How it works @@ -197,6 +197,9 @@ missing tool, or server configuration), it falls back silently to SQLAlchemy's s | MS SQL Server | `bcp` utility | `bcp` on PATH; SQL or Windows/Kerberos auth | 100 rows | | DuckDB | `read_csv()` | built-in | 100 rows | +The pages for [PostgreSQL](databases/postgresql.md#bulk-loading), [SQL Server](databases/sql-server.md#bulk-loading), +[MySQL](databases/mysql.md#bulk-loading) and [DuckDB](databases/duckdb.md#bulk-loading) give the setup details. + The threshold column means that batches smaller than that number always use `executemany` (avoiding temp-file overhead for small inserts). PostgreSQL's `COPY` is in-protocol and has no file overhead, so there is no threshold. diff --git a/docs/index.md b/docs/index.md index 3a72127..45223e9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,9 +1,9 @@ --- -title: "Loading XML into a relational database" -description: "xml2db is a Python package that automatically maps an XSD schema to relational database tables and loads XML files into them, with no custom mapping rules required." +title: "Load XML into a SQL database with Python" +description: "xml2db is a Python package that turns an XSD schema into SQL tables and loads XML files into them, with no mapping rules to write." --- -# Loading XML into a relational database +# Load XML into a SQL database with Python `xml2db` is a Python package that parses and loads XML files into a relational database: @@ -11,21 +11,153 @@ description: "xml2db is a Python package that automatically maps an XSD schema t * it can handle complex XML files which cannot be denormalized into flat tables * it works out of the box, without any custom mapping rules, while still configurable. -`xml2db` fits naturally into an ETL or [ELT (Extract, Load, Transform)](https://docs.getdbt.com/terms/elt) pipeline. It loads -XML files into a relational model that stays close to the source data while remaining easy to query as flat database -tables. +`xml2db` fits naturally into an [ELT (Extract, Load, Transform)](https://docs.getdbt.com/terms/elt) pipeline: it +loads all the content of XML files into tables that stay close to the source, and the transformations happen later, +in the database. + +See the [Getting started](getting_started.md) guide for installation and the Python API, and the +[command line reference](cli.md) for all commands. + +## From XML to tables + +Take this file, where each invoice has a customer and repeated lines, described by an XSD schema: + +??? example "invoices.xsd" + + ``` xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ``` + +``` xml title="invoices.xml" + + + AliceFR + X129.90 + Y7124.00 + + + BobDE + X129.90 + + +``` + +A single command, with no mapping to write: + +``` bash +xml2db import invoices.xml invoices.xsd --connection-string "duckdb:///invoices.duckdb" +``` + +gives an `invoice` table, a `line` table and an `invoice_line` table linking them, plus tables for the `invoices` root +element. The `customer` element has a single occurrence, so its fields become columns of `invoice`. The `X1` line appears twice in the file but is stored once. + +=== "invoice" + + | pk_invoice | id | date | customer_name | customer_country | + |-----------:|---------|------------|---------------|------------------| + | 1 | INV-001 | 2026-10-01 | Alice | FR | + | 2 | INV-002 | 2026-10-02 | Bob | DE | + +=== "line" + + | pk_line | sku | quantity | price | + |--------:|-----|---------:|------:| + | 1 | X1 | 2 | 9.9 | + | 2 | Y7 | 1 | 24.0 | + +=== "invoice_line" + + | fk_invoice | fk_line | + |-----------:|--------:| + | 1 | 1 | + | 1 | 2 | + | 2 | 1 | + +The tables are then queried with plain SQL joins: + +``` sql +SELECT i.id, i.customer_name, l.sku, l.quantity, l.price +FROM invoice i +JOIN invoice_line il ON il.fk_invoice = i.pk_invoice +JOIN line l ON l.pk_line = il.fk_line; +``` + +## Load first, transform later + +Parsing XML is slow compared to running SQL on tables already in a database. Since `xml2db` loads every element and +attribute, you do not need to decide upfront which data to extract and how to reshape it: + +* explore the loaded tables with SQL before choosing transformations; +* when a new question comes, or the transformation logic changes, rerun the SQL on the loaded tables; +* keep the `xml2db` tables as a raw layer, and build the tables people query on top of them with + [dbt](dbt.md), SQL views or stored procedures. + +The same applies when the final tables live elsewhere: load into [DuckDB](databases/duckdb.md), transform with SQL, +then [export the result](databases/export.md) to files, lakehouse tables or other databases. + +## When to use xml2db + +Many XML files do not need `xml2db`. The table below compares it with common alternatives. -The raw data can then be transformed using [DBT](https://www.getdbt.com/), SQL views, or stored procedures to produce -more user-friendly tables. +| Situation | Usually enough | What xml2db adds | +|---|---|---| +| A few fields from flat, record-like XML | `pandas.read_xml`, a short `lxml` script, or the XML functions of your database | Little: these are simpler | +| One repeated record element with shallow nesting | XML readers that turn each record into a row, such as Spark's XML data source or DuckDB XML extensions | Normalized tables, with no nested columns to explode | +| Deep nesting, repeated elements at several levels, large XSD | Hand-written mappings, one per element | Tables, keys and loading code generated from the XSD, no mapping to maintain | +| Recurring feed of files following the same schema | A custom script per feed | The same load for every file, with redundant content stored once | +| Needs not settled yet: which fields, which aggregations | Extracting the fields you expect to need, and parsing the files again when needs change | Everything loaded once: new questions are answered with SQL | +| All the data must be kept, or written back to XML | Hand-written mappings, with the risk of dropping fields | Every element and attribute kept, with a round trip back to XML | -See the [Getting started](getting_started.md) guide for installation, CLI usage, and the Python API. +`xml2db` needs an XSD schema. Files are parsed in memory: files around 500 MB work in practice. Recursive schemas, +wildcards and mixed content are only partly supported, see the [caveats](how_it_works.md#caveats). ## Supported backends Built on `sqlalchemy`, `xml2db` supports multiple database backends. Integration tests cover PostgreSQL, MySQL, -MS SQL Server, and DuckDB. You may need to install a connector package (e.g. `psycopg2` or `psycopg` for PostgreSQL, -`pymysql` or `mysqlclient` for MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). See -[How it works](how_it_works.md#bulk-loading) for which drivers enable native bulk loading. +MS SQL Server, and DuckDB. Each backend has its own page with connection strings, bulk loading and configuration +advice: + +* [PostgreSQL](databases/postgresql.md) +* [SQL Server](databases/sql-server.md) +* [MySQL and MariaDB](databases/mysql.md) +* [DuckDB](databases/duckdb.md) +* [Files, lakehouses and other databases, through DuckDB](databases/export.md) + +To build transformed tables on top of the loaded data, see [Transform with dbt](dbt.md). To only generate +`CREATE TABLE` statements from an XSD, see [XSD to SQL tables](xsd-to-sql.md). ## Visualising data models @@ -33,72 +165,30 @@ MS SQL Server, and DuckDB. You may need to install a connector package (e.g. `ps [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) to represent tables and their relationships. You can see this representation and others through the interactive configuration panel in your browser (`xml2db serve`). -It looks like this: +For the invoices schema above, it looks like this: ```mermaid erDiagram - Unavailability_MarketDocument ||--o{ TimeSeries : "TimeSeries*" - Unavailability_MarketDocument ||--|{ Reason : "Reason*" - Unavailability_MarketDocument { - string mRID - string revisionNumber - NMTOKEN type - NMTOKEN process_processType - dateTime createdDateTime - string sender_MarketParticipant_mRID - NMTOKEN sender_MarketParticipant_marketRole_type - string receiver_MarketParticipant_mRID - NMTOKEN receiver_MarketParticipant_marketRole_type - string unavailability_Time_Period_timeInterval_start - string unavailability_Time_Period_timeInterval_end - NMTOKEN docStatus_value - } - TimeSeries ||--o{ Available_Period : "Available_Period*" - TimeSeries ||--o{ Available_Period : "WindPowerFeedin_Period*" - TimeSeries ||--o{ Asset_RegisteredResource : "Asset_RegisteredResource*" - TimeSeries ||--o{ Reason : "Reason*" - TimeSeries { - string mRID - NMTOKEN businessType - string biddingZone_Domain_mRID - string in_Domain_mRID - string out_Domain_mRID - date start_DateAndOrTime_date - time start_DateAndOrTime_time - date end_DateAndOrTime_date - time end_DateAndOrTime_time - NMTOKEN quantity_Measure_Unit_name - NMTOKEN curveType - string production_RegisteredResource_mRID - string production_RegisteredResource_name - string production_RegisteredResource_location_name - NMTOKEN production_RegisteredResource_pSRType_psrType - string production_RegisteredResource_pSRType_powerSystemResources_mRID - string production_RegisteredResource_pSRType_powerSystemResources_name - float production_RegisteredResource_pSRType_powerSystemResources_nominalP + invoices ||--|{ invoice : "invoice*" + invoices { } - Available_Period ||--|{ Point : "Point*" - Available_Period { - string timeInterval_start - string timeInterval_end - duration resolution + invoice ||--|{ line : "line*" + invoice { + string id + date date + string customer_name + string customer_country } - Point { - integer position - decimal quantity - } - Asset_RegisteredResource { - string mRID - string name - NMTOKEN asset_PSRType_psrType - string location_name - } - Reason { - NMTOKEN code - string text + line { + string sku + integer quantity + decimal price } ``` +Each box is a table with its columns and XSD types. The asterisk on `line*` means that a relationship table, +`invoice_line`, links the two tables, since a line can be shared by several invoices. + ## How to contribute to this project `xml2db` is developed and used at the [French energy regulation authority (CRE)](https://www.cre.fr/) to process complex diff --git a/docs/overrides/main.html b/docs/overrides/main.html deleted file mode 100644 index 59ac178..0000000 --- a/docs/overrides/main.html +++ /dev/null @@ -1,5 +0,0 @@ -{% extends "base.html" %} - -{% block extrahead %} - -{% endblock %} \ No newline at end of file diff --git a/docs/xsd-to-sql.md b/docs/xsd-to-sql.md new file mode 100644 index 0000000..1618a11 --- /dev/null +++ b/docs/xsd-to-sql.md @@ -0,0 +1,107 @@ +--- +title: "XSD to SQL: generate database tables from an XML schema" +description: "Generate SQL CREATE TABLE statements from an XSD schema for PostgreSQL, SQL Server or MySQL with xml2db, and shape the tables with a config file." +--- + +# XSD to SQL: generate database tables from an XML schema + +`xml2db` derives a relational schema from an XSD file. You can generate the `CREATE TABLE` statements without +connecting to a database or loading any XML file, to review the schema, share it, or create the tables with your own +deployment tools. + +## From the command line + +``` bash +xml2db render invoices.xsd --format ddl --db-type postgresql --output invoices.sql +``` + +`--db-type` accepts `postgresql`, `mssql`, `mysql` and `mariadb`. Without it, the statements use generic SQL. + +For the invoices schema of the [introduction](index.md#from-xml-to-tables), this gives, among other tables: + +``` sql +CREATE TABLE line ( + pk_line SERIAL NOT NULL, + sku VARCHAR(1000), + quantity INTEGER, + price DOUBLE PRECISION, + xml2db_record_hash BYTEA, + CONSTRAINT cx_pk_line PRIMARY KEY (pk_line), + CONSTRAINT line_xml2db_record_hash UNIQUE (xml2db_record_hash) +) + +CREATE TABLE invoice ( + pk_invoice SERIAL NOT NULL, + id VARCHAR(1000), + date VARCHAR(16), + customer_name VARCHAR(1000), + customer_country VARCHAR(1000), + xml2db_record_hash BYTEA, + CONSTRAINT cx_pk_invoice PRIMARY KEY (pk_invoice), + CONSTRAINT invoice_xml2db_record_hash UNIQUE (xml2db_record_hash) +) + +CREATE TABLE invoice_line ( + fk_invoice INTEGER NOT NULL, + fk_line INTEGER NOT NULL, + FOREIGN KEY(fk_invoice) REFERENCES invoice (pk_invoice), + FOREIGN KEY(fk_line) REFERENCES line (pk_line) +) +``` + +The same output is shown in the DDL tab of `xml2db serve`, next to the diagram, and updates as you edit the +configuration. + +## How the schema is built + +* Each complex type becomes a table, with an integer primary key named `pk_
`. +* Simple elements and attributes become columns, with types derived from the XSD types and their restrictions + (`maxLength`, for instance). +* A child element that occurs at most once is merged into its parent as prefixed columns, such as `customer_name`. +* A repeated child element gets its own table. By default, identical records are stored once, identified by the + `xml2db_record_hash` column, and a relationship table such as `invoice_line` links parents and children. + +[How it works](how_it_works.md#building-a-data-model) describes these rules in detail. + +## Adjusting the schema + +The configuration file changes names, types and structure. For instance, this configuration renames a column, sets an +exact decimal type, and stores lines without deduplication, which replaces the `invoice_line` table with a foreign key +to the parent invoice: + +``` yaml title="model_config.yml" +tables: + invoice: + fields: + customer_name: + rename: customer + line: + reuse: false + fields: + price: + type: Numeric(12, 2) +``` + +``` bash +xml2db render invoices.xsd --format ddl --db-type postgresql --config model_config.yml +``` + +See [Configuring your data model](configuring.md) for all options, and the database pages for backend-specific +advice: [PostgreSQL](databases/postgresql.md), [SQL Server](databases/sql-server.md), +[MySQL and MariaDB](databases/mysql.md), [DuckDB](databases/duckdb.md). + +## From Python + +``` py title="Write the DDL to a file" linenums="1" +from sqlalchemy.dialects import postgresql +from xml2db import DataModel + +data_model = DataModel(xsd_file="invoices.xsd", db_type="postgresql") +with open("invoices.sql", "w") as f: + for statement in data_model.get_all_create_table_statements(): + f.write(f"{statement.compile(dialect=postgresql.dialect())};\n") + for statement in data_model.get_all_create_index_statements(): + f.write(f"{statement.compile(dialect=postgresql.dialect())};\n") +``` + +When `xml2db` loads data, it creates missing tables itself: generating the DDL is not a required step. diff --git a/mkdocs.yml b/mkdocs.yml index 1850517..dab3361 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -3,12 +3,20 @@ site_url: https://cre-dev.github.io/xml2db repo_url: https://github.com/cre-dev/xml2db repo_name: cre-dev/xml2db copyright: Copyright © 2026 Commission de régulation de l'énergie -site_description: xml2db is a Python package to load XML files into a relational database. It handles complex XML and - works out of the box, without any custom mapping rules. +site_description: xml2db is a Python package that turns an XSD schema into SQL tables and loads XML files into them, + with no mapping rules to write. nav: - Introduction: "index.md" - Getting started: "getting_started.md" + - Databases: + - PostgreSQL: "databases/postgresql.md" + - SQL Server: "databases/sql-server.md" + - MySQL and MariaDB: "databases/mysql.md" + - DuckDB: "databases/duckdb.md" + - Export through DuckDB: "databases/export.md" + - Transform with dbt: "dbt.md" + - XSD to SQL tables: "xsd-to-sql.md" - Configuring: "configuring.md" - How it works: "how_it_works.md" - CLI usage: "cli.md" @@ -20,7 +28,7 @@ nav: theme: name: material - custom_dir: docs/overrides + custom_dir: overrides features: - navigation.instant - navigation.instant.prefetch @@ -54,6 +62,8 @@ markdown_extensions: pygments_lang_class: true - pymdownx.inlinehilite - pymdownx.snippets + - pymdownx.tabbed: + alternate_style: true - pymdownx.superfences: custom_fences: - name: mermaid diff --git a/overrides/main.html b/overrides/main.html new file mode 100644 index 0000000..ef35ffa --- /dev/null +++ b/overrides/main.html @@ -0,0 +1,28 @@ +{% extends "base.html" %} + +{% block extrahead %} + + {#- Open Graph and Twitter tags for link previews on social networks and chat apps -#} + {%- if page and page.meta and page.meta.title -%} + {%- set preview_title = page.meta.title -%} + {%- elif page and page.title -%} + {%- set preview_title = page.title -%} + {%- else -%} + {%- set preview_title = config.site_name -%} + {%- endif -%} + {%- if page and page.meta and page.meta.description -%} + {%- set preview_description = page.meta.description -%} + {%- else -%} + {%- set preview_description = config.site_description -%} + {%- endif %} + + + + + {%- if page and page.canonical_url %} + + {%- endif %} + + + +{% endblock %} diff --git a/pyproject.toml b/pyproject.toml index 48f5a3c..fa08fd5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,17 +4,35 @@ build-backend = "setuptools.build_meta" [project] name = "xml2db" -version = "0.15.0" +version = "0.15.1" authors = [ { name="Commission de régulation de l'énergie", email="opensource@cre.fr" }, ] -description = "Import complex XML files to a relational database" +description = "Load complex XML files into a relational database, with tables generated from the XSD schema" readme = "README.md" requires-python = ">=3.11" +keywords = [ + "xml", "xsd", "xml-schema", "etl", "elt", "data-engineering", "sql", "ddl", + "sqlalchemy", "postgresql", "mysql", "mariadb", "sql-server", "duckdb", +] classifiers = [ + "Development Status :: 4 - Beta", + "Environment :: Console", + "Intended Audience :: Developers", + "Intended Audience :: Information Technology", + "Intended Audience :: Science/Research", "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", + "Programming Language :: SQL", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", + "Topic :: Database", + "Topic :: Scientific/Engineering :: Information Analysis", + "Topic :: Text Processing :: Markup :: XML", ] dependencies = [ "sqlalchemy>1.4", diff --git a/src/xml2db/model.py b/src/xml2db/model.py index 920db4b..717b117 100644 --- a/src/xml2db/model.py +++ b/src/xml2db/model.py @@ -197,6 +197,11 @@ def _create_table_model( table_config = self.tables_config.get(table_name, {}) if table_config is None: table_config = {} + if is_root_table and not table_config.get("reuse", True): + raise DataModelConfigError( + f"'reuse: false' is not supported for the root table '{table_name}': the root table is always " + f"deduplicated, so that loading the same file twice stores it once" + ) if table_config.get("reuse", True): return DataModelTableReused( table_name, diff --git a/tests/test_config.py b/tests/test_config.py index d6333e5..8fa74d7 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -168,6 +168,7 @@ def test_invalid_transform_value_raises(self): with pytest.raises(DataModelConfigError, match="Invalid 'transform'"): self._make_model({"transform": "none"}) + def test_transform_false_per_field_auto_overrides_global(self): # Global false but explicit per-field "auto" should restore default elevation m_off = self._make_model({"transform": False}) @@ -185,6 +186,12 @@ def test_transform_false_per_field_auto_overrides_global(self): ) +def test_root_table_without_reuse_raises(): + # the root table is always deduplicated, so that loads are idempotent + with pytest.raises(DataModelConfigError, match="root table 'orders'"): + DataModel(xsd_file=ORDERS_XSD, model_config={"tables": {"orders": {"reuse": False}}}) + + # --------------------------------------------------------------------------- # Field-level transform: "auto" and choice_transform: "auto" # ---------------------------------------------------------------------------