diff --git a/CHANGELOG.md b/CHANGELOG.md index 80988a6..61428c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,7 @@ versions may change the store format and the API; the release notes say how to u - Under `--executor ids`, `BIND`s of strings and IRIs run on term ids too (#11): `STR`, `CONCAT`, `IRI`, `STRLANG`, `STRDT` and `REPLACE` with a constant pattern, each computed once per distinct input. On 133k orders, `BIND(CONCAT(STR(?c), "/order") AS ?k)` takes 45 instead of 91 ms, and with a join on an `IRI` made that way 57 instead of 196 ms. See [the store](docs/store.md). - Under `--executor ids`, a `REPLACE` whose pattern or flags are expressions without variables (`REPLACE(STR(?x), STR(ex:), "")`) runs on term ids too (#11), computed once, so the queries that rewrite IRIs of one namespace into another stay on ids. - Under `--executor ids`, `/` runs on term ids too (#11), in `BIND`, filters and orders: a quotient of integers or decimals is the evaluator's (exact, as Jena's), of doubles a double, and division by zero an error. On 133k orders, `BIND(?q / 4 AS ?h) FILTER(?h > 20)` takes 33 ms instead of 191. A decimal quotient that doesn't end within 18 fraction digits (`1 / 3`) is still left to the evaluator. +- Under `--executor ids`, a `GROUP BY` over a `BIND` with few distinct inputs counts the rows per input first, as under `spareval`, and computes the expression once per input instead of once per row (#11): `BIND(?q * 2 AS ?k) … GROUP BY ?k` over 133k orders takes as long as under `spareval` (11 ms) instead of 16 to 21. - Under `--executor ids`, `ABS`, `CEIL`, `FLOOR` and `ROUND` of doubles, and `xsd:integer` and `xsd:decimal` casts of numbers, run on term ids too (#11), with the evaluator's answers. A rule such as `BIND(ROUND(?amount * 1000) / 1000 AS ?r)` over doubles no longer sends the query to the evaluator. - Under `--executor ids`, `IF` and `COALESCE` run on term ids too (#11), in `BIND` and filters: the condition a comparison of numbers or dates, `!`, `&&`, `||` or `BOUND`, the branches numbers or the strings and IRIs the id executor makes. On 133k orders, `BIND(IF(?q > 50, "big", "small") AS ?k)` takes 35 instead of 67 ms, and `BIND(COALESCE(?q * 2, 0) AS ?t)` after an `OPTIONAL` 46 instead of 173 ms. - Under `--executor ids`, `ORDER BY` an expression the id kernels compute (`DESC(?q * 3)`, `ABS(?x)`, `?d - 1`) runs on term ids too (#11), as a column of the computed values. On 133k orders, `ORDER BY DESC(?q * 3) ?o LIMIT 10` takes 15 ms instead of 117, and with a join and `LIMIT 100` 20 ms instead of 245. Rows with equal keys may come in another order than under `spareval`. diff --git a/crates/tripletrail/src/count.rs b/crates/tripletrail/src/count.rs index ae126fe..977fbe9 100644 --- a/crates/tripletrail/src/count.rs +++ b/crates/tripletrail/src/count.rs @@ -674,6 +674,18 @@ impl Pass<'_> { ], }; } + // Over a `BIND`, counting the rows per distinct input first computes the + // expression once per input value, not once per row: under the id executor + // the grouped counts below would compute it for every row (#11). + let computed = matches!(*inner, G::Extend { .. }) + || matches!(&*inner, G::Filter { inner, .. } if matches!(**inner, G::Extend { .. })); + if computed + && crate::executor_ids() + && let Some(group) = self.eager_group(&inner, &variables, &aggregates) + { + self.done += 1; + return group; + } // Without keys, counts are answered above; MIN, MAX and SAMPLE here. if (!variables.is_empty() || count_aggregates(&aggregates).is_none()) && let Some(values) = diff --git a/crates/tripletrail/tests/id_rows.rs b/crates/tripletrail/tests/id_rows.rs index 262da3f..0d00995 100644 --- a/crates/tripletrail/tests/id_rows.rs +++ b/crates/tripletrail/tests/id_rows.rs @@ -797,6 +797,20 @@ fn binds_on_ids_are_the_evaluators() { /// Groups a count pass reads back from a table of ids (1,024 or more): the id executor answers /// the query from that table (#11), with the evaluator's answers. const GROUPS: &[(&str, bool)] = &[ + // Over a `BIND` with few distinct inputs: counted per input first, the expression then + // computed per input (#11). + ( + "SELECT ?k (COUNT(*) AS ?n) WHERE { ?o ex:q ?q BIND(?q * 2 AS ?k) } GROUP BY ?k", + false, + ), + ( + "SELECT ?k (SUM(?q) AS ?t) (MAX(?q) AS ?m) WHERE { ?o ex:q ?q BIND(?q - 3 AS ?k) FILTER(?k > 1) } GROUP BY ?k", + false, + ), + ( + "SELECT ?c (COUNT(DISTINCT ?k) AS ?n) WHERE { ?o ex:by ?c ; ex:q ?q BIND(?q / 2 AS ?k) } GROUP BY ?c", + false, + ), ( "SELECT ?c (COUNT(*) AS ?n) WHERE { ?o ex:by ?c } GROUP BY ?c", true, diff --git a/docs/server.md b/docs/server.md index 6ffe336..5249d8a 100644 --- a/docs/server.md +++ b/docs/server.md @@ -206,7 +206,7 @@ tripletrail serve --store oxigraph --location /data/tripletrail --dataset kg - `--executor` (`TRIPLETRAIL_EXECUTOR`, `spareval` by default) picks what answers a `SELECT` whose pattern runs on term ids (#11). `ids` answers it straight from the id tables (triple patterns, joins, `OPTIONAL`, `MINUS`, `UNION`, `FILTER (NOT) EXISTS`, the filters on ids, - `BIND` and `SELECT (expr AS ?v)` of copies, constants, arithmetic, `STR`, `CONCAT`, `IRI`, `STRLANG`, `STRDT` and `REPLACE`, `VALUES`, sub-selects (with `DISTINCT`, an order and a slice), the `GROUP BY` counts and sums kept as ids, `ORDER BY` of variables and of the expressions the kernels compute, `DISTINCT`, `REDUCED`, `OFFSET` and `LIMIT` with an order), decoding + `BIND` and `SELECT (expr AS ?v)` of copies, constants, arithmetic with division, `ABS`, `CEIL`, `FLOOR`, `ROUND`, `xsd:integer` and `xsd:decimal` casts, `IF`, `COALESCE`, `STR`, `CONCAT`, `IRI`, `STRLANG`, `STRDT` and `REPLACE` (the full list and its limits are in [the store](store.md)), `VALUES`, sub-selects (with `DISTINCT`, an order and a slice), the `GROUP BY` counts and sums kept as ids, `ORDER BY` of variables and of the expressions the kernels compute, `DISTINCT`, `REDUCED`, `OFFSET` and `LIMIT` with an order), decoding only the terms of the answer, each once, on the query threads; anything else still goes to the evaluator. SPARQL JSON, CSV and TSV answers are written from the ids: each distinct term is written once and copied into the rows where it occurs (the bytes are those of the usual diff --git a/docs/store.md b/docs/store.md index 5cce386..ae6f1dc 100644 --- a/docs/store.md +++ b/docs/store.md @@ -82,7 +82,10 @@ there: `Engine::new_oxigraph()` or `TRIPLETRAIL_STORE=oxigraph`, `tripletrail se without `GROUP BY`: IRIs and simple strings compared by their dictionary keys, integers and booleans by value; other values (blank nodes, language strings, other typed literals, mixed literal kinds) leave the group to the evaluator. Under `ORDER BY` of counts with `LIMIT k`, - only the first k groups are kept, before their other aggregates are computed. BSBM 1M: + only the first k groups are kept, before their other aggregates are computed. A `GROUP BY` + over `BIND`s with few distinct inputs (`BIND(?q * 2 AS ?k) … GROUP BY ?k`) counts the rows + per input first and computes the expression once per input, under `--executor ids` too + (#11). BSBM 1M: `group-by-complex-aggregate` 54 → 16 ms, `group-by-implicit-string-min` 11 → 4 ms. Inline integers, decimals, dates and date-times (#125) are read as values from their ids (`crates/tripletrail/src/idvalue.rs`, #170): `<`, `<=`, `>`, `>=` and `=` between them and