Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ versions may change the store format and the API; the release notes say how to u
- Under `--executor ids`, `BIND`s of strings and IRIs run on term ids too (#11): `STR`, `CONCAT`, `IRI`, `STRLANG`, `STRDT` and `REPLACE` with a constant pattern, each computed once per distinct input. On 133k orders, `BIND(CONCAT(STR(?c), "/order") AS ?k)` takes 45 instead of 91 ms, and with a join on an `IRI` made that way 57 instead of 196 ms. See [the store](docs/store.md).
- Under `--executor ids`, a `REPLACE` whose pattern or flags are expressions without variables (`REPLACE(STR(?x), STR(ex:), "")`) runs on term ids too (#11), computed once, so the queries that rewrite IRIs of one namespace into another stay on ids.
- Under `--executor ids`, `/` runs on term ids too (#11), in `BIND`, filters and orders: a quotient of integers or decimals is the evaluator's (exact, as Jena's), of doubles a double, and division by zero an error. On 133k orders, `BIND(?q / 4 AS ?h) FILTER(?h > 20)` takes 33 ms instead of 191. A decimal quotient that doesn't end within 18 fraction digits (`1 / 3`) is still left to the evaluator.
- Under `--executor ids`, a `GROUP BY` over a `BIND` with few distinct inputs counts the rows per input first, as under `spareval`, and computes the expression once per input instead of once per row (#11): `BIND(?q * 2 AS ?k) … GROUP BY ?k` over 133k orders takes as long as under `spareval` (11 ms) instead of 16 to 21.
- Under `--executor ids`, `ABS`, `CEIL`, `FLOOR` and `ROUND` of doubles, and `xsd:integer` and `xsd:decimal` casts of numbers, run on term ids too (#11), with the evaluator's answers. A rule such as `BIND(ROUND(?amount * 1000) / 1000 AS ?r)` over doubles no longer sends the query to the evaluator.
- Under `--executor ids`, `IF` and `COALESCE` run on term ids too (#11), in `BIND` and filters: the condition a comparison of numbers or dates, `!`, `&&`, `||` or `BOUND`, the branches numbers or the strings and IRIs the id executor makes. On 133k orders, `BIND(IF(?q > 50, "big", "small") AS ?k)` takes 35 instead of 67 ms, and `BIND(COALESCE(?q * 2, 0) AS ?t)` after an `OPTIONAL` 46 instead of 173 ms.
- Under `--executor ids`, `ORDER BY` an expression the id kernels compute (`DESC(?q * 3)`, `ABS(?x)`, `?d - 1`) runs on term ids too (#11), as a column of the computed values. On 133k orders, `ORDER BY DESC(?q * 3) ?o LIMIT 10` takes 15 ms instead of 117, and with a join and `LIMIT 100` 20 ms instead of 245. Rows with equal keys may come in another order than under `spareval`.
Expand Down
12 changes: 12 additions & 0 deletions crates/tripletrail/src/count.rs
Original file line number Diff line number Diff line change
Expand Up @@ -674,6 +674,18 @@ impl Pass<'_> {
],
};
}
// Over a `BIND`, counting the rows per distinct input first computes the
// expression once per input value, not once per row: under the id executor
// the grouped counts below would compute it for every row (#11).
let computed = matches!(*inner, G::Extend { .. })
|| matches!(&*inner, G::Filter { inner, .. } if matches!(**inner, G::Extend { .. }));
if computed
&& crate::executor_ids()
&& let Some(group) = self.eager_group(&inner, &variables, &aggregates)
{
self.done += 1;
return group;
}
// Without keys, counts are answered above; MIN, MAX and SAMPLE here.
if (!variables.is_empty() || count_aggregates(&aggregates).is_none())
&& let Some(values) =
Expand Down
14 changes: 14 additions & 0 deletions crates/tripletrail/tests/id_rows.rs
Original file line number Diff line number Diff line change
Expand Up @@ -797,6 +797,20 @@ fn binds_on_ids_are_the_evaluators() {
/// Groups a count pass reads back from a table of ids (1,024 or more): the id executor answers
/// the query from that table (#11), with the evaluator's answers.
const GROUPS: &[(&str, bool)] = &[
// Over a `BIND` with few distinct inputs: counted per input first, the expression then
// computed per input (#11).
(
"SELECT ?k (COUNT(*) AS ?n) WHERE { ?o ex:q ?q BIND(?q * 2 AS ?k) } GROUP BY ?k",
false,
),
(
"SELECT ?k (SUM(?q) AS ?t) (MAX(?q) AS ?m) WHERE { ?o ex:q ?q BIND(?q - 3 AS ?k) FILTER(?k > 1) } GROUP BY ?k",
false,
),
(
"SELECT ?c (COUNT(DISTINCT ?k) AS ?n) WHERE { ?o ex:by ?c ; ex:q ?q BIND(?q / 2 AS ?k) } GROUP BY ?c",
false,
),
(
"SELECT ?c (COUNT(*) AS ?n) WHERE { ?o ex:by ?c } GROUP BY ?c",
true,
Expand Down
2 changes: 1 addition & 1 deletion docs/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ tripletrail serve --store oxigraph --location /data/tripletrail --dataset kg
- `--executor` (`TRIPLETRAIL_EXECUTOR`, `spareval` by default) picks what answers a `SELECT`
whose pattern runs on term ids (#11). `ids` answers it straight from the id tables (triple
patterns, joins, `OPTIONAL`, `MINUS`, `UNION`, `FILTER (NOT) EXISTS`, the filters on ids,
`BIND` and `SELECT (expr AS ?v)` of copies, constants, arithmetic, `STR`, `CONCAT`, `IRI`, `STRLANG`, `STRDT` and `REPLACE`, `VALUES`, sub-selects (with `DISTINCT`, an order and a slice), the `GROUP BY` counts and sums kept as ids, `ORDER BY` of variables and of the expressions the kernels compute, `DISTINCT`, `REDUCED`, `OFFSET` and `LIMIT` with an order), decoding
`BIND` and `SELECT (expr AS ?v)` of copies, constants, arithmetic with division, `ABS`, `CEIL`, `FLOOR`, `ROUND`, `xsd:integer` and `xsd:decimal` casts, `IF`, `COALESCE`, `STR`, `CONCAT`, `IRI`, `STRLANG`, `STRDT` and `REPLACE` (the full list and its limits are in [the store](store.md)), `VALUES`, sub-selects (with `DISTINCT`, an order and a slice), the `GROUP BY` counts and sums kept as ids, `ORDER BY` of variables and of the expressions the kernels compute, `DISTINCT`, `REDUCED`, `OFFSET` and `LIMIT` with an order), decoding
only the terms of the answer, each once, on the query threads; anything else still goes to
the evaluator. SPARQL JSON, CSV and TSV answers are written from the ids: each distinct term
is written once and copied into the rows where it occurs (the bytes are those of the usual
Expand Down
5 changes: 4 additions & 1 deletion docs/store.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,10 @@ there: `Engine::new_oxigraph()` or `TRIPLETRAIL_STORE=oxigraph`, `tripletrail se
without `GROUP BY`: IRIs and simple strings compared by their dictionary keys, integers and
booleans by value; other values (blank nodes, language strings, other typed literals, mixed
literal kinds) leave the group to the evaluator. Under `ORDER BY` of counts with `LIMIT k`,
only the first k groups are kept, before their other aggregates are computed. BSBM 1M:
only the first k groups are kept, before their other aggregates are computed. A `GROUP BY`
over `BIND`s with few distinct inputs (`BIND(?q * 2 AS ?k) … GROUP BY ?k`) counts the rows
per input first and computes the expression once per input, under `--executor ids` too
(#11). BSBM 1M:
`group-by-complex-aggregate` 54 → 16 ms, `group-by-implicit-string-min` 11 → 4 ms.
Inline integers, decimals, dates and date-times (#125) are read as values from their ids
(`crates/tripletrail/src/idvalue.rs`, #170): `<`, `<=`, `>`, `>=` and `=` between them and
Expand Down
Loading