Some relations (such as Datomic's :db/ident, or relations carrying metadata) are comparatively small but can induce massive skew, especially once reverse indices are involved. This can lead to straggling workers and lots of exchange data.
It might therefore be useful to introduce a new dimension of attribute configuration
Distribution := Sharded | Shuffled | Broadcasted
that would allow us to configure small, critical relations as Broadcasted, while keeping their would-be-skewed join partners entirely local, or shuffled randomly across workers.
The query engine might then need to enforce some new rules for what types of distribution are allowed to go together in a join.
Some relations (such as Datomic's
:db/ident, or relations carrying metadata) are comparatively small but can induce massive skew, especially once reverse indices are involved. This can lead to straggling workers and lots of exchange data.It might therefore be useful to introduce a new dimension of attribute configuration
that would allow us to configure small, critical relations as
Broadcasted, while keeping their would-be-skewed join partners entirely local, or shuffled randomly across workers.The query engine might then need to enforce some new rules for what types of distribution are allowed to go together in a join.