Distributed streaming with finite memory

Abstract

We introduce three formal models of distributed systems for query evaluations on massive databases: Distributed Streaming with Register Automata (DSA), Distributed Streaming with Register Transducer (DST), and Distributed Streaming with Register Transducer and Join (DSTJ). These models are based on the key-value paradigm, where the input is transformed into a dataset of key-value pairs, and on each key a local computation is performed on the values associated with that key resulting in another set of key-value pairs. Computation proceed in a constant number of rounds, where the result of the last round is the input to the next round, and transformation to key-value pairs is required to be generic. The difference between the three models is in the local computation part. In DSA it is limited to making one pass over its input using a register automaton, while in DST it can make two passes: in the first pass it uses a finite-state automaton and in the second it uses a register transducer. The third model DSTJ is an extension of DST, where in the local computation a Cartesian product between two sets is allowed. We obtain the following results: (1) DSAs can evaluate the first-order queries over bounded degree databases; (2) DSTs can evaluate the semijoin algebra; (3) DSTJs can evaluate the whole relational algebra; (4) DSTJs are strictly stronger than DSTs, which in turn, are strictly stronger than DSAs; (5) within DSAs, DSTs and DSTJs there is a strict hierarchy w.r.t. the number of rounds.