My name is Ganesh. I am Apache Beam contributor and I've been working on FlareDB, a streaming database for running batch and streaming data pipelines. It uses Apache Beam as programing interfere for writing data pipelines and running it on FlareDB's engine, instred of building its own new API or using SQL. Leveraging Apache Beam allows FlareDB to run pipelines in java, python go and SQL.
FlareDB is built on a unified worldview where stream processing and database storage are not two separate systems, but two sides of the same coin. Traditional data architectures separate stream processing engines from databases. Streaming systems such as Apache Flink and Google Cloud Dataflow excel at stream processing, while databases provide durable storage and query capabilities. This separation requires pipelines to move data from one system into another before it can be queried, analyzed and served. FlareDB eliminates this divide by building its execution engine directly around the Stream-Table Duality.
Streams and tables are fundamentally the same data expressed in two different relative states:
Streams are data in motion: They represent a continuous, append-only history of transformations. A stream captures the evolution of data over time the individual delta changes, transactions, and events flowing through the system. Transformations on data produce streams.
Tables are data at rest: They act as the conceptual resting place where those changes accumulate. A table allows you to observe the dataset as a whole at a specific point in time. It represents the current materialized state computed from the stream’s history.
This is a fundamental idea in databases. Every change to a table is first recorded as an append-only log (or write-ahead log), while the table itself represents the accumulated state of those changes. Streams and tables are simply two different views of the same data.
FlareDB applies these principles to Apache Beam.
Repo: https://github.com/flare-db/flare-db
Like to hear, comments and feedback : )