The motivation is that queries that are trivial in a SQL database are painfully impractical over any other data source. Most query languages are tied way too closely to a data format or representation, and fail miserably at the boundaries to other systems or representations. Yes, we can e.g. plug SQL into APIs but there's a severe impedance mismatch there -- we end up forced into all sorts of unintuitive situations and poor UX because SQL was never meant for APIs.
Based on experiences I had at work (some of them covered in that talk!), it felt like an "LLVM for data sources" could be useful -- that's what Trustfall is. Yes, it includes a query language. That's because SQL and GraphQL and the like just weren't a good fit to demonstrate true datasource-agnostic queries: we want a modern GraphQL-like user experience, with SQL-like expressiveness but without limitations that make it hard to plug in APIs and ML tool. But the query language is just one layer of the system, and it's entirely possible to replace it with another language parser while keeping the rest of the benefits.
A key benefit of Trustfall is that you can write queries over any data sources, and then change the underlying implementation to add optimizations or to migrate to a new system without the queries needing to change. For example, cargo-semver-checks, a Rust semver linter I built on top of Trustfall, supports 4 separate format versions of the underlying data source it uses with the same set of queries, and I was recently able to speed up its execution by over 2000x with a tiny amount of code and effort and without even a single query needing to change -- just by tweaking how the data adapter executed queries to plug in some indexes where they mattered: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
Excited to see what you build, please post a link or email / DM me on Twitter if you'd rather not post publicly? The docs are still sparse (I wasn't planning to do a Show HN for another couple of months, but someone beat me to it today) but I'd be happy to help with anything. There are Python and Rust bindings, and feel free to open issues for anything that's confusing:
The motivation is that queries that are trivial in a SQL database are painfully impractical over any other data source. Most query languages are tied way too closely to a data format or representation, and fail miserably at the boundaries to other systems or representations. Yes, we can e.g. plug SQL into APIs but there's a severe impedance mismatch there -- we end up forced into all sorts of unintuitive situations and poor UX because SQL was never meant for APIs.
Based on experiences I had at work (some of them covered in that talk!), it felt like an "LLVM for data sources" could be useful -- that's what Trustfall is. Yes, it includes a query language. That's because SQL and GraphQL and the like just weren't a good fit to demonstrate true datasource-agnostic queries: we want a modern GraphQL-like user experience, with SQL-like expressiveness but without limitations that make it hard to plug in APIs and ML tool. But the query language is just one layer of the system, and it's entirely possible to replace it with another language parser while keeping the rest of the benefits.
A key benefit of Trustfall is that you can write queries over any data sources, and then change the underlying implementation to add optimizations or to migrate to a new system without the queries needing to change. For example, cargo-semver-checks, a Rust semver linter I built on top of Trustfall, supports 4 separate format versions of the underlying data source it uses with the same set of queries, and I was recently able to speed up its execution by over 2000x with a tiny amount of code and effort and without even a single query needing to change -- just by tweaking how the data adapter executed queries to plug in some indexes where they mattered: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
Excited to see what you build, please post a link or email / DM me on Twitter if you'd rather not post publicly? The docs are still sparse (I wasn't planning to do a Show HN for another couple of months, but someone beat me to it today) but I'd be happy to help with anything. There are Python and Rust bindings, and feel free to open issues for anything that's confusing:
https://pypi.org/project/trustfall/
https://crates.io/crates/trustfall_core