Spent my 2025 building https://orchestera.com as a side project. The premise is simple - to give Data Engineers and Data Scientists the opportunity to spin up Apache Spark clusters on Amazon EKS without them needing to know all the infrastructure details, how Spark works on Kubernetes, auto-scaling etc.
The platform I am building allows users to launch Spark on Kubernetes in their own AWS account without adding any markup costs to the CPU/Memory on EC2 instances. For example, AWS EMR offering adds a 25% markup cost on top of the EC2 instance pricing. Databricks markup is even higher ranging anywhere from 30% to 100% markup.
The platform I am building allows users to launch Spark on Kubernetes in their own AWS account without adding any markup costs to the CPU/Memory on EC2 instances. For example, AWS EMR offering adds a 25% markup cost on top of the EC2 instance pricing. Databricks markup is even higher ranging anywhere from 30% to 100% markup.
To learn more, check out the docs for the project at https://docs.orchestera.com/