Modern Spark is nice enough that I will use it to work with massive excel files locally! If someone can use Pandas they can use Spark, and you get the added benefit of distribution if necessary and more resiliancy.
I think it's likely that some subsection of Spark users are the type to over engineer a project, but I'm also confident they'd over engineer a much simpler framework as well.
I think it's likely that some subsection of Spark users are the type to over engineer a project, but I'm also confident they'd over engineer a much simpler framework as well.