Dagster with Nick Schrock
Play • 1 hr 14 min

Data engineering is difficult. 

Companies want to be able to maximize the value they get from their large data sets, but there are so many steps required for deriving that value that most companies feel like they are always far behind the ideal state of where they could be. 

The cloud makes it cheap to save data. Tools like Spark and Snowflake give us usable APIs for simplifying the data processing. Workflow engines like Airflow help us visualize a complex execution path of a data pipeline. With all of this tooling, why is it so hard to make use of our data?

Nick Schrock is the creator of Dagster, an open source system for building modern data applications. Nick is also the CEO of Elementl, a company that he is building around Dagster. Before creating Dagster, Nick worked at Facebook, where he co-created GraphQL. 

Nick returns to the show to discuss modern data engineering, and why it continues to be so difficult for engineers to be productive with their data.

More episodes
Search
Clear search
Close search
Google apps
Main menu