Prefect 2.0 + Fivetran = A Perfect Pair
How these two tools integrate into a seamless data pipeline
When I first became an analytics engineer I was tasked with creating a modern data stack from scratch. My team needed to consolidate data collected from our e-commerce website with data from external sources such as Facebook, Bing Ads, Google sheets, and email. Marketing, growth, and product teams wanted to better understand where our traffic was coming from and how that related to how much we were spending.
I had the freedom to choose an ingestion, transformation, and orchestration tool, along with a data warehouse to act as our single source of truth. I found that Fivetran offered a wide range of data sources for ingestion along with top-notch security practices. They made data ingestion seamless! I discovered Prefect after a very frustrating run with Airflow, falling in love with the intuitive UI and ability to handle upstream and downstream dependencies.
Fast forward to a year and a half later and I’ve been using these two tools almost every day in conjunction with Snowflake as my data warehouse and dbt as my transformation tool. It’s genuinely a data stack made in heaven! Fivetran runs all of our data syncs, dbt then transforms this source data into our core data models, and Prefect is the tool that orchestrates it all! Prefect’s ability to handle dependencies…
