A little background, I am an old school ETL Architect who has been the industry since 2007.
I have seen Hadoop come and go (ish) and Spark slowly settling into what it is now.
I was recently helped our team migrate from Teradata to Snowflake as our data warehouse. We also use Spark and Kafka to enhance our DW applications.
Our current ETL jobs run as Informatica mappings, after the transition to Snowflake, we have used native SQL as opposed to mappings. We have also started leveraging Python for ETL development, we have been using it to connect to API's and extract data, create run time SQL's and issue the same in Snowflake.
For now, we have continue to use Informatica as an orchestration tool - albeit a very expensive one.
I was wondering if you have had any experience/opinion in using AirFlow as an orchestration tool.
I ran into your YouTube channel and was pleasantly surprised - I have subscribed and looking forward to your live videos from now on.