DevOps, Development and Operations is very important for data engineers and data scientists. But how exactly are Data Engineering and DevOps related?
If you work as a data engineer, you will build pipelines.
They don't just exist and someone has to code them. So it's the data engineer's job - your job - to actually code these things and then bring them into production and run them.
So you do the development, but you usually do the operations as well.
So at some point you're responsible that your code, your pipelines work. So you do the operations for those platforms accordingly.
Sometimes in smaller companies, the platform stuff, the running of the tools, is also the data engineer's job, sometimes not.
Sometimes there are people. Sometimes you're on platform-as-a-service tools from AWS - from a cloud vendor - or software as a service, and then you don't have to worry about so many tools.
But in general, I would say, yes, devops is a big part of data technology.
How exactly do you think Data Engineering and DevOps are related? Write it down in the comments!
>> created by Mira Roth
My free 100+ pages Data Engineering Cookbook
Follow us on LinkedIn
Check out my YouTube
Check out my full video on YouTube!