"Since I started exploring Data Engineering, it has been overwhelming. In the end I have the feeling of giving up."
This is a message that reached me from a viewer on YouTube.
And that's exactly how I feel sometimes! Sometimes I feel a bit overwhelmed by the whole thing. Because there is so much going on. All the technology and Data Science hype.
There is always something new on the horizon. And there's always something going on.
You always have new tools - for example, you have Spark, then you have Spark 2.0 and then all of a sudden you have Spark 3, or Kafka - Kafka Streaming, Kafka Connect and so on...
And this is exactly what I find to be overwhelming. And that's why I can understand the viewer that he sometimes has the feeling he has to give up.
How do you see the whole thing? Is it sometimes similar to you?
A short look at the story
The whole thing started a few years ago. There were database developers, database guys, web interface specialists and yeah.
You knew where you wanted to go and then you could get into this one topic and become an expert - or at least feel like an expert ;-)
But let's do a Google trend analysis for this. Let's just see how the whole industry moved.
So here is the trend analysis on the topic of Big Data. If you look at this, you can see that a few years ago, everyone was talking about Big Data and how Big Data revolutionizing everything. You can see very clearly from this trend how big Big Data is and how big it has become.
And then, however, it went down again after the peak around 2014.
So, let's bring Hadoop into play here.
Everyone suddenly started talking about Hadoop. Everyone should learn Hadoop.
And you can see the similar trend here as well. So it is almost the same.
And that's the thing that leads to all the confusion that I mentioned in the beginning. There was a time when people said, "Okay, let's look at Hadoop and become a Hadoop expert. Because we used to be database experts and now we're Haoop experts."
But then that disappeared again. And now people need to look in different areas...
Focus yourself!
And this is exactly the point why I want to give you this following tip. I hope it helps you to avoid such confusion.
Focus! Focus on certain technologies. Focus on certain tools.
And don't learn everything! It makes absolutely no sense to do everything and try to learn everything. It is not at all possible to learn everything.
There is a big Data Science world and you just have to figure out which direction you want to go and then you have to focus on that!
What do you think about this whole thing? Can you understand my way of thinking or do you sometimes feel like me? Let me know in the comments!
>> created by Mira Roth
My free 100+ pages Data Engineering Cookbook
Follow us on LinkedIn
Check out my YouTube
Check out my full video on YouTube!