What do Toronto hiring managers need from their data engineers?
(and how I'm going to give it to them!)
If you're anything like me, you want to learn skills that have actual demand in the market. It benefits no one, not you, not the employer, not the customers, to learn skills that are irrelevant to your job market.
If you're a hiring manager and you're reading this, I suspect you want your data engineering candidates to actually have the skill set needed to build and maintain the infrastructure at your company. I'm here to help.
To find relevant in demand skills for the Toronto job market, I spent one week surveying 120 Toronto job postings for data engineers. Yes, companies are even hiring during this COVID pandemic!
The Findings Here are the most requested skill sets in their job posts:
My first challenge when creating portfolio of work was to decide which technologies to build my project around. Constraints include my hardware setup, a 5 year old bargain laptop, as well as keeping costs from cloud service providers as low as possible. Spark, a part of the Hadoop ecosystem of big data technologies, is the second most requested skill set. While it was tempting to create a project using the Hadoop ecosystem, Amazon Web Services are the market leader for cloud data engineering in Canada and hadoop has been stable or in decline.
Therefore, to meet market demand, I needed to integrate Spark, a highly requested skill, into my AWS project. There are 2 ways this can be done. Firstly, I can run an EMR cluster with Hadoop and spark. Unfortunately EMR clusters cannot simply be paused and this would increase my costs. Secondly, I can install Spark on a EC2 instance. I have decided to go with the second option after taking the costs into consideration.
What's Next In the following next 2 blog posts I will detail my proposed pipeline setup, as well as data set selection for this project, which you can find the link to below:
See you in the next blog post! You can find my linked in here: https://www.linkedin.com/in/steven-aranibar-8891a2103/ And you can contact me at cloudengineertoronto@protonmail.com