Background The article is written as a part of the Data Engineering coaching from Team Data Science. The objective is to share the newcomer perspective on IoT and data lifecycle in the IoT industry directed towards newcomers. Feel free to correct in case if some/all parts have little/no sense. You can find the first article ‘So, you want to be a smart farmer’ where I laid the foundation of the project here . A network of things The Internet of Things (IoT) is a network of physical devices that are able to connect to a network and exchange data. The devices, which form the IoT, are equipped with sensors, objects that measure a physical quantity and convert it into typical electrical signals or optical signals. On a high-level, the IoT architecture technology mainly consists of five major components - sensors; gateways and networks; storage; and Analytics/presentation level. Can you hear me or communication in a network of things Sensors have different communication capabilities - device-to-device, where IoT devices are directly connected and communicate with each other, and device-to-program communication where output data from the IoT device is fed into the output program. The example of the device-to-device communication in our agriculture IoT project could be communication between a temperature sensor and heating device/blower where the temperature sensors sense the change in the temperature level and if it detects the decrease in the temperature level below the threshold, it will send the signal to the microcontroller and controller will start the heater, running it until the moment when the temperature reaches the set limit. If, on the opposite, the temperature is above the threshold value, the controller will start the blower to bring the temperature down. The device-to-program communication in our project will be the web-portal where we would display the soil data. Communication in the IoT network requires special networking protocols: Lower level communication protocols for low cost and short-range communication: wifi zigbee cellular (2G, 3G, or LTE) bluetooth Higher-level communication protocols that are optimized for high-latency or unreliable networks: MQTT COAP XMPP HTTP (REST) From theory to reality As per MarketsandMarkets, the global Internet of Things market size is forecasted to grow from USD 170.6bn in 2017 to USD 561.0bn by 2022 with a Compound Annual Growth Rate (CAGR) of 26.9% during the forecast period (source: Internet of Things (IoT) Market by Software Solution (Real-Time Streaming Analytics, Security Solution, Data Management, Remote Monitoring, and Network Bandwidth Management), Service, Platform, Application Area, and Region - Global Forecast to 2022). Globally, Discrete Manufacturing, Transportation & Logistics and Utilities industries are projected to spend $40B each on IoT platforms, systems, and services, followed by Healthcare. The worldwide agriculture IoT market is expected to grow from USD 12.7bn in 2019 to USD 20.9bn by 2024, at a CAGR of 10.4%. Europe and the Americas will lead all other markets (source: Agriculture IoT Market by Offering (Hardware, Software, & Services), Application (Precision Farming, Precision Forestry, Livestock Monitoring, Fish Farm Monitoring and Smart Greenhouse), Application, and Geography - Global Forecast to 2024). The IoT communication protocol market is expected to grow from USD 11.44 billion in 2015 to reach USD 15.80 billion by 2022, at a CAGR of 4.66% during the forecast period, with the Wi-Fi technology holding the largest share followed by Bluetooth. Region-wise penetration, North America has been the leader in the IoT communication protocol market (source: IoT Communication Protocol Market by Connectivity Technology (Wi-Fi, Bluetooth, Zigbee, Bluetooth Smart), End-Use Application (Consumer Electronics, Automotive & Transportation, Building Automation, Healthcare), Region - Global Forecast to 2022). To be continued…

Why smart farming? The agriculture sector plays an important role in generating employment and contribution to the gross domestic product (GDP) globally and on a regional- and country- levels. 27% or approx. 1 in every four people in the world work in agriculture, as per WorldBank. Or we look at the regional data, almost one in every two people (53%) in Sub-Saharan Africa works in the agriculture industry. Agriculture, forestry, and fishing contributes 3.4 percent of World GDP in 2018, while in Sub-Saharan Africa, the contribution reached 16% of GDP. Farmers in the developed world are increasingly using sensors to figure out what, when and where to grow it to improve the agriculture output. But agriculture still remains underdeveloped in Africa in terms of the amount of potentially arable land and post-harvest losses as in many cases farmers use traditional farming methods and techniques. So, when it was time to decide on the project for the Team Data Science program , I decided to study IoT sensors in agriculture. How can you do smart farming? The legend goes as follows, historically, a farmer, an owner of several plots has been measuring the humidity level and temperature manually. Apart from having done this manual effort in visiting the fields to monitor the crop, he also has a risk of overwatering his plants, thus, getting fungus developed. He approached the IT company, requesting to set up sensors and a system that automatically monitors changes of temperature and humidity and alerts him regarding the parameter changes via an SMS, so that he can take corrective actions on time ( a primary objective of the project ). As a complimentary service, the IT company offers the portal which displays the collected data from different sites to perform further analysis and monitor the soil condition over time ( secondary objective ). Functional requirements can be summarized as follows: 1. as the data is streamed continuously, the architecture should be fit for the continuous data ingestion and data processing. "Data is to be collected at 20-minute intervals at SoilSCAPE (Soil moisture Sensing Controller and oPtimal Estimator) project sites in four states (California, Arizona, Oklahoma, and Michigan) in the United States. SoilSCAPE uses wireless sensor technology to acquire high temporal resolution soil moisture and temperature data at up to 12 sites over varying durations. At its maximum, the network is to contain over 200+ wireless sensor installations (nodes), with a range of 6 to 27 nodes per site. The soil moisture sensors (EC-5 and 5-TM from Decagon Devices) are to be installed at three to four depths, nominally at 5, 20, and 50 cm below the surface. Soil conditions (e.g., hard soil or rocks) may have limited sensor placement. Temperature sensors are to be installed at 5 cm depth at six of the sites." (source: Soil Moisture Profiles and Temperature Data from SoilSCAPE Sites, USA ). 2. Some validation should be done to review the incoming data in terms of its completeness i.e. number of data points missing per a site. And in case if the data gets missing, an SMS should be sent. 3. A system should also be able to send the SMS when a temperature/humidity level is below/above the threshold level, identified by an agronomist. 4. The dashboard should show row metrics coming from the sensors such as temperature, humidity sliced by the site and individual sensor levels. 5. Indirect metrics such as a rolling window average temperature and humidity levels should be calculated and sliced by the site and individual sensor levels. 6. Outliers/anomalies should be calculated and sliced by site- and individual sensor- levels. 7. The dashboard should display a map with the average temperature and humidity levels plotted against the overall yield data. To be continued… P.S. In the next series of articles I will be sharing my reading summary on the topic in general as well as my understanding of the data lifecycle in the IoT industry. I would say that the articles would be written from the newcomer perspective directed towards newcomers, so, please feel free to correct in case if some parts have little/no sense.

Hello, I am a web and android developer with 4+ years of experience, and I decided to enter in the world of Big Data, adopting the role of Data Engineer. In the university I studied some Machine Learning courses but only theoretical stuff. For those already experienced, that do you think about the courses in Coursera about data engineering, for someone who wants to learn tools that will enable him/her to get the first job as Data Engineer? Is it worth? Or should I start reading Hadoop and Spark books and start doing proyects. Put in other words, How can an already experienced programmer get a job as a Junior Data Engineer?