So I was curious and wanted to know what are the tools of choice these days for on-prem and cloud based platforms. Based on my research, I found out the following: For cloud, it might be platform dependent. For on-prem hive,spark,nifi,airflow,kafka seem to be good. I would like to know which tools are in demand in your opinon. Also, would be nice if some comments on how docker, kubernetes conncect with these data tools. i heard the enw version of spark supports both along with yarn.

Introduction The purpose of this project is to solve a business problem given a dataset as input. The case is about solving the abandonment cart to increase sales conversion and gain real-time insights for the e-commerce industry. the main goals are: Analyze the business case and the given dataset identify the most suitable approach to solve the given business problems implement/build and deploy the target architecture and data pipeline Analyze the output conformity to the given business problems Share the implementation details via blog posts and YT videos. What we will build? data pipeline for streaming and batch processing CI/CD chain to set up the infra and used managed services Data visualization dashboard for business users What we will learn? How to choose the right option for data storage for streaming and batch processing How to choose the right service/product for data processing to fit business requirement How to implement streaming and batch processing data pipelines How to implement machine learning models How to communicate final results Business requirement: 1- Business problem#1 retargeting customers with abandonment cart within 3 min maximum of the session expiration The use case basically is to increase sales conversion by reducing cart abandonment. the output. the retargeting format is to send a personalized email to the subscriber offering free shipping. 2- Business problem#2 Identifying products with the most sales attraction in the future in order to anticipate the supply The use case basically is to increase sales conversion by reducing cart abandonment. the output. 3- Business problem#3 Calculate Customer lifetime value 4- Business problem#4 visualize sales data per product within 30 to 45 seconds once the payment is done Technical requirements: 1- set up the data pipeline for streaming and batch processing Ingest sales data in realtime clean up the data / ensure data quality (missing values and negative ones) prepare the cleaned data to : Business analysts for visualizations data scientists and ML engineers to build ML models ie: probability estimate of a sale for each product instant reaction to retarget customer with abandonment cart by making new offer to the subscriber and reach it via email in the next 3 minutes (free coupon or free shipping offered) Archive historical data to be re analyzed by data scientists for future ML models 2- set up the CI/CD chain 1- automate infra and managed services creation and destruction 2- setup continuous integration and continuous delivery chain Dataset 1- Description This data contains behavior data for 5 months (Oct 2019 – Feb 2020) from a medium cosmetics online store. Each row in the file represents an event. All events are related to products and users. Each event is like many-to-many relation between products and users. Note: if this dataset is too small for you, you can try larger dataset from multi-category store . There are different types of events. See below. Semantics (or how to read it): User user id during session usersession added to shopping cart (property event type is equal cart) product productid of the brand of category code ( category code ) with price at event_time 2- Structure event_time The time when the event happened at (in UTC). event_type Events can be: view - a user viewed a product cart - a user added a product to shopping cart remove from cart - a user removed a product from the shopping cart purchase - a user purchased a product Typical funnel: view => cart => purchase. product_id ID of a product category_id Product's category ID category_code Product's category taxonomy (code name) if it was possible to make it. Usually present for meaningful categories and skipped for different kinds of accessories. brand The downcased string of brand names . Can be missed. price Float price of a product. Present. user_id Permanent user ID. user_session Temporary user's session ID. Same for each user's session. Is changed every time user comes back to the online store from a long pause. Target Architecture The architecture below is the target to build on GCP. component choices are explained on further articles in the series

When it comes to choosing the storage option for your data pipeline, an initial analysis for business and the technical requirement is needed before proceeding. This article belongs to 3-parts series about our case study analysis(you can check the case study details here): Part I – 3-steps procedure to identify your storage options it’s basically about explaining the approach to analyze those requirements. Part II – Applying the 3-step procedure to our use case#1 Part III – Applying the 3-step procedure to our use case#2 Step #1: Business problems analysis This is the entry point for the analysis and consists of : defining your use cases and business problem you would like to solve.  use cases need to be specific and to address one unique problem. let’s explain it with the following example : saying that “I would like to visualize my sales data” is not precise and considered as a bad formulation.  instead, if I say “I would like to visualize sales data per product within 30 to 45 seconds once the payment is done” is much more specific and gives indications about data availability time to take it in consideration when designing the data pipeline  Define Service Level Objectives(SLO)/ metrics and KPI. different types of SLO can be identified(1):  end 2 end SLO per stage / per component SLO timeliness vs skewness vs completeness SLO we will concentrate on End2End SLO for our use cases  Step#2: Technical requirements Once your use cases are defined as well as SLO, we can settle them in terms of technical requirements. to do that we need to answer the following questions: 1- What are my data sources? 2- The data is Structured vs Unstructured or Both? 3- What are the data models I will receive? relational key-value column-Oriented Document oriented 3- What are the volumes of such data (per second/per hour/per day)? what are the volumes in busy hours? 4- What are the End2end availability time?  and then take in consideration the following rules  separate storage and compute: data need to be stored in a way to separate it from the compute layer  scalability: your data storage need to scale when necessary data retention policy & costs Step#3 identify storage option This final step is to map the requirements identified in step#2 with data storage options according to your context if you are using public cloud providers or other options. for now, we are going to use GCP and storage options offered by this platform GCP offers various storage options and choosing the appropriate option is up to you. here the general guidance: Storage options offered by GCP : Source:(2) 1- CloudSQL Cloud SQL is a fully-managed database service that makes it easy to set up, maintain, manage, and administer your relational databases on Google Cloud Platform(3). You can use Cloud SQL with  MySQL ,  PostgreSQL , or  SQL Server 2- Datastore Datastore is a NoSQL document database built for automatic scaling, high performance, and ease of application development(4) 3- Bigtable A fully managed, scalable NoSQL database service for large analytical and operational workloads(5). 4- BigQuery Serverless, highly scalable, and cost-effective multi-cloud data warehouse designed for business agility.(6) 5- Cloud Spanner Fully managed relational database with unlimited scale, strong consistency & up to 99.999% availability.(7) Additionally, to that, there is also “temporary” storage on message broker systems and for that let’s mention: 6- Pub/sub Messaging and ingestion for event-driven systems and streaming analytics.(8) Integrated with Dataflow and BigQuery to form the Google Cloud-native Stream Analytics solution Auto-scaling and auto-provisioning with support for up to 100 GB/second Independent quota and billing for publishers and subscribers Global message routing to simplify multi-region systems Push and pull message delivery 7- Cloud Storage Cloud Storage is a service for storing your  objects  in Google Cloud. An object is an immutable piece of data consisting of a file of any format. You store objects in containers called  buckets . All buckets are associated with a  project , and you can group your projects under an  organization .(9) How to choose the appropriate option? Below a diagram explaining globally how to choose storage option on GCP among differents options listed above and according to your analysis in step#2 Diagram Source(10) If your use requires Google Cloud Storage as a choice, you need to consider adding to that the file format for your data. here are a full explanation and analysis about choosing the appropriate format . Sources: (1):https://landing.google.com/sre/workbook/chapters/data-processing/ (2):https://www.coursera.org/learn/gcp-big-data-ml-fundamentals/lecture/EY31t/choosing-the-right-approach (3):https://cloud.google.com/sql/docs (4): https://cloud.google.com/datastore/docs/concepts/overview (5):https://cloud.google.com/bigtable (6):https://cloud.google.com/bigquery (7):https://cloud.google.com/spanner (8):https://cloud.google.com/pubsub (9):https://cloud.google.com/storage/docs/introduction (10):https://www.coursera.org/learn/gcp-big-data-ml-fundamentals/lecture/s3wa2/approach-move-from-on-premise-to-google-cloud-platform