Learn Data Engineering
at LearnDataEngineering.com
Search

How to write Kinesis data stream to DynamoDB


In this blog, my data stream is being written to Dynamo DB.


The latest data available in streams through the producer will trigger events to the Lambda function. The function will write data to Dynamo DB.


How the data is produced? Kindly refer to my earlier blogpost

https://www.teamdatascience.com/post/how-to-process-simple-data-stream-and-consume-with-lambda


NoSQL vs Relational



NoSQL database organizes data so that its shape in the database corresponds with what will be queried.


RDBMS system does reshaping of data when a query is processed. For example, SQL joins uses more computation power to denormalize the data and produce end results.


In Relation model( look at the above simple 1 to N relationship), entities are normalized and used complex queries to get results


1 to N relationship can be achieved in a single table to address the above query,

username (partition key) and order_id(sort Key)


Partition Key(PK): Used as an input to the internal hash function and the output is a partition in which the item will be stored.

example, personid is unique for each item


Sort Key(SK): Items with the same partition key are stored in sort order by the sort key.

example, Alex(PK) has two different orders us123,us124(SK)



NoSQL DB's


No SQL datastores are a key-value, document, Graph or column-based type data stores. I would like to give a quick snapshot of various NoSQL stores