DynamoDb Schema Design as an Expert Way



Watch my YouTube video for explanation :









Please look into the class notes here for your references:

DynamoDB Schema Design

    
**Limit the access pattern
**Throttling  meaning the reads are more than defined RCU
**data latency , storage
Use NOSQL WorkBench
Create Directly in DynamoDB
Use PartiQL

 
[Partition Key]           {Primary Key}

[Partition Key,Sort Key]  {composite primary key}

Partition Key  - Storage Index

Sort Key  - Ordering 


dept, opening_date
1, 12-03-2022
2, 22-09-2020
2, 21-12-2020
3, 17-06-2021
4, 16-05-2021



Partitions  --->partition 1   
1, 12-03-2022
3, 17-06-2021
partition 2
    2, 22-09-2020
2, 21-12-2020
Case I
when we lookup of a known global unique key

empid, joining _date
1, 16-05-2021
2, 21-12-2020
3, 17-06-2021
use the partition key


Case II
when the key is non unique OR when the range like queries are done based on some other value

dept , opening_date
1, 12-03-2022
2, 22-09-2020
2, 21-12-2020
3, 17-06-2021
4, 16-05-2021
use partition key and sort key 

Throttling 
         ** when burst in read operations
** hot key case
** uneven data distribution
** wrong partition key , sort key

Provisioning Resources
Read Capacity Units  -3000    [strongly consistent, eventually consistent and transactional]
Write Capacity Units -1000    [standard , transactional] 
 



Strategies to prevent Throttling - 
Prefixes/Suffixes, 
Composite Partition key, 
DynamoDB Accelerator(DAX)

Local Secondary Indexes and Global Secondary Indexes

Important Links:

Adaptive Capacity 
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-esign.html#bp-partition-key-partitions-adaptive

Choose the right partition Key 
https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/


Fore more Abinitio, AWS and data engineering videos please subscribe , view , like and share my YouTube channel 

Click DataPundit


Comments

Post a Comment

Popular posts from this blog

Image
  Metaprogramming in Abinitio 1. How to create output DML dynamically based in INPUT DML 2. How to create intermediate transform( XFR) dynamically based on INPUT DML      Please See my you tube video for explanation : PART 1 PART 2 For Example, let us say, we have an Input DML as below-: cat input.dml record string(",") name; decimal(",") age; decimal(",") salary; string(",") aadharid; date("yyyy-mm-dd")(",") joiningdate; string(",") mobileno=NULL; string(",") dlno=NULL; string(",") passportnumber=NULL; end; (1.1) prepare a new DML based on input DML i.e. add fields, remove fields, modify type Graph based parameters: ===================================================================== input_dml=$DIR/input.dml  [interpretation = constant] input_dml_info =$[read_type(input_dml)]   [interpretation = pdl] /* Define functions in AB_DML_DEFS */ AB_DML_DEFS= $[ out:: dynamic_dml_gen(input_dml_info)= beg...
Read more

Abinitio Interview Question # 1 - Write Multiple Files in Abinitio

Image
 Question :  I get records of various region in huge volume ,  we want to write the records in region specific files.   How to achieve that? View the explanation in the below video: Answer: -WMF WMF: filename::get_filename(in)= begin let string("") l_filename=string_substring(in.data.region); filename::string_concat($AI_SERIAL,"/",l_filename.dat); end; out::reformat(in)= begin out::in.data; end; OUTPUT DML-: DML record string(int) filename; string(integer(4)) file_contents; end; RMF: filename::get_filename(in)= begin filename::string_concat($AI_SERIAL,"/",in.filename); end; out::reformat(read,filename,in)= begin out.file_contents::read; out.filename::filename; end; Please view my youtube channel  DataPundit
Read more

How to implement SCD type 2 in Abinitio

Image
SCD-type 2 in Abinitio | SCD type 2 in Abinitio | How to implement SCD type 2 in Abinitio Lets take an example dataset to understand the SCD type 2 in Abinitio. For example . we have an input dataset which is already loaded in as dimension table already. vehicle dimension dataset sk, veh_cd,     curr_driver,hist_driver,start_dt,         end_dt,             curr_flag 1, veh01, dr0,  dr0,            19000101000000,39991231235959,Y 2, veh02, dr1,  dr1,            19000101000000,39991231235959,Y 3, veh03, dr2,  dr2,           19000101000000,39991231235959,Y 4, veh04, dr3,  dr3,           19000101000000,39991231235959,Y 5, veh05, dr5,          dr4,           19000101000000,20210220235959,N 6, veh05, ...
Read more