Posts

Batch vs Continuous Graphs in Abinitio - A Revision

Image
                                     Batch vs Continuous Graphs in Abinitio - A Revision watch my video on Batch versus Continuous Graphs Batch vs Continuous Graphs in Abinitio - A Revision We will talk about below- a. Batch versus Continuous long running , wait | continuous running , real time response best for analytical application | best for real time transactional applications run limited no of times a days | can run forever of can shutdown/stop at specified time b. Type Of Continuous Graphs a.  request/response  b.  publish/subscribe c. Continuous Graph is used when? 1. input is not a fixed set of data but a regular flow , perhaps coming at rates near real time 2. results are need to be made available before the large batch of data accumulate; low latencies    are desired    3. the overhead of graph startup and shutdown to process small batch of incoming data dominates     processing time         4. recovery model suggests it - you need to recover based on group of rec
Post a Comment
Read more

MAX CORE in JOIN in Abinitio | in memory join in abinitio

Image
                                     MAX CORE in JOIN in Abinitio  | in memory join in abinitio case when all non driving data size is <=max_core all non-driving data will be read and placed in max_core (private memory)              while doing so it creates an index of the input records to quickly locate the records         driving input record is read , if it find the match in-memory then output rec produced and marked for later use that match found at least once                                if not found match then based on the join type it may go in unused port or produce output with appropriate values Once all driving records are read , and keys matched in memory then output produced else based on join type it behaves. case when the all non driving data >max_core While reading non driving data if size of the index and non driving data size goes beyond the max_core  it starts spilling non-driving records to disk. whi
Post a Comment
Read more

MAX CORE in Rollup in Abinitio | Rollup algorithm for unsorted data

Image
                        MAX CORE in Rollup in Abinitio | Rollup algorithm for unsorted data  hash Tabale willl have KEY VALUE record along with pass through filed and values Roll Up reads the records and keep on adding/updating the Hash Table untill it reaches to the MAX_CORE limit Then it read record and if the key is there in the HASH table it updates that records else it has to spill to disk It does so untill all inputs have been read. Spillage Depends on -  The size of the group ( distinct key value group)+ the pass through fields + data distribution of the incoming data How much spillage : after the max core everything will  be spilled to disk  2nd Iteration of Spillage - when the distinct keys groups are huge , then it may not hold the data in the hash table again                  so it may spill 2nd time as well. For More Abinitio, AWS, Database content please visit my youTube channel. https://www.youtube.com/@dat
Post a Comment
Read more

MAX_CORE for Abinitio , sort algorithm for SORT component in abinitio

Image
             MAX_CORE for Abinitio , sort algorithm for SORT component in abinitio How SORT sorts the data           a.1 sort area ---> temp on disk which is sorted   a.2 buffer area for more input records        buffer-size  is 1/3 of max core ( if max_core is 100 MB the each buffer is approximatly 33 MB)   so normally only 2 buffer are used the , 3rd buffer is not used(Reserved)  but in rare circumstances it may be used.      Spillage:   so in normal cases every input record read from will be placed in some temp file on disk   therefore the spillage is equal to the size of the input data      a.3 once all temp files got created , the sort algo , reads them all and try merging and produce the final output.   For More Abinitio, AWS, Database content please visit my youTube channel. https://www.youtube.com/@datapundit @datapundit
Post a Comment
Read more

Abinitio Interview Question 43

Image
                                                Abinitio Interview Question 43 cust_id, card_type 111,  platinum 111,  gold 111,  silver 222,  ruby 222,  gold 222,  platinum 222,  diamond 333,  iron 333,  copper 333,  gold Find out the customer  who has all 3 - diamond,platinum and gold called premium customer. constant string1  =diamond constant string2  =platinum constant string3  =gold //let string(20)[3]  my_vec=[vector 'diamond','platinum','gold']; type temporary_type= begin decimal("") cnt; string(",") ctypes; end; out::initialize(in)= begin temp.cnt::0; end; out::rollup(temp,in)= begin temp.cnt::if(in.card_type member [vector 'diamond','platinum','gold']) temp.cnt+1 else temp.cnt; temp.ctypes::string_concat(temp.ctypes,in.card_type); end; out::finalize(temp,in)= begin out.cnt::temp.cnt; out.cust_id::in.cust_id; out.card_types::temp.ctypes; end; out::output_select(out)= begin out::if(out.cnt==
Post a Comment
Read more

Dynamic Subgraph in Abinitio

Image
                                       Dynamic Subgraph in Abinitio => Dynamic Subgraph -> subgraph that can be updated independntly of the parent graph or the graph which uses it -> parent graph maintains a pointer to the dynamic sub graph but the contents of the dynamic sub graph not loaded until parent graph runs graph interface - contract between parent graph and synamic subgraph .mgi Set of Requirements for Dynamic Subgraph: Input parameters, ports, propagation direction for record formats layout attributes phases if it can have more than 1 Step 1. Create .mgi i.e. the graph interface file File-> New -> Graph Interface Graph Interface Editor-> Parameters         Implementation: <path of the dynamic graph .mp file>         Error_group         Log group     <other custom parameters> Ports     < insert as many input /output ports required> Propagation Rules First Port – Second Port Save as lets us Say testdy
Post a Comment
Read more

Linked Subgraphs in Abinitio

Image
                                                    Linked Subgraphs in Abinitio => Embeded Subgraph         -> its a ongoing simple subgraph which is not existed independet of parent graph      -> used to enhance redability and maintaince of the graph    => Linked  Subgraph    How to Create     -> can be created using embeded sub graph     -> create sub grraph , label it     -> define parameters      -> File -> Save Subgraph "subgraph-name" as ...     -> saved in Components folder of the Sandbox    How to use      -> can be used in multiple graphs -> drag it in parent graph -> configure the parameters -> if any new version getting created then the parent also needs amendment For More Abinitio, AWS, Database content please visit my youTube channel. https://www.youtube.com/@datapundit @datapundit
Post a Comment
Read more

Micrograph In Abinitio

Image
                                                                   Micrograph In Abinitio Watch my youtube videos for complete understanding on micrograph Micrographs and Transactionability Chapter1, Introduction   a. What are micrographs . a kind of batch graph . it is executed as a single process  . they deployed to create micrograph object .mg file . Normally they are used in service oriented architecture  a) transaction processing b) web services processing . called by RUN MICROGRAPH component b. Why we need micrographs     c. How to Setup Micrographs d. Restrictions on Micrographs    component folding 1. DB components  2. file path name absolute 3. flows and layout - straight flow and serial layput 4. components must be foldable 5. graphs scripts - do not use these start and end scripts 6. no nesting  7. phase break , single phase , no checkpoint 8. run mode - micrograph 9. partion_path , this_par
Post a Comment
Read more

Abinitio Interview Question 46

Image
                                               Abinitio Interview Question 46 1  Calculate the difference of date in days date_difference_days(date2,date1) date1=(date(YYYY/MM/DD)) '20230521' date2=(date(YYYY/MM/DD)) '20230523' 2 2  Calucate the Month Start Date and Month End date for the given date let (date("YYYYMMDD")) l_date = '20230521'; l_month=date_month(l_date) l_year=date_year(l_date) month_start_date=(date("YYYYMMDD"))string_concat((string(""))decimal_lpad(l_year,4),(string(""))decimal_lpad(l_month,2),"01"); let string(",")[] leap_year_vec=[vector 31,29,31,30,31,30,31,31,30,31,30,31]; let string(",")[] year_vec=[vector 31,28,31,30,31,30,31,31,30,31,30,31]; let decimal("") is_leap_year = if(l_year/4 ==0) 1 else 0; if(is_leap_year) then month_end_date=(date("YYYYMMDD"))string_concat((string(""))decimal_lpad(l_year,4),(string(""))decimal_lpad(l_mo
Post a Comment
Read more

Abinitio Interview Question 45

Image
                                  Abinitio Interview Question 45 Get the unique values from the input data Input code, country 1, India 2, Mumbai 1, USA 2, Newyork 1,          UK 2, Edinburgh 2, London output code, countries 2, India,Mumbai 2, USA,Newyork 2, UK,Edinburgh,London type temporary_type= record string(",") l_countries; decimal("") cnt; end; temp::initialize(in)= begin temp.l_countries::in.country; temp.cnt::0; end; out::key_change(in1,in2)= begin out::if(in1.code==1); end; out::rollup(temp,in)= begin out.l_countries::if(temp.cnt!=0) string_concat(temp.countries,',',in.country); out.cnt::temp.cnt+1; end; out::finalize(temp,in)= begin out.code::in.code; out.countries::temp.l_countries; end; For More Abinitio, AWS, Database content please visit my youTube channel. https://www.youtube.com/@datapundit @datapundit
Post a Comment
Read more

Abinitio Interview Question 44

Image
                                                                   Abinitio Interview Question 44 Take the 2 dates and create end date for first and last quarter. (First date of the quarter , Last Date of the quarter) There will be 2 output fields in the output file. DATE1 = 25102022 DATE2 = 24082023 fqdt=25102022 ==> 01102022 lqdt=24082023 ==> 30092023 Yr_part1=2022 = string_Substring(DATE1,5,4); Yr_part2=2023 = string_Substring(DATE2,5,4); let string(",")[int] quarter_dates1=["0101"+Yr_part1,'0104'+Yr_part1,'0107'+Yr_part1,'0110'+Yr_part1]; let string(",")[int] quarter_dates2=["3112"+Yr_part2,'3009'+Yr_part2,'3006'+Yr_part2,'3103'+Yr_part2]; for (i,i<4) begin qstartdt = if((date("DDMMYYYY))quarter_dates1[i]<(date("DDMMYYYY))fqdt) (date("DDMMYYYY))quarter_dates1[i]; qenddt = if((date("DDMMYYYY))quarter_dates2[i]>(date("DDMMYYYY))lqdt) (date(&
Post a Comment
Read more

Abinitio Interview Question 40

Image
                                                                           Abinitio Interview Question 40 Watch the youtube video seqNo, deptName,citi1, citi2 1, XYZ, Pune, Guwahati 2, ABC, Mumbai, Chennai Output: seqNo,deptName,city 1, XYZ, Pune 1, XYZ, Guwahati 2, ABC, Mumbai 2, ABC, Chennai input.dml record string("\n") str=''; end; input.dml record string("\n") str=''; end; output.dml record string(",") seqNo=''; string(",") deptName=''; string(",") city=''; end; out::length()= begin out::length(string_split(in.str),',') -2; end; out::normalize(in,index)= begin out.seqNo::string_split((in.str),',')[0]; out.deptName::string_split((in.str),',')[1]; out.city::string_split((in.str),',')[index+2]; end; For More Abinitio, AWS, Database content please visit my youTube channel. https://www.youtube.com/@datapundit @datapundit
Post a Comment
Read more

Abinitio Interview Question 35

Image
  Abinitio Interview Question 35 Input: Col1|Col2 A   | A A   | B B   | C B   | D B   | E Output: Col1,Col2, Seq1,Seq2 A,  A, 1, 1.1 A,  B, 1, 1.2 B,  C, 2, 2.1 B,  D, 2, 2.2 B,  E, 2, 2.3 INPUT--> Replicate -->  ROLLUP on Col1--->assign seq no=Seq1 ----->A,1    B,2  |                  | Inner |                 JOIN  -->  SCAN -->OUTPUT               | | |-----------------------------------------------------> SCAN type temporary_type record string("") sub_seq; decimal("") cnt; end; out::initiaize(in)= begin out.sub_seq=''; out.cnt=1 end; out::scan(temp,in)= begin out.sub_seq::string_concat(in.Seq1,".",(string(""))temp.cnt); out.cnt::temp.cnt+1; end; out::finalize(temp,in)= begin out.Col1::in.Col1; out.Col2::in.Col2; out.Seq1::in.Seq1; out.Seq2::temp.sub_seq; end; For More Abinitio, AWS, Database content plea
Post a Comment
Read more

Abinitio Interview Question 36

Image
                                                            Abinitio Interview Question 36 EmployeeNo, Prodcut E01, Car E01, Laptop E02, Car Result EmployeeNo | Car_Ind | Laptop_Ind E01 | 1 | 1 E02 | 1 | 0 type temporary_type= begin decimal("") car_ind; decimal("") laptop_ind; end; out::intitialize(in)= begin car_ind::NULL; laptop::NULL; end; out::rollup(temp,in)= begin out.car_ind ::if(in.Product=='Car' or temp.car_ind==1) 1 else temp.car_ind; out.laptop_ind::if(in.Product=='Laptop' or temp.laptop_ind==1) 1 else temp.laptop_ind; end; out::finalize(temp,in)= begin out.Name::in.Filedvalue; out.car_ind::first_defined(temp.car_ind,0); out.laptop_ind::first_defined(temp.laptop_ind,0); end; For More Abinitio, AWS, Database content please visit my youTube channel. https://www.youtube.com/@datapundit @datapundit
Post a Comment
Read more

Abinitio Interview Question 42

Image
  Abinitio Interview Question   42  Watch my YouTube Video for this as below: Abinitio Interview Question 42 Input 1 Output 1 1,2 1,2,3 1,2,3,4 1,2,3,4,5 input -->  Normalize(length=5)           (normalize) out.val::in.val+index -->   1 2 3 4 5   Scan {} -> type temporary_type= begin string(int) str; decimal("") cnt; end; out::initialize(in)= begin temp.str::''; temp.cnt::0; end;   out::scan(temp,in)= begin temp.str::if(temp.cnt==0)'1' else string_concat(temp.str,",",temp.cnt+1);  temp.cnt::temp.cnt+1; end; out::finalize(temp,in)= begin out.str::temp.str; end; ->output Output 1 1,2 1,2,3 1,2,3,4 1,2,3,4,5 For More Abinitio, AWS, Database content please visit my youTube channel. https://www.youtube.com/@datapundit @datapundit
Post a Comment
Read more

Multilevel Normalization for Nested Vector in Abinitio - Volume2:

Image
                              Multilevel Normalization for Nested Vector in Abinitio  Fixed Length Nested Vector string(2)[6][3][2] =      [vector         [vector  vector ['a','b','c','d','e','f'], vector ['g','h','i','j','k','l'], vector ['m','n','o','p','q','r'] ], [vector  vector ['1a','1b','1c','1d','1e','1f'], vector ['1g','1h','1i','1j','1k','1l'], vector ['1m','1n','1o','1p','1q','1r'] ] ];   input-> Normalize level->Normalize level2 --> Normalize level3-> output Read the Data in the following Format: record    record record record string(1) vovel; end [6] level_1; end [3] level2; end [2] level3; end;    Ei
Post a Comment
Read more