Batch vs Continuous Graphs in Abinitio - A Revision

                             Batch vs Continuous Graphs in Abinitio - A Revision


watch my video on Batch versus Continuous Graphs


Batch vs Continuous Graphs in Abinitio - A Revision We will talk about below- a. Batch versus Continuous

long running , wait | continuous running , real time response best for analytical application | best for real time transactional applications run limited no of times a days | can run forever of can shutdown/stop at specified time b. Type Of Continuous Graphs

a.  request/response 

b.  publish/subscribe
c. Continuous Graph is used when?

1. input is not a fixed set of data but a regular flow , perhaps coming at rates near real time


2. results are need to be made available before the large batch of data accumulate; low latencies 
  are desired
  
3. the overhead of graph startup and shutdown to process small batch of incoming data dominates 
   processing time
   
   
4. recovery model suggests it - you need to recover based on group of records, 
not based on groups of components ( phases)    this can be case the amount of work per record is 
huge.
   
5. your application design is inherently continuous , for example service graph

for example,

SORT ,  sorts the records within each unit of work.
Roll Up (in-memory), produces the output based on each unit of work
Data base components , data base commit happens at the end of each unit of work

For More Abinitio, AWS, Database content please visit my youTube channel.

https://www.youtube.com/@datapundit

@datapundit


 




Comments

Post a Comment

Popular posts from this blog

Abinitio Interview Question # 1 - Write Multiple Files in Abinitio

Image
 Question :  I get records of various region in huge volume ,  we want to write the records in region specific files.   How to achieve that? View the explanation in the below video: Answer: -WMF WMF: filename::get_filename(in)= begin let string("") l_filename=string_substring(in.data.region); filename::string_concat($AI_SERIAL,"/",l_filename.dat); end; out::reformat(in)= begin out::in.data; end; OUTPUT DML-: DML record string(int) filename; string(integer(4)) file_contents; end; RMF: filename::get_filename(in)= begin filename::string_concat($AI_SERIAL,"/",in.filename); end; out::reformat(read,filename,in)= begin out.file_contents::read; out.filename::filename; end; Please view my youtube channel  DataPundit
Read more

Next In Sequence in ABinitio | How next_in_sequence() works in MFS

Image
https://youtu.be/JdDDpjZcv-4 next_in_sequence() working in Abinitio: Definition -      within a single process(component) , this gives successive number.   next_in_sequence()>1 i.e. all records whose occurance is 2 and more in the file.      for example dataset below:    employeeName ram shyam ravi hanuman shekhar seeta geeta shayama vidushi vishakha If we process by condition next_in_sequence()>1 then the output will be [first record will be removed] shyam ravi hanuman shekhar seeta geeta shayama vidushi vishakha 2. next_in_sequence() with reformat  - rely -  FieldA 1a 2a 3a 4a 5a 6a 7a Example 1 (serial processing)--> Reformat   count=2      select:   next_in_sequence()<900      Transofom1:   out:reformat(in)=   begin   out.*::in.*;   out.Seq-No::next_in_sequence();   end;      Transform2:   out:reformat(in)=   begin   out.*::in.*;   out.Seq-No::next_in_sequence();   end;      Output_indexes()=   begin   out::if(next_in_sequence()%2==0) [vector 0,1] else [vector 0,1];   e
Read more
Image
  Metaprogramming in Abinitio 1. How to create output DML dynamically based in INPUT DML 2. How to create intermediate transform( XFR) dynamically based on INPUT DML      Please See my you tube video for explanation : PART 1 PART 2 For Example, let us say, we have an Input DML as below-: cat input.dml record string(",") name; decimal(",") age; decimal(",") salary; string(",") aadharid; date("yyyy-mm-dd")(",") joiningdate; string(",") mobileno=NULL; string(",") dlno=NULL; string(",") passportnumber=NULL; end; (1.1) prepare a new DML based on input DML i.e. add fields, remove fields, modify type Graph based parameters: ===================================================================== input_dml=$DIR/input.dml  [interpretation = constant] input_dml_info =$[read_type(input_dml)]   [interpretation = pdl] /* Define functions in AB_DML_DEFS */ AB_DML_DEFS= $[ out:: dynamic_dml_gen(input_dml_info)= beg
Read more