Abinitio Interview Questions 24 Abinitio Parallelism Advanced


Abinitio Interview Questions 24 Abinitio Parallelism   Advanced


Watch my YouTube video for explanation :

            


Abinitio Parallelism

component parallelism
The AI component who increases component parallelism
                        replicate
input file
reformat
FBE
PBE
dedup sorted
split

The AI component who decreases component parallelism
gather
                join
fuse
combine
concatenate

pipeline parallelism

                        SORT
ROLLUP
JOIN
Dedup Sorted
Phases / Check Points
component folding
Continuous Graphs:  Checkpoint-->  Compute points
 


data parallelism

types of layouts:

    data layout
    processing layout

file:serial
mfile:mfs /Adhoc multi file

dynamic layout:
fixed-depth dynamic layout    => mfile:dynamic:$DEPTH_OF_PARALLELISM 
for ex. mfile:dynamic:16:/data/warehouse/2023/dec
(note:Can be used to read and write)
variable-depth dynamic layout => mfile:dynamic:-1:$DATA_PATH:[mb per partition]:[max-depth]
for example mfile:dynamic:-1:/data/warehouse/2022/dec
for ex. mifile:dynamic:-1:s3a:////bucket/data:100:16
(note: can only be used for data reading)


Fore more Abinitio, AWS and data engineering videos please subscribe , view , like and share my YouTube channel 

Click DataPundit




Comments

Post a Comment

Popular posts from this blog

Abinitio Interview Question # 1 - Write Multiple Files in Abinitio

Image
 Question :  I get records of various region in huge volume ,  we want to write the records in region specific files.   How to achieve that? View the explanation in the below video: Answer: -WMF WMF: filename::get_filename(in)= begin let string("") l_filename=string_substring(in.data.region); filename::string_concat($AI_SERIAL,"/",l_filename.dat); end; out::reformat(in)= begin out::in.data; end; OUTPUT DML-: DML record string(int) filename; string(integer(4)) file_contents; end; RMF: filename::get_filename(in)= begin filename::string_concat($AI_SERIAL,"/",in.filename); end; out::reformat(read,filename,in)= begin out.file_contents::read; out.filename::filename; end; Please view my youtube channel  DataPundit
Read more

Next In Sequence in ABinitio | How next_in_sequence() works in MFS

Image
https://youtu.be/JdDDpjZcv-4 next_in_sequence() working in Abinitio: Definition -      within a single process(component) , this gives successive number.   next_in_sequence()>1 i.e. all records whose occurance is 2 and more in the file.      for example dataset below:    employeeName ram shyam ravi hanuman shekhar seeta geeta shayama vidushi vishakha If we process by condition next_in_sequence()>1 then the output will be [first record will be removed] shyam ravi hanuman shekhar seeta geeta shayama vidushi vishakha 2. next_in_sequence() with reformat  - rely -  FieldA 1a 2a 3a 4a 5a 6a 7a Example 1 (serial processing)--> Reformat   count=2      select:   next_in_sequence()<900      Transofom1:   out:reformat(in)=   begin   out.*::in.*;   out.Seq-No::next_in_sequence();   end;      Transform2:   out:reformat(in)=   begin   out.*:...
Read more
Image
  Metaprogramming in Abinitio 1. How to create output DML dynamically based in INPUT DML 2. How to create intermediate transform( XFR) dynamically based on INPUT DML      Please See my you tube video for explanation : PART 1 PART 2 For Example, let us say, we have an Input DML as below-: cat input.dml record string(",") name; decimal(",") age; decimal(",") salary; string(",") aadharid; date("yyyy-mm-dd")(",") joiningdate; string(",") mobileno=NULL; string(",") dlno=NULL; string(",") passportnumber=NULL; end; (1.1) prepare a new DML based on input DML i.e. add fields, remove fields, modify type Graph based parameters: ===================================================================== input_dml=$DIR/input.dml  [interpretation = constant] input_dml_info =$[read_type(input_dml)]   [interpretation = pdl] /* Define functions in AB_DML_DEFS */ AB_DML_DEFS= $[ out:: dynamic_dml_gen(input_dml_info)= beg...
Read more