Abinitio Interview Questions 29 Dynamic Layout

    
                            Abinitio Interview Questions 29 Dynamic Layout


Watch my YouTube video for explanation :

                


What is Dynamic MFS layout - video done
        
What is it:
fixed depth
variable depth
dynamic mfs - build-mfs -dynamic -fixed depth dynamic layout
How it is being done:
fixed depth
mfile:dynamic:n  OR mfile:dynamic:$DEPTH 
variable depth
mfile:dynamic:-1:data-path[:MB-per-partition[:max-depth]]
-1 meaning its COP decided the depth of paralleism at runtime not the user
Creating Dynamic Single Directory MFS:
build-mfs -dynamic -singledir mfs-depth 64 -mfs-mount s3://my-bucket/mfs-64way
laypout:  s3://my-bucket/mfs-64way  OR mfile:dynamic:64
What are advantages of Dynamic Layout:
a. Migration  
b. Promotion
c. Collaboration and Resue

For more Abinitio, AWS and data engineering videos please subscribe , view , like and share my YouTube channel 

Click DataPundit

Comments

Post a Comment

Popular posts from this blog

Image
  Metaprogramming in Abinitio 1. How to create output DML dynamically based in INPUT DML 2. How to create intermediate transform( XFR) dynamically based on INPUT DML      Please See my you tube video for explanation : PART 1 PART 2 For Example, let us say, we have an Input DML as below-: cat input.dml record string(",") name; decimal(",") age; decimal(",") salary; string(",") aadharid; date("yyyy-mm-dd")(",") joiningdate; string(",") mobileno=NULL; string(",") dlno=NULL; string(",") passportnumber=NULL; end; (1.1) prepare a new DML based on input DML i.e. add fields, remove fields, modify type Graph based parameters: ===================================================================== input_dml=$DIR/input.dml  [interpretation = constant] input_dml_info =$[read_type(input_dml)]   [interpretation = pdl] /* Define functions in AB_DML_DEFS */ AB_DML_DEFS= $[ out:: dynamic_dml_gen(input_dml_info)= beg...
Read more

Abinitio Interview Question # 1 - Write Multiple Files in Abinitio

Image
 Question :  I get records of various region in huge volume ,  we want to write the records in region specific files.   How to achieve that? View the explanation in the below video: Answer: -WMF WMF: filename::get_filename(in)= begin let string("") l_filename=string_substring(in.data.region); filename::string_concat($AI_SERIAL,"/",l_filename.dat); end; out::reformat(in)= begin out::in.data; end; OUTPUT DML-: DML record string(int) filename; string(integer(4)) file_contents; end; RMF: filename::get_filename(in)= begin filename::string_concat($AI_SERIAL,"/",in.filename); end; out::reformat(read,filename,in)= begin out.file_contents::read; out.filename::filename; end; Please view my youtube channel  DataPundit
Read more

How to implement SCD type 2 in Abinitio

Image
SCD-type 2 in Abinitio | SCD type 2 in Abinitio | How to implement SCD type 2 in Abinitio Lets take an example dataset to understand the SCD type 2 in Abinitio. For example . we have an input dataset which is already loaded in as dimension table already. vehicle dimension dataset sk, veh_cd,     curr_driver,hist_driver,start_dt,         end_dt,             curr_flag 1, veh01, dr0,  dr0,            19000101000000,39991231235959,Y 2, veh02, dr1,  dr1,            19000101000000,39991231235959,Y 3, veh03, dr2,  dr2,           19000101000000,39991231235959,Y 4, veh04, dr3,  dr3,           19000101000000,39991231235959,Y 5, veh05, dr5,          dr4,           19000101000000,20210220235959,N 6, veh05, ...
Read more