AWS Integration with Abinitio | How to access AWS S3 using Abinitio



Watch my YouTube video for explanation :



Please look into the class notes here for your references:


AWS Integration with AI

Storage - S3, DB (RDS , Redhsift)

How CO>operating System connects to the cloud object storage S3

Configuring access to Amazon S3

1. JAVA , java 8+
2. Configure access credential for the respective cloud store
     
a. default credential provider chain
b. aws credential profile file with named profile 
c. Multiplexer file


Multiplexer
     .  to interact with apachee hadoop copatible file system
.  to calculate the resource required for containerized jobs 

AB_JOB_MUX_CONFIGURATION
AB_MUX_URI_SCHEMES


a. default credentials
In Linux
~/.aws/credentials
In Windows
C:\Users\Username\.aws\credentials

----------------------------------------------
AWS Credential profile file with named profiles
[default]
aws_access_key_id=<default-access-key>
aws_secret_access_key=<default-secret-key>

[custom]
aws_access_key_id=<custom-access-key>
aws_secret_access_key=<custom-secret-key>
[custom2]
aws_access_key_id=<custom-access-key>
aws_secret_access_key=<custom-secret-key>
----------------------------------------------
Multiplexer File
<configuraion>
<property>
<name>fs.s3a.access.key</name>
<value><Key-ID></value>
<description> aws access key id </description>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value> <Secret Key> </value>
<description> AWS Secret Key</description>
</property>
<property>
<name> fs.s3a.endpoint </name>
<value> <s3-enpoint-url></value>
<description> amazon s3 endpoint to connect to s3</description>
</property>
</configuraion>
---------------------------------------------


   
Building Graph to access S3 
File Components:

Input File
RMF
Read Parallel

Output File
WMF

aws s3 ls s3://

aws s3 cp <one bucket> <another bucket>
aws s3 cp <one bucket> <local path>
aws s3 rm  <bucketpath>

m_ls s3://hostname//<BUCKET-NAME>/revenue/jan_11022023.dat  
m_ls gs://hostname//<BUCKET_NAME>/revenue/jan_11022023.dat

We can use Read Multiple Files/Write Multiple Files for Reading/Writing data from/to S3 
(config-file)

For Vendor Specific Component(Amazon S3 API ):
credential-provider=default/profile

S3 Subscribe,
S3 Publish,
Batch S3 Subscribe


Fore more Abinitio, AWS and data engineering videos please subscribe , view , like and share my YouTube channel 

Click DataPundit



  
  
  

Comments

Post a Comment

Popular posts from this blog

Abinitio Interview Question # 1 - Write Multiple Files in Abinitio

Image
 Question :  I get records of various region in huge volume ,  we want to write the records in region specific files.   How to achieve that? View the explanation in the below video: Answer: -WMF WMF: filename::get_filename(in)= begin let string("") l_filename=string_substring(in.data.region); filename::string_concat($AI_SERIAL,"/",l_filename.dat); end; out::reformat(in)= begin out::in.data; end; OUTPUT DML-: DML record string(int) filename; string(integer(4)) file_contents; end; RMF: filename::get_filename(in)= begin filename::string_concat($AI_SERIAL,"/",in.filename); end; out::reformat(read,filename,in)= begin out.file_contents::read; out.filename::filename; end; Please view my youtube channel  DataPundit
Read more
Image
  Metaprogramming in Abinitio 1. How to create output DML dynamically based in INPUT DML 2. How to create intermediate transform( XFR) dynamically based on INPUT DML      Please See my you tube video for explanation : PART 1 PART 2 For Example, let us say, we have an Input DML as below-: cat input.dml record string(",") name; decimal(",") age; decimal(",") salary; string(",") aadharid; date("yyyy-mm-dd")(",") joiningdate; string(",") mobileno=NULL; string(",") dlno=NULL; string(",") passportnumber=NULL; end; (1.1) prepare a new DML based on input DML i.e. add fields, remove fields, modify type Graph based parameters: ===================================================================== input_dml=$DIR/input.dml  [interpretation = constant] input_dml_info =$[read_type(input_dml)]   [interpretation = pdl] /* Define functions in AB_DML_DEFS */ AB_DML_DEFS= $[ out:: dynamic_dml_gen(input_dml_info)= beg...
Read more

Next In Sequence in ABinitio | How next_in_sequence() works in MFS

Image
https://youtu.be/JdDDpjZcv-4 next_in_sequence() working in Abinitio: Definition -      within a single process(component) , this gives successive number.   next_in_sequence()>1 i.e. all records whose occurance is 2 and more in the file.      for example dataset below:    employeeName ram shyam ravi hanuman shekhar seeta geeta shayama vidushi vishakha If we process by condition next_in_sequence()>1 then the output will be [first record will be removed] shyam ravi hanuman shekhar seeta geeta shayama vidushi vishakha 2. next_in_sequence() with reformat  - rely -  FieldA 1a 2a 3a 4a 5a 6a 7a Example 1 (serial processing)--> Reformat   count=2      select:   next_in_sequence()<900      Transofom1:   out:reformat(in)=   begin   out.*::in.*;   out.Seq-No::next_in_sequence();   end;      Transform2:   out:reformat(in)=   begin   out.*:...
Read more