MAX CORE in JOIN in Abinitio | in memory join in abinitio

                             MAX CORE in JOIN in Abinitio  | in memory join in abinitio

case when all non driving data size is <=max_core

all non-driving data will be read and placed in max_core (private memory)

             while doing so it creates an index of the input records to quickly locate the records 

  

    driving input record is read , if it find the match in-memory then output rec produced and marked for later use that match found at least once

                               if not found match then based on the join type it may go in unused port or produce output with appropriate values

Once all driving records are read , and keys matched in memory then output produced else based on join type it behaves.

case when the all non driving data >max_core

While reading non driving data if size of the index and non driving data size goes beyond the max_core 

it starts spilling non-driving records to disk.

while spilling it creates partition of the data based on key on disk.

when all reading of non=driving records completed - some are in memory and some others are on disk as partitioned temp files

Now, the JOIN starts reading on driving data records , if join matches in memory then out put records produced

   If not then the driving records also placed into the disk as portioned temp files on disk because we dont have rec in memory and it may be in temp non driving data

   so in this way everything driving records been read.

   

Further, we have some driving data on disk and some non driving on disk as partitioned temp files.

JOIN picks same key portioned file and join and produced output

join behaves same as we have similar thing to join if this way 

For More Abinitio, AWS, Database content please visit my youTube channel.

https://www.youtube.com/@datapundit

@datapundit