Next In Sequence in ABinitio | How next_in_sequence() works in MFS

https://youtu.be/JdDDpjZcv-4

next_in_sequence() working in Abinitio:

Definition - 

 

  within a single process(component) , this gives successive number.

  next_in_sequence()>1 i.e. all records whose occurance is 2 and more in the file.

  

  for example dataset below:

  

employeeName

ram

shyam

ravi

hanuman

shekhar

seeta

geeta

shayama

vidushi

vishakha

If we process by condition next_in_sequence()>1 then the output will be [first record will be removed]

shyam

ravi

hanuman

shekhar

seeta

geeta

shayama

vidushi

vishakha

2. next_in_sequence() with reformat  - rely - 

FieldA

1a

2a

3a

4a

5a

6a

7a

Example 1 (serial processing)-->

Reformat

  count=2

  

  select:

  next_in_sequence()<900

  

  Transofom1:

  out:reformat(in)=

  begin

  out.*::in.*;

  out.Seq-No::next_in_sequence();

  end;

  

  Transform2:

  out:reformat(in)=

  begin

  out.*::in.*;

  out.Seq-No::next_in_sequence();

  end;

  

  Output_indexes()=

  begin

  out::if(next_in_sequence()%2==0) [vector 0,1] else [vector 0,1];

  end;

 

Output1:

FieldA|Seq-No

1a, 3

2a, 7

3a, 11

4a, 15

5a, 19

6a, 23

7a, 27

Output1:

FieldA|Seq-No

1a,4

2a,8

3a, 12

4a, 16

5a, 20

6a, 24

7a, 28

 

The overall impact of various call of next_in_sequence() in a particular component will be varied based on  

  

  1. run tme behaviour of the compnenent

  2. how many ways parallel the component is executing

  

  

2. next_in_sequence() with partitioned flows

Example 2 (parallel processing)=

(next_in_sequence()*number_of_partitions())- (number_of_partitions()-1) + this_partition()

  1*4 -3 +0 =1  --> Partition 1

  2*4 -3 +0 =5  --> Partiton  5

  

  2*4 -3 +1 =6  --> Partiton  2

  

the next_in_sequence() defined in select and output_indexes,transformN will be shared

For exercise ->

4 way - select -> output_indexes ->reformat ->O/P

Discuss JOIN component as well.

4. with JOIN/RollUp/SCAN component (multi input port component)

   A. if the data process is in-memory or sorted 

   (For example place next_in_sequence() condition in select parameter

   

   B. if the data processing is in parallel

   

   C. If the data has duplicate - how many times the join/rollup/scan gets called 

      i.e. runtime behaviour of the component

   

5. Common Use cases:

    a. to reverse the order of data 

b. to assign surrogate key

c. ranking globally OR within groups/partitions

Thanks

Datapunditeducation@gmail.com

for more useful videos in the field of Abinitio and cloud , please visit datapunditeducation