Combine data using the Funnel stage - Data Warehousing Data Warehousing Reading Time: 6 minutes

It is a processing stage that supports n inputs and 1 output.

Combining Source 1 and Source 1 without any key dependency

we will cover these topics: hide 1) Funnel requirements 2) Funnel types 3) Copy 3.1) job:- funnel, copy Generator, and copy 3.1.1) Sequential file 0 4) ↓ 5) ↓ 5.1) Sequential file 1 5.2) Column Generator 5.3) Copy 5.4) Funnel 5.4.1) Output 5.5) Job:- (Not Applicable) 5.5.1) Sequential file 1 5.5.2) Sequential file 0 5.6) lookup 5.7) Column Generator 5.8) output 5.9) Data set 0 5.9.1) Output 5.10) Conditions 5.11) Constraints 5.12) Job:- (Not Applicable) 5.12.1) Sequential file 0 5.12.2) Sequential file 1 5.12.3) Sequential file 2 5.12.4) Ref2 5.13) Lookup 5.13.1) Output 5.14) What to read next?

Funnel requirements

  1. Structure of Source 1 and Source 2 should be the same
  2. Column Names should be the same in Source 1 and Source 2.

Funnel types 

We have 3 Funnel types

  1. Continuous funnel  Random
  2. Sequence follows Linux order
  3. Sort funnel  based on key column values

 Copy 

It is having multi-purpose

job:- funnel, copy Generator, and copy

Sequential file 0

 ↓

Load

  ↓

E id             E Name      Add

11                   A             HYD

22                    E             SEC  

Sequential file 1

E id         E Name    Gender         Sal                 Add

33              B                 1                 23000             HYD

22             C                  0                 22000               Del  

Column Generator 

Column to Generate        = COUNTRY

Column to Generate        = COMPANY

Output

Do Mapping    (link 13)  

Copy 

outputDump (link)à do Mapping

Now,OutputFunnel  (link)Do MappingRemove   DSlink 13.COMPANY / DSlink 13.COUNTRYGo to  Columnsif any column Name changes have to be mode, do it here.

Funnel 

funnel Type = Continuous funnelOutputDo Mapping

Output 

Data set 0

E idE NameAdd
33BHYD
44CDEL
11AHYD
22ESEC
E idE NameGenderSalAddCountryCompany
33B123000HYDaaaaaa
44C022000DELbbbBbb

  Data set

Job:-  (Not Applicable)

Sequential file 1 

Load primary.txt

Sequential file 0 

Load ref.txt

In both  primary.txt and ref.txt   Columns   State

lookup 

àMap State Drag and Drop Required ColumnsGo to ConstraintsSet reject  

Column Generator 

Column to Generate      = STATE – DESCOutputàDo MappingColumnsRight click on STATE – DESCAlgorithm = CycleValue   = not applicableApplyClose Compile And RUN

output 

Data set 0

E idE NameStateStatusState – desc
11SUMANAP1001Andhra Pradesh
22MOONTN1000Tamil Nadu
33SUMANAP1111Andhra Pradesh
55SUMANAP1101Andhra Pradesh
66MOONJK1001Jammu Kashmir
77MOONSUNAP1011Andhra Pradesh
E idE NameStateStatusState – desc
44MOONSTARNN1011Not Applicable
88MOONMP1111Not Applicable

  Data set 1

Output 
E idE NameStateStatusState – desc
11SUMANAP1001Andhra Pradesh
22MOONTN1000Tamil Nadu
33SUMANAP1111Andhra Pradesh
55SUMANAP1101Andhra Pradesh
66MOONJK1001Jammu Kashmir
77MOONSUNAP1011Andhra Pradesh
44MOON STARNN1011Not Applicable
88MOONMP1111Not Applicable

Inclined to build a profession as Datastage Developer? Then here is the blog post on, explore Datastage Training

Conditions 

While using lookup, We Can Write the conditions in Constraints

That is Generally when we take 2 sequential files

1.primary and 1 reference file, we can Write Conditions for the only the primary file

Constraints

Link Name               Condition           Condition Not Met                   Lookup failure

Ref                                                                              Drop                                      Drop

Right-click Input column   primary

  1. job

Job:- (Not Applicable)

Sequential file  0   

 Load primary.txt columns  state

  Sequential file 1

Load ref.txt   columnsstate  

Sequential file 2 

Load rfe2.txt  columns  state  

Ref2 
StatusStatus- desc
1001Married
1000Divorced
1111Single

  Lookup

 Drag State from primary to ref

Drag Status from primary to ref 1

Now, Drag and Drop required ColumnsClick on Constraints  

Link NameconditionCondition Not metLookup failure
refDropDrop
2.ref1 DropDrop

  Right clickInput Column ref ref. State –desc  = “Andhra  Pradesh”

Now, Compile And RUN  

Output
E idE NameStateStatusState – desc 
11SUMANAP1001Andhra PradeshMarried
33SUMANAp1111Andhra PradeshSingle