File stages used to read and write data from files.
Following are some common file stages used into DataStage.
Sequential File Stage
The sequential file Stage is a file Stage. It is the most
common I/O Stage used in a DataStage Job. It is used to read data from or write
data to one or more flat Files. It can have only one input link or one Output
link. It can also have one reject link.
While handling huge volumes of data, this Stage can itself
become one of the major bottlenecks as reading and writing from this Stage is
slow.
Sequential files should be used in following conditions When
we are reading a flat file (fixed width or delimited) from UNIX environment,
which is FTPed from some external systems When some UNIX operations has to be
done on the file don’t use sequential file for intermediate storage between
jobs. It causes performance overhead, as it needs to do data conversion before
writing and reading from a UNIX file.
In order to have faster reading from the Stage the number of
readers per node can be increased (default value is one).
Dataset Stage
The Data Set is a file Stage, which allows reading data from
or writing data to a dataset. This Stage can have a single input link or single
Output link. It does not support a reject link.
It can be configured to operate in sequential mode or
parallel mode. DataStage parallel extender jobs use Dataset to store data being
operated on in a persistent form. Datasets are operating system files which by
convention has the suffix .ds Datasets are much faster compared to sequential
files. Data is spread across multiple nodes and is referred by a control file.
Datasets are not UNIX files and no UNIX operation can be
performed on them. Usage of Dataset results in a good performance in a set of
linked jobs. They help in achieving end-to-end parallelism by writing data in
partitioned form and maintaining the sort order.It also preserve partitions.
Dataset is having following parts:
Descriptor file: contains metadata, data location, but NOT
the data itself
Data file(s): Contains data in Native format
C:/IBM/Information Server / Server/data set/ file. Ds
Control file (or) header file : Resides in operating system.
File set stage
The File Set stage is a file stage. It allows you to read
data from or write data to a file set. The stage can have a single input link,
a single output link, and a single rejects link.
It only executes in parallel mode. advantage of using
fileset over a sequential file is “it preserves partitioning scheme”. The
amount of data that can be stored in each destination data file is limited by
the characteristics of the file system and the amount of free disk space
available. The number of files created by a file set depends on 1) the number
of processing nodes in the default node pool. 2) The number of disks in
the export or default disk pool connected to each processing node in the
default node pool. 3) The size of the partitions of the data set.
Lookup file set stage
The Lookup File Set stage is a file stage. It allows you to
create a lookup file set or reference one for a lookup. The stage can have
a single input link or a single output link. The output link must be a
reference link. The stage can be configured to execute in parallel or
sequential mode when used with an input link.
External source stage
The External Source stage is a file stage. It allows you to
read data that is output from one or more source programs. The stage calls
the program and passes appropriate arguments. The stage can have a single
output link, and a single rejects link. It can be configured to execute in
parallel or sequential mode.
External Target stage
The External Target stage is a file stage. It allows you to
write data to one or more source programs. The stage can have a single
input link and a single rejects link. It can be configured to execute in
parallel or sequential mode.
Complex Flat File stage
The Complex Flat File (CFF) stage is a file stage. You can
use the stage to read a file or write to a file, but you cannot use the same
stage to do both. As a source, the CFF stage can have multiple output
links and a single reject link. You can read data from one or more complex flat
files, including MVS™ data sets with QSAM and VSAM files. You can also read
data from files that contain multiple record types. The source data can contain
one or more of the following clauses:
v GROUP
v REDEFINES
v OCCURS
v OCCURS DEPENDING ON
CFF source stages run in parallel mode when they are used to read multiple
files, but you can configure the stage to run sequentially if it is reading
only one file with a single reader.
By using CFF, we can read ASCII or EBCDIC (Extended Binary
coded Decimal Interchage Code) data. We can select the required columns and can
omit the remaining. We can collect the rejects (bad formatted records) by
setting the property of rejects to “save” (other options: continue,
fail). We can flatten the arrays(COBOL files).
As a target, the CFF stage can have a single input link and
a single reject link. You can write data to one or more complex flat files. You
cannot write to MVS data sets or to files that contain multiple record types.