Introduction
to Integration Interfaces
An integration interface is an Oracle Data Integrator object stored that
enables the loading of one target datastore with data transformed from one or
more source datastores, based on declarative rules implemented as mappings,
joins, filters and constraints.
An integration interface also references the Knowledge Modules (code templates) that will be used to generate the integration process.
Datastores
A datastore is a data structure that can be used as a source or a target in an integration interface. It can be:
a table stored in a relational database
an ASCII or EBCDIC file (delimited, or fixed length)
a node from a XML file
a JMS topic or queue from a Message Oriented Middleware
a node from a enterprise directory
an API that returns data in the form of an array of records
Regardless of the underlying technology, all data sources appear in Oracle Data Integrator in the form of datastores that can be manipulated and integrated in the same way. The datastores are grouped into data models. These models contain all the declarative rules –metadata - attached to datastores such as constraints.
Declarative Rules
The declarative rules that make up an interface can be expressed in human language, as shown in the following example: Data is coming from two Microsoft SQL Server tables (ORDERS joined to ORDER_LINES) and is combined with data from the CORRECTIONS file. The target SALES Oracle table must match some constraints such as the uniqueness of the ID column and valid reference to the SALES_REP table.
Data must be transformed and aggregated according to some mappings expressed in human language as shown in Figure: Example of a business problem.
An integration interface also references the Knowledge Modules (code templates) that will be used to generate the integration process.
Datastores
A datastore is a data structure that can be used as a source or a target in an integration interface. It can be:
a table stored in a relational database
an ASCII or EBCDIC file (delimited, or fixed length)
a node from a XML file
a JMS topic or queue from a Message Oriented Middleware
a node from a enterprise directory
an API that returns data in the form of an array of records
Regardless of the underlying technology, all data sources appear in Oracle Data Integrator in the form of datastores that can be manipulated and integrated in the same way. The datastores are grouped into data models. These models contain all the declarative rules –metadata - attached to datastores such as constraints.
Declarative Rules
The declarative rules that make up an interface can be expressed in human language, as shown in the following example: Data is coming from two Microsoft SQL Server tables (ORDERS joined to ORDER_LINES) and is combined with data from the CORRECTIONS file. The target SALES Oracle table must match some constraints such as the uniqueness of the ID column and valid reference to the SALES_REP table.
Data must be transformed and aggregated according to some mappings expressed in human language as shown in Figure: Example of a business problem.
Data
Flow
Business rules defined in the interface are automatically converted into a data flow that will carry out the joins filters, mappings, and constraints from source data to target tables.
By default, Oracle Data Integrator will use the Target RBDMS as a staging area for loading source data into temporary tables and applying all the required mappings, staging filters, joins and constraints. The staging area is a separate area in the RDBMS (a user/database) where Oracle Data Integrator creates its temporary objects and executes some of the rules (mapping, joins, final filters, aggregations etc.). When performing the operations this way, Oracle Data Integrator behaves like an E-LT as it first extracts and loads the temporary tables and then finishes the transformations in the target RDBMS.
In some particular cases, when source volumes are small (less than 500,000 records), this staging area can be located in memory in Oracle Data Integrator's in-memory relational database – In-Memory Engine. Oracle Data Integrator would then behave like a traditional ETL tool.
Figure: Oracle Data Integrator Knowledge Modules in action shows the data flow automatically generated by Oracle Data Integrator to load the final SALES table. The business rules will be transformed into code by the Knowledge Modules (KM). The code produced will generate several steps. Some of these steps will extract and load the data from the sources to the staging area (Loading Knowledge Modules - LKM). Others will transform and integrate the data from the staging area to the target table (Integration Knowledge Module - IKM). To ensure data quality, the Check Knowledge Module (CKM) will apply the user defined constraints to the staging data to isolate erroneous records in the Errors table.
Business rules defined in the interface are automatically converted into a data flow that will carry out the joins filters, mappings, and constraints from source data to target tables.
By default, Oracle Data Integrator will use the Target RBDMS as a staging area for loading source data into temporary tables and applying all the required mappings, staging filters, joins and constraints. The staging area is a separate area in the RDBMS (a user/database) where Oracle Data Integrator creates its temporary objects and executes some of the rules (mapping, joins, final filters, aggregations etc.). When performing the operations this way, Oracle Data Integrator behaves like an E-LT as it first extracts and loads the temporary tables and then finishes the transformations in the target RDBMS.
In some particular cases, when source volumes are small (less than 500,000 records), this staging area can be located in memory in Oracle Data Integrator's in-memory relational database – In-Memory Engine. Oracle Data Integrator would then behave like a traditional ETL tool.
Figure: Oracle Data Integrator Knowledge Modules in action shows the data flow automatically generated by Oracle Data Integrator to load the final SALES table. The business rules will be transformed into code by the Knowledge Modules (KM). The code produced will generate several steps. Some of these steps will extract and load the data from the sources to the staging area (Loading Knowledge Modules - LKM). Others will transform and integrate the data from the staging area to the target table (Integration Knowledge Module - IKM). To ensure data quality, the Check Knowledge Module (CKM) will apply the user defined constraints to the staging data to isolate erroneous records in the Errors table.
Oracle Data
Integrator Knowledge Modules contain the actual code that will be executed by
the various servers of the infrastructure. Some of the code contained in the
Knowledge Modules is generic. It makes calls to the Oracle Data Integrator
Substitution API that will be bound at run-time to the business-rules and generates
the final code that will be executed.
At design time, declarative rules are defined in the interfaces and Knowledge Modules are only selected and configured.
At design time, declarative rules are defined in the interfaces and Knowledge Modules are only selected and configured.
At run-time, code is generated and every Oracle Data Integrator API call in the Knowledge Modules (enclosed by <% and %>) is replaced with its corresponding object name or expression, with respect to the metadata provided in the Repository. The generated code is orchestrated by Oracle Data Integrator run-time component - the Agent – on the source and target systems to make them perform the processing, as defined in the E-LT approach.
ReplyDeleteThe tutorial is very helpful for a first timer like me, by the way if you could have given a
tutorial on creating a free blog, it would have been very helpful for newbies like me.
click for info