Project

Data Transformation, Monitoring and integration to Box

Objective

Design a data pipeline ensuring intermediate data transformation step for transforming the data set to be analyzed and to make it more suitable for subsequent analytical processing. The Data Pipeline should include,  Transformation step to handle column/attribute level changes.  Configuration to handle full load or CDC on target table.  Write/Append Transformed data to IBM Db2 table.  Write/Append Transformed data to BOX folder.  Device a design to handle errors and capture error details. Ensure no data loss by retry mechanisms or create a notification system and audit ETL processes.

Outcome

Integrate Data available in IBM cloud object storage to Box.