Project - | Data Transformation, Monitoring and integration to Box |
Objective | Design a data pipeline ensuring intermediate data transformation step for transforming the data set to be analyzed and to make it more suitable for subsequent analytical processing.
The Data Pipeline should include,
Transformation step to handle column/attribute level changes.
Configuration to handle full load or CDC on target table.
Write/Append Transformed data to IBM Db2 table.
Write/Append Transformed data to BOX folder.
Device a design to handle errors and capture error details. Ensure no data loss by retry mechanisms or create a notification system and audit ETL processes. |
Outcomes | Integrate Data available in IBM cloud object storage to Box. |
Apply by Date | 09/10/2023 |
Applied Teams | 5 / 1 |
Duration | 5 months |
College | | 1. Vishwakarma Institute of Information Technology |
|
Tools-Technologies | DB2, Java, Python, REXX |
Mentor | Venkata H Sonti |