Connect to Build

Name
Project - Source Data Ingestion Into IBM Cloud Object Storage Design a data pipeline to ingest data from REST APIs. Ensure you use a REST client and a mechanism to schedule batch data processing using cron expressions. The Data Pipeline should include,  Data extraction from API’s/Db2 table.  Loading data to IBM COS (Object Storage) as a Landing Area.  Batch data load monitoring mechanism (verification and validations)  Device a design to handle errors and capture error details. Ensure no data loss by retrying mechanisms or creating a notification system and audit ETL processes. In progress
Project - Data Transformation, Monitoring and integration to Box Design a data pipeline ensuring intermediate data transformation step for transforming the data set to be analyzed and to make it more suitable for subsequent analytical processing. The Data Pipeline should include,  Transformation step to handle column/attribute level changes.  Configuration to handle full load or CDC on target table.  Write/Append Transformed data to IBM Db2 table.  Write/Append Transformed data to BOX folder.  Device a design to handle errors and capture error details. Ensure no data loss by retry mechanisms or create a notification system and audit ETL processes. In progress

Project - Source Data Ingestion Into IBM Cloud Object Storage

Design a data pipeline to ingest data from REST APIs. Ensure you use a REST client and a mechanism to schedule batch data processing using cron expressions. The Data Pipeline should include,  Data extraction from API’s/Db2 table.  Loading data to IBM COS (Object Storage) as a Landing Area.  Batch data load monitoring mechanism (verification and validations)  Device a design to handle errors and capture error details. Ensure no data loss by retrying mechanisms or creating a notification system and audit ETL processes.
In progress

Project - Data Transformation, Monitoring and integration to Box

Design a data pipeline ensuring intermediate data transformation step for transforming the data set to be analyzed and to make it more suitable for subsequent analytical processing. The Data Pipeline should include,  Transformation step to handle column/attribute level changes.  Configuration to handle full load or CDC on target table.  Write/Append Transformed data to IBM Db2 table.  Write/Append Transformed data to BOX folder.  Device a design to handle errors and capture error details. Ensure no data loss by retry mechanisms or create a notification system and audit ETL processes.
In progress