Project - | Source Data Ingestion Into IBM Cloud Object Storage |
Objective | Design a data pipeline to ingest data from REST APIs. Ensure you use a REST client and a mechanism to schedule batch data processing using cron expressions.
The Data Pipeline should include,
Data extraction from API’s/Db2 table.
Loading data to IBM COS (Object Storage) as a Landing Area.
Batch data load monitoring mechanism (verification and validations)
Device a design to handle errors and capture error details. Ensure no data loss by retrying mechanisms or creating a notification system and audit ETL processes. |
Outcomes | Design a pipeline to perform data ingestion from APIs |
Apply by Date | 20/11/2023 |
Applied Teams | 1 / 1 |
Duration | 4 Months |
College | | 1. Vishwakarma Institute of Information Technology | 2. Vishwakarma University |
|
Tools-Technologies | DB2, Java, Python, REXX |
Mentor | Muralidhar Boddepalli |