Project

Source Data Ingestion Into IBM Cloud Object Storage

Objective

Design a data pipeline to ingest data from REST APIs. Ensure you use a REST client and a mechanism to schedule batch data processing using cron expressions. The Data Pipeline should include,  Data extraction from API’s/Db2 table.  Loading data to IBM COS (Object Storage) as a Landing Area.  Batch data load monitoring mechanism (verification and validations)  Device a design to handle errors and capture error details. Ensure no data loss by retrying mechanisms or creating a notification system and audit ETL processes.

Outcome

Design a pipeline to perform data ingestion from APIs