Project - Source Data Ingestion Into IBM Cloud Object Storage
Design a data pipeline to ingest data from REST APIs. Ensure you use a REST client and a mechanism to schedule batch data processing using cron expressions.
The Data Pipeline should include,
Data extraction from API’s/Db2 table.
Loading data to IBM COS (Object Storage) as a Landing Area.
Batch data load monitoring mechanism (verification and validations)
Device a design to handle errors and capture error details. Ensure no data loss by retrying mechanisms or creating a notification system and audit ETL processes.