Project

Challenges in Utilizing Spot Instances with Distributed Cluster Environments

Objective

This project aims to discuss the difficulties and risks associated with using Spot Instances in a distributed cluster environment setup. Spot Instances offer cost savings, but are subject to interruptions and unpredictable pricing, making it challenging to use them in a distributed cluster environment where data reliability and availability are critical. This project highlights the need for careful planning and implementation to successfully utilize Spot Instances in distributed cluster environments.

Outcome

A scheduler to spin spot instances and a queuing mechanism to batch up the nodes for distributed processing