As of today, Qubole Data Service (QDS) now supports a new way to reserve AWS Spot instances, and Qubole customers will be able to configure Master and Worker nodes to be Spot blocks.
Spot blocks are instance types that can be reserved for a predefined duration–between 1 to 6 hours. Since Qubole customers’ average cluster life cycle is under 6 hours, this gives them the option to guarantee uninterrupted workload execution with a relatively cheaper price, but higher stability compared to pure Spot nodes. Compared to Spot nodes, Spot blocks provide continuous computation time, especially for applications that are not stateless and have no checkpoints generated in their execution.
For detailed documentation on using AWS Spot instances and Spot blocks in QDS, click here.
In QDS, you have the flexibility of choosing different cluster compositions for both Master and Auto-scaling nodes.
Original Cluster Compositions
| Master / Min Worker | Auto-scaling Worker | |
| #1 | On-Demand | On-Demand |
| #2 | On-Demand | Spot |
| #3 | Spot | Spot |
New Cluster Compositions Including Spot Blocks
| Master / Min Worker | Auto-scaling Worker | |
| #4 | Spot Blocks | On-Demand |
| #5 | Spot Blocks | Spot |
| #6 | Spot Blocks | Spot Blocks |
In the above:
- If you know the cluster’s life cycle is within 6 hours and have an estimate about the duration of your workloads, you may choose #4 over #1 (same for #5 over #2) to leverage Spot blocks to achieve the same stability that On-Demand instances have as well as benefit from the lower cost of Spot blocks.
- #6 is the only composition we would recommend using Spot blocks for auto-scaling. The reason being when auto-scaling kicks in, we need to set a duration for the Spot blocks request. If you choose Spot blocks for Master, Qubole will know the time remaining and use that time for autoscaling requests. For example, let’s say you choose #6 as your cluster configuration, and the duration for Master Spot blocks is set to 6 hours. In this case, after running for a couple of hours, let’s assume an autoscaling event occurs–when this happens, QDS will determine that the cluster life cycle only has 4 hours left (6-2=4). Therefore, when requesting Spot blocks for autoscaling, it will put in a request for 4.
Pricing
Spot block pricing is based on the requested duration and the available capacity; it’s typically cheaper than On-Demand nodes, but usually costs more than Spot nodes. Spot blocks and Spot instances are priced separately; view current spot pricing on AWS and select the “Defined Duration for Linux” tab to learn more.
Note: Spot blocks work very similarly to Spot nodes where users specify bidding parameters including max price. QDS uses 100% On-Demand price to bid for Spot blocks to make sure that the total cost doesn’t exceed what you might pay for On-Demand nodes in the worst scenario.
In Summary
Spot Blocks provide a balance between cost and reliability for users who have the need to keep the cluster running for a definite period of time while keeping the costs at a minimum.
If you’re interested in QDS, sign up today for a free trial!
