Migrate benchmark/MLOps GPU pipeline to Terraform/OpenTofu + cloud provider study (Alibaba vs DigitalOcean)
Summary
Move the GPU compute used by the benchmark/MLOps pipeline (currently run on ad-hoc hardware) to Infrastructure-as-Code using Terraform / OpenTofu, so benchmark environments are reproducible, disposable, and cost-controlled. As part of provider selection, this issue records a study comparing Alibaba Cloud vs DigitalOcean GPU offerings.
Motivation
- The MovieLens benchmark (
examples/06_benchmarks/movielens.ipynb) currently depends on whatever machine happens to be available (last run: RTX 5090 Laptop, 24 vCPU, 30 GB RAM). Not reproducible. - Manual GPU boxes are easy to leave running and bleed cost (both providers keep billing while powered off — you must destroy the instance to stop the meter).
- IaC makes each run spin up → run → destroy, fully declarative and repeatable.
Proposed change
- Add a
terraform/(orinfra/) module that provisions a GPU instance and tears it down, using OpenTofu-compatible providers. - Both target providers have first-class OpenTofu support via official providers:
aliyun/alicloudanddigitalocean/digitalocean(both in the OpenTofu registry). - Wire the benchmark run into the provisioned box: setup env → execute notebook → collect metrics → destroy.
Cloud provider study: Alibaba Cloud vs DigitalOcean
IaC support
Both support Terraform and OpenTofu via official providers. Caveat: DigitalOcean's A100 lives on the Paperspace side, which is not covered by the official digitalocean provider (weak IaC story — only a separate early-stage Paperspace/paperspace provider, not in the OpenTofu registry). Core GPU Droplets (H100/H200/L40S/RTX Ada) are fully covered.
Billing
Both bill per-second (DigitalOcean: 60-second minimum), no hour rounding. Both keep charging while powered off → must destroy to stop the meter.
GPU comparison (inference / light-training class)
Alibaba T4 (gn6i) |
DigitalOcean RTX 4000 Ada | |
|---|---|---|
| Architecture / memory | Turing 2018 / 16 GB | Ada 2023 / 20 GB |
| On-demand price | ~$1.20/hr | $0.76/hr |
- No true "same GPU" for training: the T4 exists only on Alibaba; DigitalOcean has no T4. For training, the only NVIDIA overlap is the A100 (Alibaba ~$2.40/hr vs DigitalOcean/Paperspace ~$3.18/hr); H100/H200 are DigitalOcean-only.
- Estimated cost of one full benchmark run (10–15 min wall on ML-100k): **$0.15–0.30** — negligible either way; RTX 4000 Ada is ~35–40% cheaper and faster than the T4.
- The real cost driver is idle time, not run time. Note ~1/3 of the pipeline is CPU/Spark-bound (ALS, SAR, BPR), so vCPU count matters as much as the GPU for ML-100k.
Tasks
- Add an OpenTofu module provisioning a GPU instance (parametrized by provider).
- Add teardown/destroy step + guard against orphaned instances.
- Script: setup env → run benchmark notebook → export metrics → destroy.
- Decide provider/GPU (recommend DigitalOcean RTX 4000 Ada for cost/reproducibility; A100 if larger training is needed).
- Document usage in
examples/06_benchmarks/README.md.
References
Source: recommenders-team/recommenders