Vultr

The compute layer.

Tiger supplies the data forks; Vultr supplies the machines that replay against them. Five pre-warmed workers turn one click into five parallel timelines, a London box keeps Tempo running live, and a GPU was rented by the minute to test whether training needed one. It didn't.

Cloud compute

vc2 instances: 1 vCPU workers in New Jersey, a 2 GB backend in London, and an NVIDIA A16 rented by the minute.

Snapshots

One bootstrapped image, cloned into every worker.

Instance tags

tempo-worker marks the pool; anything untagged is never dispatched to.

v2 API

Create, list, lease and tear down the fleet from Python.

In Tempo

How Tempo uses Vultr

Five workers replay in parallel

Five vc2-1c-1gb instances in ewr, beside Tiger's us-east-1
Pre-warmed all day: a cold boot takes ~40 s to active and ~80 s to SSH, too slow for a click
One click takes one Tiger zero-copy fork and sends five shifted timelines to five workers at once
Each worker is a stateless FastAPI app under systemd: POST /replay returns the P&L, GET /health reports liveness
A shared bearer token guards /replay, so no one can aim a public worker at their own database

Lease-based scheduling

VultrClient lists instances tagged tempo-worker over the v2 API, following cursor pagination
Idle means Vultr reports active and no job holds a lease
lease() → mark_busy() → mark_idle(): one job per worker, always released
When every worker is busy, jobs queue, so five timelines still finish on three workers
Without Vultr credentials it falls back to the same worker app in-process, labelled local-1..n

Snapshot to fleet in one command

build_worker_image.py: rsync the repo, bootstrap, verify /health and the token, snapshot, delete the build box
create_workers.py: clone five workers from the snapshot; idempotent, with --count, --status and --delete
create_backend.py: a separate lhr box running the dashboard and the poller, never tagged as a worker
time_vultr.py benchmarks create → active → SSH-ready
Snapshot ID, SSH key and worker token are written straight into .env

A GPU by the minute, to answer one question

gpu_box.py: one NVIDIA A16 (vcg-a16-6c-60g-16vram, 6 vCPU, 60 GB, 16 GB VRAM) in lhr: create, check nvidia-smi and torch.cuda, report uptime and cost, delete
The plan is easy to miss: GET /v2/plans?type=vcg lists only L40S boxes with no locations; the A16 only appears under type=all
Why a GPU at all: the walk-forward backtest retrains a model for every one of 100 weeks, and a parameter sweep multiplies that. Would GPU training make it faster, and would it forecast better?
The answer, measured on the same box: 500 models in 17.4 s on the A16 against 28.0 s on its CPU, 1.6× faster, and the forecasts came out the same. Gradient-boosted trees on tabular features don't need one
So the live system stays CPU-only (90 s to train on a laptop), and the GPU is created when there's a question to answer and deleted when idle: three boxes, about 29 box-minutes, $1.44, each billed a one-hour minimum