> ## Documentation Index
> Fetch the complete documentation index at: https://vastai-80aa3a82-auto-cli-sdk-preview-pr-510.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# vastai run benchmarks

Benchmark a template against one or more GPUs

## Usage

```bash theme={null}
vastai run benchmarks [OPTIONS]
```

## Options

<ParamField path="--template_hash" type="string">
  the template to benchmark; provide either --template\_hash or --template\_id
</ParamField>

<ParamField path="--template_id" type="integer">
  the template to benchmark; provide either --template\_hash or --template\_id
</ParamField>

<ParamField path="--gpus" type="string">
  comma-separated GPU names to benchmark (e.g. RTX\_4090,RTX\_3090). Prefix a name with a count to set GPUs per instance, e.g. "2x RTX\_4090" (overrides --num\_gpus)
</ParamField>

<ParamField path="--num_gpus" type="integer">
  GPUs per instance for tokens without an Nx prefix (default 1); overridden by inline Nx in --gpus
</ParamField>

<ParamField path="--timeout" type="integer" default="1800">
  max seconds to wait for a benchmark before giving up (default 1800)
</ParamField>

<ParamField path="--no-cache" type="boolean">
  re-measure instead of reusing recent cached benchmark results
</ParamField>

<ParamField path="--yes" type="boolean">
  Skip confirmation prompt
</ParamField>

## Description

Rents one instance per GPU in parallel, measures perf, tears down.
Each rental runs for up to --timeout seconds and costs real money.

Specs benchmarked recently (same template, GPU, and count, by any
user) are served from the benchmarks table as an estimated perf, and
you're asked whether to reuse that or run a fresh benchmark. Pass
\--no-cache to always re-measure, or -y to always reuse the cache.

Perf/$uses the GPU$/hr, so cached and freshly measured rows are
comparable. Cached rows recorded before that price was tracked show
a perf with no price.

## Examples

```bash theme={null}
# auto-sweep the default GPUs against TGI
    vastai run benchmarks --template_hash 79ebdd2ebfb9d42cedf7a221c42d37a5

    # specific GPUs against vLLM
    vastai run benchmarks --template_hash 393fa8572e6c73c927c8275fe4dffd53 --gpus RTX_4090,RTX_3090

    # multi-GPU configurations via inline Nx prefix against ComfyUI
    vastai run benchmarks --template_hash 40ef49becc953aa910ee05bd4653b9b3 --gpus "2x RTX_4090, 2x RTX_3090"

    # default count for tokens without an Nx prefix
    vastai run benchmarks --template_hash 79ebdd2ebfb9d42cedf7a221c42d37a5 --gpus RTX_4090,RTX_3090 --num_gpus 2

    # shorter timeout (10 min), skipping the cost prompt
    vastai run benchmarks --template_hash 393fa8572e6c73c927c8275fe4dffd53 --timeout 600 -y

    # ignore cached results and re-measure everything
    vastai run benchmarks --template_hash 79ebdd2ebfb9d42cedf7a221c42d37a5 --no-cache

    # raw JSON output for piping into another tool
    vastai run benchmarks --template_hash 40ef49becc953aa910ee05bd4653b9b3 --raw
```

## Global Options

The following options are available for all commands:

| Option          | Description                                           |
| --------------- | ----------------------------------------------------- |
| `--url URL`     | Server REST API URL                                   |
| `--retry N`     | Retry limit                                           |
| `--raw`         | Output machine-readable JSON                          |
| `--explain`     | Verbose explanation of API calls                      |
| `--api-key KEY` | API key (defaults to `~/.config/vastai/vast_api_key`) |