sglang/ GB200 perf

dsr1-fp8-1k1k-max-tpt

concurrency 1,024 · 21 hours ago

cron
passed
re-run
Commit

Vectorize _create_custom_4d_mask in CustomQwen2Decoder (#27527)

ckvermaAI·22 hours ago
PR #27527 · Vectorize _create_custom_4d_mask in CustomQwen2Decoder
Run
GitHub Actions27999739491-1
Slurm job5333
GPUs48·prefill 16 / decode 32
ISL / OSL1024 / 1024

Metrics

28 captured

best_of

1.00

burstiness

1.00

completed

10,240

duration

248

max_concurrency

1,024

mean_e2el_ms

23,740ms

mean_itl_ms

1,175ms

mean_tpot_ms

24.05ms

mean_ttft_ms

1,592ms

median_e2el_ms

23,306ms

median_itl_ms

1,201ms

median_tpot_ms

24.03ms

median_ttft_ms

934ms

num_prompts

10,240

output_throughput

38,080tok/s

p99_e2el_ms

32,664ms

p99_itl_ms

1,410ms

p99_tpot_ms

24.80ms

p99_ttft_ms

8,699ms

peak_output_tokens_per_s

43,312s

request_throughput

41.30tok/s

std_e2el_ms

2,560ms

std_itl_ms

198ms

std_tpot_ms

0.64ms

std_ttft_ms

2,028ms

total_input_tokens

9,440,068

total_output_tokens

9,440,624

total_token_throughput

76,158tok/s