Pinned Loading

1. inference-performance inference-performance Public

   vLLM on 2× T4: 2.3× throughput, 300×+ lower p99 TTFT, and +93% goodput with tensor parallelism.

   Shell

2. gpu-attention-performance gpu-attention-performance Public

   CUDA attention kernel optimization on Tesla T4: 1.81× speedup via shared-memory tiling, profiled with Nsight Compute.

   Cuda

3. quantization-benchmark quantization-benchmark Public

   FP16 vs GPTQ INT8 vs INT4 on a T4: up to 2.2x throughput, 37% lower cost per token. vLLM benchmark.

   Python

4. vllm-inference-operator vllm-inference-operator Public

   Kubernetes operator for vLLM: one LLMService CRD gives GPU-aware deployment, KEDA autoscaling and observability.

   Go