Tabnine Deployment Options

Tabnine AI code assistant: System requirements

Tabnine can be deployed in one of the following ways:

Single/Multi-Tenant SaaS
Private cloud / On-prem installation using open-weight models
Private cloud / On-prem installation using private API endpoints

Single/Multi-Tenant SaaS

This deployment allows you to utilize Tabnine’s private LLM endpoints to support both Chat and Agentic workflows.

Models

These utilize the following families of LLMs for both Chat and Agent:

GPT
Claude
Gemini

Hardware Requirements

None.

Private Cloud / On-Prem Installation Using Open-Weight AI Models

You can also power Tabnine by supporting open-weight models that are installed on-premises or on one of the private clouds mentioned above.

Models

For Self-Hosted (SH) customers, your hardware needs depend on whether or not you already have any open-weight models within your infrastructure.

The following models are no longer be supported, starting version 6.2.0:

Tabnine-protected
Gemma
Qwen 2.5 or lower

Open-Weight Models that Tabnine supports as part of an On-Prem installation

Poolside/Laguna-M1

Poolside/Laguna-XS.2

MiniMax up to M2.5

GLM up to 5.2

Qwen/Qwen3.6-27B

Devstral-Small-2-24B-Instruct-2512

In the absence of a pre-configured model, Tabnine provides on-premises installation services for supported models:

Open-Weight Models that Tabnine Offers to Install On-Prem

Poolside/Laguna-M1

Poolside/Laguna-XS.2

GLM 5.2

The following models are supported by Tabnine, but operate under a specific license and might require the customer to seek an approval from their owner:

Devstral 2 (123B parameters)
Minimax M3 and M2.7

Hardware Requirements

Installation requirements vary based on your specific use case. Please refer to the tables below to ensure optimal performance for both agentic workflows and chat features.

Agent + Chat

Agent + Chat

≤100 Users

101-500 Users

501-1000 Users

1001-2000 Users

Poolside/Laguna-M.1

Recommended

4 H100

4 B200

8 B200

16 B200

Minimum

4 H100 / 1 B200

4 B200 / 16H100

8 B200

16 B200

Poolside/Laguna-XS.2

Recommended

4 H100

6 H100

8 H100

Minimum

2 H100

3 H100

4 H100

6 H100

GLM-5.2

Recommended

8 H200

8 B200

16 B200

Minimum

8 H200

16 H200

16 B200

MiniMax-M2.5

Recommended

2 B200

4 B200

8 B200

16 B200

Minimum

2 H200

4 H200

8 H200

16 H200

Qwen/Qwen3.6-27B

Recommended

2 B200

4 B200

8 B200

Minimum

4 H100

8 H100

4 B200

Devstral-Small-2-24B-Instruct-2512

Recommended

2 B200

4 B200

8 B200

Minimum

2 H100

3 H100

6 H100

12 H100

Chat Only

Chat Only

≤100 Users

101-500 Users

501-1000 Users

1001-2000 Users

Poolside/Laguna-XS.2

Recommended

4 H100

6 H100

8 H100

Minimum

2 H100

3 H100

4 H100

6 H100

Devstral-Small-2-24B-Instruct-2512

Recommended

2 B200

Minimum

2 H100

4 H100

MiniMax-M2.5

Recommended

2 B200

3 B200

Minimum

2 H200

2 H200 or 4 H100

4 H200 or 8 H100

8 H200

GLM-4.7

Recommended

2 B200

4 B200

6 B200

Minimum

8 H100

2 B200

4 B200

Qwen-3-Coder-480B-A35B-Instruct

Recommended

2 B200

4 B200

8 B200

Minimum

8 H100

4 B200

8 B200

Qwen-3-30B

Recommended

2 B200

Minimum

2 H100

GPU Availability by Cloud Provider

GPU

AWS

Azure

GCP

H100

p5.4xlarge (H100 80GB)

NC40ads_H100_v5 (H100 94GB)

a3-highgpu-1g (H100 80GB)

H200

p5en.48xlarge (8×H200 141GB)

ND96isr_H200_v5 (8×H200 141GB)

a3-ultragpu-8g (8×H200 141GB)

B200

p6-b200.48xlarge (8×B200 HBM3e)

ND128isr_NDR_GB200_v6 (4×Blackwell 192GB)

a4-highgpu-8g (8×B200 HBM3e)

If you wish to use an open-weight model that is not included on this list, please contact our support team for a custom assessment.

Open-Weight Model Installation

Laguna M.1

Automated Startup Script:

#!/bin/bash
set -e

# 1. Install Docker
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg software-properties-common
curl -fsSL https://get.docker.com | sudo sh

# 2. Install NVIDIA Container Toolkit (required for --gpus all)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 3. Persistent HF cache
sudo mkdir -p /home/ubuntu/data

# 4. Launch vLLM container (Optimized for 4xH100)
sudo docker run -d --gpus all \
  --name laguna-m-1-vllm \
  -p 0.0.0.0:8000:8000 \
  -v /home/ubuntu/data:/hf_cache \
  -e HF_HOME=/hf_cache \
  -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_SERVER_DEV_MODE=1 \
  -e VLLM_USE_DEEP_GEMM=0 \
  -e VLLM_USE_FLASHINFER_MOE_FP16=1 \
  -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
  vllm/vllm-openai:v0.22.0 \
  --model poolside/Laguna-M.1-FP8 \
  --served-model-name poolside/Laguna-M.1-FP8 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --async-scheduling \
  --max-model-len 180000 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --override-generation-config '{"max_new_tokens": 16384}' \
  --trust-remote-code

1. Namespace Creation

kubectl create namespace llm-inference

2. Deployment Manifest - Optimized for 4xH100

apiVersion: apps/v1
kind: Deployment
metadata:
  name: laguna-m-1-vllm
  namespace: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: laguna-m-1-vllm
  template:
    metadata:
      labels:
        app: laguna-m-1-vllm
    spec:
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.22.0
          args:
            - "--model"
            - "poolside/Laguna-M.1-FP8"
            - "--served-model-name"
            - "poolside/Laguna-M.1-FP8"
            - "--port"
            - "8000"
            - "--tensor-parallel-size"
            - "4"
            - "--enable-expert-parallel"
            - "--tool-call-parser"
            - "poolside_v1"
            - "--reasoning-parser"
            - "poolside_v1"
            - "--enable-auto-tool-choice"
            - "--default-chat-template-kwargs"
            - '{"enable_thinking": true}'
            - "--async-scheduling"
            - "--max-model-len"
            - "180000"
            - "--max-num-seqs"
            - "128"
            - "--gpu-memory-utilization"
            - "0.95"
            - "--kv-cache-dtype"
            - "fp8"
            - "--override-generation-config"
            - '{"max_new_tokens": 16384}'
            - "--trust-remote-code"
          ports:
            - containerPort: 8000
          env:
            - name: HF_HOME
              value: "/hf_cache"
            - name: SAFETENSORS_FAST_GPU
              value: "1"
            - name: VLLM_SERVER_DEV_MODE
              value: "1"
            - name: VLLM_USE_DEEP_GEMM
              value: "0"
            - name: VLLM_USE_FLASHINFER_MOE_FP16
              value: "1"
            - name: VLLM_FLASHINFER_MOE_BACKEND
              value: "throughput"
          resources:
            limits:
              nvidia.com/gpu: 4
          volumeMounts:
            - name: cache-volume
              mountPath: /hf_cache
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: cache-volume
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "16Gi"

3. Service Manifest

apiVersion: v1
kind: Service
metadata:
  name: laguna-m-1-service
  namespace: llm-inference
spec:
  selector:
    app: laguna-m-1-vllm
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

4. Verification Commands

kubectl logs -f deployment/laguna-m-1-vllm -n llm-inference
kubectl get pods,svc -n llm-inference

In-cluster endpoint: http://laguna-m-1-service.llm-inference.svc.cluster.local/v1.

Notes

For Blackwell (B200), switch the model to poolside/Laguna-M.1-NVFP4

Poolside/Laguna-XS.2

Execution Script:

read -r -d '' STARTUP_SCRIPT <<'EOF' || true
#!/bin/bash
set -e
# 1. System Preparation & Docker Install
apt update
apt install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
usermod -aG docker $USER
# 2. Local Cache Directory
mkdir -p /home/ubuntu/data
chown -R $USER:$USER /home/ubuntu/data
# 3. Execution (Optimized for 8x H100)
docker run -d --gpus all \
  --name vllm-minimax \
  -p 0.0.0.0:8000:8000 Thank you for reaching out.
sudo docker run -d --gpus all \
  -p 0.0.0.0:8000:8000 \
  -v /root/.cache/huggingface:/root/.cache/huggingface \
  -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_SERVER_DEV_MODE=1 \
  -e VLLM_USE_DEEP_GEMM=0 \
  -e VLLM_USE_FLASHINFER_MOE_FP16=1 \
  -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
  vllm/vllm-openai:v0.22.0 \
  --model poolside/Laguna-XS.2-FP8 \
  --served-model-name poolside/Laguna-XS.2-FP8 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --trust-remote-code \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --async-scheduling \
  --max-model-len 131072 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --override-generation-config '{"max_new_tokens": 16384}'
EOF

1. Create Namespace

kubectl create namespace llm-inference

2. Deployment Manifest (poolside-laguna.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: laguna-vllm
  namespace: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: laguna-vllm
  template:
    metadata:
      labels:
        app: laguna-vllm
    spec:
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.22.0
          args:
            - "--model"
            - "poolside/Laguna-XS.2-FP8"
            - "--served-model-name"
            - "poolside/Laguna-XS.2-FP8"
            - "--port"
            - "8000"
            - "--tensor-parallel-size"
            - "4"
            - "--enable-expert-parallel"
            - "--trust-remote-code"
            - "--tool-call-parser"
            - "poolside_v1"
            - "--reasoning-parser"
            - "poolside_v1"
            - "--enable-auto-tool-choice"
            - "--default-chat-template-kwargs"
            - '{"enable_thinking": true}'
            - "--async-scheduling"
            - "--max-model-len"
            - "131072"
            - "--max-num-seqs"
            - "128"
            - "--gpu-memory-utilization"
            - "0.95"
            - "--kv-cache-dtype"
            - "fp8"
            - "--override-generation-config"
            - '{"max_new_tokens": 16384}'
          env:
            - name: HF_HOME
              value: "/hf_cache"
            - name: SAFETENSORS_FAST_GPU
              value: "1"
            - name: VLLM_SERVER_DEV_MODE
              value: "1"
            - name: VLLM_USE_DEEP_GEMM
              value: "0"
            - name: VLLM_USE_FLASHINFER_MOE_FP16
              value: "1"
            - name: VLLM_FLASHINFER_MOE_BACKEND
              value: "throughput"
          resources:
            limits:
              nvidia.com/gpu: 4
          volumeMounts:
            - name: cache-volume
              mountPath: /hf_cache
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: cache-volume
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "16Gi"

GLM 5.2

Automated Startup Script:

#!/bin/bash
set -e

# 1. Install Docker
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg software-properties-common
curl -fsSL https://get.docker.com | sudo sh

# 2. Install NVIDIA Container Toolkit (required for --gpus all)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 3. Persistent HF cache
sudo mkdir -p /home/ubuntu/data

# 4. Launch vLLM container (Optimized for 8xH200)
sudo docker run -d --gpus all \
  --name glm-5-2-vllm \
  -p 0.0.0.0:8000:8000 \
  -v /home/ubuntu/data:/hf_cache \
  -e HF_HOME=/hf_cache \
  -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_SERVER_DEV_MODE=1 \
  vllm/vllm-openai:v0.23.0 \
  --model zai-org/GLM-5.2-FP8 \
  --served-model-name zai-org/GLM-5.2-FP8 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8 \
  --max-model-len 180000 \
  --async-scheduling \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --override-generation-config '{"max_new_tokens": 16384}' \
  --trust-remote-code

1. Namespace Creation

kubectl create namespace llm-inference

2. Deployment Manifest — Optimized for 8xH200

apiVersion: apps/v1
kind: Deployment
metadata:
  name: glm-5-2-vllm
  namespace: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: glm-5-2-vllm
  template:
    metadata:
      labels:
        app: glm-5-2-vllm
    spec:
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.23.0
          args:
            - "--model"
            - "zai-org/GLM-5.2-FP8"
            - "--served-model-name"
            - "zai-org/GLM-5.2-FP8"
            - "--port"
            - "8000"
            - "--tensor-parallel-size"
            - "8"
            - "--kv-cache-dtype"
            - "fp8"
            - "--max-model-len"
            - "180000"
            - "--async-scheduling"
            - "--reasoning-parser"
            - "glm45"
            - "--tool-call-parser"
            - "glm47"
            - "--enable-auto-tool-choice"
            - "--override-generation-config"
            - '{"max_new_tokens": 16384}'
            - "--trust-remote-code"
          ports:
            - containerPort: 8000
          env:
            - name: HF_HOME
              value: "/hf_cache"
            - name: SAFETENSORS_FAST_GPU
              value: "1"
            - name: VLLM_SERVER_DEV_MODE
              value: "1"
          resources:
            limits:
              nvidia.com/gpu: 8
          volumeMounts:
            - name: cache-volume
              mountPath: /hf_cache
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: cache-volume
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "32Gi"

3. Service Manifest

apiVersion: v1
kind: Service
metadata:
  name: glm-5-2-service
  namespace: llm-inference
spec:
  selector:
    app: glm-5-2-vllm
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

4. Verification Commands

kubectl logs -f deployment/glm-5-2-vllm -n llm-inference
kubectl get pods,svc -n llm-inference

In-cluster endpoint: http://glm-5-2-service.llm-inference.svc.cluster.local/v1.

MiniMax-M2.5

Automated Startup Script:

#!/bin/bash
set -e

# 1. Install Docker
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg software-properties-common
curl -fsSL https://get.docker.com | sudo sh

# 2. Install NVIDIA Container Toolkit (required for --gpus all)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 3. Persistent HF cache
sudo mkdir -p /home/ubuntu/data

# 4. Launch vLLM container (Optimized for 2xB200)
sudo docker run -d --gpus all \
  --name minimax-m2-5-vllm \
  -p 0.0.0.0:8000:8000 \
  -v /home/ubuntu/data:/hf_cache \
  -e HF_HOME=/hf_cache \
  -e VLLM_ATTENTION_BACKEND=FLASHINFER \
  -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
  -e VLLM_USE_FLASHINFER_MOE_FP16=1 \
  -e VLLM_USE_FLASHINFER_MOE_FP8=1 \
  -e VLLM_USE_FLASHINFER_MOE_FP4=1 \
  -e VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 \
  -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_SERVER_DEV_MODE=1 \
  -e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 \
  vllm/vllm-openai:v0.14.0 \
  MiniMaxAI/MiniMax-M2.5 \
  --served-model-name MiniMaxAI/MiniMax-M2.5 \
  --port 8000 \
  --max-model-len 180000 \
  --max-num-seqs 56 \
  --tool-call-parser minimax_m2 \
  --reasoning-parser minimax_m2 \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --swap-space 16 \
  --enable-expert-parallel \
  --trust_remote_code

1. Namespace Creation

kubectl create namespace llm-inference

2. Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minimax-m2-5-vllm
  namespace: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minimax-m2-5-vllm
  template:
    metadata:
      labels:
        app: minimax-m2-5-vllm
    spec:
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.14.0
          args:
            - "MiniMaxAI/MiniMax-M2.5"
            - "--served-model-name"
            - "MiniMaxAI/MiniMax-M2.5"
            - "--port"
            - "8000"
            - "--max-model-len"
            - "180000"
            - "--max-num-seqs"
            - "56"
            - "--tool-call-parser"
            - "minimax_m2"
            - "--reasoning-parser"
            - "minimax_m2"
            - "--enable-auto-tool-choice"
            - "--tensor-parallel-size"
            - "2"
            - "--gpu-memory-utilization"
            - "0.9"
            - "--swap-space"
            - "16"
            - "--enable-expert-parallel"
            - "--trust_remote_code"
          ports:
            - containerPort: 8000
          env:
            - name: HF_HOME
              value: "/hf_cache"
            - name: VLLM_ATTENTION_BACKEND
              value: "FLASHINFER"
            - name: VLLM_FLASHINFER_MOE_BACKEND
              value: "throughput"
            - name: VLLM_USE_FLASHINFER_MOE_FP16
              value: "1"
            - name: VLLM_USE_FLASHINFER_MOE_FP8
              value: "1"
            - name: VLLM_USE_FLASHINFER_MOE_FP4
              value: "1"
            - name: VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8
              value: "1"
            - name: SAFETENSORS_FAST_GPU
              value: "1"
            - name: VLLM_SERVER_DEV_MODE
              value: "1"
            - name: TORCH_ALLOW_TF32_CUBLAS_OVERRIDE
              value: "1"
          resources:
            limits:
              nvidia.com/gpu: 2
          volumeMounts:
            - name: cache-volume
              mountPath: /hf_cache
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: cache-volume
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "16Gi"

Service Manifest

apiVersion: v1
kind: Service
metadata:
  name: minimax-m2-5-service
  namespace: llm-inference
spec:
  selector:
    app: minimax-m2-5-vllm
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

Verification Commands

kubectl logs -f deployment/minimax-m2-5-vllm -n llm-inference
kubectl get pods,svc -n llm-inference

Qwen/Qwen3.5-27B

Execution Script:

#!/bin/bash
set -e

# 1. Install Docker
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gnupg software-properties-common
curl -fsSL https://get.docker.com | sudo sh

# 2. Install NVIDIA Container Toolkit (required for --gpus all)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 3. Persistent HF cache
sudo mkdir -p /home/ubuntu/data

# 4. Launch vLLM container (Optimized for 2xB200)
sudo docker run -d --gpus all \
  --name qwen3-6-27b-vllm \
  -p 0.0.0.0:8000:8000 \
  -v /home/ubuntu/data:/hf_cache \
  -e HF_HOME=/hf_cache \
  -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_SERVER_DEV_MODE=1 \
  -e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  vllm/vllm-openai:v0.22.0 \
  --model Qwen/Qwen3.6-27B-FP8 \
  --served-model-name Qwen/Qwen3.6-27B-FP8 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --max-model-len 262144 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --async-scheduling \
  --trust-remote-code

1. Namespace Creation

kubectl create namespace llm-inference

2. Deployment Manifest — Optimized for 2xB200

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen3-6-27b-vllm
  namespace: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: qwen3-6-27b-vllm
  template:
    metadata:
      labels:
        app: qwen3-6-27b-vllm
    spec:
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.22.0
          args:
            - "--model"
            - "Qwen/Qwen3.6-27B-FP8"
            - "--served-model-name"
            - "Qwen/Qwen3.6-27B-FP8"
            - "--port"
            - "8000"
            - "--tensor-parallel-size"
            - "2"
            - "--reasoning-parser"
            - "qwen3"
            - "--tool-call-parser"
            - "qwen3_xml"
            - "--enable-auto-tool-choice"
            - "--max-model-len"
            - "262144"
            - "--max-num-seqs"
            - "128"
            - "--max-num-batched-tokens"
            - "16384"
            - "--gpu-memory-utilization"
            - "0.95"
            - "--kv-cache-dtype"
            - "fp8"
            - "--enable-prefix-caching"
            - "--async-scheduling"
            - "--trust-remote-code"
          ports:
            - containerPort: 8000
          env:
            - name: HF_HOME
              value: "/hf_cache"
            - name: SAFETENSORS_FAST_GPU
              value: "1"
            - name: VLLM_SERVER_DEV_MODE
              value: "1"
            - name: TORCH_ALLOW_TF32_CUBLAS_OVERRIDE
              value: "1"
            - name: VLLM_ALLOW_LONG_MAX_MODEL_LEN
              value: "1"
          resources:
            limits:
              nvidia.com/gpu: 2
          volumeMounts:
            - name: cache-volume
              mountPath: /hf_cache
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: cache-volume
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "16Gi"

3. Service Manifest

apiVersion: v1
kind: Service
metadata:
  name: qwen3-6-27b-service
  namespace: llm-inference
spec:
  selector:
    app: qwen3-6-27b-vllm
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

4. Verification Commands

kubectl logs -f deployment/qwen3-6-27b-vllm -n llm-inference
kubectl get pods,svc -n llm-inference

In-cluster endpoint: http://qwen3-6-27b-service.llm-inference.svc.cluster.local/v1.

Last updated 24 days ago

Was this helpful?