LLM Priority Scheduling
LLM Priority Scheduling lets admins assign a numeric priority (1–100) to each team, controlling which teams are served first when the self-hosted model is under heavy load.
Availability: Self-hosted deployments with at least one self-managed (OpenAI-compatible) model configured. Required role: Admin or Manager.
LLM Priority Scheduling lets admins assign a numeric priority (1–100) to each team, controlling which teams are served first when the self-hosted model is under heavy load. Lower numbers mean higher priority.
When the model's request queue is full, requests from higher-priority teams are served before lower-priority ones. Priority only affects queued requests — it cannot interrupt a generation that is already in progress.
How LLM Priority Scheduling Works
Tabnine passes each request's priority value to the vLLM engine via its built-in priority scheduling policy. A team with priority 1 is served first. A team with priority 50, the default, receives standard handling. A team with priority 90 is most likely to see a "model busy" message during peak load. Teams with the same priority value are handled in arrival order (FIFO).
Priority applies to both chat and agent requests routed through a priority-aware self-managed model. In-flight agentic flows are never interrupted; priority enforcement only applies at the next user prompt boundary. If a user belongs to multiple teams, the priority of their active team is used.
LLM Priority Scheduling is also designed to support automation workflows. In addition to interactive configuration, the capability is exposed in an API-first way so organizations can manage team priorities through their own provisioning, operations, or incident-response tooling. This is especially useful in self-hosted environments where priority levels may need to change quickly during peak load or scheduled events.
Enabling Priority Scheduling
In the admin console, go to Settings → AI Features. Locate the Priority Scheduling toggle. This toggle is only visible when at least one self-managed model is configured.
Turn the toggle On.
When enabled, an LLM Priority column appears on the Teams → Features page. Turning the toggle off hides the column but preserves all stored priority values; turning it back on restores them.
Setting Team Priorities
Go to Teams → Features (v6.4.0). In the LLM Priority column, enter a value from 1 (highest priority) to 100 (lowest priority) for each team. All teams default to 50. Changes are saved immediately. To set the same priority for multiple teams at once, select the teams using the checkboxes and use the bulk actions bar.
Example Configuration
Platform
12
Yes
Yes
1
SRE / On-Call
6
Yes
Yes
10
Backend
15
Yes
Yes
50
Frontend
8
Yes
Yes
50
Interns
3
No
No
90
What End Users See
End users do not see or control their priority level. When a request is rate-limited due to model capacity, the user sees the following message in the IDE:
"The AI model is currently at capacity. Please try again in a moment."
Important Notes
Priority does not preempt running requests. If a lower-priority request is already being processed, a higher-priority request will not interrupt it — priority determines only the order of the waiting queue.
In-flight agentic cycles are protected: if an agent flow is mid-cycle between turns, it is treated as highest priority and will not be blocked by incoming requests.
Lower-priority flows are held only at their next user prompt boundary.
Newly created teams default to priority 50. Disabling the toggle stops priority enforcement so all requests are treated equally, but stored priority values are preserved.
Prerequisites
At least one self-managed model (OpenAI-compatible) must be configured in the admin console. The Priority Scheduling toggle is hidden when only Tabnine-managed models are in use.
SME verification required: Confirm with engineering whether the requirement that the self-managed model be served by a vLLM instance configured with --scheduling-policy priority is a customer-facing configuration step. If it is managed internally, remove this note before publishing.
Last updated
Was this helpful?