Model configuration and API¶
Model Controls¶
Ollama, vLLM and vLLM-Omni share one local-model form for both Create and Edit. Model source, hardware, runtime and memory sections expose the selected engine's supported controls; one collapsed Advanced section contains shared CPU controls and engine-specific options. External providers keep their provider form. Existing activations retain their saved engine, source and hardware target when edited; only changed fields are submitted with the original revision fence. Polling updates availability without replacing unsaved input.
With automatic NVIDIA pool assignment the calculator and slider describe one physical GPU, not total pool VRAM. A numeric budget above the known per-card physical limit disables Save/Add and is also rejected by the API, even with memory-risk acceptance. An invalid older saved budget remains visible for an explicit correction; opening Edit does not silently replace it. An explicit multi-GPU split retains its separate group-total control. See memory accounting.
The Name field validates local and external models before submission: 1–63 lowercase letters, numbers or hyphens, starting and ending with a letter or number. Inline feedback explains an invalid or already-used name and blocks creation. Surrounding whitespace is removed before saving; names remain fixed when editing. The API also validates the name when creating a resource.
Create → Location: Local → Inference Engine: (Experimental) vLLM-Omni selects
the catalog's experimental Omni profiles for NVIDIA CUDA, AMD ROCm, Intel XPU
and CPU. GPU choices use schedulable Kubernetes resources, not chip/driver or
Strix-Halo allowlists. Shared mode uses one slot; exclusive mode offers the
one-/two-GPU stage plans. CPU does not request a GPU. A matching runtime image
is still required; CPU/XPU need an explicit image in Advanced.
Context and concurrent sessions use the common runtime section. The memory
section contains system RAM and the GPU fraction; Advanced contains the
optional image and thinker offloading as well as CPU reservation/limit.
RAM estimates are advisory; actual node capacity and slot checks stay.
Model source offers HF search and direct repository paths. Quantization,
architecture, stage completeness and config.json are not pre-approved.
Any valid HF reference can be tried without an online configuration check;
the runtime reports actual load failures in Status/Logs. Selection is not a
hardware/model support guarantee. The selected repository is saved in
local.url and retained on edit.
Location distinguishes only Local and External; vLLM-Omni is offered
alongside the other local engines when the catalog advertises a Realtime profile.
It still persists as engine: VLLM with local.realtime, not a new backend engine
type. Ordinary KV precision, tensor/pipeline parallelism, MiB VRAM controls and
vision-attention overrides are not mapped into the shipped duplex stage plan;
the API rejects these fields for Omni. The shared interface does not change that
runtime contract. Installed Realtime models reuse Edit, Logs, Start/Stop, Restart and
Remove. Speech testing uses LiteLLM's existing Realtime Playground, not a new
dashboard conversation UI. See Realtime, including the pinned
development-runtime and hardware acceptance boundaries.
Local model creation and editing include one collapsed Advanced section.
Its CPU subsection is independent of memory controls. Blank fields use the
engine defaults advertised by the compute-target catalog. Users can set a CPU
reservation and optional limit in logical cores; limit 0 removes the CPU
quota. Use automatic CPU settings removes stored overrides. Saving only CPU
changes preserves RAM/VRAM and model identity, and may restart the model Pod.
See the CPU scheduling policy for
defaults.
An unschedulable model shows Kubernetes' actual scheduling reason (for example,
Insufficient cpu) instead of only waiting for a ready replica.
For vLLM on AMD, the Deployment subsection of Advanced exposes the Vision attention backend. It defaults to vLLM Triton attention for new AMD models; Automatic (vLLM default), PyTorch SDPA + AOTriton, and FlashAttention (AMD Triton) remain selectable. The compute-target catalog owns the options and eligible targets. Info icons explain runtime dependencies; selecting an option is not a successful hardware/model test. Editing saves only the changed selection and may restart the runtime. Switching engine or hardware in Create clears the AMD-specific draft. This setting does not change the memory reservation, context, KV precision, or text decoder backend. See vision attention deployment for mappings and the distinction between an omitted setting and explicit Auto.
Local and external models are runtime requests stored as ModelActivation
resources in namespace ai-system.
The Create button beside Installed Models opens the model form directly
in that section. The form starts with a persistent location dropdown containing
Local and External. The external choice reveals the existing provider form.
For a local model, inference engine and hardware appear as additional dropdowns
above the model form. Every completed selection remains visible and can be
changed directly; there are no wizard steps or back buttons. Engine and hardware
choices are filtered by current cluster capability. Unavailable CPU or
accelerator targets are omitted instead of being presented as disabled choices.
The engine dropdown sorts regular engines alphabetically first (Ollama,
vLLM), followed by (Experimental) vLLM-Omni, also alphabetically. The first offered engine is the
default for a new form unless the API advertises an available default engine.
These are display labels only; catalog availability and
persisted engine identifiers are unchanged.
The ordinary local Create form detects the Chat or Embedding task from the selected preset or registry metadata. It shows the result without a Type dropdown. If metadata is insufficient, Model task becomes a required manual choice; changing the model clears that choice. External creation and existing model editing retain their explicit type controls. Realtime uses its own catalog profile. Detection does not certify that an engine can load the model. See the API task contract.
Ordinary Create and Edit both offer optional Max Output Tokens; leaving it blank uses the runtime default. Clearing a stored limit sends an explicit reset. Omni does not expose this field because its current duplex adapter rejects it.
For vLLM, Model source offers three persistent choices:
- Hugging Face search accepts a model name or prefix and provides normal
full-width dropdowns, one below the other, for the matching model and its
original or quantized artifacts. A stable model-family row provides Qwen,
DeepSeek, GLM, Llama, and Gemma searches, while a second row comes from
Hugging Face's live
trendingScoreorder. Additional result pages can be loaded without losing the current selection. - Tested preset retains the allowlisted catalog flow and its Precision / Quantization dropdown.
- Direct Hugging Face URL retains the existing
hf://escape hatch.
The discovery backend returns only public, non-gated, non-disabled repositories.
Quantizations must declare a direct quantized relationship to the exact
selected model; name similarity, adapter, fine-tune, and merge relationships do
not qualify. Publisher, formats, parameter metadata, revision, trust
classification, and runtime-compatibility guidance remain visible after
selection. Revision displays the full SHA returned for the currently selected
artifact, including when a different quantization is selected. It is discovery
metadata, not a guarantee that the runtime download is pinned to that commit.
If the registry returns no revision, the dashboard does not invent one.
When Hugging Face publishes usedStorage, the dashboard also shows
the repository download size. The artifact resolver and estimator read the
selected repository's public config.json, including nested language-model
configuration such as text_config; this is necessary because Hugging Face
search and detail payloads can omit the advertised context limit. When the
configuration advertises a maximum context, that value becomes the initial
Context Size while remaining editable. If a quantization repository does
not publish its own context metadata, discovery inherits the directly related
base model's value and labels it accordingly. A new dynamic model starts
with Max Num Seqs = 1. A dynamic result is labelled experimental until the
chosen runtime has loaded it; the dashboard does not claim that arbitrary
community artifacts are validated. Selecting an artifact writes its hf://
repository to a custom activation and reuses the same server-side memory
estimator as a direct URL.
GGUF and MLX repositories are excluded from the current vLLM selection because
that path cannot select the required GGUF file/plugin and does not run MLX.
For Ollama, Model source offers three persistent choices:
- Ollama Library provides stable Qwen, DeepSeek, GLM, Llama, Gemma, and Mistral family shortcuts, a live popular-model row, prefix search, and two full-width dropdowns for model and tag/quantization.
- Tested preset retains the validated catalog variants.
- Direct Ollama model reference retains the
ollama://model:tagescape hatch.
The tag selection displays the download size and advertised context from the
public Ollama tag page and starts with Max Num Seqs = 1. After selection, the
registry manifest supplies exact model-layer bytes and the estimator reads only
a bounded GGUF-header range for architecture dimensions. It can also refine the
quantization from source metadata. Cloud-only tags are not
offered. The public Ollama Library is an HTML interface rather than a documented
catalog API, so the backend uses a bounded, cached adapter and leaves presets and
direct references available when discovery fails. Hugging Face discovery remains
separate because the current runtime contract pulls ollama:// registry models
and cannot import an arbitrary Hugging Face GGUF repository automatically.
vLLM accepts hf:// model references; Ollama uses ollama://
references.
Ordinary vLLM uses upstream 0.31.0 for all four target images, pinned by digest;
vLLM-Omni is versioned separately. Ollama uses immutable 0.40.3 images for CPU,
NVIDIA and AMD ROCm, retaining the same registry tags and saved model controls.
See Ollama upgrades
before changing an installed runtime. Registry search and memory estimates are not
a runtime compatibility certification. Check model architecture, quantization
and target hardware against the selected runtime when startup fails.
cpu is available on a compatible Ready Linux node. nvidia-gpu, amd-gpu,
and intel-gpu are shown only when the matching provider is Ready and
Kubernetes reports its allocatable resource. Intel automatically resolves
gpu.intel.com/xe or gpu.intel.com/i915. vLLM supports all four targets.
Ollama supports CPU, NVIDIA, and AMD; Intel is absent from the Ollama choices
because no validated KubeAI/Ollama Intel profile is bundled.
For an experimental AMD profile, the selected engine needs current host evidence and an adopted runtime configuration; GPU validation remains optional. Upstream provider readiness and an allocatable resource alone do not satisfy that additional gate.
Beside Context Size, the local form shows a KV Cache dropdown. Ollama offers F16, Q8, and Q4. vLLM offers model precision on every target and FP8 only for CUDA/NVIDIA and ROCm/AMD. The options come from the selected compute target's live API contract, so CPU and Intel never show an unsupported FP8 choice. Changing the cache format immediately reruns both the normal memory estimate and, when selected, the CPU-offloading estimate.
An Ollama pod is not sufficient by itself for a Ready model. KubeAI addresses
the runtime with the ModelActivation name, while the downloaded registry tag
can have a different name. The Magic Stick Operator therefore waits for the
source download and verifies or repairs that runtime alias before the model is
published to LiteLLM. During this phase the installed-model status remains
Starting instead of exposing a route that would return 404 model not found.
The dashboard calls POST /api/models/estimate-memory for every supported
vLLM/Ollama local combination: vLLM on CPU, NVIDIA, AMD, and Intel, plus Ollama
on CPU, NVIDIA, and AMD. vLLM estimates use public HuggingFace weight and model-configuration
metadata. Ollama estimates use the exact runtime-layer byte total from the
public registry manifest and attention, recurrent-state, GQA, hybrid-layer, and
context dimensions from the GGUF header. Only the first bounded header range is
requested; model tensors are not downloaded by the estimator. If a registry
proxy cannot serve that range, the API retains the conservative manifest-only
calculation and reports that fallback explicitly.
Both the RAM and VRAM controls use unreserved memory (total memory - active
model reservations) as their 100-percent slider maximum. The separate live
free-memory value does not change that planning limit. Minimum and recommended
values are marked on the same scale. If either estimate exceeds unreserved
capacity, its marker remains visible in a gray overflow section to the right of
the slider. The slider stays bounded; the numeric field also accepts larger
budgets, with an explicit memory-risk warning before creation.
For Split one model across several selected NVIDIA GPUs, Create and Edit
show a Total VRAM budget (MiB), including total Minimum, Recommended and
100% markers. The maximum is the smallest selected card's unreserved budget,
rounded down to 100 MiB per card, multiplied by the number of selected cards.
The input divides that total equally into per-card reservations; each share
must still fit its physical GPU. Replicate model copies retains a budget
for each full copy, and a single GPU retains its individual budget.
The React dashboard's collapsible Breakdown separates model weights, the base KV-cache
estimate, recurrent state where applicable, hybrid-allocator safety, engine
runtime components, recommendation headroom, and download size. Split-model
GPU components are totals across the selected cards, matching the calculator;
their explanations show the per-card formula and group multiplier. Host runtime
for a split Pod and download bytes are counted once. Replication keeps full
per-copy estimates. For vLLM hybrid models, the UI labels the
architecture-derived KV value as Theoretical KV cache and shows the extra
compatibility budget independently instead of presenting their sum as physical
cache use. Ollama hybrid models show attention KV and their context-independent
recurrent state separately. CPU vLLM runtime reserve is split into compile/warm-up headroom, the
multimodal processor cache, and a quantization working copy when applicable.
Download size is storage/network information and is explicitly not included in
the memory total. Every displayed recommendation and selectable reservation is
rounded upward to a 100 MiB planning step. The safe slider maximum is rounded
down to the same step. Each summary/Breakdown value has an info (i) button:
hover or focus previews its explanation; click/Enter pins it, and Escape,
the close button, or an outside click dismisses it. Overlays show the formula,
substituted current inputs, binary units and rounding. Cache explanations include
bytes per token per sequence and assumed KV precision, independent of weight
quantization. Quantized-cache overlays also show the exact format storage rule,
the F16/native-16-bit baseline, and the resulting saving. Ollama Q8_0 uses
34 bytes per 32-value block and Q4_0 uses 18 bytes per block, including their
two-byte scales; only attention cache changes. Runtime allowances and fallback heuristics are labelled as estimates,
not measurements. The API supplies additive calculations entries containing
formula, substitution, and notes; an older API shows an honest unavailable
message instead of invented dimensions.
For CPU targets, the selected value is stored as
spec.local.memoryRequiredMi. The operator rounds it up to a 16 MiB unit and
turns it into the model pod's Kubernetes requests.memory. For accelerator
targets, the selected VRAM remains scheduling/planning metadata; old Ollama
models without an explicit offloading policy retain their automatic fit.
Live memory metrics currently come
from NVIDIA DCGM, so AMD and Intel estimates can show minimum, recommendation,
and breakdown without an adjustable maximum until matching memory metrics are
available.
For CPU vLLM, the dashboard API derives spec.local.kvCacheMemoryBytes
server-side from public model architecture metadata, context size, and maximum
parallel sequences. The browser cannot supply an arbitrary runtime value. The
operator passes that value to vLLM as --kv-cache-memory-bytes; 512 MiB remains
only as a compatibility fallback for older or directly created resources that
do not contain the derived field.
GPU models also show System RAM estimate / reservation with serving RAM, an estimated startup range and a conservative recommended budget. Startup includes transfer buffers and a reclaimable file-cache envelope; these are not additional permanent model weights or measured RSS. One split Pod has one RAM budget, while replication shows per-copy and combined recommendations. New multi-GPU/offloading configurations use this recommendation when available; saved or manually edited RAM is preserved. The recommendation button is explicit, and a value below the provisional startup minimum needs memory-risk acceptance. Single-GPU models without CPU offloading retain their existing resource profile and show advisory RAM figures rather than claiming that a new limit was applied. Unverified capacity or missing weight metadata stays Unknown.
Opening Edit keeps the saved RAM/VRAM budgets visible while the estimate loads. A failed estimate shows an actionable error and blocks saving; it does not replace the saved budget with a guessed value. Realtime compute choices identify both the node and CPU/GPU target, including when both are advertised on one node.
vLLM GPU cache estimates use local tensor-rank heads (including GQA/MLA replication) or pipeline-local layers. Recurrent state and QSA side caches are itemized. Ollama uses an estimated whole-layer placement envelope. Per-worker runtime reserve follows local shards, with uneven placement separated from weights; the total estimate is labelled Estimated even when checkpoint byte metadata is exact. Special N-gram/PLE host placement requires runtime-specific verification and is not automatically enabled by this calculator.
The installed-model card distinguishes KV requested from KV active. The second value remains pending until the operator observes a Ready replica created with the requested vLLM argument or Ollama environment. This confirms the applied pod configuration rather than claiming a separate engine-memory measurement; a runtime that rejects the mode never receives a misleading active label.
For NVIDIA GPU models, CPU offloading → Use additional system RAM is an
opt-in setting below the VRAM control. Keep VRAM and host RAM separate: the new
Host RAM reservation slider/number field includes offloaded weights and
host runtime, while Use recommended RAM allocation adjusts only host RAM.
Budgets use 100 MiB steps. The host ceiling is the largest eligible node's
allocatable RAM after workload requests, with pending requests deducted
conservatively; it is not the cluster-wide CPU gauge. Unknown capacity,
uncertain estimates, or inadequate budgets show a warning, not a disabled
Add Local Model button. Clicking the warning-styled button accepts the
memory risk and stores spec.local.allowMemoryRisk: true. The API/controller
then allow estimated under-reservation and unverifiable/oversubscribed capacity
without raising the selected budgets or removing requests/limits. Pods may remain
Pending, fail to load, or OOM/restart. Invalid inputs, unsupported hardware/engine
combinations, missing metadata needed to derive an offloading plan and replica
restrictions are still errors. Manual RAM choices survive recalculation, and changing engine or
hardware resets opt-in so a previous policy cannot silently carry over.
The expanded breakdown separates estimated GPU/RAM weights and KV placement,
host/GPU runtime, and headroom. vLLM offloads weights, not KV; Ollama always uses
GPU-first auto-fit and chooses the exact maximum layer placement when loading.
The proportional preflight split is not a VRAM hard limit. The installed
model card shows the host reservation separately and displays Ollama's /api/ps
RAM/VRAM buffer reports when available. Missing measurements remain explicitly
unknown, never zero. These reports are not process RSS or the Kubernetes
reservation. See CPU offloading behavior and limits.
The estimator prefers exact public Safetensors file sizes, checks the requested
context against the model's advertised maximum, and adds a CPU-specific
start-up envelope for working tensors, compilation/warm-up, and the default
multimodal processor cache. Hybrid attention models are labelled Estimated
and receive a conservative cache compatibility factor because current vLLM
hybrid cache grouping can allocate more memory than the pure full-attention
formula suggests. The shared API returns both values as
theoreticalKvCacheMi and hybridAllocatorSafetyMi; runtime components are
returned in runtimeDetails, while recommendedReserveMi is the separate
recommendation headroom. CPU reservations below the computed minimum require
the explicit memory-risk acceptance described above. API/CLI clients that omit
local.allowMemoryRisk retain the stricter preflight checks.
This applies to both CPU Ollama and CPU vLLM, on creation and editing:
the minimum is rounded up to the same 100 MiB step as the form. An explicit
Ollama risk acceptance retains the requested budget without requiring public
registry metadata; it does not inject vLLM cache parameters or bypass invalid
RAM values. Legacy Ollama definitions without an explicit RAM budget keep their
runtime defaults.
Accelerator availability uses the Ready, schedulable node's allocatable vendor
resource as the final runtime signal. An enabled vendor operator that is
temporarily reported as Reconciling by Flux therefore does not hide NVIDIA,
AMD, or Intel once Kubernetes already publishes the matching GPU resource. If
the resource is still absent, the operator phase remains the blocking reason.
Above the installed-model list, the Models screen renders only a small
Compute memory heading and one semicircular gauge per compute device. Ordinary
CPU/discrete-GPU gauges have two rings: violet for unreserved memory
(total - active model reservations), cyan for current availability. The full
arc represents that reading's capacity. On unified-memory CPUs both rings
use Linux-visible RAM; the unreserved value excludes system headroom. The
center shows current availability, or the unreserved budget if live metrics are
missing. A small legend and an info symbol replace inline explanation paragraphs.
CPU totals are aggregated across Ready,
schedulable appliance nodes, reservations come from active CPU models and
GPU models with explicit CPU offloading via ModelActivation.status.memoryRequiredMi
(falling back to the requested value), and current availability
comes from fresh host /proc/meminfo MemAvailable samples when available.
Ordinary nodes retain the Kubelet summary/metrics API fallback. Unified-memory
nodes never use that fallback: driver-owned GPU allocations can be absent from
Kubelet working-set accounting, overstating both CPU and shared GPU availability.
NVIDIA gauges use one DCGM record per physical GPU. With the default device
plugin, Kubernetes exposes a whole-GPU request but not the chosen GPU UUID on
the ModelActivation; the dashboard packs planned vramRequiredMi reservations
deterministically across detected devices. That packing is not a binding.
With opt-in NVIDIA DRA, local.gpuDevice binds the model to a verified card and
its node UID. Its reservation is charged only to that card, never a sibling or
a replacement node. The actually-free inner ring always comes from DCGM.
local.gpuDevices extends this to a same-node NVIDIA group: one slot and
the per-card VRAM reservation are charged to each selected UUID, not every GPU on
the node. Eligibility uses DRA node identity and known positive physical
gpuCapacityMi, with gpuCapacitySource: nvidia-dra-inventory. GPU models and
capacities may differ; equal per-card budgets are bounded by the smallest
unreserved budget and every card's physical capacity.
DCGM's totalMi can differ between samples or from the inventory's capacity;
it is not the grouping criterion. Missing inventory capacity blocks
a group, while missing live readings alone do not. Live readings and planned
reservations retain their separate memory meanings.
In Split one model, the dialog shows the combined VRAM budget and
converts it to the per-card local.vram/local.vramMi contract when saving.
Existing saved per-card budgets need no migration. System RAM/CPU are reserved
once for the Pod.
In Replicate model copies, the full model runs on each selected GPU and
system RAM/CPU are reserved per copy. The UI shows the combined host-RAM budget
and the per-copy ready state; Logs can select an individual copy. NVIDIA GPUs
shows all physical cards as checkboxes. Checked cards determine the selected
count and saved identities. Polling never
silently substitutes a missing or full card. Multi-GPU settings and vLLM
parallelism appear under Advanced. See GPU sharing
for supported combinations, memory semantics and current acceptance limits.
ResourceSlice capacity without DCGM yields a known total but unknown free
memory. AMD and Intel device-plugin resources are also listed individually. Until their installed
operator supplies a compatible memory exporter, unavailable readings use a
dashed ring and —, not an invented zero, total, percentage, or free value.
This preserves an honest UI while keeping the response contract
ready for additional vendor metric adapters.
An explicitly configured unified-memory AMD profile shows one GPU, not separate fixed/shared GPUs. Fresh, PCI-matched KFD evidence determines the ordinary allocation domain and GPU planning capacity. GTT/TTM bounds alone do not prove model capacity. Fixed GPU allocations are outside Linux RAM; only their host runtime requests count there. Dynamic GPU allocations consume Linux RAM and intersect its remaining budgets after safety headroom. Missing metrics and unverified GPU cgroup accounting remain explicit.
GPU gauges add an outer segmented model-slot ring (gold: free, grey: occupied)
and exact free/total counts, independently of memory. Fully occupied GPUs remain
visible but disabled in the Hardware selector, with a no free slots hint.
For multiple NVIDIA device-plugin cards, an aggregate Node slots ring is
shown once above their memory gauges; the runtime chooses the physical card.
NVIDIA DRA instead gives each card its own ring and a NVIDIA GPUs checkbox list in
Create / Edit with name, PCI address, VRAM and free slots. Editing an active model
may reuse its own card's slot. Stopped models have no such credit. A lost card or
changed node UID remains visibly blocked rather than selecting another card.
The model form refreshes every 15 seconds; if its selected GPU becomes full,
submission is disabled without clearing the form or switching hardware.
The API also checks current slots before writing a local model; accepting a
memory-estimate risk does not bypass slot exhaustion. CPU models are unchanged.
See slot accounting for counting rules.
On unified-memory hosts, Models → Compute Memory keeps one GPU gauge with four memory rings inside the slot ring: dedicated unreserved (violet), dedicated free (cyan), shared unreserved (blue), shared free (green). The compact legend groups each pair under its own capacity. These are separate scales, not an additive model budget or two deployment targets.
Dedicated free uses PCI-matched AMD mem_info_vram_total - mem_info_vram_used;
dedicated unreserved is attributed only to a confirmed firmware-reserved model
allocation domain. Without live metrics, the free ring stays unknown and the
center falls back to the applicable unreserved budget. Shared free is
max(0, min(Linux MemAvailable, dynamic GPU ceiling - mem_info_gtt_used)).
Linux availability already reflects those GPU allocations; they are not
subtracted from MemAvailable a second time. This is remaining reported
capacity, not a guarantee that any individual allocation will fit.
For example, 38.2 GiB Linux availability, a 108 GiB shared ceiling and 78.7 GiB
reported GTT usage yield min(38.2, 108 - 78.7) = 29.3 GiB shared free.
A separate 95 GiB reservation leaves a 13 GiB GPU planning budget; it does not
change that measured free value. Calculations use bytes before conversion to
MiB; compact gauge labels round GiB for display.
Shared unreserved is limited by remaining Linux budgets after system headroom
and model requests; when the confirmed model domain is shared/GTT, remaining
GPU reservations also constrain it. Missing pool metrics, unknown domains and
non-matching pool IDs are not replaced with another node's values. The dynamic
ceiling is part of Linux-visible RAM, not an additional bank or protected reserve.
A confirmed shared/GTT domain remains valid when its Linux-capped capacity is
smaller than the fixed firmware reservation; that comparison alone must not
erase the unreserved budget. The host corroborates the domain using raw KFD/GTT
evidence before applying capacity bounds.
The lightweight host sampler refreshes every 30 seconds. The API requires a
sample no older than 90 seconds matching the Node UID, boot ID and kernel;
missing, invalid or stale counters stay unknown, never zero usage. Current
sample time and the formula are available through the info symbol.
Hovering or focusing the info symbol previews the explanation. Clicking/tapping pins it open; Escape, the close button or an outside click dismisses it. The popup includes installed RAM, fixed GPU reservation, Linux-visible RAM, dynamic ceiling, driver capacity, data sources and accounting caveats. The full inventory remains in System → Hardware → GPU nodes; it is not a separate card on Models. Unknown driver capacity remains unknown. Only separately confirmed hardware controls change memory settings. See the inventory contract. CLI hardware output and the physical TUI's Models/Hardware views expose the same values and the same non-additive relationship. The dynamic ceiling and Kubernetes requests are not protected reservations against other processes; the UI does not promise guaranteed dynamic AI capacity.
The preset selector is populated from ConfigMap/magicstick-model-presets and
shows only variants compatible with the selected engine and target. Each
variant may declare defaultArtifact and artifacts[]; the artifact entries
carry their own checkpoint/tag, precision, quantization method, and planning
budget. The selected ID is stored in spec.local.artifact and displayed on the
installed-model card. Existing resources without that field continue to use the
variant default. For catalog presets, the URL is derived from the artifact and
remains read-only; selecting Custom is the explicit path for a user-supplied
URL. An artifact compatibilityNote, for example an FP8 hardware requirement,
is shown directly below the dropdown. Such a note is a warning rather than a
replacement for accelerator-generation acceptance testing.
qwen2505bcpu remains the portable smoke preset. The expanded Qwen3.5,
Qwen3.6, and Qwen3.8 catalog uses official Hugging Face checkpoints plus
target-specific FP8/GPTQ/AWQ artifacts where the runtime supports them, and
explicit Q4_K_M/Q8_0/BF16 Ollama tags on CPU, NVIDIA, and AMD. Qwen3.5 4B fills the useful
capacity gap between the 2B and 9B presets. qwen3827b retains its validated
single-NVIDIA-GPU AWQ default and adds official FP8 and BF16 alternatives.
Selecting a preset and artifact fills its target-specific context, output-token,
concurrency, memory, and runtime values before the activation is submitted. The
model catalog propagates consumer limits into managed KubeOpenCode templates.
The default dashboard is GPU-neutral. External models do not require or activate
GPU or KubeAI modules. CPU model creation remains available without any GPU
driver and lets the operator install KubeAI on demand. An unavailable target is
disabled with a concrete reason. The API enforces the same live availability
check and engine/target compatibility matrix and returns HTTP 409 if a client
attempts to bypass the UI. It resolves the engine-specific resource profile
server-side instead of accepting arbitrary args, environment values, or profile
names from the browser.
The Models screen treats unavailable device metrics as a neutral, per-device state. Runtime removal remains outside the memory display and is exposed only when no local model still depends on the automatically enabled runtime.
The System → System Status screen renders all three GPU providers even on a CPU-only
appliance. Each card shows the pinned operator version, driver mode, detected
and compatible nodes, management owner, allocatable resource count, phase, and
the controller's non-sensitive explanation. NotRequired means no matching
hardware was found and the vendor operator consumes no cluster resources;
Installing lasts until the vendor resource becomes allocatable.
The Services screen uses this hardware lifecycle rather than the Flux revision
of the operator module. For NVIDIA it keeps the service in Installing after
the extended resource appears until the DCGM exporter can provide GPU telemetry.
This prevents a successfully applied chart from being presented as an
operational GPU before the Models screen can observe the device.
Catalog-only models are read-only in the Models screen. Remove actions are shown
only for ModelActivation rows that the dashboard can delete.
Operators and administrators can Stop and Start every managed model from
its installed-model card. This uses the existing spec.enabled flag for Ollama,
vLLM, Realtime, and external providers, preserving the activation and all saved
settings. Local runtimes are removed on Stop; their resources become available
as the Pods and allocations are released. Start recreates the runtime with its
saved settings through the normal readiness and scheduling checks. For external
providers, Stop only withdraws the Magic Stick route; it does not shut down the
remote service. Realtime also provides a Restart action.
The button follows the desired enabled state, prevents duplicate requests, and
waits for runtime removal before allowing Start. Writes carry the current
configuration revision; errors remain visible without removing the model card.
For ordinary Ollama/vLLM runtimes, Removing continues until the KubeAI Model
and remaining runtime Pods are gone. A terminal Pod or a withdrawn catalog
entry alone is not proof of completed cleanup; runtime shutdown errors must
be investigated rather than hidden by removing finalizers or extending every timeout.
See stopping and resuming models.
Operators can open Edit on every managed installed model. Local edits keep the model reference, inference engine, hardware target, name, and namespace fixed while exposing the applicable engine settings: context/output limits, sequence concurrency, KV-cache/memory/offloading controls for vLLM/Ollama, and Realtime stage settings. The memory estimate is recalculated against capacity with the model's own existing reservation removed, so an unchanged deployment is not counted twice. External edits cover provider parameters and can replace an API key; leaving the key blank preserves the existing Secret. Save changes remains disabled until a parameter differs. Saving reconciles the activation and can restart its runtime. The object identity and configuration generation captured when the dialog opens prevent an older form from overwriting a concurrent change or a recreated model. Status-only updates do not invalidate the form. Writes still use an atomic Kubernetes resource-version precondition; one status-only race can be retried without accepting a concurrent spec edit.
Administrators can open Logs on each local installed-model card. The dialog
refreshes a bounded tail from every declared runtime and init container and also
shows the previous container run after a restart. It remains useful while a
model is starting; if the runtime controller has not created a Pod, or
Kubernetes has no output yet, the dialog says so without turning that expected
state into a dashboard failure. External providers have no local Pod and
therefore no Logs button.
Because model input and provider output can be sensitive, this API and control
are restricted to magicstick-admin. The backend accepts only a
ModelActivation name, validates the target namespace, uses KubeAI model labels
plus its controller owner reference, strips terminal control sequences, and
never accepts a browser-supplied namespace, Pod, or container name.
A local model first shows WaitingForPod while no model Pod exists. After two
minutes without a Pod it becomes Degraded with reason ModelPodCreationStalled,
and a hint to inspect KubeAI/controller admission errors. Retries continue;
the status recovers automatically when a Pod appears. This timeout does not
apply to downloads or engine startup in an existing Pod.
An owned Pod that has permanently failed is automatically replaced with bounded
backoff. Its original failure appears in the model message; after five failed
recovery attempts the model stays Degraded until its configuration is reviewed.
A local model with a Pod is shown as Starting until KubeAI reports a ready vLLM or Ollama replica.
If a container in an exactly owned model Pod restarts unsuccessfully three or
more times without readiness, the model instead becomes Degraded with reason
ModelRuntimeCrashLoop and a hint to inspect runtime logs, startup-probe events,
the model reference and registry access. This also covers an Ollama download
failure that leaves the Pod in Running / CrashLoopBackOff. Explicit OOM,
image-pull backoff and container-configuration failures are reported separately.
Kubernetes retries continue; the status check does not delete Pods. Readiness
clears the failure, and a replacement Pod starts with its own restart count.
A healthy, slow download is not subject to a blanket startup timeout.
The status message includes the ready-replica count, for example 0/1 replicas
ready. Ready therefore means both that the local runtime is serving its
health endpoint and that the generated catalog has published the model.