Unattended Agents on oMLX and gentle-shell

Introduction

This article addresses a concrete question: is it possible to complete real work from beginning to end using local models, consumer hardware, and only open-source technology?

To investigate it, we developed a documentation pilot: turning PDF manuals and older screenshots into wiki documentation, with images of the new ADempiere version and an assistant that answers questions. The implementation combined the oMLX inference server, the Gentle Shell agent harness, and open-weight models from the Qwen family, running on a personal workstation.

I am the lead maintainer of ADempiere, an open-source project for ERP, CRM, and SCM. Its official repository provides public access to the source code and development history, giving readers context for the project behind this pilot. I also lead a company specializing in the development, support, and professional maintenance of solutions based on this technology. That experience helped us choose a concrete, useful task with verifiable results.

The purpose was not to compare this configuration with frontier models or tools such as Claude Code or Codex, but to establish whether it could complete the proposed work, identify the obstacles, and document the conditions needed to make it possible.

Below, I share the experience, the problems encountered, the solutions developed, and the results obtained, together with the limitations of the evidence.

Executive summary

We tested whether local models on consumer hardware, using open-source tools, could carry out useful documentation work without constant human intervention. The pilot used Qwen models served by oMLX on an Apple M5 Max with 128 GB of memory, with pi and gentle-shell coordinating agents working on an ADempiere wiki.

• For oMLX: TurboQuant batched-decode and restored-prefix prefill fixes improved measured performance. A proposed pause-and-resume stack kept requests alive under forced memory pressure.
• For gentle-shell: A todo-driven system-prompt rewrite caused substantial cache loss. The tested fix moves todo state into conversation messages to keep the system prompt stable.

With the proposed server and harness fixes applied during the pilot, the project met its objective: the documentation was completed from beginning to end in wiki format, with images of the new ADempiere version and an assistant that answers questions, using local models, consumer hardware and open-source tooling. The benchmarks document improvements in performance, memory use and stability that supported that outcome; they do not independently audit the wiki's content. Historical batch results, newer upstream comparisons and an unpaired live run answer different questions.

Results are limited to one Apple consumer workstation configured for local AI and one model family. Upstream merged PR2 as #4442 and PR1's fix as #4453 on 2026-10-12; the other fixes are proposals, not claims of upstream adoption, and none is a guarantee for smaller machines.

Results and limits at a glance

Read each result with its baseline. Repeated requests within one run are not independent benchmark repetitions. Follow the evidence links for exact settings, logs and commit identities.

At a glance

The use case

The setup is one Apple M5 Max (128 GB) serving one local model to several pi coding-agent sessions. pi runs the gentle-shell harness (package gentle-pi) on top, which adds the agent workflow: orchestration, a todo tool, review integration and subagent delegation. Subagents run as separate pi processes, each with the same harness. The sessions read whole source files and take screenshots. Prompts are large: across 1,260 recorded requests the median was 107k tokens and the p90 168k. Each turn re-sends the growing conversation, so the prefix cache decides whether a turn costs 1 second or several minutes.

The goal was simple to state: leave the machine working until every job finishes. That means using free memory for concurrency, never throwing away completed work, and pausing rather than aborting when memory is tight.

Field illustration from the later fork.5 recordings, not a benchmark of the original configuration.

What broke on stock v0.7.0

• Several concurrent long prompts reached the memory guard's hard watermark. Requests were aborted mid-prefill, all prefill progress was lost, and the agent stopped waiting for a human.
• Raising concurrency made things worse. In a 3-request mixed benchmark, concurrent finished in 538 s against 246 s serial, with 6 aborts.
• When batched decode did run, short rows in a left-padded TurboQuant batch produced degenerate output.
• Prefix-cache reuse was high on average but collapsed in bursts: single-turn losses of 70k–170k tokens.

How it developed

Premises and results

Start with the newer re-validations below. The original comparisons are retained as historical evidence; they are not measurements of today's upstream version.

Re-validation on newer upstream main

Field illustration of concurrent decode from the fork.5 recordings. The paired B=3 measurements below are separate evidence; this B=5 clip is not their recording.

The evidence above was measured on b7c7929e with the v0.7.0 app runtime. It was re-checked on newer upstream main commits in two rounds: PR1's premise on 25aebb3e (this subsection), then the whole fork against main c0b1056b (next subsection). Every branch proposed for upstream is now rebased on main 013bc3f3 (2026-10-10), with its focused tests and CI's command re-run there (Proposed contribution). The upstream commits between 25aebb3e and 013bc3f3 do not touch the TurboQuant files PR1 changes.

PR1 on 25aebb3e. PR1 was rebased on 25aebb3e at the time, so its premise, P3, was re-run on that tree. main 25aebb3e and PR1 (61bd8a06, two commits on 25aebb3e) ran from one venv built from main's pins: mlx 0.32.3, mlx-vlm 0.7.7, mlx-lm 94cdcae1, transformers 5.17.0. Only the source tree changed between them. The prompts are about 20k / 45k / 135k tokens, byte-identical to the earlier runs, with 512 new tokens and greedy sampling. Three B=3 rows are restored from the prefix cache, decode together, and are compared with the same prompts restored at B=1. One run per build, one server at a time, with the live app quit.

• Performance. Neither tree aborts or degenerates. PR1 makes batched decode 5.4× faster per row with MTP on and 12× faster with MTP off, cuts the B=3 phase from 345 to 66 s and from 780 to 65 s, and lowers the process peak by 11–15 GB. main's 0.66 tok/s with MTP off is what the microbenchmark predicts: about 86 ms per TurboQuant layer on the padded path, times 15 layers. B=1 decode is about 40 / 29 / 17 tok/s with MTP on, the same on both trees, so batched rows remain slower than single rows.
• Identity against B=1, MTP off. The B=1 reference is deterministic here: a second B=1 run is identical on both trees. main's batched rows leave it at about token 1–2 on the 20k and 45k rows; PR1's at about token 8–10, and at about token 45 on the 135k row. Neither is token-identical. Upstream has no logprob dump, so these divergences cannot be classified as near-tie noise or as a defect. The earlier departure on main is consistent with padded-row attention, not proven by it.
• Identity, MTP on. Two B=1 runs already differ from each other, so the B=3 divergences say nothing about batching.
• What changed since the earlier evidence. On b7c7929e with the v0.7.0 app runtime (mlx-vlm 0.7.1), main degenerated (4-gram about 0.33 / 0.29 / 0.18 at 121 GB) in one run and aborted (6 memory aborts at 120 GB) in another. Neither reproduces on 25aebb3e. Two things differ: the 52 upstream commits in between (including #3127's exact rollback and memory-guard and decode-fairness changes) and the runtime. With one run per build, which of the two removed it is unknown.

The whole fork against main c0b1056b

The fork was rebased again, onto upstream main c0b1056b (the commit tagged v0.7.1.dev1). Both trees ran from one venv built from main's pins (mlx 0.32.3, mlx-vlm 0.7.7), one server at a time, with the live app quit. The fork (6fbb3274) ran with the fork app's settings; main ran with plain settings. The forced-pressure scenarios are the C7 ones (below): three greedy prompts of about 130k / 115k / 100k tokens started 0.5 s apart, hard_threshold 0.8, the same custom ceiling for both trees. Pooled-buffer reclaim is always on now, because upstream removed the switch that disabled it. One run per scenario.

• Identity after a pause, MTP off. Four forced decode pauses per request (12 in total, at output offsets 100, 323, 1044 and 1706) give token-identical output for 3/3 requests.
• Park during a packed prefill. Upstream's packed prefill now prefills concurrent prompts in one forward. A row parked while packed resumed, and all 3 requests finished with 0 aborts. Outputs differ from the alone references only at near-ties (an exact tie and a 0.125-nat margin), and a control at 56 GB without any park diverges at the same tokens. The row that was never parked is token-identical with and without its companion's park.
• A fork defect found and fixed during this round. At 42 GB the first image-decode run rejected a lone request that fits alone, on a memory reading that still held a just-finished request's buffers. The fix re-measures before deciding and renews the request's eviction budget when the rows it waited on leave (strict TDD); the re-run passed at 42 and 44 GB.

Premises, fork against main c0b1056b, same venv:

P6's first measurement (3 runs, 256 decode tokens each, not interleaved) showed the fork's decode 7.1% slower, the same with every fork setting off. It did not reproduce. A re-measurement with 1,024 decode tokens and three interleaved sessions per tree, MTP on, gives the medians in the table, and a per-verify-cycle cost of +0.9% (standard error 1.4%) at 110k after controlling for drafted tokens. With MTP off both trees decode at 16.8–17.0 tok/s at 110k and 24.9–25.4 at 20k. The first gap came from MTP acceptance in a short sample (1.90 against 2.01 tokens per cycle) plus drift between runs that were not interleaved, so it was closed as noise without a bisect.

Re-validation with the prefill fix: fork 4aa66150 against main 013bc3f3

The runs above predate the restored-prefix prefill fix, which changes how a resumed park or a re-issued request continues over restored TurboQuant blocks. So the forced-pressure scenarios and P1/P2 ran again on the fork with the fix (4aa66150), against main at 013bc3f3, with the same venv, ceilings, prompts and settings. Every resume was checked in the log for the fix's dequantized … restored TurboQuant layers line, or its Kept … quantized fallback, and for the reservation that follows it. One run per scenario.

• Image prefill is where the fix shows. Each image prompt restores a 2,048-token block from the paged cache. Before the fix, that small quantized prefix kept the whole 100k–130k prefill on the quantized path: TTFT 1,797 / 1,267 / 2,130 s with two parks. With the fix: 249 / 460 / 637 s. The dense layers are now in the prefill reservation, so admission runs the prompts one after another. At 42 GB a prompt admitted beside another, both dense, was parked under hard pressure 4.6 s later, resumed alone, dequantized again and completed, with a 37 GB peak. That is a park and resume across the dense step on hardware, with no abort.
• Re-issue after a cancel (P1, 2 runs per tree). The fork keeps 73,728 tokens and re-issues in 80.0 and 82.0 s. main keeps nothing and re-issues in 192.0 and 193.0 s, the same as its cold prefill. Before the fix the same fork code took 198.9 s. The log shows dequantized 15 restored TurboQuant layers (73728 tokens) for a 36302-token suffix prefill in 0.42s, then a reservation across 16 caches (+2.22 GB).
• Two concurrent 130k prompts (P2). 2/2, 0 aborts, peak 53 GB, wall 574 s, against 51 GB and 564 s before the fix. No regression; no restore is involved.
• Identity after a pause, MTP off. 12 forced pauses per run, all resumed exactly. Two requests are token-identical; the third departs at a 0.125-nat near-tie. Its drift starts at token 4, before its first pause, after a one-off prefill throttle. The commit before the fix gives the same tokens in the same session (logprob-identical, 2,000/2,000), so neither the fix nor the pause path causes it.
• Packed prefill. Output with pressure equals the no-pressure control (the 42 GB run differs from it at one exact tie), and both depart from the alone references at the same tokens, as before. This session never parked a packed row, so that path was not re-exercised.

Upstream defect: slow suffix prefill after a restored TurboQuant prefix

Field illustration of concurrent prefill from the earlier fork.5 recordings, not a recording of the restored-prefix fix or its paired benchmark below.

Tracing P1's missing TTFT gain on c0b1056b led to a second upstream TurboQuant defect. After a paged-SSD prefix hit, the uncached suffix of a prompt prefilled about 2.4× slower per token than a cold prefill of the same positions. It reproduces on main, so it is not a fork bug, and with TurboQuant off the two take the same time.

Mechanism

• A paged-SSD hit restores TurboQuant layers as quantized TurboQuantKVCache objects (#1842), and the scheduler keeps them quantized while it prefills the suffix.
• Every chunk of every TurboQuant layer then attends to the whole quantized prefix through the tiled quantized-attention path, never dense SDPA, so the cost grows with the prefix length.
• The prefill capacity reservation reports across 1 caches after such a restore instead of 16, because only the unquantized last layer is a plain KVCache it can plan. That line is a symptom, not the cause.

Fix

Before a suffix that needs more than one prefill chunk, the scheduler copies the restored TurboQuant layers into dense KVCache layers once, so suffix chunks run the same dense path as a cold prefill. At the end of prefill only the suffix is quantized and appended to the original TurboQuant cache, so the restored prefix codes are reused bit for bit instead of being quantized again. The step runs only when the dense layers, sized for the whole prompt, fit under the prefill peak target, the same gate as the prefill reservation; otherwise it logs that it kept the quantized prefix. A one-chunk suffix, such as a chat turn, keeps the current path. Two commits with 16 tests, on main 013bc3f3.

Results

Qwen3.8-27B, TurboQuant 4-bit KV on 15 of 16 KV layers, one server at a time from the pinned venv. A = 73,787 tokens, stored; B = A + 36,358 tokens with a 73,728-token prefix hit; positions [73,728, 110,086) of B compared with a cold prompt from the same session. One run per configuration.

• TTFT of B: 198.8 s on main, 87.9 s with the fix. The one-off dense copy of 15 layers × 73,728 tokens takes 0.43 s.
• Quality (MTP off, greedy, 512 tokens, per-token logprobs from a local debug hook): cold against cold is identical. Restored output, with or without the fix, departs from cold only at near-ties (a 0.125-nat margin, one bf16 logprob step), at the same tokens, because the quantized prefix itself shifts the distribution. With the fix and without it, the restored outputs agree for the first 239 and 97 tokens, then part at exact ties.
• Memory: the reservation after a restore now plans the dense layers (16 caches, +2.23 GB instead of 1 cache, +0.14 GB), and the memory guards see them as current usage. The prefill peak is 26.29 GB against 25.22 GB for a cold prompt of the same length; the difference is the quantized prefix, about 1 GB, kept until the end of prefill.
• Live smoke on fork.7 (compiled kernels): a 41,323-token prompt with 24,576 tokens restored logged dequantized 15 restored TurboQuant layers (24576 tokens) for a 16746-token suffix prefill in 0.16s and a reservation across 16 caches. TTFT was 20.6 s, against 53.9 s for the same prompt cold.

The strategy: pause, don't abort

The most valuable idea of the project was a change of question. When memory reaches the hard limit, the stock server aborts in-flight requests, which throws away every token they prefilled and generated. We asked whether the server could pause a request instead. The paged SSD cache already existed and makes a pause cheap.

The first version, the park, covered only chunked prefill. A forcing benchmark (P7) showed that it works, and also where it stops. The next spike hit a decoding request, which could not be paused, and all three requests were aborted. That led to the full cycle. It pauses and resumes requests in prefill and in decode, image prompts included, and rejects a request that can never fit with a structured error. Below is what the code does, checked against the implementation (omlx/scheduler.py, omlx/process_memory_enforcer.py, omlx/settings.py).

The first version: park a prefill

1. Save progress while prefilling (D1, prefill_progress_commit_blocks). Every N completed blocks (4 blocks of 2048 tokens in the fork app), the prefilled blocks are stored in the paged SSD cache under the request's own block table. This needs the SSD cache. Without it the commit is skipped and a park resumes cold.
2. Park instead of abort (T7, memory_pressure_park). At hard pressure (not emergency, and within a budget of 30 deferral polls), the memory enforcer asks the scheduler to park a victim before it considers an abort. The scheduler drops the victim's live KV cache, which frees the memory. It keeps the client stream, prompt, sampling parameters and arrival time. It pins the committed blocks against eviction, both in the in-memory block pool and in the SSD index. The request stays registered and waiting, and the client gets no error.
3. Resume from cold cache. In the first version, a parked request was re-admitted after park_resume_ok_polls (default 3) consecutive ok polls, when the monitor went silent, or when nothing else could be admitted. The re-admission does a normal prefix lookup, which restores the committed blocks, and prefill continues from the last committed boundary. The full cycle replaced this resume rule with a memory-aware one (below).

Forcing it: the P7 park benchmark

The setup was three concurrent long prompts of about 130k, 115k and 100k tokens, started 0.5 s apart, on an isolated server. It used a low custom memory ceiling (42 GB, hard watermark at 80%), and the pooled-buffer reclaim was disabled so pressure reaches the park decision. We ran it once with text-only prompts and once with an image early in each prompt, on the full fork and on upstream main, one server at a time.

07:04:13  Hard memory pressure: parking a prefill victim on 1 scheduler(s) instead of aborting (poll 1/30)
07:04:13  Parked request e2f2… at 129024 tokens committed (park 1/3), reason: hard memory pressure
07:04:14  Resumed parked request e2f2…: cached=129024
07:04:23  Park skipped for e2f2…: decoding, not in chunked prefill (memory pressure falls back to abort)

P7 proved the path works: a prefilling request was paused, its memory freed, and it resumed from the SSD cache in about half a second with every committed token restored. It did not save a batch, because decoding rows and image prompts still fell back to abort.

The full cycle

1. Single victim (C1). When nothing is pausable, abort one victim (largest live KV, newest on ties) instead of all. The enforcer waits for that memory to drain before it picks another. Emergency and model-unload keep abort-all. Opt-in: OMLX_MEMORY_ABORT_SINGLE_VICTIM.
2. Decode pause and resume (C2). The pending sampled token is emitted first. The row's KV is extracted and stored as an exact SSD chain, keyed to the request and pinned, and the row leaves the batch while its stream stays open. The sampler, logits processors, stop-sequence position, parser state and remaining max_tokens are stashed. Resume restores the chain and re-inserts the row with no prefill, and the RNG is not reseeded. A singleton Lightning MTP row is first handed to standard decode. Batched MTP, DFlash, VLM-MTP and SpecPrefill rows are excluded.
3. Memory-aware resume (C3). A paused request resumes when estimated free memory covers its need: the KV of its final length plus a transient. It never resumes at hard pressure. FIFO among paused requests, and the head-of-line hold reserves the room.
4. Never fits vs fits later (C4). If weights + KV(prompt + max_tokens) + transient exceed capacity, the request gets HTTP 400 context_exceeds_capacity with seven numeric fields and retryable: false. A request that fits alone waits instead. GET /v1/omlx/capacity and the X-oMLX-Max-Context-Tokens header expose the numbers.
5. Harness compaction (C5, client side). The pi extension compacts and retries on context_exceeds_capacity, and sets its compaction threshold from the capacity endpoint. It has unit tests but no live validation yet.
6. Image prompts (C6). A text-only suffix after a cached last image is prefilled in the chunked path, so it can be paused. This route is taken only after checking that the suffix's mRoPE positions are arange + delta on every plane. Opt-in: OMLX_MEDIA_TEXT_SUFFIX_CHUNKED.

Pause and resume are behind memory_pressure_park (default off). The never-fits gate, the fits-later wait and a progress-based admission-stall clock sit behind memory_scheduling (auto | on | off, default auto). auto turns them on only when the memory guard has an explicit ceiling (the custom tier); otherwise admission stays as on main. The C7 runs all set a custom ceiling, so their evidence holds under the default.

C7: the extended benchmark, and what failed first

C7 extends P7 to decode-phase pressure, image prompts, a never-fits request and greedy identity after a pause. The target is 0 aborts, with one server at a time and the live app quit. The first run (branch e80eb1f7, 42 GB ceiling, hard 0.8, one run per pair) did not meet it:

Every pause logged restored N tokens exactly (stall 1.6–3.5 s), so the mechanics worked. The gaps were in policy and wiring. Each was traced to a root cause in the server logs and fixed test-first, one commit each:

Two intermediate runs show the progression:

• After D1–D3 (31cab33d, 48 GB): never-fits passed, image decode finished 3/3 with 0 aborts, and no abort-all fired. But text decode (twice) and image prefill finished 2/3: the third request failed on the admission-stall timeout, which led to D5.
• After D5–D6 (f3c11356): text and image decode finished 3/3. Image prefill at 46 GB still finished 2/3, which led to D7.

Final results

Same model, machine and scenarios, one run per pair. main (b7c7929e) ran the same scenarios at 48 GB. The branch's image-prefill run used 46 GB, which is harsher for the branch.

Never fits (42 GB, 31cab33d), a 60k-token prompt with max_tokens 2,000,000:

Greedy identity after a pause. B=1 at 56 GB, so there is no memory pause, and forced decode pauses at output offsets 100, 323, 1044 and 1706. Each test run is compared with a run without pauses.

Premises against main, measured at e80eb1f7, before D1–D8:

Reproduction commands

The harness is benchmarks/unattended/bench_premises.py on the fork branch bench/p7-park (95dd47e8). It overlays a checkout's omlx/ on the app runtime. Run one oMLX server at a time, with every other instance quit. The fork variant sets OMLX_MEMORY_PRESSURE_PARK=true and OMLX_PREFILL_PROGRESS_COMMIT_BLOCKS=4.

# branch build: export once, so every command below gets the same env
export OMLX_BENCH_FORK_REPO=<pause-resume checkout>
export OMLX_BENCH_EXTRA_ENV=OMLX_DISABLE_PRESSURE_RECLAIM=1,OMLX_MEMORY_ABORT_SINGLE_VICTIM=true,OMLX_MEDIA_TEXT_SUFFIX_CHUNKED=true

# references for one scenario: alone, 56 GB
python benchmarks/unattended/bench_premises.py --variant fork --premise C7 \
  --c7-scenario decode --runs 1 --p7-ceiling-gb 56 --c7-refs-only --out refs/ref-decode

# pressure run: decode | image-decode at 48 GB, image-prefill at 46 GB
python benchmarks/unattended/bench_premises.py --variant fork --premise C7 \
  --c7-scenario decode --runs 1 --p7-ceiling-gb 48 --c7-refs-from refs/ref-decode --out out/branch-decode

# never fits
python benchmarks/unattended/bench_premises.py --variant fork --premise C7 \
  --c7-scenario never-fits --runs 1 --p7-ceiling-gb 42 --out out/branch-never-fits

# identity with MTP off: references, then forced pauses
python benchmarks/unattended/bench_premises.py --variant fork-nomtp --premise C7 \
  --c7-scenario identity --runs 1 --p7-ceiling-gb 56 --c7-refs-only --out out/idrefnm
python benchmarks/unattended/bench_premises.py --variant fork-nomtp --premise C7 \
  --c7-scenario identity --runs 1 --p7-ceiling-gb 56 --c7-refs-from out/idrefnm \
  --c7-force-pause-at auto --out out/idforcenm

# main build: re-export, then run the same scenario commands (with main's own --out paths)
export OMLX_BENCH_FORK_REPO=<main checkout>
export OMLX_BENCH_EXTRA_ENV=OMLX_DISABLE_PRESSURE_RECLAIM=1

# unit and rig tests, then upstream's CI selection (needs pytest-xdist, as CI installs it)
pytest tests/test_scheduler_decode_pause.py tests/test_memory_resume_thrash.py \
  tests/test_memory_external_prefill.py tests/test_scheduler_capacity_vlm.py \
  tests/test_scheduler_forced_decode_pause.py tests/test_scheduler_solo_admission.py
pytest tests/ -m "not slow and not integration" -n 3 --dist loadgroup

Our runs also passed OMLX_BATCHED_DECODE_HOLD_MAX_S=120, OMLX_MID_SYSTEM_POLICY=inline_note and OMLX_THINKING_RETENTION_GUARD_TOKENS=20000 to both builds. Those belong to other fork features, and main ignores them. Scenario prompts: about 130k, 115k and 100k tokens, started 0.5 s apart, greedy, max_tokens 2000 (32 for image prefill).

Live production run (fork.5)

The C7 results come from forced pressure on an isolated server. We then ran the packaged fork as the daily server for real unattended coding-agent work: pi with the gentle-shell harness, one orchestrator session and four gentle-shell subagents doing browser-driven documentation capture for a documentation project, plus a second, unrelated agent session. Every request went to the local model through the profile pin.

Open requests are those with an open stream, running or queued, and waiting are the queued ones (both from /api/status). System free memory comes from macOS memory_pressure and includes the agents' browsers and other apps. Cache efficiency is the dashboard's cumulative figure at the end of each level.

• Concurrency raised live. At a limit of 5 the queue almost emptied (average waiting 2.2 → 0.1), while cache efficiency held at 94.5–95.4%.
• Batched decode at B=4 and B=5. On upstream main b7c7929e with the same v0.7.0 app runtime, P3 at B=3 degenerated or aborted. On newer main 25aebb3e it completes, but padded batched decode runs at about 1.5 tok/s per row (P3 re-validation).
• Cold starts when a wave arrives at once. Each wave's first requests (about 18k tokens, mostly the shared system prompt) arrived together. Of 8, 5 prefilled cold in parallel (58–129 s) and 3 reused 16,384 shared tokens (87–91%). After that, every subagent turn was append-only.
• Upstream prefill throttle. Upstream's adaptive prefill throttle paused chunked prefills 21 times across 14 requests to resize chunks. The highest process memory it logged was 46.6 GB. These are not the fork's pause and resume.
• Memory scheduling stayed on throughout. The log shows "Memory scheduling: off" lines in groups of four or eight (00:27, 03:02, 03:08, 04:19, including times with no settings change). They come from the admin settings page, which builds four throwaway enforcers (one per memory tier, default mode auto) to preview the tier ceilings; the live enforcer stayed on, which the capacity endpoint and the context header confirm. The misleading lines are fixed in the fork (04bb7899, proposed upstream with slice G): only the live enforcer logs its state, and the admin settings API reports the effective state as memory_scheduling_effective. The zero solo, admission-wait and never-fits counts therefore reflect a workload that never approached the memory limits, not disabled features.

Harness findings from the same sessions

The same sessions also surfaced harness-side prompt changes. They are listed in the gentle-shell section.

Field footage: sustained load

Screen recordings from the same night (about 04:28–04:43, shortly after the measured window), sped up. The oMLX dashboard is shown in its Spanish locale. Terminal and IDE content is pixelated for privacy; the dashboard numbers are as recorded. This was ordinary load, so no pause or resume appears on screen; the pause/resume evidence is the forced-pressure C7 runs above. Use the playback controls to pause, replay or inspect the recordings. The other recordings accompany the use case, batched decode and prefill sections.

What was built

The cache-reuse investigation

With aborts gone, the cost moved to re-prefill. We attributed every large cache break in the recorded traffic to a cause. The breaks came from the prompt changing in the middle, which invalidates everything after it.

The largest cause was in the harness. How it was found, what it cost, the fix and its validation are in the gentle-shell section.

The server-side change A1 is complementary. pi also sends system-prompt changes as mid-conversation system messages, and oMLX's strict fallback merged those back into the head. Any client doing the same benefits from A1.

preserve_thinking: true, measured again

The earlier result. On the 8-bit model with DFlash speculative decoding, keeping all past thinking gave near-perfect reuse, but at about 53k tokens of context the model started emitting literal </think> and raw tool-call XML as text, its draft acceptance fell from 65% to 54%, and the session did not recover. A2 was designed around that result. It keeps thinking only within the current agent loop and strips at real user messages, exactly as the template does with preserve_thinking: false. A deterministic guard strips once when retained loop thinking exceeds N tokens, then stays append-only. The guard is set to 30k.

The question. With false, every user-role message strips the reasoning of the previous segment, so the prompt shrinks and prefill restarts at the first assistant message of that segment. pi sends harness notices (todo updates, subagent results and wakes, background-job notices) as user-role messages, so they trigger the same strip. That was the last cache loss left in the parent session after the gentle-shell fix. true had never been tested for long on the current model (4-bit oQ4e with Lightning MTP), so we measured it on 2026-10-11.

Method. The installed fork.7 app was the only server running. The setting was changed at runtime through the admin API (PUT /admin/api/models/<id>/settings, which applies without a reload and persists), after a backup of the model settings. The client was the installed gentle-shell in pi RPC mode. Each pi request was joined to its server log line by its prompt and cached token counts.

• P1, harness scenario. One parent session with a todo list, a foreground subagent, a background subagent and its wake, a background job and its notice, and two follow-up prompts. true was compared against a false run of the same driver on the same server.
• P2, long realistic session. A fixed script of 27 real coding prompts on a scratch clone of the oMLX fork: read and explain code, find call sites, write and run unit tests, fix them, write a log parser with tests, use git. It made 99 tool calls in each run. The true run used its 110-minute budget after 27 prompts. The false control then ran the same 27 prompts. We counted degeneration markers in assistant text, tool arguments and reasoning: literal <think>/</think>, <function=, <parameter=, <tool_call>, visible-text repetition loops, length stops and tool-call parse errors. Reasoning repetition-detector signals are reported separately below.

Fixed settings in every run: thinking forced on (budget 3,000 tokens), the 30k retention guard on, MTP on, TurboQuant 4-bit KV, temperature 1.0.

Primary-record audit. Recounting tool_errors in preserve-thinking/p2-false-turns.jsonl gives 7, correcting the earlier reported 9. The corresponding p2-true-turns.jsonl contains 3 thinking_loops detector signals; the original summary counted only loops in visible text. The detector flags lines of at least 30 characters repeated at least four times within one message. These signals remain unclassified and may include false positives; they are not confirmed degeneration. The local records were audited without rerunning the experiment and are not currently linked as a public archive.

In P1, true made all 11 parent requests full-prefix hits, against 4 block-aligned divergences in 13 with false. The uncached tokens fell from 6,208 to 1,803 and the summed prefill from 17.2 to 8.1 s. The subagent and job sessions behaved the same under both settings.

Findings.

• No confirmed visible degeneration with true on this model up to 200k context; reasoning repetition signals remain unclassified, about 4× the point where the 8-bit DFlash setup broke. MTP acceptance stayed flat, and spot checks at 90k, 115k, 121k and 196k quoted code and file:line references from context correctly. The one slip was at 125k: the model ran a command in a documentation path quoted in the harness system prompt, then corrected itself on the next call. This is one run per setting at temperature 1.0, so it is evidence, not proof.
• The cache gain is real but small at these segment sizes. true removed every boundary divergence, but each turn here added only 1–15k tokens, so a divergence cost a few seconds. The minutes-long re-prefill seen with false in daily use (about 560 s at 135k) needs a turn that reads tens of thousands of tokens of files before the next user message. This script did not produce one.
• true has a cost: the context fills about 1.5× faster, because all reasoning stays in it. The model also reasoned about 2.4× more in that run, and decode slows as context grows. The same 27 prompts took 1.8× longer, and summed prefill was about equal, because true prefilled a much longer context. Part of the extra reasoning may be sampling variance.
• Confound: the 30k retention guard stayed on in both runs. With true it never fired; the prompt never shrank.

Decision. No literal degeneration markers were observed with true in this one run on this model; reasoning repetition-detector signals remain unclassified, and these observations do not establish the absence of quality risk. On 2026-10-11 it was switched on in daily use for further observation on workloads where single turns read large files, accepting the measured context-growth and wall-time tradeoffs. The things to watch are degeneration markers, boundary TTFT, how often sessions reach compaction, and wall time per task. The A2 guard stays at 30k as a backstop.

Live result

• Turns adding a new image: 10 of 10 had a 97.4–99.7% hit, so only the new tool result and image were prefilled.
• Consecutive turns after the client fix: 31 of 31 strictly append-only.
• Session-level cache efficiency on the dashboard: 95.2–95.3%.

gentle-shell: making the harness cache-friendly

This section is for the gentle-shell maintainers. The largest single cache loss in this report was in the harness, not in oMLX. While todos were open, gentle-shell's todo extension rewrote the system prompt on every user prompt. Over about 1.5 days of real use it caused about 1.05M re-prefilled tokens, roughly 80–85 minutes of GPU time, and turns that took 2–6 minutes instead of seconds. We improved the harness with the same method as the server, and the fix is ready on fix/todo-prompt-cache-1367, tracked in #1367.

The harness's existing tests did not expose the cache cost: the todo tool behaved correctly and no request failed, but some agent turns took minutes instead of seconds. We identified the cause by exercising the harness against a local inference server whose cache behaviour we could observe. Prompt-stability tests can catch the rewrite itself; server measurements establish its operational cost.

The process

What it cost on the server

This is about 1.5 days of real use, all on one machine. Figures marked ≈ are derived from measured rates.

On a hosted API, a changed prefix may reduce cached input and increase latency or cost, depending on that provider's cache rules and pricing. No hosted-provider cache or billing experiment was run here; the measured effects are specific to the tested local setup.

How the invalidation was identified

1. The symptom on the dashboard. During normal use, the oMLX panel kept showing long prefills (the blue progress bar) on turns that should have been nearly free. One example: 43.2k tokens at about 300 tok/s on an 80k prompt. A concurrent request's decode sat frozen behind it ("last token 46 s ago").
2. The server's own diagnosis. The prefix-cache log for that request reads:

   prefix cache restore: cached=36864 suffix=43171
   re-prefills 43171 of 80035 tokens; closest stored sequence shares the first 37735

   The same session diverged again later at 37,739 tokens. A divergence at the same place every few turns points to something rewriting the prompt at a fixed position, not to eviction.
3. Ruling out the server cache. SSD eviction was checked against the losses: the cache holds about 8 hours of history, and none of the large losses lined up with evictions. Model reloads were hours away. The block grid explains at most 2,047 tokens per event.
4. The client-side diff. Comparing the recorded payloads of two consecutive requests from that session showed the first difference inside the system message, at character 70,384 of about 104,700. That is roughly token 37.7k, matching the server's divergence point:

     …pending: install dependencies and verify loading.
   + (stale: 2 turns without an update — bring the list up to date now)
     </addendum>

5. Attribution across sessions. Doing the same across all recorded sessions attributed about 945k re-prefilled tokens in the main sessions and about 107k in subagents to system-prompt or tool-section changes. The todo addendum was the main driver.
6. Confirmation. After the fix, the same session's system prompt stopped carrying todo state at 00:26:37. Every later turn was append-only, apart from one expected cache break at the switch.

The fix: keeping the system prompt stable

The gentle-shell harness appended a todo block to the system prompt on every user prompt. Once the list went stale it added a counter, (stale: N turns without an update …), that changed on every prompt. The block sat around token 30–35k of a ~41k system prompt, followed by other sections. In affected parent sessions, every tick or todo edit broke reuse of the rest of the system prompt and the conversation after it. This does not apply to every subagent: on the upstream versions examined in the re-tests, children and background jobs with GENTLE_PI_AGENTS_CHILD=1 do not load the todo hook. System-prompt or tool-section changes recorded in child traffic must therefore not all be attributed to this hook. Single turns lost 70k–110k tokens of cache this way.

We fixed it in the harness. The system prompt is now byte-stable within a session. Todo state travels as a persisted conversation message, sent only when the list actually changes. The stale reminder fires once, with no counter. After a compaction, the state is re-injected once. The fix is tested (45/45 todo tests, with the unrelated failures reproducing on a clean base), was used daily, and is proposed upstream in gentle-shell #1367. The validation below ran on the fix branch fix/todo-prompt-cache-1367 at 955523df (2026-10-05, gentle-shell 4.0.0), not on upstream main.

• Recorded requests carrying a system-prompt mutation: 21 of 61 before, 0 of 90 after.
• Consecutive turns that were strictly append-only after the fix: 31 of 31.
• Server-side re-prefills above 20k tokens: they went from 1–2 per hour to 0.

Re-tests on upstream gentle-shell main

We re-checked upstream gentle-shell main at aa2c0389 during the live production run and posted the results to #1367 on 2026-10-07. Eight subagents sent 344 requests with one system prompt per session; 336 of 336 consecutive pairs were append-only, and per-session prefix-cache hits were 95.5–98.4%. The todo block appeared in no system prompt, and the problem did not recur. For accuracy: aa2c0389 still appends the todo block at before_agent_start, and its trigger (open todos plus a new user prompt) did not occur in this run. The issue stays open for its author and maintainers to decide.

Re-test on 2026-10-10: the problem is still present on upstream main, and the fix had to be applied. We re-tested gentle-shell main at 43541a0d with a test that drives the todo extension's public hooks and with a live pi session against oMLX. Upstream main still appends ## Todo list and its stale counter to the system prompt in before_agent_start (extensions/gentle-todo.ts). In a session with open todos, every new user prompt changed the system prompt: the block appeared, then the counter went from stale: 2 to 3 and 4. Live, the session produced 4 system messages, 3 of them "Updated system prompt section" updates carrying the todo block. Resuming a session also added a todo block (+515 characters) that the opening system prompt never had. The recent reminder change (#1924) adds a request-local reminder message, which is cache-friendly, but the block in the system prompt remains.

The trigger is the main (parent) session. gentle-shell subagents and background jobs do not load the todo hook, which is why the earlier subagent-only window showed no change. Each update is sent as a system message, and the Qwen chat template accepts a system message only at the start, so it is folded into the system head right after the base prompt (about 4,050 tokens). Everything after that point has to be prefilled again. The oMLX log of the live run shows it:

The session was short, so each break cost only a few hundred tokens here. In a 100k-token session the same break re-prefills almost the whole conversation on every user prompt while todos are open.

The todo list still has to change; the point is where the change goes. The fix does not freeze the todo state. It appends todo updates at the end of the conversation, as hidden messages sent only when the list actually changes, instead of rewriting the system prompt, so the cached prefix is preserved. The stale reminder is sent once, with no counter. We rebased the fix onto 43541a0d (it keeps #1924's reminder). With it, every request reused its full prefix except the expected re-render of the previous answer, and the 63 todo tests pass. It also passed an independent review with no blocking findings. The re-test, the cache table and the rebased fix (branch fix/todo-prompt-cache-1367) were posted to #1367 on 2026-10-10, correcting the 2026-10-07 comment above.

Status on 2026-10-11. Upstream main at d42264b4 still appends the todo block to the system prompt in before_agent_start (appendSystemPromptOnce in extensions/gentle-todo.ts). The fix branch is one commit on 43541a0d. The same patch applies unchanged to 563fede6, where it runs in daily use. It will be rebased onto current main before a pull request is opened.

Subagents and background jobs

On upstream main, the trigger is the parent session: subagents and background jobs run with GENTLE_PI_AGENTS_CHILD=1 and do not load the todo hook. To check the whole harness with the fix, we drove a gentle-shell build that carries the same patch (identical git patch-id to 4b929257, applied on upstream 563fede6) in pi RPC mode against the live server. One parent session ran a todo list, a foreground subagent, a background subagent and its wake, a background job and its notice, and two follow-up prompts.

• The parent kept one system message, the one at session start. Todo snapshots (4) and the subagent and job notices travelled as custom conversation messages.
• Each child session kept one system message.
• Of 18 requests, 11 reused the whole cached prefix and 2 were the cold starts of the parent and the first child. The second child started from a 2048-token block boundary of the first child's cache, as expected. The other 4 were parent requests that diverged at a block boundary; none of them was caused by a system-prompt change.

Harness notices and thinking retention

Those 4 parent divergences happened right after harness notices: a todo update, a subagent result or wake, a job notice. pi sends these as user-role messages, so the Qwen template treats each one as a new user query. With preserve_thinking: false it then strips the reasoning of the previous segment, the prompt shrinks, and prefill restarts at that segment's first assistant message. On the server side, preserve_thinking: true removes the loss: the same scenario had 11 of 11 parent requests reuse their whole prefix, and the uncached tokens fell from 6,208 to 1,803 (measurement). For harness authors this is worth knowing. A notice that arrives as a user-role message is a cache boundary for templates that strip reasoning at the last user query, so how often notices are sent matters.

Other harness findings

Running the harness and the server together during the live production run (fork.5) surfaced more prompt changes that neither side shows on its own. They belong to gentle-shell, not oMLX. The first two are already raised in #1367: the YOLO directive as a follow-up question (related to #1029, volatile RDD status in the system prompt), and the memory addendum toggle in the 2026-10-07 comment. The last two are not reported yet. On aa2c0389 the todo block is still injected into the system prompt at before_agent_start; it did not fire in this window because no new user prompt arrived while todos were open.

1. Turning on gentle-shell's YOLO mode injects its directive into the system prompt. In the orchestrator session (2026-10-06 22:53, before this window) that cost one re-prefill of 34,379 tokens: 111.9 s of prefill, against about 1.6 s for the next cache-hit turn.
2. An Engram memory addendum was added to and removed from the second session's system prompt twice, and toggled once in the orchestrator's. Each toggle invalidates the cache from that point on.
3. The gentle-ai-worker role lacked the native agent_browser tool, while the harness guard requires it for browser work.
4. Squad workers launched by a third-party extension did not use the profile's pinned local model. They started on a cloud model and failed for lack of credentials.

Why it matters for harness projects such as gentle-shell

• Prefix and KV caching is how both local servers (oMLX, llama.cpp, vLLM, MLX) and hosted APIs keep long agent sessions affordable. A harness that rewrites an earlier prompt prefix can increase uncached input and latency. The impact depends on each backend's caching rules, pricing and workload.
• Ordinary todo-behaviour tests missed this interaction. A prompt-stability invariant can catch the rewrite without a server; end-to-end measurements are needed to quantify its cache and latency impact.
• The same loop separated the causes cleanly: one was in the harness (fixed on a gentle-shell branch; still present on upstream main on 2026-10-10 and 2026-10-11), two were in the server (A1, A2), and the rest were expected (compaction, real user turns).

A proposal for gentle-shell

• A prompt-stability invariant test. Drive N turns through the harness with fakes, and assert that each rendered request is a strict prefix-extension of the previous one, except at declared boundaries (a real user message, compaction). The #1367 fix already includes the narrow version of this for the todo tool. It could be generalized to every extension that touches the system prompt (RDD status, YOLO directive, memory).
• An optional end-to-end check against a local server. Run a scripted session against a local OpenAI-compatible server that reports cached tokens (oMLX does, through usage.prompt_tokens_details.cached_tokens). Flag any turn whose cached-token count drops below the previous prompt length.
• A recorder mode. An opt-in payload recorder makes this kind of diagnosis reproducible for users who report "it got slow".

Open findings

Decode fairness does not cover multimodal prompts

Any request with an image anywhere in its history carries vlm_inputs_embeds and prefills in the synchronous external loop. That happens even when every image is already inside the restored cached prefix and the uncached suffix is pure text. Both chunked branches require vlm_embeds is None, so concurrent decode freezes for the whole prefill: 25–46 s observed, and 155 s on cold 43k prefills. The fix is now built as part of the pause-resume cycle (C6, opt-in OMLX_MEDIA_TEXT_SUFFIX_CHUNKED). It routes a text-only suffix that follows a cached last image through the chunked path, after checking that the suffix positions are t + rope_delta on every mRoPE plane. Tiny-model tests show identical output with the route on and off. Its effect on the decode freezes has not been measured on hardware yet.

TurboQuant decode attention at depth

A regression over 975 single-row requests gives about 250–295 ns per context token per MTP cycle, against roughly 32 ns for a pure KV read at this model's KV size. The intercept is 70–75 ms against a weight roofline near 30 ms. These are estimates from logs. We have not yet done kernel-level profiling.

MTP depth-1 verify uses stock qmm

Depth-1 verify (M=2) runs in 97.5% of cycles, and no verify-qmm route accepts M=2: the routes cover 3–6, 4–8 and 7–24. We estimate 10–20 ms per cycle from this. MTP is still net positive in every context bucket (1.3–1.45×).

How it was validated

Evidence status. The benchmark figures, traffic counts, test-suite outcomes and pilot completion in this report are reported by the author, not an independent reproduction. A reference audit checked published link destinations, internal anchors, available code and tests, arithmetic, and the measured-to-published commit mapping. Link availability or test-source existence does not prove a run passed. This page does not currently link an immutable archive of the primary run logs, result files and exact configuration for each measurement, or an independent wiki-content audit. PR1's non-identical outputs and PR2's unexercised hardware fallback remain limitations; branch links can move, so use the commit hashes for fixed evidence.

• Test-first. Each behaviour came with a regression test that failed before the change and passed after. The fork's full suite has 17,063 passed and 439 skipped, plus three environment-only failures (a local asset model without a chat template).
• Independent reviews. Each slice was reviewed separately, with four review lenses on the high-risk slices. Findings fed back as small follow-up commits, for example graceful degradation when a VLM tokenizer has no encode.
• Benchmarks against upstream. The premise harness ran upstream and fork servers in isolation, and checked that the live app was idle. It flagged contaminated runs, discarded them and reran them.
• Recorded-traffic replay. A client-side recorder captures exact provider payloads, and a replayer re-sends them in order to an isolated server to measure reuse per build.

Source code and commit references

Every commit and branch named in this report links to the fork on GitHub, e-Evolution/omlx (gentle-shell references link to e-Evolution/gentle-shell). Upstream commits are reachable from the fork with the same hashes. The PR branches are fix/turboquant-padded-batched-decode (PR1) and fix/turboquant-restored-prefix-prefill (PR2).

The fork branches were published on 2026-10-11. Before publishing, local machine paths were replaced in benchmark defaults, one test fixture and one docstring (output directory, cache volume name). Comparison of all 17 measured-to-published Git tree pairs found no differences under omlx/. The differences are confined to benchmark path defaults, supporting documentation and examples, the path fixture and the C7 docstring reference. The complete trees are therefore not byte-identical; reproduction requires explicit local path settings. This corroborates source continuity, not the historical measurements themselves. The commit hashes changed. The report shows the published hashes; the table maps each to the hash that was measured, which also appears in the logs. Hovering a published hash shows the same mapping. The earlier PR1 commit 61bd8a06, replaced by the rebased branch, is kept under the tag report/61bd8a06.

Proposed contribution

To oMLX

We would like to send these upstream as small pull requests, each with its own regression tests and before/after numbers on the same hardware, following docs/CONTRIBUTING.md. Upstream routes feature proposals to GitHub Discussions > Ideas, and blank issues are disabled, so the proposal is posted as Ideas discussion #4440 (2026-10-11). The two TurboQuant fixes, PR1 and PR2, are bug fixes under 1,000 changed lines, so they were opened directly on the same day: PR1 as #4441 and PR2 as #4442. On 2026-10-12 the maintainer merged PR2 as #4442, with a follow-up for the dense dtype when no KV layer is plain and the embedding is quantized, and closed PR1 in favor of #4453, which extends the #4408 kernel to take each row's left padding and lists us as co-author. The pause-resume cycle is over 1,000 changed lines and adds a subsystem, so it was first posted for feedback on #4440; on 2026-10-12 its stack was opened as twelve pull requests, #4478 (ready for review) to #4489 (drafts). Each slice states its dependency and merge order. The order is a proposal and we will adapt it to your preferences.

Every branch below is rebased on upstream main 013bc3f3 (2026-10-10, MLX 0.32.3, mlx-vlm 0.7.7). On that base, CI's command (-m "not slow and not integration" -n 3 --dist loadgroup, Python 3.11, a venv with main's pins) gives 16,928 passed on main, 16,956 with PR1 (+28 tests) and 16,944 with PR2 (+16), with the same 3 environment-only failures (test_gemma4_rendering.py). Before opening them, PR1 and PR2 were also checked against the newer main e02ae514 (2026-10-11): both merge without conflicts, and CI's command gives 16,938 passed on main, 16,966 with PR1 and 16,954 with PR2, with the same 3 failures. The whole stack was rebuilt on the same base from the fork, every slice from P-a to S9, with the memory_scheduling gate (G) before S4; each slice applies on the one before, and its focused tests and CI's command pass at its tip (two load-sensitive tests that pass alone aside). For the pull requests, the stack was rebased on main f3ad34d2 (2026-10-12), and following upstream #4396 its two environment-only switches became settings (memory.abort_single_victim, scheduler.media_text_suffix_chunked) and the forced-pause test knob was dropped. CI's command gives 17,119 passed on that main and 17,175 (P-a) to 17,580 (S9) at the slice tips, with the same 3 failures; one load-sensitive cluster test also failed at S2 and S4 and passes alone 3/3 there.

About two thirds of the cycle's changed lines are tests. For S1–S9 and G, the per-slice counts in the table below add up to 8,632 test lines out of 12,961 (67%); with the prerequisites P-a and P-b, 11,658 of 17,652 (66%). The scheduler is concurrent (the MLX thread, the memory enforcer and the SSD writer interleave), so every edge case found on the real model has a regression test on the real scheduler and enforcer rather than on fakes. That is why it is split into chained pull requests, each carrying its own tests.

Line counts are additions plus deletions of each slice against the slice before it, on main 013bc3f3. "Memory scheduling" means off unless an explicit memory ceiling is set or memory_scheduling=on. G adds that gate before any path reads it, and S4, S7 and S8 consult it from their first commit, so with every flag off each prefix of the stack behaves as main on these paths; each gated slice has tests that drive it with the gate off and expect main's behaviour. The C7 evidence on c0b1056b (re-validation) covers P-b, S1–S3, G and S6–S9, the MTP-off identity run covers S2, the never-fits run covers S4, and S5 has unit and tiny-model identity tests only. The rebuild cherry-picked the fork's commits in slice order with mechanical resolutions; the one test that used the fork's defaults file (G's admin-state test) now sets the mode in settings.json, and a test-fixture fix for upstream #4312 moved into S6, whose tests need it. These fork commits are in no pull request: the packaging commits (defaults bundle, gate removal, --allow-pin-mismatch), the defaults-file layer and the shared on/off env reader that goes with it, the benchmarks/unattended harness (linked as reproduction only), and the fork-side test that combines PR2 with the prefill park (it follows once both are on main).

Later, separate oMLX proposals

oMLX bug reports we would file (or comment on)

• TurboQuant left-padded batched decode attends over padding columns (short-row norm ratio 0.32 in synthetic tests) and is slow (about 1.5 tok/s per row at B=3 on 25aebb3e), with a reproducer; PR1 is the fix. Degenerate output and aborts were seen only on b7c7929e with the v0.7.0 app runtime. Related: #4110, #3767, #2784.
• Slow suffix prefill after a paged-SSD restore of a TurboQuant prefix (2.4× a cold prefill of the same positions on c0b1056b); PR2 is the fix. Related: #1842, #3933.
• Decode fairness bypassed for multimodal prompts. Related: #3209.
• Lightning MTP single-request output is not bit-reproducible on reruns. It diverged on 1 of 3 runs on upstream.
• A1 and A2 context added to the existing threads (#3608, #1966, #3378, #900, #1003) rather than new issues.

Before each oMLX PR we will

• rebase onto current main (013bc3f3 at the last rebase, 2026-10-10: MLX 0.32.3, mlx-vlm 0.7.7);
• run CI's command: python -m pytest tests/ -m "not slow and not integration" -n 3 --dist loadgroup on Python 3.11, plus Black and Ruff at 88 columns;
• re-run the relevant benchmark against main with run counts.

To gentle-shell

gentle-shell requires an issue with status:approved before a pull request. #1367 is open with status:needs-review, and we have asked for approval so that the fix can go in as a pull request, or be folded into another change if the maintainers prefer.