{"version":1,"source":"repository-snapshot","updatedAt":"2026-09-23T11:36:53Z","totalRecordCount":437,"recordCount":11,"filters":{"q":"latency","repo":"","limit":30},"records":[{"id":"manaflow-ai/cmux#13754","repository":"manaflow-ai/cmux","number":13754,"title":"ci: seed the compilation cache on the runner pull requests restore it from","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13754","mergedAt":"2026-09-22T23:56:19Z","updatedAt":"2026-09-22T23:57:15Z","labels":[],"metricHints":["| **Compile app-host test product** | **19.9 min** |","| Resolve Swift packages | 2.9 min |","| Package compiled product | 1.8 min |","| Cache Swift packages | 1.7 min |","| Checkout | 1.1 min |","| *job total* | *29.3 min* |","> Runner pools lay the workspace out differently, and a seed built under another layout cannot hit, so it should be a cache miss and not a download.","Different pools, different workspace path, different fingerprint, guaranteed miss. `nightly.yml` already carries the requirement directly above the job — *\"Match the admission job's runner and Xcode. The cache key carries the toolchain and the build paths, so a mismatch is a miss, not a wrong hit.\"* The admission job later gained a PR-specific runner and ...","Point the seeder at the runner the pull request path uses. No cache key, schema, build input or product identity changes — this only puts the seed where the consumer looks.","FAIL: refresh-test-compilation-cache must run on the same runner pull request admission uses,","`tests/test_ci_test_compilation_cache_seed.sh` already pinned the shared script, the shared build paths and the shared key prefix — every ingredient of the key except the runner that determines `$PWD`. This adds that.","The first seed after this lands still has to be written by a scheduled nightly run before any PR can hit it, so the improvement appears on the next `17 */6 * * *` cycle, not immediately. I have not measured the warm compile time, because no warm compile has ever happened on this path — the 19.9 min figure is the cold cost this removes the cause of, not a ...","Both jobs are on Blacksmith macOS, which is free on this repo, so this is pull-request latency rather than spend.","Fixes the Xcode compilation cache so pull requests can restore the nightly seed from main instead of compiling cold. The seeder ran on a different runner pool than pull request admission, and since the cache key hashes the workspace path, the seed never matched — the compile step alone took 19.9 minutes per PR.","- Point the seeder at the runner pull request admission selects (`vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15'`); no cache key or build input changes.","The first seed after this lands appears only after the next scheduled nightly run, so PRs benefit starting with the following cycle, and the warm compile time is still unmeasured."]},{"id":"manaflow-ai/cmux#13711","repository":"manaflow-ai/cmux","number":13711,"title":"CI: add free GitHub-hosted macOS background lane","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13711","mergedAt":"2026-09-22T17:38:05Z","updatedAt":"2026-09-22T17:38:06Z","labels":[],"metricHints":["Every macOS job shares one pool. Blacksmith is sponsored, but PR CI queues on it for 30–60+ min at peak, and the overflow is paid Warp (`MACOS_RUNNER_15` currently points at `warp-macos-15-arm64-6x`). Some of the work in that queue has nobody waiting on it. The repo is public, so standard GitHub-hosted macOS runners are free with unlimited minutes (about ...","New lane variable **`MACOS_RUNNER_BACKGROUND`**, used as `${{ vars.MACOS_RUNNER_BACKGROUND || 'macos-15' }}`. It is unset, so the lane runs on free GitHub-hosted `macos-15`. An admin can repoint the whole lane with one variable edit.","Members (off the PR/merge path, and small enough for 3 cores / 7 GB):","| Job | Trigger | Workload | Recent runtime on the paid pool |","| `build-ghosttykit.yml` | dispatch only | Zig test + universal xcframework | 5.4 min Blacksmith (after a **22 min queue**), 6.2–6.5 min Warp |","| `cmux-tui-artifacts.yml` macOS legs (aarch64 and x86_64 darwin) | push to main, after merge / dispatch | Rust release build | ~4.5 min each on Warp |","`cmux-tui-build-package.yml` is unchanged. The two callers above pass the lane through its `macos_runner` input, and the release (`cmux-tui-release`) and merge-gate (`cmux-tui.yml`) callers keep their own runner. `build-ghosttykit` gets a `verify_build` dispatch input. It builds and packages even when the release already exists and never re-uploads. Its t...","Why `macos-15` and not `macos-26`: the self-hosted fleet has the `macos-26` label (see `test-ios.yml`), and the fleet guard already rejects it. The `macos-15` image ships Xcode 26.3 (macOS 26.2 SDK) next to its default 16.4, so `CMUX_CI_XCODE_APP_MACOS_15=/Applications/Xcode_26.3.app` also works there.","- Release, nightly app and signing: `release.yml`, `nightly.yml`, `cmux-tui-release`, `ios-app-store`. These are urgent, and the docs keep them on `MACOS_RUNNER_*`. `refresh-*-compilation-cache` is also excluded because the cache has to match the runner that consumes it.","- Full app or app-host XCTest builds (too heavy for 7 GB): `tmux-corpus` terminal-nightly, `iroh-release-gate` (both jobs build `cmux-unit`), `ios-streamed-validate`, `ios-screenshots`, `test-ios`, `ci-macos-compat`.","## Latency trade-off","- **cmux-tui-artifacts darwin legs:** about 4 min on warp-6x today; expect roughly 12–16 min on hosted macos-15. The run's critical path is the ~12 min x86_64 Linux musl leg, so the whole run grows by about 0–5 min plus hosted queueing (about 5 concurrent jobs). `release.yml` and `reload-build.yml` resolve the newest published cmux-tui commit without a fa...","- **build-ghosttykit:** a PR that bumps ghostty now waits about 22 min for the xcframework instead of about 6.","- A GitHub-hosted macOS label (`macos-NN`, `macos-latest`, `-intel`/`-xlarge`…) is allowed in a runner-selection position only as the exact `vars.MACOS_RUNNER_BACKGROUND || 'macos-15'` fallback. That covers `runs-on`, matrix `os`/`runner`, `macos_runner:` and list items. It also closes an existing gap, where `vars.X || 'macos-15'` used to pass. The existi...","`build-ghosttykit.yml` dispatched on this branch with `verify_build=true`: [run 35755233641](https://github.com/manaflow-ai/cmux/actions/runs/35755233641). It ran on the GitHub-hosted `macos-15` label and was picked up after **9 s in the queue**, compared with the 22 min Blacksmith wait above. Wall time was **22.5 min**, about 4x the paid 6-vCPU runner an..."]},{"id":"manaflow-ai/cmux#13378","repository":"manaflow-ai/cmux","number":13378,"title":"ci: extract reusable guard workflow","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13378","mergedAt":"2026-09-21T12:59:59Z","updatedAt":"2026-09-21T12:59:59Z","labels":[],"metricHints":["This is the topology half of RFC #13095 slice 1. The guard latency behavior change already landed separately in #13097; this PR keeps those job bodies, checkout depths, runner labels, permissions, commands, and route meanings unchanged.","The accepted #13097 optimization provides the before/after measurement for this slice:","| Metric | Before | After #13097 |","| Guard wall-clock critical path | 385.5 s | 124.3 s |","| Linux guard runner-minutes | 6.43 min | 7.49 min |","Before: https://github.com/manaflow-ai/cmux/actions/runs/35477597285/job/105989493676","After: https://github.com/manaflow-ai/cmux/actions/runs/35478932148","- history: 124.2 s","- main guard suite: 116.8 s","- CLI/profiling guards: 109.9 s","- source lints: 98.4 s","So #13097 cut the measured guard wall-clock by about 68% while adding about 1.06 Linux runner-minutes from parallel setup/checkout overhead. The critical path is roughly 2m04s; the history lane is the longest remaining group.","- guard-only test routing after the file move","- existing app-host and Ghostty workflow-body guards after ownership moves","Extracts the four Linux guard jobs from `ci.yml` into the reusable `ci-guards.yml` workflow, replacing them with a single `guards` call. Job bodies, checkout depths, runner labels, permissions, commands, and route meanings are unchanged."]},{"id":"manaflow-ai/cmux#13589","repository":"manaflow-ai/cmux","number":13589,"title":"#13531: harden cold-task cleanup ownership and recovery","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13589","mergedAt":"2026-09-22T06:18:20Z","updatedAt":"2026-09-22T06:18:20Z","labels":[],"metricHints":["Finishes the remaining #13531 acceptance gaps on current `main` after #13532 introduced the retire-then-reap lifecycle.","- reject symlink/path-confused active, retired, and generation directories before cold-task retirement or reclamation;","- when `inflight.json` is absent after a helper crash, require the durable lease's exact native run ID and process group to be settled before retiring its cold generation;","## Foreground latency","| pre-fix leave-active baseline | 0.00009 ms | 0.00010 ms |","| atomic retire (`os.replace`) | 0.00454 ms | 0.01481 ms |","| recursive delete (illustrative avoided foreground cost) | 71.70 ms | 75.95 ms |","- `python3 tests/test_dev_fleet_warm_slot.py`: 38 tests passed in 7.64s.","- `python3 tests/test_benchmark_dev_fleet_warm_slots.py`: 5 tests passed in 0.15s.","- The repeated-cold test runs four cold fallbacks, observes zero active cold generations after every completion, then reclaims the four retired generations in two bounded passes of two.","- All cache navigation and cleanup use no-follow file descriptors rooted at the machine state; lease-supplied paths are never used, and directory identity is verified before and after every operation.","- Telemetry stays advisory: one `events.jsonl` plus a 16 MiB archive, crash-torn tails repaired on append, no `fsync`, no writer thread.","* Cleanup now avoids affecting shared warm data and validates cache generations before removal.","* Cleanup can resume after preemption without blocking foreground task execution."]},{"id":"manaflow-ai/cmux#13646","repository":"manaflow-ai/cmux","number":13646,"title":"ci: throttle main pushes by nightly published commit age","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13646","mergedAt":"2026-09-23T01:25:39Z","updatedAt":"2026-09-23T11:32:30Z","labels":[],"metricHints":["- Push-to-main macOS job time was 7,197 min, or **about 1,030 min/day**. Three hung `build-nightly-app` jobs (996, 229 and 161 min) account for part of that; without them it is about 830 min/day. The whole workflow, including schedules and dispatches, used about 1,270 min/day.","- Each publish costs about 51 macOS minutes at the median: app build 29, three sign/notarize jobs at 7 each, Ghostty helper 4.","- **70 of 101 publishes were replaced by another publish within 60 minutes.** The median gap between publishes was 44 minutes, which matches the median trigger-to-publish time of 43 minutes. On busy days the pipeline runs back to back.","A replay of the same pushes, using the 43-minute pipeline and the 08:47 catch-up:","The expected result is about 55–60% fewer push builds, **roughly 450–600 macOS min/day**. The lower bound is about 280 min/day: 14.4 publishes a day becomes 9 or fewer, at 51 minutes each.","- **Latency.** The two-hour threshold uses the published commit age, not publication time; build duration and push timing also affect delivery latency. In a quiet period it waits until the next push or 08:47 UTC. Anyone who needs a build sooner can dispatch the workflow.","- **Compilation cache seeds (#13182).** This does not affect the Debug seed that compile admission reads, because only the `17 */6` schedule writes it. The Release seed (`xcode-compilation-release-*`) that the PR release-build lane restores is written by push builds and by the 6-hour schedule. It will be refreshed every 2 hours or so instead of every 45 m...","- **Why not `cancel-in-progress` for pushes?** It was tried and reverted in 9fdf3d0f84. Merges land about 25 minutes apart at the median, and a publish takes 43 minutes, so cancelling would keep publishes from finishing and throw away partial builds. Run 35484670961 lost 157 macOS minutes when its signing jobs were cancelled.","- **#12610** skips builds whose changes are only web, docs or tests. That change is complementary: it avoids builds that change nothing, and this one limits how often builds that do change something run. Both edit `decide`, so whichever lands second needs a small rebase.","Throttles push-triggered Nightly builds on `main` so the macOS pipeline is no longer busy around the clock. A push on `main` now skips while the commit the `nightly` tag points at is younger than `NIGHTLY_PUSH_MIN_INTERVAL_HOURS` (default 2; `0` restores publish-every-push). RC branches, manual dispatches, and both schedules decide exactly as before, and ...","- Expect ~55–60% fewer push builds, about 9/day instead of 21, saving roughly 450–600 macOS minutes per day.","- A merge can wait up to ~2 hours for a Nightly; the 08:47 UTC schedule still publishes whatever `main` HEAD is then.","Validation after review fixes: fixed-clock behavioral suite passes, including exact interval boundaries; mutation checks confirm the non-default-interval and same-commit cases detect regressions. Release guard structure, 28 guard-routing tests, execution registry validation, and all required CI checks pass."]},{"id":"manaflow-ai/cmux#13658","repository":"manaflow-ai/cmux","number":13658,"title":"ci: route pull-request macOS jobs to Blacksmith, keep overflow for the rest","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13658","mergedAt":"2026-09-22T13:46:48Z","updatedAt":"2026-09-22T13:46:48Z","labels":[],"metricHints":["From a 24h measurement (618 runs → 1,973 jobs, weighted to full run volume; method in #13652), overflow macOS minutes by triggering event:","| **`pull_request`** | **6,535** | **56.5%** |","| `workflow_dispatch` | 2,782 | 24.0% |","| `push` | 1,922 | 16.6% |","| `schedule` | 330 | 2.8% |","This PR covers roughly **5,900 min/day, about 51% of all paid macOS minutes**, without touching a repository variable.","2. **It is the least latency-critical.** PR runs are cancelled on supersession by design; ~25% of all macOS minutes in the window were spent on runs that were cancelled mid-flight.","3. **It is the burst-prone lane, and bursts are why overflow exists.** Expected concurrent macOS jobs run at ~11.6 mean but spike to ~61 at peak (p95 ~31). Moving the burstiest lane onto sponsored capacity is what reduces the need to reach for paid fan-out in the first place.","**`swift-package-tests` is deliberately excluded.** Its `runs-on` line is pinned by an exact-string assertion in `tests/test_ci_self_hosted_guard.sh` because of the dual-SDK sequence (build the Release Ghostty helper under SDK 15, then select Xcode 26 for package tests). Loosening a safety guard for ~574 min/day was not a good trade, and `docs/ci-runners....","- `tests/test_ci_self_hosted_guard.sh` (includes `no workflow falls back to a Warp runner`)","- #13652 — the measurement, and the repository-variable changes that cover the remaining 49%","Routes pull-request macOS jobs to Blacksmith to cut paid overflow usage, covering roughly 51% of overflow minutes. All other events keep following the repository variable exactly as before.","- Excluded `swift-package-tests` (pinned runner-line guard) and `release-build` (macOS 26, gate)."]},{"id":"manaflow-ai/cmux#13580","repository":"manaflow-ai/cmux","number":13580,"title":"Bound warm-slot telemetry without per-event fsync","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13580","mergedAt":"2026-09-22T04:32:36Z","updatedAt":"2026-09-22T04:32:36Z","labels":[],"metricHints":["- Bound retention to `events.jsonl` plus one 16 MiB archive at `events.jsonl.1`.","- Repair one crash-torn JSONL tail before the next append; archive-only restart states recreate the current journal.","There is no telemetry writer thread or shutdown drain. Each call holds `events.lock` through tail repair/rotation/append and closes its descriptor before returning, so shutdown and rotation have no queue race.","| metric | before | after | reduction |","| p50 event append | 0.0698 ms | 0.0592 ms | 15.2% |","| p95 event append | 0.1438 ms | 0.1002 ms | 30.3% |","| mean event append | 0.0847 ms | 0.0672 ms | 20.7% |","| median 500-event batch | 43.74 ms | 33.42 ms | 23.6% |","| event-journal fsyncs/event | 1 | 0 | 100% |","| event-journal appends/event | 1 | 1 | 0% |","The overlay filesystem makes `fsync` unusually cheap, so the latency delta understates the persistent-storage savings. The deterministic write reduction is one durability barrier removed from every telemetry event.","Fixes #13533. Bounds warm-slot event telemetry to one current journal plus one 16 MiB archive and removes the durability barrier from every event, cutting benchmark append latency by roughly 15–30%.","- Rotates with `os.replace()` under `events.lock`, repairs a crash-torn JSONL tail before the next append, and recreates the current journal when only the archive survives a restart. There is no writer thread or shutdown drain."]},{"id":"manaflow-ai/cmux#13398","repository":"manaflow-ai/cmux","number":13398,"title":"dev: seed agent builds from warm main generations","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13398","mergedAt":"2026-09-21T12:51:10Z","updatedAt":"2026-09-22T01:21:18Z","labels":[],"metricHints":["The 24-hour physical worker trial intentionally follows this repo-side change. Historical #13091 measurements remain context; the new runner exists so expansion decisions use current worker-class measurements.","- `scripts/dev-fleet-warm-slot.py` implements reservations that survive the task-creation-to-build handoff: one warmer per Mac, task-first preemption, complete-tree input classification, toolchain invalidation, bounded concurrent reservations, and stale-lease recovery that also reaps descendants after a forced kill.","- Worker identities and filesystem paths are validated before use, and path traversal refusals are covered by tests.","- `scripts/benchmark-dev-fleet-warm-slots.py` measures hit rates, foreground latency from task arrival, cache growth, and fallback, quarantine, and recovery counts across cold, warm, stale, changed-input, toolchain, and preemption scenarios.","- `docs/dev-fleet-warm-slots.md` documents the generation receipt, worker commands, safety contract, and 24-hour trial policy."]},{"id":"manaflow-ai/cmux#13530","repository":"manaflow-ai/cmux","number":13530,"title":"perf: keep recursive warm-slot disk scans off foreground path","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13530","mergedAt":"2026-09-22T01:00:24Z","updatedAt":"2026-09-22T01:00:24Z","labels":[],"metricHints":["Normal warm-slot `warm` / `task-run` execution recursively walked the entire cache tree before and after native work solely to populate disk-growth telemetry. As DerivedData grows, that observation cost can become foreground latency.","- leaves ordinary warm/task execution measurement-free by default;","Moves recursive cache-size measurement off the foreground `warm`/`task-run` path so large DerivedData trees can't make routine telemetry a latency tax.","`warm` and `task-run` no longer walk the cache tree before/after native work by default. A new `--measure-disk` flag opts back into byte-growth measurement, and the physical benchmark passes it so cache-growth evidence stays available. Receipts omit disk fields unless measured; regression tests cover both default-off and opt-in behavior. Related: #13091."]},{"id":"manaflow-ai/cmux#13541","repository":"manaflow-ai/cmux","number":13541,"title":"fix(iOS): preserve tagged App Group when signing supports it","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13541","mergedAt":"2026-09-22T02:31:34Z","updatedAt":"2026-09-22T02:32:31Z","labels":[],"metricHints":["- only after Xcode reports the specific `com.apple.security.application-groups` provisioning-profile mismatch, and only for Debug + ASC API-key + provisioning updates, retry without the App Group","- derive the retry entitlements from `Config/cmux.entitlements` and `Config/NotificationService.entitlements` at runtime, removing exactly `group.dev.cmux.ios`","- `dogfood readiness writes a secret-free identity and latency receipt`"]},{"id":"manaflow-ai/cmux#13272","repository":"manaflow-ai/cmux","number":13272,"title":"ci: report cache restore route, state and duration","url":"https://redirect.github.com/manaflow-ai/cmux/pull/13272","mergedAt":"2026-09-20T19:59:56Z","updatedAt":"2026-09-20T19:59:56Z","labels":[],"metricHints":["The cache-restore composite exposes only an exact-hit flag, which cannot explain whether a restore used a prefix, failed, or selected a different action route on another runner. Add one structured `CMUX_CACHE_RESTORE` receipt and job-summary entry per restore, including requested backend, selected action route, runner, key/matched key, outcome and elapsed...","Validation: five focused Python tests and the existing read-only cache guard pass, including 18 provider/outcome cases and execution of the actual composite clock/report shell with output-boundary fixtures. The composite-wiring test fails against the previous action and passes after the change. It checks default/output preservation, arbitrary key quoting,...","No latency or dollar saving is claimed: the improvement is trustworthy cache-state evidence for future cohorts. Duration covers lookup, transfer and extraction, and hard runner termination may prevent a final receipt."]}],"links":{"curated":"/api/work-impact","publicPage":"/work/impact","compactIndex":"/data/impact-pr-candidates-v1.json"}}