Changelog

[Unreleased]

[0.3.10] - 2026-09-18

Post-0.3.9 computer-use reliability pass: focus/keylogger chain bugs found in live OpenCode testing, plus the ML outcome ledger (desktop-as-teacher). Includes a full local computer-use data reset for clean testing (datalog, training store, memory, transformer weights; ONNX models kept).

Fixed

• focus_window no longer reverse-maximizes windows — FocusWindow called ShowWindow(SW_RESTORE) unconditionally, which un-maximizes a maximized window every time it is focused (OpenCode observed browsers/editors snap out of maximize). Restore is now applied only when IsIconic (minimized); otherwise SW_SHOW keeps the current maximized/normal placement.
• Keylogger/replicate chain replay missing tool — keylogger_stop emitted "tool":"_focus" and eventsToSmartSteps emitted FocusWindow steps with an empty Tool. Chain dispatch then failed with unknown tool: _focus / unknown tool: (confirmed in chain_log). Fixes:
  • keylogger emits focus_window_by_title
  • smart steps carry Tool: focus_window_by_title + focus_window field
  • chain treats focus-only steps (empty Tool + focus field) as successful focus after auto-focus
  • toolDispatch gains focus_window_by_title, _focus (legacy alias), keylogger_start/stop/status, replicate
• Local tests no longer require live APPDATA assumptions for ledger schema — ml_predictions is created on demand; supervised samples never include unknown/pending.

Added

• ml_predictions outcome ledger (desktop as teacher) — predictions from ml_query / agent_suggest (transformer or statistical) are inserted as immutable rows (engine, query/ocr/window hashes, pred_tool, pred_x/y, confidence, model_version). Only resolution columns update: status, outcome_source, actual_*, verification_data, resolved_at.
• Outcome resolver on click — ResolveMLPredictionsForAction scores pending preds after a real click:
  • hit — click succeeded within hit radius (~3% screen, min 80px) of a pending coord
  • miss — click failed near a pending pred (or same-tool retry nearby)
  • unknown — pending rows older than 10m (timeout); never treated as failure
  • recovered — a miss followed within ~45s by a successful nearby same-tool action
  • Successful clicks far from predictions leave rows pending (agent may have ignored ML) — no poison labels
• ml_outcome_status tool + ml_status.outcome — hit/miss/unknown/recovered counts, hit_rate, recovery_rate, unknown_rate, last_hour_hit_rate, eligible_train.
• Resolved-only supervised signal — SupervisedLedgerSamples returns only hit|miss|recovered; transformer train pool prefers training_pairs.success=1; unknown/pending never train.
• Chain aliases for keylogger/replicate replay — focus_window_by_title, _focus, keylogger_start/stop/status, replicate (executes provided steps).

Changed

• Clean-slate computer-use data for testing — local %APPDATA%\go-mcp-computer-use training/datalog/memory/ML weights archived then cleared so 0.3.10 testing starts from zero pairs (ONNX models in models\ retained). Agents must re-collect via normal use; agent_train / scripts/eval-ml.ps1 -Train rebuild from new data only.

Build / verification

• scripts/build.ps1 — OK (version 0.3.10, Zig cc + CGO)
• scripts/lint.ps1 — go vet clean + build OK
• go test -short ./internal/actions — pass (includes focus/keylogger chain alias tests)

[0.3.9] - 2026-09-18

Fixed

• Transformer training never saw real click coordinates — production training_pairs.command_json stores args as a nested JSON string with mixed-case keys ({"args":"{\"X\":700,\"Y\":400}","tool":"click"}). ml/trainer.decodeCoords only unmarshaled top-level lowercase x/y, so on live datalogs every click/hover coord target was (0,0) (measured: 800/800 samples). Fix: ml/dataloader.NormalizeCommandJSON / ApplyNormalization unwrap string-or-object args, accept X/Y/from_x/to_x case-insensitively, and fill Sample.CoordX/Y + FromCoordX/Y. Trainer decodeCoords uses the same unwrap path; makeTargetFromSample prefers loader-normalized pixels. Unit tests cover the production nested-string shape.
• Transformer unusable after restart (empty tokenizer) — MLEngine.LoadModel created a fresh unfitted tokenizer and never loaded vocab. tokenizer.Encode returns nil when unfitted, so Forward failed with token len 0 != maxLen 128 while IsReady() could still be true; predictions silently fell through to the statistical engine. Fix: training persists vocab.bin next to model.gob (tokenizer Save/Load now wired in production), plus ml_meta.json (vocab size, ArgDim/WindowDim/FromCoordDim, tool list, holdout metrics). LoadModel refuses ready-state without vocab and surfaces a clear last_error.
• Vocab size vs embedding table mismatch — fitted vocabs on real OCR exceeded the hardcoded VocabSize=2000 (observed 2421), so high token IDs were dropped in embedding lookup. modelConfigFor() sizes the model from the fitted tokenizer (rounded up) and is shared by Train / LoadModel / online / finetune paths.
• Spatial features were always zero — prepareBatch called encoder.Encode(0, 0) for every sample, and inference used an all-zero coord vector. Training now encodes the sample’s decoded click/drag pixels; inference sets a context point from the live cursor via predict.Engine.SetContextPoint.
• Online training destroyed the tokenizer — trainFromBuffer refit the vocab on a 32-sample replay batch then ran TrainEpoch over the full SQLite DB, scrambling token IDs against existing embeddings. Online updates now train on replay-derived samples without Fit, keep embedding IDs stable, and re-save vocab.bin + report real eval loss/accuracy (checkpoint accuracy is no longer hardcoded 0.0).
• Honest tool-accuracy metric — trainer.Accuracy previously took argmax over toolStart (tools + coord + arg dims), mixing continuous outputs into classification and inflating scores. It now scores argmax over tool logits only, skips unknown tools, and Train logs click_acc / non_click_acc alongside overall holdout accuracy and the majority-class baseline.
• agent_train only rebuilt statistical indexes — it now also trains the Go transformer (writes model.gob + vocab.bin + ml_meta.json) and returns ml_status in the tool payload.
• scripts/lint.ps1 failed when launched from scripts/ — Get-Content VERSION resolved relative to the script directory. Now uses $PSScriptRoot\..\VERSION and builds ..\cmd\mcp-server.

Added

• ml_status on agent ML tools — agent_train, agent_suggest, and chain_predict return transformer health: ready, model_loaded, vocab_loaded, vocab_size, paths, last_eval_loss, last_eval_accuracy, majority_baseline, last_error, and a source verdict (transformer

  statistical_preferred

  unavailable).

• source field on PredictedAction — predictions are labeled transformer or statistical so agents know which engine produced a coordinate/command guess.
• Honest transformer gate — MLEngine.Predict returns nil when outputs are near-uniform (no signal) or when last holdout tool accuracy is below the majority-class baseline, so PredictActions falls back to the statistical adaptive engine instead of shipping junk coords.
• cmd/ml-eval + scripts/eval-ml.ps1 — local eval harness: tool distribution, coord-decode sanity (with_coords %), majority baseline, optional -Train, ml_status dump, and smoke predict on real datalog OCR. Exit code 2 when model accuracy is below majority baseline.
• Class balancing for click-heavy logs — balanceTools upsamples minority tools (cap 48) before augmentation so a 70%+ click datalog does not collapse the model to “always click”. Tool one-hot targets are amplified (2.0) so MSE is not drowned by zero coord/arg dims.
• Production-shaped ML tests — ml/dataloader/normalize_test.go and ml/trainer/decode_coords_test.go lock the nested-args schema that unit tests previously missed (tests used {"x":...} while logs used {"args":"{\"X\":...}"}).

Changed

• README ML section is truthful — the Go-native transformer is described as experimental with vocab/meta persistence, majority-baseline reporting, and an explicit preference for the statistical engine (ml_query/ml_teach) when the neural path underperforms. OCR/UIA/statistical priors remain the reliable locators.
• agent_train tool description — documents dual training (adaptive indexes + transformer artifacts) and ml_status in the response.
• docs/reference/tools.md regenerated after handler/description updates.
• Live retrain on a real datalog (800 pairs) — after the data-path fix, with_coords went from 0% → 70.4% for transformer targets; model.gob + vocab.bin + ml_meta.json load correctly across process restarts. Holdout tool accuracy on noisy OCR remains at or near the majority baseline — the neural path is gated and labeled rather than silently preferred. Statistical ml_query/agent_suggest now actually learn token→coord averages from unwrapped production args.

Build / verification

• scripts/build.ps1 — OK (mcp-server.exe, Zig cc + CGO, version 0.3.9)
• scripts/lint.ps1 — go vet clean + build OK
• go test — internal/actions short suite + ml/dataloader, ml/trainer, ml/predict, ml/tokenizer, ml/transformer pass

[0.3.8] - 2026-08-27

Fixed

• Transformer training never saw real click coordinates — production training_pairs.command_json stores args as a nested JSON string with mixed-case keys ({"args":"{\"X\":700,\"Y\":400}","tool":"click"}). ml/trainer.decodeCoords only unmarshaled top-level lowercase x/y, so on live datalogs every click/hover coord target was (0,0) (measured: 800/800 samples). Fix: ml/dataloader.NormalizeCommandJSON / ApplyNormalization unwrap string-or-object args, accept X/Y/from_x/to_x case-insensitively, and fill Sample.CoordX/Y + FromCoordX/Y. Trainer decodeCoords uses the same unwrap path; makeTargetFromSample prefers loader-normalized pixels. Unit tests cover the production nested-string shape.
• Transformer unusable after restart (empty tokenizer) — MLEngine.LoadModel created a fresh unfitted tokenizer and never loaded vocab. tokenizer.Encode returns nil when unfitted, so Forward failed with token len 0 != maxLen 128 while IsReady() could still be true; predictions silently fell through to the statistical engine. Fix: training persists vocab.bin next to model.gob (tokenizer Save/Load now wired in production), plus ml_meta.json (vocab size, ArgDim/WindowDim/FromCoordDim, tool list, holdout metrics). LoadModel refuses ready-state without vocab and surfaces a clear last_error.
• Vocab size vs embedding table mismatch — fitted vocabs on real OCR exceeded the hardcoded VocabSize=2000 (observed 2421), so high token IDs were dropped in embedding lookup. modelConfigFor() sizes the model from the fitted tokenizer (rounded up) and is shared by Train / LoadModel / online / finetune paths.
• Spatial features were always zero — prepareBatch called encoder.Encode(0, 0) for every sample, and inference used an all-zero coord vector. Training now encodes the sample’s decoded click/drag pixels; inference sets a context point from the live cursor via predict.Engine.SetContextPoint.
• Online training destroyed the tokenizer — trainFromBuffer refit the vocab on a 32-sample replay batch then ran TrainEpoch over the full SQLite DB, scrambling token IDs against existing embeddings. Online updates now train on replay-derived samples without Fit, keep embedding IDs stable, and re-save vocab.bin + report real eval loss/accuracy (checkpoint accuracy is no longer hardcoded 0.0).
• Honest tool-accuracy metric — trainer.Accuracy previously took argmax over toolStart (tools + coord + arg dims), mixing continuous outputs into classification and inflating scores. It now scores argmax over tool logits only, skips unknown tools, and Train logs click_acc / non_click_acc alongside overall holdout accuracy and the majority-class baseline.
• agent_train only rebuilt statistical indexes — it now also trains the Go transformer (writes model.gob + vocab.bin + ml_meta.json) and returns ml_status in the tool payload.
• scripts/lint.ps1 failed when launched from scripts/ — Get-Content VERSION resolved relative to the script directory. Now uses $PSScriptRoot\..\VERSION and builds ..\cmd\mcp-server.

Added

• ml_status on agent ML tools — agent_train, agent_suggest, and chain_predict return transformer health: ready, model_loaded, vocab_loaded, vocab_size, paths, last_eval_loss, last_eval_accuracy, majority_baseline, last_error, and a source verdict (transformer

  statistical_preferred

  unavailable).

• source field on PredictedAction — predictions are labeled transformer or statistical so agents know which engine produced a coordinate/command guess.
• Honest transformer gate — MLEngine.Predict returns nil when outputs are near-uniform (no signal) or when last holdout tool accuracy is below the majority-class baseline, so PredictActions falls back to the statistical adaptive engine instead of shipping junk coords.
• cmd/ml-eval + scripts/eval-ml.ps1 — local eval harness: tool distribution, coord-decode sanity (with_coords %), majority baseline, optional -Train, ml_status dump, and smoke predict on real datalog OCR. Exit code 2 when model accuracy is below majority baseline.
• Class balancing for click-heavy logs — balanceTools upsamples minority tools (cap 48) before augmentation so a 70%+ click datalog does not collapse the model to “always click”. Tool one-hot targets are amplified (2.0) so MSE is not drowned by zero coord/arg dims.
• Production-shaped ML tests — ml/dataloader/normalize_test.go and ml/trainer/decode_coords_test.go lock the nested-args schema that unit tests previously missed (tests used {"x":...} while logs used {"args":"{\"X\":...}"}).

Changed

• README ML section is truthful — the Go-native transformer is described as experimental with vocab/meta persistence, majority-baseline reporting, and an explicit preference for the statistical engine (ml_query/ml_teach) when the neural path underperforms. OCR/UIA/statistical priors remain the reliable locators.
• agent_train tool description — documents dual training (adaptive indexes + transformer artifacts) and ml_status in the response.
• docs/reference/tools.md regenerated after handler/description updates.
• Live retrain on a real datalog (800 pairs) — after the data-path fix, with_coords went from 0% → 70.4% for transformer targets; model.gob + vocab.bin + ml_meta.json load correctly across process restarts. Holdout tool accuracy on noisy OCR remains at or near the majority baseline — the neural path is gated and labeled rather than silently preferred. Statistical ml_query/agent_suggest now actually learn token→coord averages from unwrapped production args.

Build / verification

• scripts/build.ps1 — OK (mcp-server.exe, Zig cc + CGO, version 0.3.9)
• scripts/lint.ps1 — go vet clean + build OK
• go test — internal/actions short suite + ml/dataloader, ml/trainer, ml/predict, ml/tokenizer, ml/transformer pass

[0.3.8] - 2026-08-27

Fixed

• YOLO UI detection now letterboxes non-square inputs instead of aspect-distorting them - preprocessYOLO previously rescaled X and Y independently to a 640x640 square, so a non-square input (e.g. the 3200x1980 virtual desktop) was squashed by 5x on X and 3.09x on Y, destroying every element’s geometry. The detector then emitted floor-confidence (~0.5) sigma-degenerate 0x0 boxes everywhere. Fix: aspect-preserving letterbox (one uniform scale min(640/w,640/h), centered gray padding) shared between the encoder and decoder via a yoloLetterbox struct, and parseYOLOOutput now inverse-maps boxes with (out - pad) / scale. Verified live: full-screen detection now yields real, varied boxes and click points with confidence 0.5-1.0 instead of thousands of 0x0 floor boxes.
• elements_query / onnx_detect no longer return garbage off-screen boxes from the letterbox padding - ONNXDetect inverse-mapped detections whose centers sat inside the 640x640 letterbox’s gray padding band back into negative or oversized source coordinates (observed: {948,842,839,236} -> click y:960 on an 860-tall window). Fix: a new clipElementToImage gate filters every detection against the captured bitmap bounds - boxes whose center lands outside the image are dropped (padding false positives), and partially off-screen boxes are clipped to the image so screen_box/click_point are always actionable. Verified by unit tests covering all reported garbage shapes.
• ocr_window / screenshot_element now report + key the captured window, not the foreground window - annotateCaptureOpts stamped window_title from getActiveWindowTitle() (the FOREGROUND window) regardless of the handle being captured, so handle-scoped captures mislabeled the wrong window and keyed per-window ML-priors from that wrong title. Fix: annotateCaptureOpts takes a windowTitle param and AnnotateWindow resolves getWindowTitle(handle) for the actual captured handle. Verified live: ocr_window(132082) (Mozilla Firefox) now reports window_title="Mozilla Firefox" (was “OpenCode”).
• Capture responses no longer dump a giant element array on every ocr/screenshot/onnx_detect call - the fused YOLO+MobileNet annotation (from 0.3.5/0.3.6) embedded every detected box into every capture response with no cap, so a busy desktop returned thousands of degenerate icon boxes (~1751 observed) and blew the payload to several hundred KB even with include_image:false. The AI was forced to grep a monolithic JSON blob. Fix: annotateCaptureOpts now runs FilterAndCapElements before returning - it drops boxes whose fused combined_confidence is below a floor (default 0.40) and caps the surviving list (default 50, highest-confidence first). Detection/ML still runs at full fidelity internally; only what’s embedded on the wire is bounded. Verified by unit tests; a 1751-element dump is now at most 50 high-trust rows.

Added

• include_image opt-out for the base64 image bloat in capture-tool responses - every ocr/screenshot/screenshot_element/ocr_window/ocr_active_window/onnx_detect/onnx_classify response embeds the base64 screenshot (~0.7-1.5MB), which wastes model context and forces truncation for text-only AIs. This adds a config flag include_image (default true, preserving current behavior) that globally strips the returned image_b64 copy while keeping OCR text, element boxes/confidence, and all metadata intact - the internal ML detection pipeline still runs on the frame; only the returned copy is removed. A per-call include_image override (true/false) on any capture tool wins over the global config, giving callers a tri-state. Exposed through set_config/get_config and persisted. Verified live: with include_image:false an ocr_window returns metadata only (no image_b64); a per-call include_image:true restores it; onnx_detect(include_image:false) still returns 1677 elements / 350 clickable with the image stripped.
• exclude_elements opt-out for text-only captures - a config flag exclude_elements (default false, preserving current behavior) plus a per-call exclude_elements override on ocr, ocr_window, ocr_active_window, screenshot, screenshot_element, and onnx_detect empties the returned element array so a text-only client never drags the detection boxes onto the wire. Like include_image, it flows through the single safeHandler choke point (stripImageIfExcluded), is persisted via set_config/get_config, and the per-call override wins over the global value.
• elements_query tool - a grep-style query over the fused capture for text-based LLMs - instead of dumping the full annotated JSON, the AI asks “where is the submit button” and gets a compact flat projection: filter by source (screen/window/region), YOLO class, MobileNet label, an x,y,w,h region, min_confidence, clickable_only, and OCR text (returns matching words with screen coords). Each returned row is just {index, class, label, confidence, clickable, screen_box, image_box, click_point} - the coordinates to click - with no base64 and no nested classifier bloat. Registration mirrors the existing onnx_* tool family.
• scripts/gen-icons.ps1 no longer writes a UTF-8 BOM into winres/winres.json - Set-Content -Encoding UTF8 (PowerShell 7) prepends a BOM, which go-winres rejects with invalid character 'ï' looking for beginning of value, breaking the resource-generation step and hence the whole build. Fix: write with -Encoding utf8NoBOM so the generated JSON parses cleanly.
• key_press key names expanded to the full Windows virtual-key surface + case-insensitive lookup + generic modifier combos - vkModMap now exposes the explicit left/right modifier variants (LCTRL/LCONTROL, RCTRL/RCONTROL, LALT, RALT, LSHIFT/RSHIFT, WIN/META/LWIN/SUPER/CMD/COMMAND, RWIN), and vkSpecialMap gained the numpad block (NUMPAD0-NUMPAD9, NUMPAD_MULTIPLY/ADD/SEPARATOR/SUBTRACT/DECIMAL/DIVIDE), the media/volume keys (VOLUME_MUTE/VOLUME_DOWN/VOLUME_UP, MEDIA_NEXT_TRACK/PREV_TRACK/STOP/PLAY_PAUSE), plus SNAPSHOT, APPS/CONTEXT, CLEAR, EXECUTE, and SLEEP. keyNameToVK now upper-cases its input so lookups are case-insensitive, and single-char punctuation resolves through the charToVK table. KeyPress generalized the old CTRL-only MOD+key prefix into any modmap modifier followed by a single letter or digit (CTRL+A, WIN+R, ALT+F4, SHIFT+1, …) and emits the requested modifier’s real VK instead of hardcoding Ctrl. Covered by new internal/actions/keyboard_test.go unit tests.

[0.3.7] - 2026-08-27

Added

• --version, --license, and --help CLI flags on the server binary - mcp-server.exe --version prints the build version (go-mcp-computer-use 0.3.7), --license prints the full Apache-2.0 text plus the NOTICE, and --help lists the available subcommands/flags. LICENSE and NOTICE are embedded into the binary via go:embed (kept in sync by scripts/build.ps1), so the license text is always retrievable from the shipped executable with no files alongside.

Changed

• Project relicensed from MIT to Apache-2.0 - LICENSE replaced with the full Apache-2.0 text, (c) 2026 coff33ninja. Added a NOTICE file wiring the Apache-2.0 attribution and clarifying that the bundled gpa_gui_detector.onnx (a converted Salesforce GPA-GUI-Detector) remains under the upstream MIT license, not Apache-2.0. README gained a license badge, a License section, and updated model-attribution wording.
• Version-info, copyright, and admin manifest now embedded in the executable - resource generation switched from akavel/rsrc (icon only, no manifest/version) to go-winres (scripts/gen-icons.ps1 + a committed winres/winres.json.template). The exe now carries a VS_VERSION_INFO block visible in Windows Explorer Properties -> Details (CompanyName coff33ninja, LegalCopyright “Copyright (c) 2026 coff33ninja. Licensed under the Apache License, Version 2.0.”, ProductName, File/Product version from VERSION) and an application manifest declaring requireAdministrator (admin is required for the tool’s UIA/UIPI automation to fully work) plus per-monitor-v2 DPI awareness. Previously the exe had no embedded manifest at all (as-invoker, no DPI declaration), so this is an explicit behavioral improvement that enforces the project’s real admin requirement at the OS level.
• License decisions: LICENSE + NOTICE are shipped with each release - .github/workflows/release.yml now attaches LICENSE and NOTICE to every versioned release alongside mcp-server.exe, so the license and third-party model attribution always accompany the binary.

[0.3.6] - 2026-08-27

Added

• onnx_classify tool — MobileNetV3 GUI element classifier — a new advisory tier that classifies UI content using mobilenetv3_small.onnx across 15 classes (button, checkbox, container, dropdown, icon_button, image, label, link, menu_item, scrollbar, slider, tab, text_input, toggle, unknown). Supported source values: screen (full screen), window (active window), region (x,y,w,h), elements (classify every YOLO-detected element crop), crop (raw base64 image). Returns label + confidence for top-N (default 3) per target. Advisory only — it returns an additional signal and never hard-blocks an action.
• MobileNet as a chain verification tier — chain verify steps accept an optional classify config ({enabled, top_n, source}). When enabled, the executed step’s result carries a classify advisory payload with the top classifications, independent of the pass/fail decision.
• MobileNet as a watcher advisory tier — the background watcher now classifies each YOLO-detected element crop and attaches classifications to its cached detection, giving the AI type+confidence context per element without affecting detection or training. Best-effort and skipped entirely when the model file is absent.
• AnnotatedCapture / AnnotateScreen / AnnotateRegion / AnnotateWindow — the fused, best-effort annotated-capture production layer. All classifier signals run on the SAME captured frame so geometry stays aligned. Any engine that is unavailable yields an empty sub-block (noted in errors) without failing the capture.
• scripts/gpa-gui-export/ — redoable GPA-GUI→ONNX conversion — a uv-based project (pyproject.toml + export.py + README.md) that downloads Salesforce/GPA-GUI-Detector model.pt via huggingface_hub, validates the single icon class layout, exports to ONNX (1,3,640,640)→(1,5,8400) opset 12, and writes gpa_gui_detector.onnx. uv sync + uv run export.py regenerates the artifact from source; it is what CI uses to produce the release asset.
• Main README “Models” section — documents the three runtime models, auto-download behavior, and how to regenerate the detector, plus a “UI-aware element detection” feature bullet.

Changed

• Every click now returns an advisory validated block — click and find_text_and_click results carry a post-click validation combined into MobileNet classification of the click target (top-N label + confidence), the element-priors DB (sample count, prior-adjusted confidence, known-location flag, learned frequency/position), and an ML-memory cross-reference (whether the model has seen this click context before). Hooked at the central Click() choke point so every click source gets it. The validated block is merged at the top level of the tool result so the AI sees a consistent shape whether or not OCR auto-verify ran. Best-effort and non-blocking: if the model or capture fails, the click still succeeds and the validation block simply carries minimal/no data.
• Internal refactor — ClassifyElements now delegates to a shared classifyElementList helper (also used by the watcher), keeping crop capture + model inference in one place.
• Annotated capture on all AI-facing capture tools — screenshot, screenshot_element, ocr, ocr_window, and ocr_active_window now always-on return a fused AnnotatedCapture (OCR text + YOLO element boxes + MobileNet per-element classification + element-priors + ML-memory) with each element exposed in BOTH bitmap-image space and virtual-screen space (screen_box), so text-bound AIs know exactly what is on screen and where to click regardless of whether a vision model is available. Screenshot tools keep the raw b64 as text content for vision AIs while the annotated map rides alongside as structured result. screenshot/screenshot_element gained an optional language param passed to the OCR signal.
• onnx_detect and onnx_classify now return the same fused annotated capture — instead of separate raw detection/classification payloads, onnx_detect and onnx_classify (screen/window/region) now produce the identical AnnotatedCapture shape as every other capture tool, so the classifier signals are aligned on the same frame with dual image_box/screen_box coords, click_point, combined_confidence, and the clickable gate — no redundant double-inference. onnx_detect honors its threshold/iou_threshold args; onnx_classify elements/crop sources return the classification wrapped in the same shape via a shim (no live capture-derived screen boxes in those cases).
• The clickable gate and confidence fusion now trust the MobileNet UI label as the authoritative signal — the YOLO proposal tier emits general COCO object classes (person, vase, …) that are essentially never interactive, so gating on the YOLO class alone always returned clickable=false even when MobileNet had correctly identified a real control (button, text_input, link, …). ElementIsClickable now accepts an element when either the MobileNet top-1 classified label OR the YOLO class names an interactive type, and CombineElementConfidence weights an interactive MobileNet label more heavily regardless of YOLO agreement. This makes the fused capture actually recommend clickable controls in the live pipeline.
• Coordinate safety + anti-over-click assurance — each annotated element carries a screen_box in virtual-screen physical pixels (the exact space click consumes, no DPI double-scaling), a click_point centroid to press, a fused combined_confidence, and a clickable gate (interactive-class whitelist + min confidence) so the AI avoids false-positive clicks on low-trust or non-interactive boxes. The capture also exposes width/height, dpi_scale, and the full virtual_screen bounds so the AI can reason about screen size and multi-monitor layout. The ml_teach/priors feedback loop weights previously-seen UI higher, so unfamiliar or moved elements are flagged rather than silently clicked.
• GUI element detector swapped from generic COCO YOLO to Salesforce GPA-GUI-Detector (single-class icon) — the previous yolo11n.onnx is a COCO-80 general object detector whose proposals (person, car, bicycle, …) flooded the watcher/priors loop with meaningless, never-interactive regions. The detector is now a UI-native, single-class (icon) ONNX export of Salesforce’s MIT-licensed GPA-GUI-Detector (fine-tuned from OmniParser), which proposes actual interactive-element boxes. The authoritative per-control UI type remains the 15-class MobileNet tier.
• Watcher/priors novelty gate now keys on the MobileNet UI label instead of the detector class — because the detector is now single-class (icon), the watcher dedup and element-priors DB would otherwise bucket every control under one key. The watcher now runs MobileNet classification BEFORE persisting crops and threads each element’s top UI label (button, text_input, …) into ElementKnownConfidently/saveElementRegionSamples, so the priors learn per-control-type locations. DetectedElement gained a mobile_net_label field carrying the authoritative class.
• Detector model auto-downloads when missing — ONNXDetect now fetches gpa_gui_detector.onnx from https://github.com/coff33ninja/go-mcp-computer-use/releases/latest/download/gpa_gui_detector.onnx on first use if absent (best-effort, once per process, serialized against concurrent watcher/tool paths). GitHub redirects that URL to the newest non-draft release’s asset, so it follows every version bump with no hardcoded tag.
• Detector model ships with releases — .github/workflows/release.yml rebuilds gpa_gui_detector.onnx in CI (via the new export project) and attaches it to each versioned release alongside mcp-server.exe.

[0.3.5] - 2026-08-27

Changed

• Action-triggered training snapshots are now cropped around the target — SaveSnapshotAfterAction (used by click, type, drag, hover, find_text_and_click, type_and_submit, select_all_and_type, and click_menu_item) now captures a 400×400 region centered on the action target instead of saving a full-screen screenshot. This gives the ML model a focused view of the element being acted on, dramatically reducing the size of training data. Older call sites fall back to full-screen capture automatically.
• Watcher training samples are now cropped around detected elements — The background watcher (which screenshots the screen every cycle to feed the priors/ML systems) now saves a small padded crop around each detected UI element instead of a full-screen 3200×1980 PNG every cycle. When no elements are detected in a frame, nothing is saved. This prevents the tens of gigabytes of near-identical full-screen images that previously accumulated.
• Watcher is AI-gated with an achievement lock — The background watcher now starts locked (watcher_locked: true by default). While locked it still runs ONNX detection and caches results for reference, but never persists training crops. The AI must first prove competence through real interactive actions (click/type/record) and then explicitly call watcher_unlock to grant the watcher permission to save crops. The unlocked state persists across reboots, so the watcher stays unlocked once earned.
• Watcher confidentiality / novelty dedup — The watcher now consults the priors system before saving a crop via a new ElementKnownConfidently check: an element is considered known when its (class, window) pair has enough samples AND its current normalized location falls within tolerance of the learned position. Familiar elements are skipped; only new/moved/uncertain elements are saved. Snapshots therefore decay as the ML gains confidence, replacing the old fragile count-only gate.
• Wallpaper guard — The watcher will never save training crops when the foreground window is the desktop/shell (e.g. “Program Manager”, empty title, Shell_*, Windows Shell Experience), regardless of the lock state. This prevents the wallpaper from ever being mined as a training signal.
• Training DB pruning actually reclaims disk — PruneOldSamples now runs VACUUM after deleting rows, and a new PruneOrphanedSamples pass (run alongside the periodic retention pruner) removes rows whose image file no longer exists on disk. This reclaims the space that stayed locked inside samples.db after the large full-screen PNG cleanup.

Added

• watcher_lock_status tool — reports whether the watcher’s training-crop capture is achievement-locked, plus the count of real-action training samples gathered (action_signal) as guidance on when unlocking is warranted.
• watcher_unlock tool — grants the watcher permission to persist training crops after the AI has gathered confident data from real interactive actions. The unlock persists across reboots.
• set_config ... watcher_locked / get_config watcher_locked — the watcher lock can be inspected and toggled through the standard config surface alongside the dedicated tools.

[0.3.4] - 2026-08-26

Fixed

• find_text_and_click multi-monitor coordinate bug — OCR returns bitmap-space coordinates (0,0 at top-left of captured image), but Click() expects virtual screen coordinates. On setups with displays above the primary (e.g. Display3 at y=-1080), this caused “y out of bounds” errors. Fixed by tracking the capture region origin (captureX, captureY) and offsetting OCR coordinates before clicking and storing in memory.
• Window-scoped OCR coordinate offset — OCRWindow returns bitmap-relative coords (0,0 at window top-left), but Click() needs virtual screen coords. Now captures the window rect and uses rect.Left, rect.Top as offset, matching the same fix applied to full-screen and region OCR paths.
• Text location memory coordinate consistency — StoreTextLocation now stores virtual screen coords (converted from OCR bitmap coords at write time). Memory retrieval uses stored coords directly without re-conversion, preventing double-offset bugs when mixing full-screen OCR and SystemFind sources.
• Stale text location memory pruning — On startup, entries with coordinates outside the virtual screen bounds are automatically deleted. Prevents old bitmap-space coordinates (from pre-0.3.4) from being reused. Also added bounds validation in the memory retrieval path — stale entries are skipped even if they haven’t been pruned yet.

Changed

• find_text_and_click scoped window OCR — When window_title is provided, now tries window-specific OCR first (via OCRWindow) before falling back to full-screen. This scopes the search to the target window, avoiding false matches from other windows on screen. Window OCR returns coordinates already in virtual screen space, so no offset conversion is needed.
• Watcher auto-starts on boot by default — watcher_auto_start now defaults to true. Previously the watcher had to be manually started with set_config after every server restart. The watcher feeds the priors system (element position statistics) which improves UI element lookup confidence. Set to false in config to disable.

Added

• Z-order layering for text location memory — TextLocation now stores z_order (window stack position: 0=topmost). FindTextAndClick captures the foreground window’s z-order at call time and uses it for memory matching via new FindTextLocationMatch / FindTextLocationAnyMatch functions. These prefer exact z-order matches (clicking the right layer) then fall back to any match. Handles overlapping windows with identical text.
• Schema migration: z_order INTEGER NOT NULL DEFAULT 0 column added to text_locations table. Existing databases auto-migrate via ALTER TABLE.
• get_config tool — read-only view of all current configuration values plus live watcher status. Returns every field that set_config can change, plus watcher_running and watcher_auto_start. Use to inspect state without modifying anything.

[0.3.3] - 2026-08-26

Added

• ml_query — ask the ML engine “where is X on this screen?” Pass a query (what you’re looking for) plus current OCR text. Returns coordinate predictions ranked by confidence, matched OCR keywords, and related commands the ML has seen. Searches both coordIndex (per-tool coordinate distributions) and wordToCmds (command frequency). Query tokens get priority matching, context tokens add breadth.
• ml_teach — feed confirmed correct answers back to the ML after every action. Pass what was being looked for, the screen OCR, which tool was used, coordinates, and success/fail. Updates coordIndex and wordToCmds directly with both query and context tokens. The learning loop: ml_query → AI acts → ml_teach reinforces. Each cycle strengthens token→coordinate associations.

Changed

• The ML feedback loop is now complete: query → predict → act → teach. Whether the AI follows an ML prediction or discovers the correct answer itself, ml_teach ensures the ML learns from every outcome — including when the user shows the AI the right answer.
• CI workflows updated from v0.2.x to v0.3.x (ci.yml, auto-tag.yml, jekyll-gh-pages.yml, mod-maintenance.yml).
• README status block rewritten — v0.2.x framed as testing/iteration ground, v0.3.x as current stable. Recording & replication and ML feedback loop documented in features section.
• docs/architecture.md — added record_replicate.go to code map, ML Loop layer in agent stack diagram, updated adaptive.go and chain.go descriptions.
• docs/ci-cd-pipeline.md — updated branch references and diagram to v0.3.x.
• _config.yml logo URL updated to v0.3.x branch.
• scripts/gen-tools-doc.go — added ml_query + ml_teach to Adaptive Agent category (now 5 tools).

Fixed

• scripts/gen-icons.ps1 — rewritten to handle encoding issues and missing dependencies gracefully. No longer fails with PowerShell parse errors when rsrc is not installed or icon path contains spaces. Build script (scripts/build.ps1) now works end-to-end again.

[0.3.2] - 2026-08-26

Added

• Timed recording no longer blocks MCP — record(duration_secs=N) now starts a background goroutine for the sleep+auto-stop, returning immediately with a confirmation. Previously time.Sleep() inside the handler caused the MCP SDK to kill the request as timed out. Both manual (duration_secs=0) and timed modes now work reliably.
• Recording feeds ML on stop — RecordStop() now calls LogEnrichPatternsFromSession() asynchronously, feeding OCR, UIA, and ML enrichment payloads back to the adaptive engine immediately when recording ends. No manual wiring required — the full loop is: record → stop → enrich → ML learns.

Verified

• 30-second timed recording test: started, returned immediately, auto-stopped after 10 seconds, keylogger inactive — no MCP timeout.
• Manual recording + record_stop: full session returned with enrichment. OCR snapshots jumped 1668→1817 (+149 from enrichment logging). ML engine received 692 command sequences, 535 click patterns, 26 double-clicks, 26 long-presses, 49 type patterns. Top sequences: "reminder"→click (260 samples), "delete"→click (154), "schedule"→click (152).
• AI replication of recorded sessions not yet tested — next validation step.

[0.3.0] - 2026-08-25

Added

• Recording plugin — record_and_replicate, record, record_stop, and replicate MCP tools with intelligent chain generation. Recording captures enriched context at each click: OCR text near the click point, UIA element identity (name, automation_id, control_type), ML-predicted coordinates, window title, mouse movements, drags, scroll, and key events.
• Typed text capture — keylogger reconstructs typed text from VK code sequences via vkToChar reverse map. Consecutive printable characters accumulate into type steps. Modifier combos (Ctrl+C, Ctrl+V) become key_press steps. Shift+key produces uppercase. Text flushes on focus change, mouse event, or non-char key.
• Window snapshot at recording start — RecordStop() calls ListWindows() + GetWindowState() for every visible window. WindowsAtStart []WindowStateInfo stored in RecordSession. Replication generates find_window → move_window / restore_window / maximize_window + focus_window using ${var} chain variables to avoid stale handles.
• Learning loop — execTool in chain.go calls LogToolCall (not LogCommand) so every chain step feeds OCR→command pairs back to the ML engine via LearnFromCommandWithContext. elapsed_ms tracked per step for AI pacing analysis.
• Smart chain generation — eventsToSmartSteps handles all event types: type, key_combo (with modifiers), click (all 5 buttons), double_click, long_press, drag, scroll, move_mouse, focus, key_down, key_up. Smart click priority: UIA invoke > OCR find_text_and_click > ML predicted coordinates > raw coordinates.
• Context menu detection — right-click generates: click(right) → wait(350ms) → uia_find(control_type=MenuItem) → find_text_and_click(OCRText) with skip_system_find: true. Menu position cache checked first — if cached item found, skips UIA/OCR and clicks cached item directly.
• All mouse buttons — keylogger captures all 5 buttons (left, right, middle, X1, X2). Middle-click → click(button=middle). X1 → key_press ["Alt","Left"] (browser back). X2 → key_press ["Alt","Right"] (browser forward). ElapsedMs tracked per click/drag for hold duration.
• Double-click detection — mergeDoubleClicks() post-process merges 2 rapid left-clicks within 400ms at same position (8px tolerance) into EnrichedEvent.Kind="double_click". Chain generation emits smart click + click with clicks: 2.
• Long-press detection — detectLongPress() post-process converts clicks with elapsedMs > 500 into EnrichedEvent.Kind="long_press". Chain generation emits smart click + wait(hold duration in ms).
• Menu position memory — global cache: menuCacheKey{Window, BucketX, BucketY} → []menuCacheEntry{ItemText, RelativeX, RelativeY, HitCount}. 50px grid bucketing for fuzzy position matching. Hit count increments on repeated observations. Chain tools: cache_menu_items, lookup_menu_items. Thread-safe via sync.RWMutex.
• ML integration for enrichment patterns — double_click, long_press, context_menu registered as distinct ML tools in coordIndex. LogEnrichPattern() captures OCR context at event coordinates and feeds LearnFromCommandWithContext + RecordResult. LogEnrichPatternsFromSession() batch-logs all enrichment events from a recording session. detectAndLogEnrichPatterns() pre-scans chain steps for enrichment patterns before execution. Wired into both RecordAndReplicate (after chain execution) and ExecuteChain (before chain execution for standalone replicate).
• uia_invoke chain step — chain engine can now execute UIA automation via the uia_invoke tool in step sequences.
• mouse_down / mouse_up chain steps — new chain tools for holding and releasing mouse buttons independently. Used by long-press replay (move→down→wait→up) and available for custom chains.
• record MCP tool — starts recording; if duration_secs > 0 auto-stops after that duration, otherwise starts manual mode (use record_stop to finish).
• replicate MCP tool — takes a recorded session JSON and replays it as a smart chain with configurable slowdown and loop.
• record_stop MCP tool — stops an active recording and returns the enriched session with OCR, UIA, and window context at each event.

Changed

• No arbitrary duration limit — AI/user decides when to stop recording. duration_secs=0 enables manual stop mode.
• All 5 recording gaps addressed: typed text, window snapshot, learning loop, smart chain generation, context menus.
• All 5 enhancements complete: double-click, long-press, context menu chain, menu position memory, ML enrichment integration.
• Double-click replay uses clicks: 2 parameter with UIA invoke path for reliable replay.
• Long-press replay uses mouse_down→wait→mouse_up sequence for actual button hold duration.
• Window snapshot captured at Record() start so replay restores the layout the user started with.
• chainUIAInvoke returns error when UIAInvoke returns invoked: false, enabling proper fallback/error reporting.
• Shift handling extended — vkToShiftedChar map covers digits (1→!, 2→@…), OEM keys (-→_, =→+, [→{…), and punctuation. Previously only converted lowercase→uppercase letters.
• LogToolCall executes synchronously to ensure OCR→command bridge state is consistent before the next chain step.
• enrichClick coordinates always come from click/drag args — no sentinel values.
• vkToShiftedChar reverse map built from charToVK shift entries in keylogger init.
• 178 tests all passing.

Fixed

• record tool null structuredContent — Record(0) in manual mode returned nil session, causing MCP SDK to reject the response. Now returns a confirmation message.

Fixed (pre-0.3.0)

• Nil-pointer in ensureWindowFocus — added state.Rect != nil guard and bounds checking before clicking the window title bar. GetWindowState could return a nil Rect on certain window types.
• CI/CD: tool count auto-patching — gen-tools-doc.go now patches the "tools", N value in server.go startup log automatically, keeping it in sync with registered tools.
• CI/CD: tool categorization — added all 153 tools to categoryForTool map with proper categories. Previously 27 tools were uncategorized.
• CI/CD: category ordering — added File Operations and Recording & Replication to categoryOrder for consistent doc generation.
• Docs: versioning and release checklist — added tool-addition checklist to docs/reference/versioning-strategy.md documenting all files that must be updated when adding tools.

Key Functions

Function	File:Line	Purpose
vkToChar	keylogger.go:60	VK→char reverse map
vkToShiftedChar	keylogger.go:65	VK→shifted char map (!, @, _, +, etc.)
isModifierVK	keylogger.go:93	Check if VK is modifier
resolveVK	record_replicate.go:298	Key name→VK code
enrichEvents	record_replicate.go:191	Raw steps→enriched events + post-processing
mergeDoubleClicks	record_replicate.go:420	Merge 2 rapid clicks → double_click
detectLongPress	record_replicate.go:444	Convert slow clicks → long_press
cacheMenuItems	record_replicate.go:33	Store menu items in position cache
lookupMenuItems	record_replicate.go:84	Retrieve cached menu items
activeModifiers	record_replicate.go:135	Collect held modifier names
eventsToSmartSteps	record_replicate.go:569	Events→chain steps (all types)
smartClickSteps	record_replicate.go:558	Smart click: UIA>OCR>ML>raw
Record	record_replicate.go:58	Start recording
RecordStop	record_replicate.go:76	Stop + enrich + window snapshot
Replicate	record_replicate.go:393	Generate chain with window restoration
LogToolCall	datalog.go:208	OCR→command bridge + ML learning
execTool	chain.go:530	Chain step execution (calls LogToolCall)
toolDispatch	chain.go:171	53+ chain tools
LogEnrichPattern	record_replicate.go:467	Capture enrichment event → ML + datalog
LogEnrichPatternsFromSession	record_replicate.go:507	Batch-log session enrichment events
detectAndLogEnrichPatterns	chain.go:253	Pre-scan chain steps → ML enrichment logging
MouseButtonDown	mouse.go:103	Send mouse button-down event (for long-press)
MouseButtonUp	mouse.go:125	Send mouse button-up event (for long-press)
chainMouseDown	chain.go:1131	Chain step: mouse_down handler
chainMouseUp	chain.go:1143	Chain step: mouse_up handler

[0.2.61] - 2026-08-25

Added

• record_and_replicate MCP tool — records mouse and keyboard events for a user-specified duration (1-60 seconds), then automatically replays them as a chain. Supports slowdown factor (1=real-time, 2=2x slower, etc.), loop count (repeat the same recording N times), and delay_ms (pause before replay). Eliminates the manual keylogger_start → wait → keylogger_stop → chain round-trip.

[0.2.60] - 2026-08-25

Fixed

• InitAbortFromConfig wired into config handlers — the convenience wrapper for ParseHotkeyString + StartAbortPoller was never called; set_config and startup both inlined the same logic. Now uses InitAbortFromConfig in all 4 call sites, eliminating duplication.
• makeSequenceTargets wired into training pipeline — the multi-step sequence target builder had zero callers; the trainer only built single-action targets. prepareBatch now fills the sequence section of the target vector when sequenceLen > 0, enabling the transformer to learn temporal action ordering.

Added

• Sequence-aware training in prepareBatch — looks ahead sequenceLen consecutive samples and fills target slots via makeSequenceTargets.
• 6 new tests: TestInitAbortFromConfig_DisabledNoop, TestInitAbortFromConfig_StartsPoller, TestMakeSequenceTargets_FillsSlots, TestMakeSequenceTargets_ZeroLenSkips, TestPrepareBatch_FillsSequenceTargets, TestPrepareBatch_NoSequenceWhenDisabled.

[0.2.59] - 2026-08-24

Added

• system_find_stats MCP tool — returns last-used timestamp and total call count for the system-find feature, enabling AI agents to observe system-find usage patterns.
• task_is_active MCP tool — returns whether a task session is currently active (between task_begin and task_end), allowing agents to check session state without calling task_end.
• Text location pruning in retention cycle — PruneTextLocations now runs alongside PruneOldSamples in the 6-hour retention pruner, cleaning stale entries from the text_locations SQLite table.

Fixed

• Retention pruner goroutine leak on shutdown — StopRetentionPruner() is now called during server shutdown, closing the background goroutine channel and preventing a goroutine leak on exit.

Tests

• Added 16 tests across 3 new test files: text_location_test.go (6), system_find_test.go (3), introspection_test.go (3), plus 4 new retention-pruner tests in training_test.go.

[0.2.58] - 2026-08-23

Fixed

• ML init no longer skipped when adaptive stats training fails — EnsureAdaptive() returned early from its startup goroutine when TrainFromDatalog() errored (e.g. transient SQLite lock when many server instances start concurrently), which meant mlEngine.LoadModel()/Train() never ran and chain_predict stayed dead for the entire server lifetime (sync.Once prevents retry). It now logs a warning and continues, with a 2s backoff before the Train() fallback.
• Augmentation actually reaches training — Train() called AugmentAll(samples, 2) but only used the result to fit the tokenizer vocabulary; TrainEpoch re-loaded raw samples from SQLite, so augmented pairs never participated in gradient updates. New Trainer.TrainSamples trains a caller-provided sample set directly.
• Checkpoint accuracy was hardcoded 0.0 — SaveCheckpoint always received accuracy 0, permanently disabling ModelVersion.CheckAndRollback (it requires best.Accuracy > 0). Checkpoints now record real holdout accuracy.

Added

• Holdout evaluation for transformer training — Trainer.Evaluate (mean MSE over a sample set, no weight updates) and Trainer.Accuracy (argmax tool-match against target tool). Train() holds out 10% of datalog pairs before augmentation so the eval set stays honest.

Changed

• Transformer trains multiple epochs on the augmented set — previously one pass over raw samples (loss moved 0.0434 → 0.0436, i.e. not at all). Now 5 epochs over ~3x augmented data with per-epoch eval. Live run on existing datalog (674 pairs): loss_eval 0.0433, accuracy 16.18% (previously unmeasured), saved as checkpoint v3.

[0.2.57] - 2026-08-10

Fixed

• Chain timeout now cancels pending steps — chainState.ctx is wired to the chain’s cancellation context, and execSteps checks ctx.Done() before dispatching each step. Previously a chain that exceeded its timeout returned failure to the caller while its goroutine kept executing later desktop actions in the background. Added TestExecuteChainTimeoutStopsLaterSteps proving steps after a timeout never run.
• Verified chain steps reuse captured OCR — execVerify now calls verifyOCRResults(before, after, cfg) with the OCR snapshots it already took instead of calling VerifyAction, which re-captured full-screen OCR twice more. This eliminates redundant full-screen OCR on every verified step (the cause of a 15s chain timeout during live testing).
• Drag uses virtual-desktop coordinates — Drag now normalizes absolute mouse input against the virtual screen bounds (via VirtualScreenBounds() using SM_XVIRTUALSCREEN/SM_YVIRTUALSCREEN/SM_CXVIRTUALSCREEN/SM_CYVIRTUALSCREEN) and sets MOUSEEVENTF_VIRTUALDESK. Previously it mapped against the primary screen origin (0,0), which broke coordinates on multi-monitor layouts with a negative origin (monitors above or left of the primary). Added TestNormalizeVirtualDesktopPointUsesVirtualOrigin.
• Virtual-desktop-aware bounds checks — coordinate/region validation (ValidateClickCoord, ValidateRegion), OCR window clamping, and SmartRegionAround now use the virtual screen bounds instead of assuming origin (0,0), so tools correctly handle negative-origin multi-monitor desktops.

Changed

• CaptureScreen now captures the full virtual desktop (all monitors) rather than the primary display only.
• VerifyAction refactored to accept an already-captured BeforeOCR; new verifyOCRResults helper shared with the chain verification path.

[0.2.56] - 2026-08-09

Changed

• Dependency security update — bumped google.golang.org/protobuf v1.28.0 → v1.36.11 and golang.org/x/net v0.46 → v0.56 in both the main and ml/ modules. Fixes the protojson infinite-loop denial-of-service advisory (CVSS 7.5) in both modules.
• Merged Dependabot dependency bumps:
  • github.com/modelcontextprotocol/go-sdk v1.6.1 → v1.7.0
  • golang.org/x/sys v0.46.0 → v0.47.0
  • modernc.org/sqlite v1.54.0 → v1.56.0
  • github.com/pdfcpu/pdfcpu v0.13.0 → v0.14.0 (main + ml/)
• Closed stale weekly dependency PRs (#8, #22, #44, #48) superseded by the individual bumps above.

[0.2.55] - 2026-07-22

Fixed

• Transformer broadcasting — biases (coordProjB, historyProjB, ff1B, ff2B) and layer norm params (ln1W, ln1B, ln2W, ln2B) changed from [MaxLen, d] matrices to [d] vectors, added via BroadcastAdd/BroadcastHadamardProd instead of same-shape Add/HadamardProd. Softmax normalization replaced ones-matrix outer-product hack with BroadcastHadamardDiv. Reason: the old [MaxLen, d] shapes meant every sequence position learned independent bias/norm parameters — not standard transformer behavior, inflates param count ~24%, breaks generalization across positions, and NAS param-count estimator disagreed with actual model size.

• Multi-monitor wiring — detectScreenConfig() now enumerates all monitors via EnumDisplayMonitors + GetDpiForMonitor and populates the spatial encoder’s Monitors slice. Previously the slice was always nil — MonitorInfo struct and monitorAt() existed but were never wired up, so multi-monitor DPI was silently ignored. Added 7 tests covering L-shape, vertical stack with negative Y, fallback outside all monitors, and negative-coordinate virtual desktops.

Added

• Sinusoidal positional encoding — fixed (non-learned) sin/cos positional encoding added to token embeddings before attention. PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d)). Reason: the transformer had zero positional encoding — it was permutation-equivariant over the sequence axis, meaning “click then type” was indistinguishable from “type then click”. Without position information, the model cannot learn temporal ordering of actions.

• Spatial encoder: per-monitor DPI — MonitorInfo struct with per-monitor rect + DPI scale, monitorAt() lookup, 4th positional pair monRelX/Y (position within the specific monitor). FeatureDim 12→14. Reason: the old encoder sampled DPI at a fixed point (0,0) and applied it globally — on dual-monitor setups with mismatched DPI scales, coordinates on the second monitor got wrong DPI baked in silently.

Changed

• NAS paramCount unified — deleted hand-rolled paramCount() in ml/nas/search.go, now calls transformer.ParamCount(cfg) directly. Reason: the old formula had a phantom MaxLen*d “positional encoding” term (no such params exist) and used 3*d*d for Q/K/V instead of the real model’s 4*d*d (Q/K/V/O), so NAS parameter-budget search was scoring against wrong model size.

[0.2.54] - 2026-07-22

Changed

• Multi-token attention — transformer now processes all MaxLen token positions simultaneously instead of averaging embeddings into a single [1,d] vector. Previously Q/K/V operated on one row — attention was trivially softmax([1,1]) = 1.0 so output was always just V, making Q/K dead weights. Now each of the MaxLen token positions gets its own embedding, coordinate, and history projection, and attention genuinely mixes information across positions.

  Attention mechanism (built entirely in the Gorgonia computation graph so gradients flow to Q/K):

  scores = Q @ Kᵀ / √d        ← [MaxLen, MaxLen]
  weights = softmax(scores)     ← row-wise via Exp → Sum(axis=1) → HadamardDiv
  output = weights @ V          ← [MaxLen, d]
  

  Pooling collapses [MaxLen, d] back to [1, d] for the output head:

  pooled = Mean(transformer_output, axis=0) → reshape to [1, d]
  logits = pooled @ headW + headB
  

  Gorgonia broadcasting constraint: Gorgonia doesn’t support element-wise broadcasting (e.g. [MaxLen,d] / [d] fails). All biases and layer norms changed from vectors to [MaxLen, d] matrices. Row-wise softmax uses trick: row_sum_2d @ ones[1,MaxLen] produces a [MaxLen, MaxLen] denominator, then HadamardDiv divides element-wise.

  Result: Q/K now genuinely participate in attention. Test config: 16K → 27K params. Loss: 0.099 → 0.082 in 20 training steps.

  • embInput shape [1,d] → [MaxLen,d] (per-token embeddings)
  • coordIn shape [1,coordDim] → [MaxLen,coordDim] (coords repeated per token)
  • coordProjB/historyProjB shape [d] → [MaxLen,d] (per-position biases)
  • Layer norm params ln1W/ln1B/ln2W/ln2B shape [1,d] → [MaxLen,d]
  • FFN biases ff1B/ff2B shape [d] → [MaxLen,FFNDim]/[MaxLen,d]
  • Mean-pool via gorgonia.Mean(x, 0) + Reshape → [1,d] before output head
• Removed Q/K L2 regularization hack (connected through attention path now)
• Removed Go-side softmax2D, computeAttentionWeights, getWeight helpers

[0.2.53] - 2026-07-22

Fixed

• Dashboard URL now prints to stderr on startup so AI can read it.
• Dashboard config check moved inside Start() — reads config directly instead of relying on ActiveConfig timing.

[0.2.52] - 2026-07-22

Added

• Web dashboard — live monitoring dashboard on random port (auto-picks free port, prints URL on startup). Shows tool usage stats, recent commands, chain history, OCR sequences, training samples, model status. Auto-refreshes every 5 seconds. Configurable via dashboard_enabled in config. Dark theme, responsive layout.

Fixed

• Model save race condition — model.Save() and GobSerializer.SaveModel() now use atomic write (write to .tmp, then os.Rename). Prevents corruption when multiple MCP instances train simultaneously.

[0.2.51] - 2026-07-21

Fixed

• training_list_samples no longer returns massive payloads. List endpoint now returns lightweight metadata only (omits onnx_detections, normalized_coords, ocr_text, window_rect). Added TrainingSampleMeta struct.
• training_cleanup_noise with max_age_hours=0 now means “delete all noise” (no age filter) instead of silently defaulting to 24 hours.
• training_cleanup_noise dry-run mode now reports deleted count (files that would be deleted) alongside freed_bytes.
• find_window now supports case-insensitive substring matching. Tries exact match first via FindWindowW, falls back to EnumWindows substring search. Handles backslashes in window titles.

[0.2.50] - 2026-07-21

Added

• §34 Custom Neural Network (12/12 items done):
  • Multi-task output head — model now predicts scroll direction (up/down/left/right) and key category (modifier/navigation/function/alpha/numeric/special) alongside tool and coordinates. predict.Prediction includes Args field with ArgPrediction struct. Trainer fills arg one-hot targets from ArgsJSON.
  • Transfer learning — MLEngine.FinetuneForApp(appName, epochs) fine-tunes the base model on app-specific samples filtered by window title. PredictForApp() uses app-specific model when available, falls back to base. LoadAppModels() restores all app models from app_models/ directory on startup.
  • Model compression — ml/compress package with magnitude-based weight pruning (Prune(), IterativePrune()) and weight quantization (Quantize(), 2–8 bit). Compression ratio utilities. Supports prune+quantize stacking.
  • Neural architecture search — ml/nas package with grid search over EmbedDim/NumHeads/NumLayers/FFNDim. Search() evaluates configs, BestConfig() picks best under parameter budget. Parameter count utility.
• §35 Action Prediction — Tool Expansion, Batch Training, Full Attention:
  • Expand tool set — mlDefaultTools expanded from 11 to 25 tools. New: move_window, maximize_window, minimize_window, close_window, find_window, set_clipboard, get_clipboard, type_and_submit, select_all_and_type, ocr, screenshot, click_menu_item, launch_app. OutputDim = tool(25) + from_xy(2) + to_xy(2) + arg(10) + window(6) = 45.
  • Batch training — TrainerConfig.BatchSize controls mini-batch size. Trainer accumulates samples into batches, calls model.ForwardBackward(target) per sample, then model.Step(lr) + model.ResetGradients() once per batch.
  • Full attention (Q/K/V/O) — Each transformer layer now has separate Q, K, V, O weight matrices (layerDef{qW, kW, vW, oW}). Scaled dot-product attention computed via Q @ (Kᵀ @ V / √d) using MatMul-only path to avoid Gorgonia shape squeeze. Layer norm weights changed from [d] vectors to [1,d] matrices for same reason.
• §35 Action Prediction — Sequence Prediction:
  • Sequence head — Config.SequenceLen controls output: primary = tool + from_xy + to_xy + arg, sequence = N×(tool + to_xy + arg). PrimaryDim(), SeqSlotDim(), TotalOutputDim() layout helpers. predict.SequencePrediction{Primary, Next[]} struct.
  • Sequence training — dataloader.LoadSequences(ctx, minLen) groups consecutive training_pairs by session_id. makeSequenceTargets() fills sequence section of target vector in trainer.
  • Sequence prediction — PredictSequence() returns SequencePrediction with primary + next-N predictions. decodePrimary() and decodeSlot() methods in predictor. Wired through MLEngine.PredictSequence() → AdaptiveEngine.PredictSequenceActions(). Helper functions: mlPredictedAction(), predictedCoordFromPrediction(), predictedArgsFromPrediction().
  • Chain integration — chain_predict MCP tool calls PredictSequenceActions(ocr_text), returns primary + future actions as structured JSON for chain execution.
  • Confidence-gated sequences — SeqConfidenceThreshold=0.15. Steps below threshold truncate the sequence, providing fallback to shorter prediction when model is uncertain.
• §35 Action Prediction — Window Context & Spatial Encoding:
  • Window category output — Config.WindowDim=6, window category one-hot in primary head. WindowCategories = [browser, editor, terminal, file_manager, dialog, other]. Predictor decodes WindowCategory + WindowConf fields. Trainer defaults to “other” category.
  • Window title as input — PredictWithWindow()/PredictSequenceWithWindow() prepend window title to OCR text before tokenization. chain_predict accepts optional window_title parameter. MLEngine + AdaptiveEngine wired.
  • Spatial relationship encoding — Encoder expanded 7→12 features: added distFromCenter, isCenter, isEdge, windowAspect, isDialog. spatial.FeatureDim=12. DefaultConfig() updated.
  • Multi-window awareness — WindowInfo{Title,Category,Active} struct. PredictWithWindows()/PredictSequenceWithWindows() encode top-3 visible windows as [ACTIVE][category]title prefix. Wired through MLEngine + AdaptiveEngine.
  • 4-coordinate output — extended output head from tool(11)+xy(2)+arg(10)=23 to tool(11)+from_xy(2)+to_xy(2)+arg(10)=25 dims. Config.FromCoordDim field added. Predictor decodes 4 normalized coordinates. Trainer fills both from/to coordinate targets normalized via spatial encoder.
  • Action template schema — decodeCoords() per-tool coordinate extraction: drag uses {from_x,from_y,to_x,to_y}, other tools use {x,y} as destination. Trainer normalizes both pairs through the spatial encoder.
  • Drag prediction — Prediction.FromCoordX/Y fields, PredictedAction.FromCoord struct, bridge populates both coordinate pairs from model output.
  • Drag-and-drop — drag tool already in mlDefaultTools with full from/to coordinate support. drag_and_drop alias handled in trainer’s decodeCoords().
  • Backward compatibility — existing models with old OutputDim (23) fail LoadParameters (param count mismatch) → gracefully recreated and retrained with new 25-dim layout.

Changed

• ml/transformer/model.go — Config.ArgDim, Config.FromCoordDim, Config.SequenceLen fields. PrimaryDim(), SeqSlotDim(), TotalOutputDim() layout helpers.
• ml/predict/predictor.go — Prediction.Args, Prediction.FromCoordX/Y fields. SequencePrediction{Primary, Next} struct. PredictSequence() method with decodePrimary()/decodeSlot(). SeqConfidenceThreshold=0.15. ArgCategories constants.
• ml/trainer/trainer.go — FinetuneEpoch(). makeTarget() fills arg one-hot + sequence targets. decodeCoords() per-tool coordinate extraction. Trainer.sequenceLen, Trainer.primaryDim, Trainer.fromCoordDim fields. makeSequenceTargets() fills sequence section.
• ml/dataloader/loader.go — Sequence, Action types. LoadSequences(ctx, minLen) interface method.
• ml/dataloader/sqlite.go — LoadSequences() implementation grouping consecutive training_pairs by session_id.
• internal/actions/ml_bridge.go — MLEngine.PredictSequence() returns SequencePredictionResult. Helper functions: mlPredictedAction(), predictedCoordFromPrediction(), predictedArgsFromPrediction().
• internal/actions/adaptive.go — PredictedAction.Args/FromCoord fields. PredictSequenceActions(ocrText) wrapper. PredictActionsWithWindow, PredictSequenceActionsWithWindow, PredictActionsWithWindows, PredictSequenceActionsWithWindows.
• ml/spatial/encoder.go — FeatureDim expanded 7→12. New features: distFromCenter, isCenter, isEdge, windowAspect, isDialog.
• ml/transformer/model.go — Config.WindowDim field. PrimaryDim() includes WindowDim. ConfigForSize() accepts windowDim, seqLen params. DefaultConfig CoordDim updated to 12.
• ml/predict/predictor.go — WindowInfo{Title,Category,Active} struct. Predictor interface extended with PredictWithWindow, PredictSequenceWithWindow, PredictWithWindows, PredictSequenceWithWindows. Prediction.WindowCategory/WindowConf fields. WindowCategories constants. encodeWindowContext() helper.
• ml/trainer/trainer.go — Trainer.windowDim field. makeTarget() fills window category one-hot (defaults to “other”).
• internal/actions/ml_bridge.go — WindowInfoForPrediction struct. MLEngine.PredictWithWindows(), PredictSequenceWithWindows(), PredictWithWindow(), PredictSequenceWithWindow(). chain_predict handler accepts optional window_title.

VERSION

0.2.50 → 0.2.51

[0.2.49] - 2026-07-21

Added

• §34 Custom Neural Network (in progress) — Phase 1–3.3 of the 4-phase ML engine expansion:
  • Sequence context — ml_bridge.Predict() now passes recent action history into the transformer model. AdaptiveEngine maintains a ring buffer of recent actions, wired through PredictWithContext().
  • Auto-detect screen config — spatial encoder reads actual screen dimensions and DPI via ScreenSize() + GetDPIScaleForPoint() instead of hardcoded 1920×1080/1.0.
  • Improved tokenizer — OCR artifact normalization (pipe chars, multi-space), [NUM] special token for numeric values, character bigram subword fallback for out-of-vocabulary words. Backward-compatible Save/Load.
  • Online learning — ReplayBuffer (10K capacity) stores every action execution. Background goroutine retrains the model every 30 seconds from the buffer.
  • Model versioning — ml/versioning package saves versioned checkpoints (model_vN.gob), auto-rolls back if accuracy regresses >5%, tracks rollback count. Persistence via gob index.
  • Data augmentation — ml/dataloader/augment.go generates synthetic training samples with DPI scaling (0.75×, 1.25×, 1.5×), OCR noise (common substitutions), and coordinate jitter (±5px). Integrated into Train() with 2× augmentation.
  • Model size presets — ConfigForSize(small/medium/large) factory for easy architecture experiments. ParamCount() utility.
  • Debug output — DebugForward() and DebugInfo struct for model diagnostics (logits, tool probs, norms).

Changed

• ml/transformer/model.go — Config.HistoryLen, Model.Forward() now accepts history tokens. New DebugForward(), ParamCount(), ConfigForSize().
• internal/actions/ml_bridge.go — stores screenCfg, replayBuf, versioner. RecordExperience() feeds online learning. StartOnlineTraining()/StopOnlineTraining() manage background goroutine. Training uses augmented data.
• internal/actions/adaptive.go — RecordCommand() now calls mlEngine.RecordExperience().
• ml/tokenizer/simple.go — bigram subword fallback, OCR normalization, [NUM] token, backward-compatible Save/Load format.

Fixed

• Flaky TestTrainer_MultipleEpochs — changed strict loss-decrease check to log (allows non-monotonic loss with small datasets).

Removed (this cycle)

• Deleted 6 persona files from repo root (AGENTS.md, SOUL.md, USER.md, IDENTITY.md, TOOLS.md, HEARTBEAT.md).
• Deleted ml/DESIGN.md — content merged into docs/reference/models-setup.md.
• §33 UPGRADE restored — all 6 tools moved to FAR (nice to have), NEXT emptied.

VERSION

0.2.48 → 0.2.49

[0.2.48] - 2026-07-21

Added

• Smart cascade text finding — find_text_and_click and wait_for_text now use a 3-tier strategy instead of brute-force scrolling:
  1. Spatial memory — checks text_locations table for where text was previously seen (position, window, confidence). Instant, no OCR.
  2. System find-text — injects Ctrl+F into the active app (Chrome, Firefox, Edge, Brave, Opera, Notepad, VS Code, Explorer) and OCRs the match highlight. ~500ms, zero scrolls.
  3. OCR + scroll — falls back to screenshot OCR with auto-scrolling only if the first two tiers miss.
• text_location_memory.go — SQLite-backed spatial RAG store. Every successful find_text_and_click auto-saves word/line text, screen position, window title, and hit count. FTS5 index for fast lookup. Confidence increases with repeated hits.
• system_find.go — OS-level find-text via SendInput Ctrl+F injection. Tracks usage stats (last used timestamp, total count). 5-second timeout guard on the full Ctrl+F workflow.
• find_text_smart.go — unified SmartFindText() and SmartFindTextAndClick() orchestrating the memory → system-find → OCR cascade. New params on find_text_and_click: window_title (for system-find), skip_memory, skip_system_find.
• WaitForTextScroll — wait_for_text now scrolls when text isn’t visible (pass max_scrolls > 0), using the same cascade.

VERSION

0.2.47 → 0.2.48

[0.2.47] - 2026-07-21

Fixed

• Go 1.26 panic on startup — updated go4.org/unsafe/assume-no-moving-gc to latest version. Exe no longer requires ASSUME_NO_MOVING_GC_UNSAFE_RISK_IT_WITH env var.

VERSION

0.2.46 → 0.2.47

[0.2.46] - 2026-07-21

Added

• Go-native ML prediction engine — real transformer neural network (Gorgonia) replaces statistical word-counting for UI automation predictions. 9-package ml/ sub-module: transformer (FFN + residuals + Adam optimizer), tokenizer, spatial encoder (7-feature DPI-aware), SQLite dataloader, training pipeline, softmax/sigmoid inference. Model trains from training_pairs SQLite table on first run, persists to model.gob, loads on restart. Falls back to existing statistical engine when ML model is unavailable.
• ml_bridge.go — integration layer wiring ML transformer into AdaptiveEngine. PredictActions() tries ML inference first, falls back to statistical engine. EnsureAdaptive() loads or trains model on startup.
• Self-improving per-machine models — each installation builds its own personalized model from local usage history. No pre-trained model shipped.
• launch.ps1 — launcher wrapper that sets ASSUME_NO_MOVING_GC_UNSAFE_RISK_IT_WITH=go1.26 automatically. Use this or add the env var to your MCP client config’s env field (all examples updated).

VERSION

0.2.45 → 0.2.46

[0.2.45] - 2026-07-19

Fixed

• Nullable union type serialization bug — the Go jsonschema library generates "type": ["null", "integer"] for *int32 fields and "type": ["null", "boolean"] for *bool fields. The opencode MCP client cannot serialize values for nullable union types, producing truncated JSON (e.g. {"x": instead of {"x": 100}). This broke screenshot, ocr, find_text_and_click, image_diff, and any tool with VerifyArgs (auto_verify, expected, pre_expected). Fixed via addToolClean wrapper that auto-generates clean schemas with plain "type": "integer" (not nullable) for all 100+ tool registrations. Recursively strips nullable unions from nested properties, array items, definitions, and additional properties.

VERSION

0.2.44 → 0.2.45

[0.2.44] - 2026-07-19

Added

• image_diff tool — pixel-level screenshot comparison. Accepts two base64 PNGs (before, after) and returns changed_pixels, total_pixels, change_ratio (0-1), mean_diff (0-255), max_diff (0-255), same (bool). Optional threshold (0-255, default 30) controls per-channel sensitivity. Optional generate_image returns a diff image with changed pixels highlighted in red. Handles mismatched dimensions by comparing the overlapping region. Lives in verify.go alongside computeTextDiff.

VERSION

0.2.43 → 0.2.44

[0.2.43] - 2026-07-19

Added

• File-based structured logging — rotating JSON log files at %APPDATA%/go-mcp-computer-use/logs/server.log. Configurable via set_config: log_file_enabled (default: true), log_file_max_size_mb (default: 10), log_file_retention (default: 7 rotated files). Logs persist across restarts.
• get_logs tool — reads server log entries from the file-based log. Supports filtering by level (min level), search (keyword), since_minutes (time window), and lines (max entries, default 50, max 500). Returns parsed log entries with timestamps, levels, messages, and attributes.
• report_issue tool — generates a GitHub issue report with system info, recent error logs, and AI-provided context. If gh CLI is available, creates the issue automatically with the bug label. Otherwise returns the full markdown body for manual submission.
• Panic recovery wrapper — safeHandler generic wrapper catches panics in tool handlers, logs the panic value + full stack trace, and returns a structured MCP error instead of crashing the process. Applied to get_logs and report_issue. Global panic recovery in main.go catches any remaining panics.
• Multi-handler slog — logs now go to both stderr (TextHandler, for MCP transport) and file (JSONHandler, for persistence) simultaneously. File handler uses the configured log_level.
• New config fields — log_file_enabled (default: true), log_file_max_size_mb (default: 10), log_file_retention (default: 7). All persist via set_config.

VERSION

0.2.42 → 0.2.43

[0.2.42] - 2026-07-19

Added

• chain_abort tool — checks if the global chain abort hotkey has been pressed since last check. Returns {aborted: true} when the configured key combo (default: Ctrl+Shift+Escape) is detected. The abort is consumed on read (auto-resets). Call before starting long chains or poll periodically.
• set_window_lock tool — locks the active chain to a specific window by handle. Screen-touching tools (click, type, OCR, etc.) will verify the locked window is foreground before executing. If window_lock_auto_focus is enabled, automatically re-focuses the locked window when it loses foreground.
• clear_window_lock tool — releases the window lock. Screen-touching tools will no longer be restricted to a specific window.
• Chain abort hotkey engine — background goroutine polls GetAsyncKeyState every 50ms (configurable) for a global hotkey combo. When all keys in the combo are held simultaneously, the abort channel closes and any running chain terminates at the next step boundary.
• Window lock-on context — verify_window_lock() is called before every screen-touching step in chains when window_lock_enabled=true. Prevents accidental actions on wrong windows when focus is stolen.
• New config fields — chain_abort_enabled (default: true), chain_abort_keys (default: “Ctrl+Shift+Escape”), chain_abort_poll_ms (default: 50), window_lock_enabled (default: false), window_lock_auto_focus (default: true). All persist via set_config.
• go-mcp-computer-use skill for Claude — new skill file that teaches Claude about this MCP server’s tools, architecture, and limitations.
• scripts/test.ps1 — PowerShell test runner with Zig CGO setup (mirrors scripts/build.ps1), -Integration flag, -Short flag, and configurable timeout.
• SaveToBytes / LoadFromBytes config helpers — enables testing config round-trips without touching the filesystem.

Fixed

• Chain goroutine leak on global timeout — ExecuteChain spawned a goroutine for execSteps but never cancelled it on timeout. The goroutine would run to completion (or hang) after the parent returned. Now uses context.WithCancel — timeout cancels the context, and the abort case also cancels.
• Chain abort channel shared across chains — previously the abort channel was never reset after being consumed, so a single hotkey press would abort all future chains. Now GetAbortChannel detects a closed channel and creates a fresh one automatically.

VERSION

0.2.41 → 0.2.42

[0.2.41] - 2026-07-18

Added

• reset_state tool — clears accumulated adaptive engine stats (timings, successes, sequences, word→cmd index, coord index) and OCR→command bridge buffer. Use between heavy batch operations to prevent state accumulation and MCP timeouts.
• dismiss_all_menus tool — presses Escape to dismiss open context menus/dialogs. OCRs before and after to detect which menus were open and whether they successfully closed. Returns {esc_pressed, menus_before, menus_still_open}.
• Error enrichment for find_text_and_click — when target text is not found, the error now includes up to 15 visible lines from OCR so the AI can see what IS on screen (e.g. "text 'Delete...' not found. Visible text: 'Back | Refresh | Save as | Print...'").

Removed

• AI_ERROR_REPORT.md — error report findings addressed in v0.2.41 (reset_state, error enrichment, dismiss_all_menus).

VERSION

0.2.40 → 0.2.41

[0.2.40] - 2026-07-15

Added

• cmd/credit-audit — MCP tool credit audit — new Go tool that measures JSON payload size and estimated token cost of every read-only MCP tool. Covers 66 probes across 14 groups (System, Display, Audio, Window, File, Perception, ONNX, UIA, Memory, Template, Training, Datalog, Agent, Layout). Features per-probe 60s goroutine timeout to survive hanging probes, -json flag for machine-readable output, and -sessions N to project hog-tool costs across N calls. Identifies the top credit hogs: training_list_samples (4.2 MB — base64 screenshots in training DB), record_screen (1.1 MB), screenshot_full (580 kB), priors_stats (121 kB).

Fixed

• loadPriorsFromDB reentrant mutex deadlock — UpdatePriorsFromDetections and UpdatePriorsForNegative held elementPriors.mu.Lock() then called loadPriorsFromDB() which tried to Lock() the same non-reentrant mutex. Split into loadPriorsFromDB() (acquires lock) and loadPriorsFromDBLocked() (assumes caller holds lock). The two write-path callers now use the locked variant directly.
• Fragile RLock→RUnlock→Lock→RLock lazy-init pattern — AdjustConfidenceWithPriors, GetPriorStats, and FindPriorPrediction each manually dropped the read lock, acquired the write lock, then re-acquired the read lock. Replaced with sync.Once for the initial load, eliminating the lock-dance and preventing future reentrant deadlocks if anyone adds a code path between the drop and re-acquire.
• TaskBegin nested lock ordering — held taskMu.Lock() while acquiring dlogMu.Lock(), creating a taskMu→dlogMu ordering dependency. Restructured to release taskMu before acquiring dlogMu, matching TaskEnd’s pattern.
• Analyze held e.mu.RLock across external lock acquisitions — Analyze() called tt.Stat() and ts.Rate() while holding e.mu.RLock(). If Stat() or Rate() ever acquired e.mu, this would deadlock. Fix: snapshot all maps under RLock, deep-copy PersistedStat values (not just pointers), release RLock, then compute stats.
• LogToolCall implicit pairMu dependency in log parameter — bridgeBufferSize() (which acquires pairMu) was called as an argument to slog.Warn, evaluated in the parameter list before the explicit pairMu.Lock() later in the function. Extracted to a variable before the log call to decouple lock acquisition ordering.

VERSION

0.2.39 → 0.2.40

Added

• get_dpi_for_point(x, y) — new MCP tool that returns DPI and scale percentage at a specific screen coordinate. Useful for determining which monitor a coordinate is on and its scaling factor in mixed-DPI multi-monitor setups. Returns {dpi, scale_percent, x, y}. Also available in chain dispatch.
• FocusHandle chain step field — new focus_handle field on chain steps accepts a handle directly (via `` capture), avoiding title re-resolution. Chain checks focus_handle first, falls back to focus_window (title-based).
• auto_verify_focus chain option — new boolean field on chain requests. When true, the chain engine tracks the last-focused window handle and re-verifies foreground state before sending input (click, type, key_press, scroll, etc.). If focus was stolen by a popup or notification, it re-focuses before acting.
• click_menu_item accepts handle — new optional handle field. When provided, bypasses title-based window lookup. Falls back to window_title if handle is 0.
• layout_validate accepts window handle — new optional window_handle field. When provided, bypasses title-based window lookup. Falls back to window_title if handle is 0.

Fixed

• FocusWindow now verifies and retries — previously discarded SetForegroundWindow return and never confirmed the window actually became foreground. Now uses a 4-attempt fallback chain: (1) SetForegroundWindow with AttachThreadInput, (2) BringWindowToTop + SW_SHOW, (3) SwitchToThisWindow (bypasses foreground lock), (4) retry SetForegroundWindow after delay. Each attempt verified via GetForegroundWindow. Returns error if all attempts fail instead of silently returning nil. This fixes the stale-focus failure where clicks land on the wrong window.
• click_menu_item no longer silently fails — when window title matches but window is obscured, the improved FocusWindow (called by browser focus helpers) now actually brings the window to foreground before OCR+click.
• ensureWindowFocus title-bar click guarded by z_order — the post-focus activation click at Top+10 was blind to always-on-top windows. Now checks ZOrder == 0 (truly topmost) before clicking, preventing accidental hits on overlapping windows.

VERSION

0.2.38 → 0.2.39

[0.2.38] - 2026-07-15

Added

• z_order field on get_window_state — reports 0=topmost, higher=deeper in Z-order stack. Uses GetDesktopWindow + GW_CHILD to find the true topmost, then walks GW_HWNDNEXT counting visible windows. The AI can compare z_order between handles to determine absolute stacking: window A at z_order=3 is above window B at z_order=12.
• uia_get_all_elements(handle, max_results) — returns all immediate child UI elements in a window (title bar, menu bar, content panes, toolbars, status bar). Uses TreeScope_Children + TrueCondition (one level deep, not recursive DOM tree) so a browser window doesn’t flood with 10K+ elements. Returns name, control_type, automation_id, bounding rect, is_enabled for each.
• uia_get_element_at_point(x, y) — identifies which UI element is at screen coordinates using UIA ElementFromPoint. Returns name, control_type, automation_id, bounding rect. Use after get_cursor_position or click to validate what was under the cursor.
• wait_for_ui_element(handle, name, control_type, timeout_ms) — polls UIA FindFirst on a window’s descendants until the element appears or timeout. Use for content verification: click a button, then wait_for_ui_element for the dialog that should appear. Default timeout 10s.
• Auto-capture element_at_point in chain — mouse-based chain steps (click, move_mouse, hover, drag) now automatically call UIAElementFromPoint at their target coordinates after execution. The UIA element is attached to the step output as element_at_point and available as a captured variable for subsequent steps.
• verify_ui step type — new chain step that verifies a UI element exists or disappears using UIA instead of OCR. Accepts element_name, control_type, handle (window scope), timeout_ms, not_exists (true = expect absence). Polls UIA FindFirst / WaitForUIElement until timeout. Complements the existing OCR-based verify step for structural post-action validation.
• if_uia step type — new conditional branch that checks UI element existence via UIA. Branches then/else based on whether an element with the given name/control_type is found. Like if (OCR), but structural rather than pixel/text-based.
• Chain-callable UIA tools — uia_find, uia_get_element_at_point, uia_get_all_elements, uia_set_text, wait_for_ui_element are now registered in the chain tool dispatch and can be called as regular chain steps with `` substitution.
• Chain-callable window tools — get_active_window, ocr_window, ocr_active_window added to chain dispatch.
• ocr_window tool — new MCP tool that extracts text from a specific window by handle using Windows OCR. Calls OCRWindow(hwnd, language) which captures the window’s bounding rect via GetWindowRectByHandle, clamps to screen bounds, then runs WinRT OCR on the region. Accepts handle (uintptr) and optional language.
• ocr_active_window tool — new MCP tool that extracts text from the current foreground window. Uses ForegroundWindowHandle() to get the active window handle, then delegates to OCRWindow. Accepts optional language parameter.
• ForegroundWindowHandle() exported helper (datalog.go) — returns the current foreground window’s HWND via GetForegroundWindow.
• clamp32() helper (ocr.go) — clamps int32 values to [lo, hi] range for safe screen bounds capping.
• list_windows now returns bounding rect — each window now includes x, y, width, height from GetWindowRectByHandle. The AI can cross-reference window bounds with list_displays monitor positions to determine which screen a window is on.
• get_active_window now returns bounding rect — same x, y, width, height fields added.

Changed

• OCRWindow now clamps to screen bounds — window rects with negative coordinates (off-screen title bars) are clamped to the virtual screen dimensions via ScreenSize() before capture, preventing “out of bounds” errors from ValidateRegion.
• LogToolCall auto-capture unchanged — remains full-screen OCRScreen to preserve the training pair bridge behavior; the AI chooses between ocr, ocr_window, ocr_active_window as needed.
• Tool descriptions updated — get_window_state documents z_order and the visible-flag vs actual-visibility distinction. focus_window tells AI to ALWAYS call before interacting with a non-foreground window. uia_find mentions it can find textboxes, address bars, search menus, title bars; advises focusing window first. ocr_window warns about minimized/obscured windows. list_windows and get_active_window descriptions mention bounding rect fields.
• Chain input schema relaxed — args AdditionalProperties now accepts any type ({} instead of {type:object}), so `` template strings in arg values pass schema validation.

VERSION

0.2.37 → 0.2.38

[0.2.37] - 2026-07-08

Added

• Per-tool enable/disable — tool_denylist config field (array of tool names) removes tools from the MCP server entirely so the AI never sees them. Uses server.RemoveTools() after registration — zero changes to existing handler code. Case-insensitive matching. Example: "tool_denylist": ["shutdown", "restart", "hibernate"]. Configurable at runtime via set_config.
• Retention policy — retention_days config field (integer, 0=disabled) auto-prunes training samples older than N days. Background pruner runs every 6 hours. Deletes both database rows and image files. Starts automatically on boot when retention_days > 0 and training_enabled is true. Configurable at runtime via set_config.
• Unit tests for v0.2.33 behaviors — 25 new tests across internal/actions/adaptive_test.go and internal/actions/datalog_test.go: uniqueTokens dedup, nearbyOCRText spatial scoping, capAndDedupeText fallback, SaveAdaptiveStat/LoadPersistedStats round-trip, Analyze() persisted fallback. 5 new tests in internal/config/config_test.go for ToolEnabled helper.
• ToolEnabled helper (config.go) — case-insensitive denylist check with empty-string safety.

Changed

• set_config tool — accepts new fields: tool_denylist (string array), retention_days (integer). Description updated to document both.
• Config struct — added ToolDenylist []string and RetentionDays int fields.

[0.2.36] - 2026-07-07

Fixed

• All handlers now return structured JSON instead of hardcoded "ok" — verifiedResult() helper and ~50+ handler functions set Content: []mcp.Content{&mcp.TextContent{Text: "ok"}} which blocked the MCP SDK from auto-populating with the structured second return value. When Content == nil, the SDK marshals the second value and sets both StructuredContent and Content (as JSON text). Fixed verifiedResult() to return nil Content when extra data is present, and changed every handler that returned explicit "ok" Content to return &mcp.CallToolResult{}, map[string]any{"ok": true}, nil (no-data tools) or &mcp.CallToolResult{}, result, nil (data tools). Verified live post-reboot: list_windows, get_system_info, get_active_window, ocr all return their structured JSON payloads instead of "ok".

[0.2.35] - 2026-07-07

Fixed

• roUninitialize never called on shutdown — COM WinRT apartment initialized by ensureRo() was never cleaned up. Added CloseWinRT() in ocr_com.go wired into main.go’s signal handler so RoUninitialize runs on graceful shutdown.
• status initializer unreachable in verifiedResult — status := "ok" was always overwritten by the if/else branches before use. Changed to single default status := "ok (verified)" with negation on !vr.Passed.
• klMouseDown and winEventHookProc dead code — leftover scaffolding from keylogger refactor. winEventHookProc type was unused (callback uses syscall.NewCallback inline); klMouseDown was shadowed by local vars in pollLoop. Removed.
• Unused state params in execVerify and execPoll — both functions received *chainState but never used it. Renamed to _ *chainState to match the pattern already used by execWait and execTool. Keeps call-site interface compatibility with execIf/execLoop which do need state.
• Inefficient string concatenation in WriteString calls — three spots in readXlsx and readPdf concatenated strings before passing to buf.WriteString(...), allocating intermediate strings. Changed to fmt.Fprintf(&buf, ...) which avoids the extra allocation.

Added

• uia_set_text MCP tool — writes text into a UI element via UI Automation’s ValuePattern.SetValue. The COM plumbing (uiaElement.setValue) existed but was never wired. Added UIASetText() in uia.go, handler and registration in server.go. Fills a backlog item from docs/meta/backlog.md.
• Prior-based coordinate prediction in FindUIElement — priors.go tracked element frequency and position per window but FindUIElement never consulted them. Added FindPriorPrediction() that returns predicted coordinates for high-confidence priors (frequency >= 70%, sample_count >= 5, StdX/StdY <= 2.0). Inserted as step 1.5 in the cascade: memory → prior → ONNX → OCR. A prior hit avoids ONNX (no GPU/CPU inference) and OCR (no PowerShell launch).
• ONNX/OCR failure logging in FindUIElement — when ONNX detection failed, the function silently fell through to OCR with no trace of the error. Same for OCR failure. Added log.Printf calls in both paths so server logs record when these subsystems are unavailable, distinguishing “not found” from “can’t detect.”
• Competitive intelligence gathered from landscape survey — cross-referenced 12 open-source and commercial projects in docs/comparison-vs-alternatives.md and extracted features worth adopting. Added a “Competitive Intelligence” section to docs/meta/plan.md with prioritized steals (quick wins → medium → high effort). Fleshed out docs/meta/backlog.md with 3 new sections (Memory & ML, Transport & Server, Browser Automation, Linux & Container) and enhanced Security & Identity, Vision, Mouse, Screen, and Processes sections — totaling ~41 new backlog items. Shout out to Cua, Agent-S, Bytebot, MS Magentic-UI, Windows-MCP, DesktopCtl, Windows MCP Server, Computer Control MCP, Browser Use, and Microsoft Windows Recall for the reference implementations and design inspiration.

Fixed

• Double screenshot in FindUIElement OCR fallback — when ONNX failed, OCRScreen("") captured the screen again even though CaptureScreen() had already been called at step 2. Changed OCR fallback to call ocrFromBase64(b64, "") directly, reusing the existing screenshot. Also performs pushRecentOCR/tryCompletePair/LogOCRSnapshot side effects that OCRScreen normally handles.
• Fragile memory deserialization in FindUIElement — the memory fast path used 4 levels of nested type assertions (any → map → float64 → int32) with silent fall-through on any mismatch. Extracted into memoryToElement() helper that returns nil on any parse failure, keeping it clear and testable.

[0.2.34] - 2026-07-07

Fixed

• Training pair pipeline died permanently after a single missed OCR window — LogToolCall only called OCRScreen("") (the auto-capture that completes a pending pair and refreshes recentOCR for the next action) inside the if ocrBefore != "" branch. If one action’s findRecentOCRBefore call missed the bridgeWindow — entirely plausible under normal agent round-trip latency between an explicit ocr() call and the action that follows it — the buffer never refreshed again. Every subsequent action would also find nothing, permanently starving training_pairs until something called an OCR tool manually to re-seed it. Moved the OCRScreen("") auto-capture out of the conditional so it now runs after every action regardless of whether that action found a prior snapshot, making the bridge self-healing instead of a single point of failure.

Changed

• bridgeWindow widened 30s → 60s — 30s was tight for realistic agent-driven latency between an OCR call and the action that follows it. Combined with the self-healing refresh above, 60s gives headroom without letting stale context linger indefinitely.

Verification

Live end-to-end test through the MCP server (ocr → click → type → key_press → click → type): before this fix, 5 real actions produced 0 training_pairs rows (the bridge had already gone dark from an earlier missed window). After the fix and a rebuild/restart, the same kind of sequence produced 5/5 pairs, with agent_train/agent_analyze showing populated timing_stats, success_rates, and top_sequences with counts correctly bounded by total_commands.

[0.2.33] - 2026-07-06

Fixed

• agent_analyze: timing_stats/success_rates resetting on restart — Stats were purely in-memory (map[string]*ToolTiming, map[string]*ToolSuccess), so every server restart wiped them, leaving agent_analyze with empty timing_stats/success_rates until new commands arrived. Added adaptive_stats SQLite table and SaveAdaptiveStat()/LoadPersistedStats()/HydratePersisted(). On startup, EnsureAdaptive() calls HydratePersisted() to seed in-memory stats from the durable table, and every RecordResult() call persists the aggregate asynchronously via go SaveAdaptiveStat(...).

• Training token inflation: uniqueTokens() dedup — tokenize() on raw OCR text produced duplicate word entries per row (same word appearing multiple times in the OCR dump contributed multiple hits to the (word, command) pair). This caused Count in top_sequences to exceed total_commands/total_sequences (which count rows, not word occurrences). Added uniqueTokens() to dedupe before insert, so each (word, command) pair gets at most 1 hit per row regardless of how many times the word appears in the OCR context.

• OCR context scoped to nearby words — findRecentOCRBefore() used to pass the ENTIRE screen’s OCR dump as ocr_before for every command, so common on-screen words (“the”, bullet points, news headlines, unrelated UI labels) all got associated with whatever command followed — producing noise-dominated predictions. Now:
  • For coordinate-based tools (click, move_mouse, drag, hover), it uses word bounding boxes (words []OCRWord) to scope context to words within 200px of the target coordinates, nearest-first, capped at 20 words.
  • For non-coordinate tools, it falls back to capAndDedupeText() which returns at most 40 unique words from the full dump instead of the raw dump.
• LogToolCall no longer double-counts timing/success — LogToolCall previously called Adaptive.RecordResult(tool, 0, errVal == nil) with a phantom 0ms duration. Callers (Click, TypeText, etc.) already call Adaptive.RecordResult with the real elapsed duration. Removed the call from LogToolCall to avoid every action being double-counted with a fake 0ms sample alongside the real one.

Changed

• pushRecentOCR() signature — Now accepts *OCRResult (with word bounding boxes) instead of just string, enabling spatial scoping in findRecentOCRBefore.
• findRecentOCRBefore() signature — Now takes tool string, argsJSON string to enable tool-specific OCR scoping.
• Datalog query limit relaxed — From clamped 50–200 range to 1–5000 range (default 50).

[0.2.32] - 2026-07-05

Fixed

• chain tool startup panic — shared sub-schema pointers — chainInputSchema() reused the same *jsonschema.Schema for then, else, and steps fields. The MCP SDK’s AddTool() requires schemas to form a tree (not a DAG) and panics on duplicate pointers. Changed to factory functions that return unique instances per call.
• Module path mismatch — go.mod declared github.com/user/go-mcp-computer-use but the repo lives at github.com/coff33ninja/go-mcp-computer-use. Updated module path and all 7 import references across the codebase. (User was lazy to update this.)
• Adaptive engine: timing_stats and success_rates never populated — RecordCommand (which calls RecordResult → RecordTiming + RecordSuccess) was defined but never called. Added Adaptive.RecordResult(tool, 0, errVal == nil) to LogToolCall so runtime success/failure is tracked per tool. Previously agent_analyze always showed empty timing_stats: {} and success_rates: {}.
• Adaptive engine: rebuildSequences merge corrupts Count and never updates Freq — When merging duplicate (word, command) entries, Count was overwritten instead of accumulated, and Freq (success ratio) was set at creation and never recalculated on merge. Added internal SuccessCount/FailCount fields to SequenceExample, fixed merge to accumulate correctly and recalculate Freq.

Added

• Chain integration tests — 7 tests build-tagged //go:build integration that start the mcp-server binary and validate chain tool end-to-end via stdio MCP protocol. Covers: simple steps, capture, loop, if/else branching, unknown tool error, timeout, and structured data output. Run with go test -tags=integration -v -count=1 -timeout 120s ./internal/actions/ -run 'TestChain_'.
• CI: chain-tests job — runs chain integration tests after lint in .github/workflows/ci.yml.
• README badges — Go version, release, CI status, Windows, MCP, last commit, PRs welcome.

[0.2.31] - 2026-07-05

Changed

• All 60+ query/result handlers now return structured data instead of “ok” — Every handler that returns meaningful data (get_volume, get_battery, list_windows, get_system_info, get_uptime, get_clipboard, get_pixel_color, list_displays, get_disk_usage, get_network_info, ocr, find_image, find_all_images, list_audio_devices, list_processes, uia_find, uia_get_text, memory_get/search/list, template_find/list/store/forget, training_, onnx_, datalog_status, chain, launch_and_wait, write_file, delete_file, find_files, list_directory, set/get_working_directory, bridge_debug, set_config, task_begin, and more) now return their structured JSON data instead of the placeholder "ok" text. The SDK auto-populates tool results from structured output when no explicit TextContent is set, so tools like get_screen_size now show {"width":1920,"height":1080} instead of "ok".
• get_screen_size, get_cursor_position — same fix applied (noticed during audit).
• chain tool schema — no longer rejected by Gemini — IfConfig.Then/Else and LoopConfig.Steps changed from []any to []ChainStep, which produced items: true and type: ["null", "array"] in the auto-generated JSON schema (both rejected by Gemini’s MCP schema validator). The chain tool now uses a manually crafted InputSchema that avoids the recursive type cycle in jsonschema-go and produces clean schema output.

[0.2.30] - 2026-07-03

Added

• Windows icon embedded in mcp-server.exe — app.ico compiled into a COFF .syso resource via rsrc (github.com/akavel/rsrc), so the binary shows a custom icon in File Explorer, taskbar, and title bar. Icon sizes: 16, 32, 48, 64, 256px with SVG source in icons/app.svg.
• icons/ directory — app.svg source, generated PNGs at 5 sizes, app.ico multi-res icon, app.rc resource script.
• scripts/gen-icons.ps1 — PowerShell script that runs rsrc to compile app.ico into cmd/mcp-server/rsrc_windows.syso, auto-installing rsrc if missing. Called from build.ps1, lint.ps1, and CI release workflow.

Security

• Bump golang.org/x/net v0.54.0 → v0.55.0 — dependency update from Dependabot patching a security vulnerability in the go_modules group.
• .github/dependabot.yml — package-ecosystem was empty ""; set to "gomod" so Dependabot properly scans go.mod for vulnerabilities.

Changed

• write_file — overwrite is now optional — changed Overwrite bool to *Overwrite *bool with omitempty. No longer required in tool schema, defaults to false when omitted.
• get_file_info — returns metadata instead of “ok” — handler now returns actual file info JSON (name, size, is_dir, mod_time, mode).

[0.2.29] - 2026-07-02

Added

• File System tools — 11 new tools for navigating and manipulating files:
  • list_directory — list files/dirs at path (name, size, is_dir, mod_time, mode)
  • read_file — read file with automatic type detection + native format parsing
  • write_file — write/overwrite files with format-aware creation and editing
  • find_files — recursive glob search (e.g. *.go, **/*.md)
  • copy_file — copy file or directory (recursive)
  • move_file — move/rename file or directory
  • delete_file — delete file/dir to Recycle Bin via SHFileOperationW (not permanent)
  • create_directory — mkdir -p (recursive directory creation)
  • get_file_info — file/dir metadata (size, mod_time, is_dir, mode)
  • set_working_directory — set working directory for relative path resolution
  • get_working_directory — get current working directory
• Format-aware read_file — auto-detects mime type by magic bytes + extension; parses:
  • Plain text (txt, json, csv, yaml, toml, md, source code, configs, etc.) via io.ReadAll
  • .docx via nguyenthenguyen/docx library, extracting text from <w:t> XML elements
  • .xlsx via xuri/excelize/v2, all sheets rendered as TSV
  • .pdf via ledongthuc/pdf, all pages with separators
  • Images (png, jpg, gif, bmp, tiff, webp) via native WinRT COM OCR (ocrNative)
  • Pagination: page and page_size params (default 8000 chars), returns page/totalPages/truncated
• Format-aware write_file — detects target extension and creates/edits:
  • Plain text — raw write via os.WriteFile
  • .docx — new file creates from scratch (ZIP+XML); overwrite preserves existing headers/footers/images by swapping <w:body> content, writes to temp + rename
  • .xlsx — new file via excelize.NewFile(); overwrite opens existing, repopulates cells from TSV content
  • .pdf — new file creates from text via go-pdf/fpdf; overwrite tries pdfcpu.FillFormFile with JSON form field data, falls back to createPdf
• File verification — FilePreCheck/FilePostVerify wired into all 5 action file tool handlers (write, copy, move, delete, create_directory) using ExpConfig / VerifyArgs pattern from v0.2.28
• Recycle Bin delete — delete_file uses SHFileOperationW with FOF_ALLOWUNDO to move items to the Recycle Bin instead of permanent os.RemoveAll
• Working directory — all file tools resolve relative paths against a configurable working directory (defaults to process CWD, changeable via set_working_directory)
• New dependencies — nguyenthenguyen/docx, ledongthuc/pdf, xuri/excelize/v2, go-pdf/fpdf, pdfcpu/pdfcpu for native document parsing and creation

Changed

• internal/actions/filesystem.go — Enhanced ReadFile with format dispatch + pagination; enhanced WriteFile with format-aware creation/editing (docx, xlsx, pdf); file verification helpers; Recycle Bin via SHFileOperationW; working directory support.
• internal/server/server.go — Updated ReadFileArgs{Path, Page, PageSize}, updated write_file tool description; file verification handlers.
• internal/actions/chain.go — Updated chainReadFile with page/page_size params.
• VERSION — bumped to 0.2.29.

Tool Count

Now at 131 total MCP tools (+11).

[0.2.28] - 2026-07-02

Added

• Auto-verify on 5 remaining high-value tools — open_url, launch_app, find_text_and_click, select_all_and_type, click_menu_item now support auto_verify and expected parameters with OCR-based post-action verification, matching the existing 6 tools.
• TrainingCatLaunch category — New "launch" training category for app launch snapshots.
• Pre-action validation (pre_expected) — All 11 verification-enabled tools now accept pre_expected with the same ExpConfig shape (text/not_text/change). Runs OCR before the action and fails fast if precondition not met — action is never executed.
• VerifyArgs embeddable struct — Replaced duplicate AutoVerify/Expected/PreExpected fields across all 11 arg structs with a single VerifyArgs embed, reducing 22 lines of redundancy.
• preVerifyCheck helper — Common pre-verify logic extracted to server.go.
• Region-of-typing OCR — type, type_and_submit, select_all_and_type now capture cursor position (GetCursorPosition) before typing and restrict verification OCR to SmartRegionAround(cursor, 400px) instead of full-screen scan.
• Window-aware OCR for click_menu_item — Verification scans only within the target window bounds (found by FindWindowByTitle + GetWindowState) instead of full screen.
• Coordinate-reuse for find_text_and_click — FindTextAndClick now returns click coordinates (int32, int32, error). The handler uses SmartRegionAround(click_pos, 400px) for post-verify instead of full-screen OCR. Pre-verify uses the specified search region.

Changed

• internal/actions/chained.go — FindTextAndClick signature changed from error → (int32, int32, error). Callers updated: server.go, chain.go, cmd/benchmark/main.go.
• internal/actions/training.go — Added TrainingCatLaunch.
• internal/server/server.go — All 11 verification handlers updated with cursor-aware region (type tools), window-bounds region (click_menu_item), coordinate-reuse region (find_text_and_click), and pre-verify checks. VerifyArgs struct replaces repetitive fields.

[0.2.27] - 2026-06-30

Added

• find_image / find_all_images ONNX + OCR fallback — When NCC template matching fails (no match, degenerate template), both tools now cascade through ONNX YOLO object detection → Windows OCR. find_image returns the highest-confidence ONNX element (or first OCR word if ONNX is empty). find_all_images returns all ONNX elements + all OCR words. The fallback captures a fresh screenshot if no screenB64 was provided (ensureScreenB64), making it robust against degenerate templates.
• findImageONNXFallback / findAllONNXFallback helpers — Extracted fallback logic with full cascade (ONNX → OCR) and screen capture self-healing.
• ensureScreenB64 helper — Captures screen on demand when the passed-in screenB64 is empty, fixing the edge case where degenerate templates bypass screen decode.
• ocr_languages tool — New OcrLanguages() function in ocr.go:145 queries WinRT COM (IOcrEngineStatics.get_AvailableRecognizerLanguages) and returns every installed OCR language with tag, display name, and native name.
• Middle mouse button — Click now supports button: "middle" via mouseEventMiddleDown/mouseEventMiddleUp (0x0020/0x0040).
• Horizontal scroll — Scroll now takes horizontal bool parameter; uses mouseEventHWheel (0x1000) flag when horizontal. chainScroll passes it through from args.

Changed

• internal/actions/template.go — FindImage and FindAllImages restructured: degenerate templates (zero-dim, no-variance) skip NCC entirely and go straight to ONNX+OCR fallback. Template-larger-than-screen still errors (unrecoverable). FindAllImages added as new NCC implementation with non-maximum suppression (overlap >50% suppressed).
• internal/actions/window_ext.go — GetWindowState now returns fullscreen boolean field. Added isFullscreen() helper that checks MonitorFromWindow + GetMonitorInfoW + WS_CAPTION style.
• internal/actions/window_ext.go — Added monitorFromWindow proc, MONITORINFO struct, WS_CAPTION constant.
• docs/tools.md → docs/reference/tools.md — auto-regenerated (2 new tools, 120 total)
• Doc audit & reorganization — Audited 20 files across docs/, dev/, .github/instructions/. Deleted docs/todo.md (dead) and dev/models-setup.md (duplicate of docs/models-setup.md). Deduplicated privacy table, accessibility block, inline tool listing (80 lines), and inline release cycle (8 steps) into cross-references. Reorganized flat docs/ into subdirs: adr/, reference/, guides/, meta/. Moved 15 files to new locations. Updated all cross-references in README.md and all 18 docs files.
• CI/CD path updates — ci.yml references docs/tools.md → docs/reference/tools.md; release.yml references docs/CHANGELOG.md → docs/meta/CHANGELOG.md; scripts/gen-tools-doc.go writes to docs/reference/tools.md; scripts/push-and-release.ps1 references docs/meta/CHANGELOG.md.
• scripts/gen-tools-doc.go — MkdirAll("docs") → MkdirAll("docs/reference"), output path changed from docs/tools.md to docs/reference/tools.md.
• New reference docs — docs/reference/windows-dll-ref.md (every syscall proc, DLL, COM interface used), docs/reference/uipi.md (UIPI elevation detection logic + call sites), docs/reference/com-patterns.md (COM/WinRT patterns: vtable dispatch, async polling, HSTRING/BSTR lifecycle, threading model, UIA tree traversal)
• Cross-reference updates — README.md, codebase-map.md, windows-dll-ref.md all updated with links to the 3 new reference docs
• docs/reference/windows-dll-ref.md — Fixed inaccurate advapi32.dll proc names (removed CheckTokenMembership/AllocateAndInitializeSid/FreeSid that were never used in uipi.go)
• docs/reference/vtable-verification.md — New doc covering COM vtable stability guarantees, SDK header verification procedure, CI/CD plan, and complete test table (13 tests, 16 unique vtbl indices, all verified)

Fixed

• find_image / find_all_images no longer error on degenerate templates — Previously a constant-color or 0×0 template returned an unrecoverable error. Now falls through to ONNX + OCR object/text detection.

VTable Verification System (vtable hardening)

• 36 vtblMethod() call sites annotated — Every COM/WinRT vtable dispatch in uia_com.go, ocr_com.go, ocr.go, winrt.go tagged with inline // N = MethodName (verified 2026-06-30, Win11 26200) and block-level // Verified YYYY-MM-DD — WinBuild SDK headers above each interface definition.
• internal/actions/vtable_test.go — 13 smoke tests with build tag //go:build vtable && windows. Covers all 16 unique vtbl indices used in production: UIA GetRootElement (vtbl 5), BoundingRect (43), FindFirst (5, 21), Conditions (21, 23, 25), GetPropValue (10), GetCurrentPattern (16), ValuePattern/Invoke/Array (3, 4, 6, 21), OCR GetLanguages (7, 6), OCR TryCreate (10), OCR TryCreateFromLanguage (9), StorageFile async pipeline (6, 8, 14, 0, 7), HSTRING round-trip, vtblMethod nil guard. All pass (~8s).
• scripts/verify-vtable-docs.go — Parses all 36 vtblMethod() call sites from source, cross-references each unique index against com-patterns.md and vtable-verification.md doc tables, checks test coverage via // vtbl: annotations. Run go run ./scripts/verify-vtable-docs.go. Confirms all 16 unique indices are documented and tested.
• CI: vtable-check job — .github/workflows/ci.yml runs go test -v -tags=vtable ./internal/actions/ -run 'TestVtable' after build, plus go run ./scripts/verify-vtable-docs.go to catch doc drift.
• go vet passes — zero warnings across all vtable code.
• scripts/verify-iid-usage.go — New Go script that parses winrt.go, scans internal/actions/ for IID references, categorizes each as used (referenced outside winrt.go), internal (only within winrt.go), or unused. Cross-references against the Status column in com-patterns.md docs. With -update, rewrites the Status column. Run: go run ./scripts/verify-iid-usage.go or go run ./scripts/verify-iid-usage.go -update.
• scripts/discover-winrt-iids.ps1 -UpdateDocs mode — Added -UpdateDocs switch that auto-generates the 51-entry IID table in docs/reference/com-patterns.md from discovered values, replacing the content between <!-- IID_TABLE_START --> / <!-- IID_TABLE_END --> markers. Includes fallback pass for statics discovered under flat keys. Now also calls verify-iid-usage.go -update to set correct usage statuses. CI runs -UpdateDocs then go run verify-iid-usage.go then git diff to catch drift.
• ci.yml vtable-check job — Added Verify WinRT IID docs are in sync with source step: runs powershell -File scripts\discover-winrt-iids.ps1 -UpdateDocs + go run ./scripts/verify-iid-usage.go, fails if docs/reference/com-patterns.md has uncommitted changes.
• docs/reference/scripts.md — New reference doc cataloging all 8 scripts with purpose, invocation, uniqueness, and cross-references to docs and CI.
• docs/reference/com-patterns.md — VTable dispatch section (§2) updated with IUnknown base column in all index tables. IID table wrapped in <!-- IID_TABLE_START/END --> markers with new Status column (used/internal/unused). Inline -UpdateDocs flag in both “run the script” instructions.
• docs/reference/vtable-verification.md — Expanded from “proposed plan” to actual test table with vtbl Indices + Verified columns, Performance note documenting why FindAll on desktop root is slow by design, and CI/CD integration reference.
• uia.go andCondition dead code fixed — CreateAndCondition (vtbl 25) was called from buildCondition but not tested; now exercised by TestVtable_IUIAutomation_Conditions.
• internal/actions/uia_ctrltype_test.go — 3 tests validating the UIA ControlTypeId map covers all 41 constants (50000–50040), no duplicates, no out-of-range values, and unknown names return nil.
• Cross-references updated — README.md, docs/reference/codebase-map.md, docs/reference/windows-dll-ref.md, docs/ci-cd-pipeline.md all linked to new vtable docs.
• VTable indices are hard-won — Every index was researched against SDK headers (UIAutomationClient.h, windows.data.h, windows.storage.h) and verified at runtime on Windows 11 (build 26200). These are the most fragile part of the codebase: an incorrect vtbl index silently calls the wrong method or crashes. The verification script and test suite exist because even a single off-by-one error can take days to diagnose. Microsoft freezes these indices for published interfaces, but when upgrading to a new Windows build, run go test -tags=vtable and go run ./scripts/verify-vtable-docs.go before shipping.

[0.2.26] - 2026-06-30

Fixed

• Chain tool: computer_use_* prefix normalization — execTool now strips the computer_use_ prefix from tool names before dispatch lookup, so all computer_use_* tools (click, type, key_press, ocr, get_screen_size, etc.) work inside chain steps.
• Chain success aggregation — result.Success now initializes to true before checking step results, fixing false-negative success: false when all steps pass.
• Keyboard modifier key case sensitivity — KeyPress normalizes modifier key names to uppercase before vkModMap/vkSpecialMap lookups, so "Ctrl", "ctrl", "CTRL" all correctly match instead of being silently skipped.
• Window focus reliability — FocusWindow uses AttachThreadInput to attach to the target window’s input thread before SetForegroundWindow, working around Windows focus-stealing restrictions for background automation processes.

Changed files

• internal/actions/chain.go — execTool adds strings.TrimPrefix(step.Tool, "computer_use_") at line 312; ExecuteChain initializes result.Success = true before failure loop at line 200
• internal/actions/keyboard.go — KeyPress normalizes keys via strings.ToUpper before modifier/special key lookups
• internal/actions/window.go — FocusWindow uses AttachThreadInput with GetWindowThreadProcessId/GetCurrentThreadId
• internal/actions/system.go — added getCurrentThreadId (kernel32) and attachThreadInput (user32) proc declarations

[0.2.25] - 2026-06-30

Fixed

• Coordinate extraction: case-insensitive key matching — getIntArg now falls back to case-insensitive key lookup when an exact match fails, fixing coordinate extraction for click and move_mouse tools which store their args with capitalized X/Y (from Go json.Marshal of struct fields) while the code searched for lowercase x/y. This caused all click coordinate data to be silently ignored by TrainFromDatalog, meaning the __learned__ aggregate and per-token coordinate index never accumulated click coordinates.

Changed files

• internal/actions/adaptive.go — getIntArg now does case-insensitive key lookup via strings.EqualFold as fallback

[0.2.24] - 2026-06-30

Changed

• Adaptive engine: __learned__ aggregate built from persisted training data — TrainFromDatalog now aggregates all coordinate samples per tool into the __learned__ key in coordIndex, so coordinate predictions survive server restart. Combined with the v0.2.23 fallback in predictCoord, agent_suggest now returns coordinate predictions for click/hover/move_mouse using the aggregate average from all training data, even before any runtime samples accumulate.

Changed files

• internal/actions/adaptive.go — TrainFromDatalog stores aggregated coords under __learned__ key per tool

[0.2.23] - 2026-06-30

Changed

• Adaptive engine: coord prediction fallback to __learned__ aggregate — predictCoord now falls back to the runtime-learned __learned__ aggregate coordinate when per-token samples in coordIndex are below the threshold of 3. This ensures agent_suggest returns coordinate predictions for click/hover/move_mouse even when the specific OCR tokens haven’t accumulated 3+ samples yet — the aggregate __learned__ accumulates across all invocations of the same tool.

Changed files

• internal/actions/adaptive.go — predictCoord now checks __learned__ in toolMap as fallback when tCount < 3, returning the aggregate coord instead of nil

[0.2.22] - 2026-06-30

Changed

• Adaptive engine: real timing_stats and success_rates — RecordResult is now called from every action tool’s defer with a captured start time, so timing_stats (mean, stddev, count, min, max) and success_rates per tool populate correctly. Previously RecordCommand was defined but never called, leaving both maps permanently empty.

Changed files

• internal/actions/datalog.go — removed Adaptive.RecordResult(tool, 0, ...) from LogToolCall (moved to per-action defer with real timing)
• internal/actions/chained.go — added start capture + RecordResult to LaunchAndWait and Hover
• internal/actions/keyboard.go — added start capture + RecordResult to KeyDown, KeyUp, KeyPress, TypeText
• internal/actions/mouse.go — added start capture + RecordResult to Click, MoveMouse, Scroll, Drag
• internal/actions/window.go — added time import + start capture + RecordResult to FocusWindow

[0.2.21] - 2026-06-30

Changed

• LogToolCall coverage: all 11 MCP action tools now instrumented — Added LogToolCall to key_down, key_up, focus_window, and launch_and_wait, completing adaptive engine training pair coverage for every non-query action tool. Previously 4 tools produced commands without OCR context pairs, leaving gaps in the training index.

Changed files

• internal/actions/keyboard.go — added LogToolCall("key_down", ...) to KeyDown, LogToolCall("key_up", ...) to KeyUp
• internal/actions/window.go — added LogToolCall("focus_window", ...) to FocusWindow
• internal/actions/chained.go — added LogToolCall("launch_and_wait", ...) to LaunchAndWait

[0.2.20] - 2026-06-30

Changed

• Adaptive engine: OCR bridge auto-complete in LogToolCall — LogToolCall now synchronously captures OCR after setting a pending training pair, ensuring every action produces a complete (ocr_before, tool, ocr_after) pair. Previously pairs only completed when the next explicit OCRScreen() call happened, causing all training sequences to cluster under “click”. Also added LogToolCall to Hover and MoveMouse which were missing it entirely.

Changed files

• internal/actions/datalog.go — LogToolCall auto-captures OCR after pending pair set
• internal/actions/chained.go — added LogToolCall("hover", ...) to Hover
• internal/actions/mouse.go — added LogToolCall("move_mouse", ...) to MoveMouse

[0.2.19] - 2026-06-30

Changed

• Keylogger rewrite: hooks → polling — Replaced WH_MOUSE_LL + WH_KEYBOARD_LL low-level hooks with GetAsyncKeyState polling loop (50ms ticker). Eliminates the system-wide input lag caused by the Go hook callback trampoline on every mouse event. The polling loop runs in a goroutine with no locked OS thread and no Windows message loop. Trade-off: scroll wheel events no longer detectable (acceptable cost for eliminating system-wide input lag).

Fixed

• CI lint failure — stale tools.md & uncategorized tools — scripts/gen-tools-doc.go was missing category entries for 4 tools (bridge_debug, introspection_analyze, task_begin, task_end), causing them to fall under “Uncategorized” and docs/tools.md to show 114 instead of 118 tools. The lint check (regenerate + diff) then failed, skipping the build job. Added "Introspection & Debugging" category, removed stale docs2/ staging output from the script, and regenerated docs/tools.md.

• yolo_dataset location inconsistency — removed stale yolo_dataset/ from repo root (empty train/val dirs). export_yolo_dataset now defaults to %APPDATA%\go-mcp-computer-use\yolo_dataset\ when output_dir is omitted. Added yolo_dataset/ to .gitignore to prevent future repo root drift.

[0.2.18] - 2026-06-29

Added

• Post-Task Introspection Engine (internal/actions/introspection.go) — three new MCP tools for task-aware self-improvement:
  • task_begin — marks task start with description, timestamps
  • task_end — closes task, mines insights from command_log between start/end: slowest tools, most failed tools, OCR stats, repeated command patterns, and improvement suggestions
  • introspection_analyze — browse completed task history with full insight data
  • Uses existing command_log + ocr_log tables — no new logging infra needed
  • task_log table added to datalog DB

Changed

• datalog_status now reports task_count in stats

[0.2.17] - 2026-06-29

Fixed

• OCR→Training bridge window — bridgeWindow increased from 3s to 30s. The OCR→AI→MCP→Click round trip regularly exceeded the original 3-second window, preventing training pair creation. Debugged via new bridgeBufferSize() and BridgeDebugInfo() diagnostic functions exposed through the bridge_debug MCP tool.

Added

• bridge_debug MCP tool — debug the OCR→command bridge state, showing recent OCR buffer contents, pending command, and timing info.

[0.2.16] - 2026-06-29

Added

• Adaptive Engine (internal/actions/adaptive.go) — pure Go statistical ML system with three components:
  • TimingTracker — rolling-window (N=100) per-tool statistics: mean, stddev, min, max. Auto-suggests adaptive delays based on historical execution time plus success-rate multiplier (1.5× by default, 3× when success rate < 50%).
  • SuccessTracker — per-tool success/failure ratios. Queried on every SuggestDelay() call to adjust timeouts.
  • SequencePredictor — TF-IDF-style word index from training_pairs. Given OCR text, tokenizes and scores each word→command mapping by historical success frequency. Returns ranked predictions with confidence (0.0–1.0) and sample size.
• MCP Resources (5) — auto-exposed to the AI client, read on every session context:
  • datalog://stats — current row counts for all four datalog tables
  • datalog://commands — 20 most recent command log entries
  • datalog://ocr — 10 most recent OCR snapshots
  • datalog://pairs — 20 most recent training pairs
  • adaptive://analysis — full adaptive engine analysis (timing stats, success rates, learned sequences)
• Agent MCP Tools (3) — AI-queryable loop for context-aware decisions:
  • agent_analyze — returns full timing stats, success rates, and top learned sequences for AI decision-making
  • agent_suggest — given OCR screen text, predicts the best next command ranked by confidence
  • agent_train — rebuilds the word→command index from current training_pairs table
• Auto training pair generation — passive OCR bridge creates triple (ocr_before, command, ocr_after) without slowing commands:
  • Ring buffer of last 5 OCR snapshots with timestamps
  • Every command auto-pairs with most recent OCR (within 3s window) as ocr_before
  • Next OCR snapshot completes as ocr_after
  • Command stored as {"tool":"name","args":"..."} JSON for robust parsing

Fixed

• datalog_query table name mismatch — switch-case expected short names ("commands", "ocr", "chains", "pairs") but the handler passed raw table names. Now accepts both forms as aliases.
• TrainFromDatalog JSON parsing — robust extractToolFromJSON helper handles both JSON {"tool":"..."} and plain string command values.

Changed

• Tool count — 111 → 114
• VERSION — bumped 0.2.15 → 0.2.16
• gen-tools-doc.go — added “Adaptive Agent” category
• LogCommand — now releases SQLite lock before OCR bridge to avoid deadlock with LogOCRSnapshot (no cross-lock ordering)
• LogTrainingPair — Command field stores structured {"tool":"name","args":"..."} JSON instead of raw args string

Documentation

• docs/tools.md — regenerated with 114 tools across categories including “Adaptive Agent”

[0.2.15] - 2026-06-29

Added

• Data logging database (internal/actions/datalog.go) — new SQLite DB at %APPDATA%/go-mcp-computer-use/datalog/datalog.db with four tables:
  • command_log — every chain/tool execution with args, success, duration, error text
  • chain_log — full chain executions with step counts, success/fail breakdown, chain JSON
  • ocr_log — OCR snapshots with full OCR text, word count, linked screenshot image path
  • training_pairs — OCR-before + command + OCR-after triples for ML sequence learning

• Automatic logging hooks — chains, individual commands, and OCR calls are logged automatically via goroutines with no performance impact on the main execution path.

• Three new MCP tools:
  • datalog_query — query any table (commands, chains, ocr, pairs) with filters (source, tool, success), returns rows as JSON
  • datalog_export — export training pairs as JSON array for downstream ML training pipelines
  • datalog_status — get row counts for all four tables

Changed

• VERSION — bumped 0.2.14 → 0.2.15
• Tool count — 108 → 111

[0.2.14] - 2026-06-29

Added

• NormalizedElement coordinate system — element positions stored as window-relative 0.0–1.0 fractions via WindowNormalizer in internal/actions/dpi.go. Layout-independent across screen resolutions and multi-monitor. Includes GetDPIScaleForWindow, Normalize/Denormalize helpers, and ProportionalRegion for computing screen-absolute OCR crops as a percentage of the active window.

• OCRProportionalWindowRegion — new OCR function in ocr.go that takes a window handle + proportional fractions, eliminating hardcoded pixel crops.

• Auto-expand tiny OCR regions — FindTextAndClick now detects crops <300px in any dimension and falls back to a generous 5%–95% of the active window. Prevents “Desktop not found” failures on small fixed-pixel regions.

• Window context in ONNX detection — DetectionOutput carries WindowTitle and Normalized []NormalizedElement alongside absolute coordinates. Computed per-active-window during inference.

• Training schema migration — training_samples table gains window_rect TEXT and normalized_coords TEXT columns. saveTrainingSampleDirect accepts and persists both normalized coords and window rect JSON.

Fixed

• NormalizeElement missing Class/Confidence copy — WindowNormalizer.NormalizeElement returned a NormalizedElement with zeroed Class and Confidence fields. Exposed by round-trip test (TestNormalizeElementRoundTrip). Now copies both fields before returning.

Changed

• Watcher cache — CachedDetection includes Normalized elements alongside absolute Elements. Training samples from watcher snaps now carry window rect context.
• VERSION — bumped 0.2.13 → 0.2.14

Tests

• Coordinate system tests — dpi_test.go with 6 tests covering: normalize/denormalize round-trip, coordinate bounds (corners, center, size), proportional region math, NormalizeElement class/confidence round-trip, and zero-size window edge case.

[0.2.13] - 2026-06-29

Fixed

• ONNX detection timeout (65s → 599ms) — root cause was not DLL incompatibility but performance:
  • parseYOLOOutput passed all 8400 raw detections through NMS at O(n²) = ~15M iterations
  • MemoryStoreDetectionElements called MemorySet 5507 times — each a separate SQLite INSERT with global mutex lock
  • Fixed: parseYOLOOutput now applies confidence threshold early (0.25), pre-filtering to ~50 boxes before NMS
  • Fixed: MemoryStoreDetectionElements rewritten with batched SQLite inserts in a single transaction, capped at 200 elements

Changed

• ONNX Runtime DLL updated — v1.20.1 → v1.26.0 to support opset 22 (required by yolo11n.onnx). Limited opset support warning is non-fatal.

[0.2.12] - 2026-06-29

Fixed

• Release binaries crash with STATUS_ILLEGAL_INSTRUCTION — Zig cc on GHA runners defaults to -march=native, generating CPU-specific instructions incompatible with older machines (Pentium Gold G5400). Pinned -mcpu=x86_64_v2 in CGO_CFLAGS so binaries run on any x86-64 CPU.
• CGO_LDFLAGS also needs -mcpu=x86_64_v2 — actions/setup-go@v5 overrides CGO_LDFLAGS with -O2 -g, dropping the CPU baseline. Both CGO_CFLAGS (compile) and CGO_LDFLAGS (link) now pin -mcpu=x86_64_v2.

Changed

• scripts/build.ps1 — added CGO_CFLAGS with -mcpu=x86_64_v2 baseline for portable builds
• .github/workflows/release.yml — same CPU baseline pin in both CGO_CFLAGS and CGO_LDFLAGS, plus -fno-sanitize=all and -Wno-error

[0.2.11] - 2026-06-29

Added

• scripts/gen-tools-doc.go — parses internal/server/server.go for mcp.AddTool calls, generates docs/tools.md with categorized 108-tool listing. CI validates freshness on every push/PR.
• scripts/push-and-release.ps1 — one-shot auto-release: reads VERSION, commits with changelog body, tags, pushes, waits for release workflow, downloads binary, replaces mcp-server.exe, restarts OpenCode Desktop as admin.
• docs/tools.md — auto-generated tool reference doc (never stale).
• docs/security.md, docs/configuration.md, docs/build.md, docs/architecture.md, docs/accessibility.md — split from monolithic README.
• Weekly module maintenance — .github/workflows/mod-maintenance.yml runs go get -u ./... + auto-PR every Monday.
• CI: go mod tidy validation — fails if go.mod/go.sum drifts from tidy state.

Changed

• README.md — collapsed 383→92 lines, links to focused docs/ split.
• Root docs moved — plan.md, todo.md, backlog.md, known-issues.md, CHANGELOG.md relocated to docs/.
• CGO mandatory — removed all -NoCGO flags, pure-Go fallback paths, and optional-CGO language across 9 files. release.yml now produces a single mcp-server.exe (CGO+Zig).
• Release workflow — single binary output, no -cgo suffix variant.
• scripts/build.ps1 — removed -NoCGO switch, always requires Zig cc.

Documentation

• README split — large sections moved into focused docs for maintainability.
• All NoCGO references removed — across plan.md, adr-002, comparison-vs-alternatives.md (formerly comparison-vs-windows-recall.md), ci-cd-pipeline.md, build.md, README.md.

[0.2.10] - 2026-06-29

Documentation

• Systematic doc audit — fixed 90 stale statements across 12 docs: tool counts (103→108 restored from actual registrations), version refs, CGO/dependency claims, category counts, missing tool listings, completed Slice 4 checkboxes, stale future-tool lists
• Architecture guide — added Part 6 to computer-use-guide: layered agent stack (LLM→MCP→Controller→Perception→Memory→World), ML vision + spatial memory, division of responsibilities, convergence of LLM+MCP+ML
• Source fix — server.go tool count hardcode corrected 103→108 to match actual registrations
• Config auto-start — watcher_auto_start config created on dev machine

Changed

• VERSION — bumped 0.2.9 → 0.2.10

[0.2.9] - 2026-06-29

Added

• scripts/build.ps1 — unified build script with -UseZig flag for CGO-enabled builds
• CI/CD: CGO + Zig cc build pipeline — CI now runs two jobs: no-CGO lint+build and CGO+Zig build. Release workflow produces both mcp-server.exe (no CGO) and mcp-server-cgo.exe (with ONNX support).
• Zig 0.16.0 support — scripts/install.ps1 updated to download Zig 0.16.0

Documentation

• README.md — documented CGO requirements for ONNX tools with Zig cc build instructions
• known-issues.md — B13: ONNX tools require CGO (documented workaround)
• Tool count docs updated — all docs updated to 108 tools, stale CGO claims corrected

[0.2.8] - 2026-06-29

Added

• key_down / key_up MCP tools — separate key hold/release for game-play sequences. Chains can now hold movement keys while dragging camera and pressing abilities, all server-side with no round-trip latency. KeyDown("W") holds the key, KeyUp("W") releases. Full VK support including modifiers, letters, digits, and special keys.

• keylogger_start / keylogger_stop / keylogger_status MCP tools — record real keyboard + mouse input (keys, clicks, drags, moves, scroll) via low-level Windows hooks (WH_KEYBOARD_LL + WH_MOUSE_LL). Output is a chain-compatible JSON sequence for AI replay. Includes timing-accurate delays between events. Mouse clicks auto-detect drag vs click by distance/time thresholds. Mouse moves throttled to meaningful position changes.

• sendVKPress helper with 50ms inter-key delay — KeyPress, TypeText, sendCharWithVK now use sendVKPress(vk) which inserts a 50ms time.Sleep between key down and key up. Fixes game engines and DirectInput applications that miss instant down/up sequences (character switch hotkeys 1-4, ability keys).

Fixed

• warnElevated false positive when both server and target are elevated — warnElevated() only checked if the foreground window was elevated, not the MCP server itself. If both are elevated (server running as Admin targeting an admin game), SendInput keyboard works fine, but the check falsely blocked it. Added isSelfElevated() — only blocks keyboard when server is non-elevated AND target is elevated.

• KeyPress modifier ordering — ["CTRL", "C"] sent C via Unicode first, then pressed Ctrl down, then released Ctrl. The key arrived before the modifier was held. Rewrote to process keys in order: modifiers are pressed immediately, target keys are sent while held, all modifiers released in reverse at end.

• Keyboard input uses VK codes instead of KEYEVENTF_UNICODE — KEYEVENTF_UNICODE synthesizes WM_CHAR messages, which many applications ignore (game engines, terminals, code editors, browser input fields). Rewrote all keyboard functions to use VK codes:
  • TypeText and TypeAndSubmit use sendCharWithVK() — maps each character to its VK code + Shift state using charToVK table (US keyboard layout). Letters, digits, punctuation all handled.
  • KeyPress sends all keys (letters, digits, special keys) as VK codes. Modifier combos like Ctrl+C now work correctly: Ctrl down → VK_C → Ctrl up.
• Drag incremental movement — was sending a single jump from start to end (mouseDown → teleport → mouseUp). Games and map UIs ignored this as a teleport. Now sends 5–50 incremental steps with 5ms delays, proportional to distance. Map panning now works correctly.

Changed

• sendUnicode removed — no longer used. All keyboard input via VK codes.
• Tool count: 103 → 108 (added key_down, key_up, keylogger_start, keylogger_stop, keylogger_status).

Documentation

• Elevation & UIPI section in README — explains admin vs non-admin behavior
• Known issues B11, B12 — documented keyboard issues and fixes

[0.2.7] - 2026-06-29

Added

• Statistical prior model (priors_stats tool) — Go-native “training” without Python. Element frequency + position distributions are learned per window from collected training samples. Priors boost confidence for expected elements (e.g., “laptop” in browser windows) and suppress unlikely ones (e.g., “tv” in code editor). Position outliers beyond 3σ are penalized.
• Prior-based confidence adjustment — ONNXDetect now calls AdjustConfidenceWithPriors() after NMS, adjusting every detection’s confidence based on learned per-window statistics. Gated by prior_adjustment config field (default: true).
• export_yolo_dataset tool — exports unused training samples (signal_level >= 1) as a YOLO-format dataset (images + normalized label files + train/val split + dataset.yaml). Users with Python can train externally via Ultralytics.
• training_cleanup_noise tool — deletes low-signal (signal_level=0) samples older than a threshold. Supports dry_run=true to preview deletions. Frees disk space from watcher noise frames.
• training_enabled config field — when set to false, disables all auto-save training snapshots (both from actions and the background watcher). Default: true.
• prior_adjustment config field — when set to false, disables prior-based confidence adjustment in ONNXDetect. Default: true.
• Priors auto-populated on save — every training sample save (raw or watcher) also updates element priors via UpdatePriorsFromDetections. Negative samples (zero elements) update frequency denominators.

Changed

• set_config tool — runtime config changes without restart. Accepts: training_enabled (stop/start background data collection), prior_adjustment, verify_bounds, log_level, watcher_enabled (start/stop watcher), watcher_interval_seconds (change polling frequency live). Changes persist to disk immediately. Enables users to disable data collection and control the watcher mid-session for privacy or debugging.
• watcher_auto_start / watcher_interval_seconds config — watcher_auto_start: true starts the background watcher on server boot with the configured interval. Default: false.
• Tool count: 99 → 103 (added priors_stats, export_yolo_dataset, training_cleanup_noise, set_config).

Fixed

• SendInput silently dropping mouse clicks — the input struct in mouse.go had an orphan _ [8]byte padding field, making unsafe.Sizeof = 48 bytes. Windows sizeof(INPUT) on x64 is 40 bytes. SendInput returns 0 when cbSize doesn’t match, so SetCursorPos moved the cursor but the click event never fired. Removed the extra padding — struct is now exactly 40 bytes.
• Network struct layout mismatches — IP_ADDR_STRING was missing _ [4]byte trailing padding (44→48 bytes). IP_ADAPTER_INFO and FIXED_INFO used [260/132]uint16 for char arrays (2x Windows size, shifting every subsequent field). Changed to [260/132]byte and added alignment padding after DhcpEnabled.
• All Windows API structs verified — audited every struct passed to Win32 via unsafe.Pointer in internal/actions/: mouseInput (32B ✓), input (40B ✓), point (8B ✓), keyboardInput (24B ✓), inputKbd (40B ✓), BITMAPINFOHEADER (40B ✓), RECT, MONITORINFOEXW, DEVMODEW, MEMORYSTATUSEX, SYSTEM_POWER_STATUS, PROCESSENTRY32W, LASTINPUTINFO, VARIANT, UiaRect, WinRect — all match Windows x64 sizes.

Changed

• Drag rewritten for raw input games — replaced SetCursorPos (invisible to DirectInput/raw input) with SendInput + MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE. Coordinates normalized to 0–65535 range. Game engines using raw input now see the movement between mouse-down and mouse-up.

Documentation

• Elevation & UIPI section added to README — explains admin vs non-admin behavior (keyboard warns, mouse silently fails), how to run elevated, and reassurance that normal apps work fine without elevation.

[0.2.6] - 2026-06-28

Added

• Training data pipeline (training_save_sample, training_list_samples, training_stats, training_mark_used) — persistent screenshot + ONNX detection storage for model fine-tuning. Images saved to categorized folders (raw/click/, raw/type/, raw/navigate/, raw/ocr/, raw/general/, watcher/elements_found/, watcher/no_elements/) with metadata in samples.db. Each sample carries a task_prompt string that the ML learns to predict during training.
• Auto-save on every UI action — click, type, scroll, drag, hover, key_press, type_and_submit, select_all_and_type, browser_navigate, browser_search, open_url, and find_text_and_click handlers (both direct MCP and chain steps) automatically capture a screenshot + ONNX detection + save to raw/{category}/ with the action description as task_prompt.
• find_ui_element tool — three-layer cascading element locator: checks memory first (cached ONNX detections by window+label), runs ONNX detection with label matching, falls back to OCR for text elements. Stores findings in memory for reuse. Saves training samples (positive + negative).
• Memory-backed element caching — every ONNXDetect call auto-stores detected elements as memory facts (memory_set, scope ui, keyed ui:{window_title}:{class}) with 1-hour TTL. AI can query memory for known element locations without re-running ML.
• Quality/signal filtering — every training sample gets a signal_level (0=noise, 1=elements found, 2=elements+task context). training_list_samples accepts min_signal filter. Noise samples (watcher frames with zero elements) are flagged for discard.

Changed

• Restructured training directories — from flat samples/{cat}_{ts}.png to raw/{cat}/{ts}.png + watcher/{cat}/{ts}.png layout. Database renamed from training.db to samples.db.
• Watcher save path — frames now save to watcher/elements_found/ or watcher/no_elements/ instead of flat references/ dir.
• ONNXDetect no longer auto-saves — removal of inline saveTrainingSampleDirect in ONNXDetect to avoid caller confusion. Watcher handles persistence; explicit calls handle the rest.

[0.2.5] - 2026-06-28

Fixed

• memory_set schema validation — MemorySetArgs.Value any generated "value": true in JSON Schema, which OpenCode’s MCP validator rejected. Fixed with explicit InputSchema using json.RawMessage + description-only schema.
• close_window Win32 API — was calling ShowWindowAsync(hwnd, 0x10) but 0x10 = WM_CLOSE is not a ShowWindow command. Changed to PostMessageW(hwnd, WM_CLOSE, 0, 0).
• onnx_status global state bug — used global modelsDir which was empty when InitONNX failed. Now calls getModelsDir() directly.

Added

• Background watcher (onnx_watch_start/stop/status/cache) — goroutine that periodically captures screen, runs ONNX detection, caches last 20 results, and auto-saves reference PNGs when detection returns zero elements.
• savePNG auto-save in detection — onnx_detect now saves a ref_<ts>.png to %APPDATA%/go-mcp-computer-use/models/references/ when detection returns zero elements (AI confusion signal).
• focus_window_by_title — finds window by title, focuses, and clicks title bar to ensure activation.
• Browser automation — browser_focus_url_bar, browser_new_tab, browser_navigate, browser_search.
• File Explorer automation — explorer_focus, explorer_open_path.
• uia_warmup config field and async UIA warmup on startup.

Changed

• Eliminated Python dependency entirely — removed convertYoloToONNX(), detectWithPython(), pythonDetectResult struct, os/exec, bytes, strings imports.
• Switched YOLO model — from HuggingFace best.pt (PyTorch, 57 MB, 7 UI classes) to Ultralytics pre-exported yolo11n.onnx (10.9 MB, 80 COCO classes).

[0.2.0] - 2026-06-27

Changed

• v0.2.x branch baseline — cut from v0.1.11 as starting point for v0.2 development. All subsequent changes on this branch increment as +0.0.1 (v0.2.1, v0.2.2, etc.).

[0.2.1] - 2026-06-27

Added

• chain tool — sequential step executor that runs multiple tools server-side without round trips. Supports tool (call any registered tool), wait (sleep N ms), and capture (save step output as `` for use in subsequent steps). Error modes: stop (halt on first error, default) or skip. Global timeout. 40+ tools dispatched.
• Variable substitution — `` in string args is replaced with captured output from earlier steps.
• ChainFromJSON — convenience entry point for programmatic chain execution from JSON string.

[0.2.4] - 2026-06-28

Added

• memory_set / memory_get / memory_search / memory_list / memory_forget tools — SQLite-backed memory store using modernc.org/sqlite (pure Go, zero CGO). Database at %APPDATA%/go-mcp-computer-use/memory.db with WAL mode, FTS5 full-text search, auto-syncing triggers, TTL support, scope isolation, and tag filtering.
• layout_validate tool — validates stored UI element layouts against the current screen. Checks window existence, position drift (with tolerance), and OCR keyword verification around element coordinates. Returns per-element confidence (ok/drifted/stale) with adjusted coordinates.
• template_store / template_find / template_list / template_forget tools — self-growing template library. template_store auto-crops a 48×48 PNG template around a coordinate from the current screen and stores it in the element_templates table. template_find uses NCC template matching (find_image) to relocate the element visually on the current screen, returning coordinates and drift. Hit count auto-increments on each successful find, enabling the system to self-train over time.
• onnx_status / onnx_detect / onnx_download tools — ONNX ML backend for UI element detection. onnx_status checks runtime and model availability. onnx_detect runs YOLO11s inference on a screenshot or full screen to detect UI elements (button, textbox, checkbox, dropdown, icon, tab, menu_item) with bounding boxes and confidence scores. Uses github.com/yalue/onnxruntime_go for native ONNX Runtime support. Requires manual download of onnxruntime.dll and model files. Falls back gracefully when runtime/models are missing.
• focus_window_by_title tool — focus management for reliable keyboard input. Finds a window by title, focuses it, and clicks its title bar to ensure activation.
• ChainStep.FocusWindow field — chain steps can specify focus_window: "window title" to auto-focus and activate the window before executing the step. The chain executor handles window lookup, focus, title bar click, then runs the step.
• browser_focus_url_bar / browser_new_tab / browser_navigate / browser_search tools — generic browser automation (Firefox, Chrome, Edge, Brave, Opera). browser_focus_url_bar focuses the URL bar (Ctrl+T for Firefox, Ctrl+L for others). browser_new_tab opens a new tab (Ctrl+T). browser_navigate opens a new tab and navigates to a URL. browser_search opens a new tab and performs a search query. Backed by BrowserFocusURLBar, BrowserNewTab, BrowserNavigate, BrowserSearch in internal/actions/browseruse.go — reusable composite functions that import existing modules instead of duplicating logic.
• explorer_focus / explorer_open_path tools — File Explorer automation. explorer_focus finds and activates an existing File Explorer window by title. explorer_open_path opens explorer at a given path, reusing existing windows when possible (Ctrl+L + path) or launching a new one. Backed by ExplorerFocus, ExplorerOpenPath, ExplorerNavigateTo in internal/actions/windowexploreruse.go.

Changed

• Replaced firefox_focus_url_bar — removed Firefox-specific function from chained.go. Replaced with generic browseruse.go that detects browser type from window title and uses browser-specific keyboard shortcuts (Ctrl+T for Firefox URL bar, Ctrl+L for Chrome/Edge).
• Refactored FocusWindowByTitle — now delegates to shared focusAndActivateWindow helper, reducing duplication across browser, explorer, and generic focus code paths.

Removed

• FirefoxFocusURLBar — removed from internal/actions/chained.go. Superseded by BrowserFocusURLBar. Tool name changed from firefox_focus_url_bar to browser_focus_url_bar.

[0.2.3] - 2026-06-28

Fixed

• TypeAndSubmit Enter via KeyPress — appended \r used sendUnicode(0x0D) which sends the CR character via KEYEVENTF_UNICODE, unreliable in Firefox/browser address bars. Replaced with KeyPress([]string{"ENTER"}) with a 50ms pause, matching the same code path used by the key_press handler.

[0.2.2] - 2026-06-28

Added

• poll step type — polls OCR at every_ms interval until ocr_contains text is found or timeout_ms elapses. Syntax: {"poll": {"every_ms": 1000, "timeout_ms": 30000, "ocr_contains": "Submit"}}.
• if step type — OCR checks for ocr_contains text, executes then or else branch. Syntax: {"if": {"ocr_contains": "Error", "then": [...], "else": [...]}}.
• loop step type — repeats sub-steps times iterations. Syntax: {"loop": {"times": 5, "steps": [...]}}.
• StepResult.Steps — nested step results for if/loop sub-steps, visible in chain output.
• UIA warmup at server startup — pre-initializes COM and creates/releases a UIA instance, absorbing the one-time 15-42s cold-start cost so handlers respond instantly.
• WarmupUIA() — exported function to pre-warm COM/UIA at server startup.

Fixed

• StepResult.Index always 0 — execWait/execTool created fresh StepResult structs discarding the loop index. Index is now set after the switch.
• SelectAllAndType uses VK codes — sendUnicode(0x01) used KEYEVENTF_UNICODE (VK_PACKET) which doesn’t trigger select-all in most apps. Replaced with sendVK(VK_CONTROL) + sendVK(VK_A) for reliable Ctrl+A.
• Variable substitution supports dotted paths — regex [a-zA-Z0-9_]+ didn’t match ``. Updated to [a-zA-Z0-9_.]+ with resolveVarPath() for nested map lookups.
• SelectAllAndType elevated warning — now calls warnElevated() before sending input, preventing silent drops on admin windows.

[0.1.11] - 2026-06-27

Added

• VERSION file + ldflags — single source of truth at project root, injected via -X main.Version, replaces hardcoded string
• CI/CD pipeline — .github/workflows/ci.yml (build + vet on push/PR), .github/workflows/release.yml (tag-triggered GitHub Release with binary + SHA256 + changelog)
• .govetallow — documents COM/WinRT unsafe.Pointer conventions for vet policy
• scripts/lint.ps1 — local CI runner: vet + build + tests

Changed

• COM types — all interface pointers stored as unsafe.Pointer instead of uintptr: uiaAuto.p, uiaCondition.p, uiaElement.p, uiaElementArray.p, bstrToGo parameter, getCurrentPattern return type
• vtblMethod — rewritten with unsafe.Pointer parameter + unsafe.Add, satisfies vet’s unsafeptr checker
• Syscall output params — all local variables receiving COM pointers via SyscallN declared as unsafe.Pointer instead of uintptr
• GUID literals — all 14 windows.GUID values in winrt.go use keyed fields
• CI workflows — use scripts/lint.ps1 instead of raw go vet

[0.1.10] - 2026-06-27

Fixed

• Keyboard VK-coded keys (Enter, Backspace, Tab, Ctrl+letter) sent via sendKey/KeyPress were silently dropped by the system — only KEYEVENTF_UNICODE path worked. Rewrote keyboard handling to send all keys through KEYEVENTF_UNICODE where possible: special keys map to Unicode control characters (Enter=0x0D, Backspace=0x08, Ctrl+A-Z=0x01-0x1A). VK fallback only for non-printable keys (arrows, F-keys, Insert, etc.)
• TypeAndSubmit and SelectAllAndType now use Unicode path instead of VK-coded KeyPress for Enter and Ctrl+A

[0.1.9] - 2026-06-27

Added

• B9: UIPI elevation detection for keyboard input (TypeText, KeyPress) — returns clear warning when foreground window is elevated (admin), instead of silently dropping input

[0.1.8] - 2026-06-27

Fixed

• B3: list_displays only returned primary monitor — monitorEnumProc gated on MONITORINFOF_PRIMARY flag, skipping all non-primary displays

[0.1.7] - 2026-06-27

Fixed

• B4: uia_get_text / uia_invoke no longer crash MCP transport — GetCurrentPattern nil check added before pattern operations

[0.1.6] - 2026-06-27

Fixed

• B2: list_audio_devices returns [] instead of null — empty PowerShell output produced nil slice which serialized as JSON null

[0.1.5] - 2026-06-27

Fixed

• B6: Wait() calculation was 1 million times too long — NtDelayExecution argument was -(ns * 10000) instead of -(ms * 10000), causing hover (and any tool calling Wait) to block for hours instead of milliseconds

[0.1.4] - 2026-06-27

Fixed

• B1: get_brightness returns clear “brightness not supported on this display” instead of parse error when display doesn’t support WMI brightness control (desktop monitors)

[0.1.3] - 2026-06-27

Fixed

• B5: screenshot_element now clamps off-screen window coordinates to screen bounds instead of rejecting them (e.g., windows with x=-8 from Aero Snap)
• Multi-monitor: ScreenSize() now returns virtual desktop dimensions (SM_CXVIRTUALSCREEN/SM_CYVIRTUALSCREEN) instead of primary monitor only, fixing coordinate validation across multiple displays

[0.1.2] - 2026-06-27

Fixed

• UIA COM and OCR WinRT apartment model conflict: changed RoInitialize from RO_INIT_SINGLETHREADED to RO_INIT_MULTITHREADED so both UIA and OCR use MTA on the same thread, preventing RPC_E_CHANGED_MODE error

[0.1.1] - 2026-06-27

Added

• Native WinRT COM OCR — replaces PowerShell OCR with direct COM calls: StorageFile.GetFileFromPathAsync → OpenAsync → BitmapDecoder.CreateAsync → GetSoftwareBitmapAsync → OcrEngine.RecognizeAsync. Zero CGO, no Windows SDK needed.
• Native COM UI Automation — replaced PowerShell UIA with direct COM calls to UIAutomationCore.dll (IUIAutomation, IUIAutomationElement, conditions, patterns). All operations via native COM.
• WinRT COM infrastructure (winrt.go) — HSTRING management, RoInitialize, RoGetActivationFactory, IAsyncInfo polling, COM helpers
• OCR falls back to PowerShell if native COM fails

Changed

• All OCR and UIA operations now use native COM instead of PowerShell — 2-8x faster
  • OCR full screen: 653→292ms (2.2x)
  • OCR region 400×400: 542→68ms (8x)
  • find_text_and_click: 809→275ms (2.9x)
• comRelease signature changed from uintptr to unsafe.Pointer for unified COM cleanup
• ADR-002 updated: project now uses native COM/WinRT, not just Win32 API

Fixed

• WindowsGetStringRawBuffer signature: actual DLL export returns buffer pointer in RAX (2 params), not as out parameter (3 params) — MSDN docs differ from Win10 10.0.26100 behavior
• All vtable reads: corrected *(*[N]uintptr)(obj) pattern (reads object data) to vtblMethod() (reads actual vtable entries)
• OCR PowerShell script: properly loads WinRT types via WindowsRuntimeSystemExtensions.GetAwaiter with MakeGenericMethod, fixing OCR on systems where WinRT async extension methods don’t resolve in PowerShell 5.1
• Go raw string literal: avoids backtick in IAsyncOperation1 by using -like` wildcard matching

[0.1.0] - 2026-06-27

Added

• Screenshot (full + region) via GDI BitBlt
• Mouse control: click, move, scroll, drag, hover
• Keyboard input: type, key_press, type_and_submit, select_all_and_type
• OCR via Windows.Media.Ocr with language support
• Template matching via normalized cross-correlation
• find_text_and_click, wait_for_text, click_menu_item, launch_and_wait
• Screen recording (duration_ms + interval_ms → base64 frames)
• Window management: list, focus, find, move, resize, minimize, maximize, restore, close, get_state
• Audio devices: list playback/recording, set default
• Clipboard: get/set with retry + timeout
• System: volume, mute, brightness, battery, disk, DPI, display info, uptime, idle
• Network: hostname, IPs, DNS, gateway, ping
• Processes: list, launch, kill
• Power: shutdown, restart, sleep, hibernate
• Per-monitor DPI awareness
• UI Automation via PowerShell: find elements, get text, invoke
• get_display_modes tool (69th tool) — enumerate all display modes
• Config file: ~/.config/go-mcp-computer-use/config.json
• Install script: scripts/install.ps1 with Zig cc support

Changed

• syscall hardening: ptr() helper for safe unsafe.Pointer conversion
• performance optimizations across all action modules
• README with comprehensive tool listing and security warning
• MCP client configs documentation for 19 agents

Security

• Added SECURITY WARNING section to README detailing dangerous capabilities