Backlog — Every Desktop Ability a Human Has

Goal: go-mcp-computer-use must be able to do literally everything a human can do on a Windows PC. Key: prompt engineering to chain tools and manage which model gets which subset.

How to Read

• HAVE = implemented as standalone MCP tools (counted by individual tool names)
• NEXT = high-impact, feasible additions
• FAR = possible but lower priority or complex
• Items within a section ordered roughly by priority
• Note: Some sections include “built-in features” (not standalone tools) marked as such — these are capabilities baked into existing tools, not separately callable MCP endpoints

---

Key Constraints (What is NOT enabled)

• No Easing — only straight-line interpolation is supported for drag (from→to); there are no bezier or eased pointer paths.
• No CLI Help — RESOLVED in 0.3.7: --help, -h, --version, and --license were added in v0.3.7. Prior to 0.3.7 the binary had no help or version flags.
• No Robotgo — the server uses raw Windows syscalls (user32!SetCursorPos, user32!SendInput) rather than the robotgo library (see ADR-002, Windows Automation Strategy).

---

1. VISION — See What’s on Screen

HAVE (13 tools)

• screenshot — full screen or region → base64 PNG
• get_pixel_color — hex color at x,y
• get_screen_size — virtual screen dimensions
• get_screen_dpi — per-monitor DPI + scale %
• get_dpi_for_point — DPI and scale at a specific screen coordinate (multi-monitor aware)
• ocr — extract text via Windows.Media.Ocr (any language)
• find_image — NCC template matching with ONNX + OCR fallback
• find_all_images — NCC template matching with ONNX + OCR fallback, returns all matches
• ocr_languages — list installed OCR languages (so agent knows what’s available)
• record_screen — frame polling at interval → base64 frame array
• ocr_window — extract text from a specific window by handle
• ocr_active_window — extract text from the current foreground window
• image_diff — pixel-level diff between two screenshots (detect changes)

NEXT

| Tool | Why | |——|—–| | image_histogram | color histogram analysis (detect dark/bright screens) | | match_template_multi | multi-pass NCC at different scales (handle DPI variation) | | color_region_analysis | average/min/max/std dev color in a region (not just single pixel) |

FAR

| Tool | Why | |——|—–| | detect_ui_elements | use OCR bounding boxes to classify buttons/text fields/lists | | barcode_reader | decode QR/DataMatrix/Code128 from screen region | | face_detection | WinRT face detection via Camera | | screen_recording_video | encode frames to actual video file (mp4) | | real_time_stream | WebSocket stream of frames for live agent view | | color_detection | find all pixels of a given color on screen | | snap_diff | compare screenshot to a reference “golden” image | | ocr_gpu_accelerated — GPU-backed OCR backend | 10x faster text extraction (from DesktopCtl) | | screen_tokenize — structured UI token output | deterministic CLI-style UI description (from DesktopCtl) |

---

2. MOUSE — Point and Click

HAVE

• click — left/right/middle, single/double at x,y
• move_mouse — move cursor to x,y
• scroll — wheel up/down/left/right (clicks + horizontal)
• drag — click-hold from→to
• hover — move + wait 300ms (for tooltips)
• get_cursor_position — current x,y

NEXT

| Tool | Why | |——|—–| | click_hold / release — separate hold/release | complex drag-and-drop, slider manipulation | | scroll_smooth — pixel-based scrolling | precise scroll in lists/canvases | | drag_relative — drag by (dx, dy) from current pos | relative drag gestures |

FAR

| Tool | Why | |——|—–| | mouse_gesture — recognize shape (L, V, Z) | gesture-based commands | | click_all_matches — click every occurrence on screen | dismiss all notifications, close all tabs | | right_click_menu — right-click + get menu items | context menu interaction | | multi_touch — simulate touch gestures (pinch, swipe) | tablet/touch scenarios | | pen_stylus — WinRT pen simulation | drawing, handwriting | | background_click — click without stealing cursor | non-disruptive automation (from Cua) | | background_type — type without focusing target window | invisible input (from Cua) |

---

3. KEYBOARD — Type and Command

HAVE (11 tools)

• type — send text string
• key_press — key combos (Ctrl+C, Alt+Tab, etc.)
• key_down — hold individual key
• key_up — release individual key
• type_and_submit — type + Enter
• select_all_and_type — Ctrl+A + type
• get_keyboard_layout — current input language (e.g. en-US)
• set_keyboard_layout — change input language
• keylogger_start — start recording input
• keylogger_stop — stop recording, return sequence
• keylogger_status — check if recording active

NEXT (4 items)

| Tool | Why | |——|—–| | type_character_by_character — type with per-char delay | simulate human typing for apps that buffer input | | type_with_modifiers — e.g. type “hello” while holding Shift | advanced text entry | | get_keyboard_state — which modifier keys are pressed | check CapsLock/NumLock/ScrollLock state | | set_keyboard_state — toggle CapsLock/NumLock | fix common input issues |

FAR

| Tool | Why | |——|—–| | text_from_clipboard_paste — paste instead of type | faster, avoids IME/input issues | | text_file_input — type contents of a text file | paste large text without clipboard | | ime_text_input — send text through IME composition | Japanese/Chinese input methods | | send_keys_advanced — SendKeys-style with pauses | legacy app compatibility |

---

4. WINDOWS — Manage Screen Real Estate

HAVE (15 tools)

• list_windows — all visible windows (handle, title, PID)
• focus_window — bring to foreground
• find_window — by title substring
• wait_for_window — poll until window appears
• move_window — set x,y,w,h
• minimize_window — minimize
• maximize_window — maximize
• restore_window — restore from min/max
• close_window
• get_window_state — visible, minimized, maximized, fullscreen, position, z_order
• get_active_window — current foreground window info (handle, title, PID, bounding rect)
• focus_window_by_title — find by title and focus (convenience wrapper)
• set_window_lock — lock chain to a window (v0.2.42)
• clear_window_lock — unlock chain (v0.2.42)
• screenshot_element — screenshot a specific window

NEXT

| Tool | Why | |——|—–| | snap_window_left / snap_window_right — snap to half screen | common multi-window workflow | | snap_window_top / snap_window_bottom / snap_window_corner | quarter-screen layouts | | set_window_z_order — bring to top/bottom/above/below | reorder windows | | set_window_transparency — alpha blend (SetLayeredWindowAttributes) | peek through windows | | set_window_title — change window title | identification | | cascade_windows / tile_windows — arrange all windows | classic Windows arrange | | find_window_by_pid — get window owned by a process | process→window mapping | | save_window_layout / restore_window_layout — save/restore named window groups | workspace management |

FAR

| Tool | Why | |——|—–| | get_window_children — enumerate child windows | UI Automation for controls | | window_screenshot_multi — screenshot all windows at once | full desktops | | scroll_window — scroll content inside a window | scroll without focus | | click_window_at — click relative to window origin | element positioning | | set_window_always_on_top | floating windows | | get_window_class_name — Win32 class name | identify control types |

---

5. VIRTUAL DESKTOPS — Multiple Desktops

HAVE

— none

NEXT

| Tool | Why | |——|—–| | list_virtual_desktops | enumerate all desktops | | switch_virtual_desktop (by index or GUID) | move between workspaces | | create_virtual_desktop | new workspace | | remove_virtual_desktop | delete workspace | | move_window_to_desktop | move window to different desktop | | get_current_desktop | which desktop is active |

FAR

| Tool | Why | |——|—–| | pin_window_to_all_desktops | show on every desktop | | get_desktop_wallpaper | background image |

---

6. PROCESSES — Run and Manage Apps

HAVE

• list_processes — name, PID, threads, parent PID
• launch_app — ShellExecute (open verb)
• kill_process — TerminateProcess by PID

NEXT

| Tool | Why | |——|—–| | launch_app_with_args — specify arguments | run commands with params | | launch_app_hidden — no window | run background processes | | launch_app_as_admin — UAC elevation | admin tasks | | launch_app_with_env — set env vars for child proc | custom environment | | wait_for_process_exit — wait until PID exits | sequential task chaining | | launch_and_wait_exit — launch + wait for exit | run-and-collect | | get_process_info — memory, CPU, command line | process details | | watch_process — watch process by name/PID, trigger chain on exit | process monitoring |

FAR

| Tool | Why | |——|—–| | set_process_priority — idle/below_normal/normal/above_normal/high/realtime | CPU management | | suspend_process / resume_process | freeze/thaw | | get_process_threads — list threads | diagnostics | | run_as_user — impersonation | different user context | | create_process_group / kill_process_tree | manage process families | | vm_sandbox_create — launch QEMU/Docker sandbox | isolated execution (from Cua) | | vm_sandbox_destroy — tear down sandbox | cleanup (from Cua) | | sandbox_exec — run command inside sandbox | safe testing (from Cua) |

---

7. FILE SYSTEM — Navigate and Manipulate Files

HAVE

• get_disk_usage — all drives (total, free, used %)
• open_file_explorer — open Explorer to path
• open_file_location — open Explorer with file selected
• list_directory — list files/subdirs (name, size, is_dir, mod_time, mode)
• read_file — format-aware (txt, docx, xlsx, pdf, images via OCR), pagination
• write_file — format-aware write (txt, docx, xlsx, pdf), requires overwrite=true for existing
• find_files — recursive glob search (e.g. *.go, **/*.md)
• copy_file — copy file or directory recursively
• move_file — move/rename file or directory
• delete_file — move to Recycle Bin (SHFileOperationW)
• create_directory — mkdir -p
• get_file_info — size, mod_time, is_dir, mode
• set_working_directory — set working directory for relative path resolution
• get_working_directory — get current working directory
• explorer_focus — focus an existing File Explorer window
• explorer_open_path — open File Explorer at specified path

NEXT

| Tool | Why | |——|—–| | append_to_file — add content to end of file | logging | | read_file_lines — read specific line range | large file navigation |

FAR

| Tool | Why | |——|—–| | download_file — HTTP download to local path | fetch remote files | | archive_create / archive_extract — zip/tar,7z | compression | | watch_directory — file system watcher | monitor for changes | | read_file_binary — read as base64 | binary file handling | | check_disk_health — SMART, error counts | diagnostics | | get_recycle_bin — list recycle bin contents | recovery |

---

8. CLIPBOARD — Copy and Paste

HAVE

• get_clipboard — text content
• set_clipboard — write text

NEXT

| Tool | Why | |——|—–| | get_clipboard_image — read image from clipboard | image copy operations | | get_clipboard_files — read copied file paths | file cut/copy | | get_clipboard_formats — list available formats | clipboard inspection | | clear_clipboard — empty clipboard | security cleanup | | clipboard_history — access Windows clipboard history (Win+V) | multi-paste |

FAR

| Tool | Why | |——|—–| | set_clipboard_image — copy image to clipboard | image editing context | | set_clipboard_files — copy files to clipboard | file operations |

---

9. AUDIO — Hear and Speak

HAVE

• get_volume / set_volume — system volume 0-100
• set_mute — toggle mute (sets to 0 or 50)
• list_audio_devices — render + capture devices
• set_default_audio_device — set default render device

NEXT

| Tool | Why | |——|—–| | get_microphone_mute / set_microphone_mute | mic control for voice | | play_sound — play a .wav file or system sound | audio feedback | | play_beep — beep at frequency/duration | alerts | | get_audio_levels — current VU meter level | volume visualization | | record_audio — record from microphone to file | voice capture |

FAR

| Tool | Why | |——|—–| | text_to_speech — SAPI/WinRT speech synthesis | speak to user | | speech_to_text — WinRT speech recognition | transcribe microphone | | get_system_sounds — list available system sounds | customization | | set_audio_device_volume — per-device volume | granular control | | audio_mixer — per-app volume control | app-specific audio |

---

10. TEXT-TO-SPEECH & SPEECH-TO-TEXT — Voice Interaction

HAVE

— none

NEXT

| Tool | Why | |——|—–| | speak — TTS via WinRT SpeechSynthesis | voice feedback to user | | speak_to_file — generate speech .wav | save speech output | | list_tts_voices — available voices | selection | | set_tts_voice — choose voice | personalization |

FAR

| Tool | Why | |——|—–| | transcribe_microphone — live speech→text | voice commands | | transcribe_file — transcribe audio file | offline transcription | | list_speech_languages — supported recognition languages | multi-lingual | | continuous_speech_recognition — real-time streaming | always-listening mode |

---

11. POWER & SYSTEM — Control the Machine

HAVE

• shutdown / restart / sleep / hibernate
• get_uptime — time since boot
• get_idle_time — time since last user input
• get_battery — percentage, charging, AC status
• get_system_info — hostname, OS, total/free RAM
• get_disk_usage — all drives
• lock_workstation

NEXT

| Tool | Why | |——|—–| | set_screen_power — turn monitor on/off | power saving | | get_power_scheme — current power plan GUID | power management | | set_power_scheme — high performance / balanced / power saver | performance control | | get_computer_name / set_computer_name | system identity | | get_timezone / set_timezone | time management | | get_system_locale — current locale | localization awareness |

FAR

| Tool | Why | |——|—–| | get_cpu_usage / get_memory_usage | performance monitoring | | get_installed_updates — Windows Update status | patch management | | get_event_log — tail system/application logs | diagnostics | | get_os_version — detailed version/build | compatibility | | screen_off / screen_on — display power state | energy saving | | get_system_uptime_seconds (already have via duration) | — |

---

12. NETWORK — Connect and Communicate

HAVE

• get_network_info — hostname, IPs, DNS, gateway
• ping — ICMP reachability

NEXT

| Tool | Why | |——|—–| | get_wifi_ssid — connected WiFi network name | connectivity awareness | | get_wifi_signal_strength — RSSI | signal quality | | list_network_adapters — adapters + status | diagnostics | | get_active_network_connections — active TCP/UDP | app network awareness | | get_firewall_status — Windows Defender Firewall state | security check | | get_proxy_settings — current proxy config | internet access | | speed_test — network speed (download/upload) | bandwidth check |

FAR

| Tool | Why | |——|—–| | set_static_ip / set_dhcp | network configuration | | enable_network_adapter / disable_network_adapter | connectivity control | | get_public_ip — external IP | internet presence | | connect_to_wifi / disconnect_wifi | wireless management | | traceroute / nslookup — network diagnostics | troubleshooting |

---

13. REGISTRY — System Configuration

HAVE

— none

NEXT

| Tool | Why | |——|—–| | registry_read — read a registry value | system settings | | registry_list_keys — enumerate subkeys | registry browsing |

FAR

| Tool | Why | |——|—–| | registry_write / registry_delete | system configuration |

---

14. ENVIRONMENT — User and System Environment

HAVE

— none

NEXT

| Tool | Why | |——|—–| | get_env — get a single env var | config awareness | | list_env — all env vars | full environment | | set_env — set env var for current process | runtime config |

FAR

| Tool | Why | |——|—–| | set_env_permanent — persistent env var | system configuration | | get_path / set_path — PATH management | executable lookup |

---

15. UI AUTOMATION — Interact with Controls

HAVE

• uia_find — find elements by name, automation_id, control_type → bounding rect + properties
• uia_get_text — read text from a UI element
• uia_invoke — click/invoke a button or control via Invoke/Toggle/Click pattern
• uia_set_text — set text in a text control via ValuePattern (v0.2.35)
• uia_get_all_elements — list all immediate child elements of a window (v0.2.38)
• uia_get_element_at_point — identify UI element at screen coordinates (v0.2.38)
• wait_for_ui_element — poll for a UI element to appear in a window (v0.2.38)

NEXT

| Tool | Why | |——|—–| | uia_get_focused_control — AutomationElement for focused element | know what’s focused | | uia_store_element / uia_recall_element — store/recall UIA element positions per window | element memory across sessions |

FAR

| Tool | Why | |——|—–| | uia_select — select from combobox/listbox | dropdown selection | | uia_get_table — read table/grid content | data extraction | | uia_scroll / uia_select_tab | control interaction | | uia_get_bounding_rect — get element screen rect | positioning |

---

16. TASKBAR & START MENU — Shell Interaction

HAVE

— none

NEXT

| Tool | Why | |——|—–| | taskbar_search — type in taskbar search box | quick access | | start_menu_open / start_menu_close | start menu access | | get_pinned_taskbar_items — list pinned apps | quick launch awareness |

FAR

| Tool | Why | |——|—–| | pin_to_taskbar / unpin_from_taskbar | taskbar customization | | get_jump_list — right-click menu for taskbar item | recent files, tasks | | action_center_open / action_center_close | notification center | | clock_flyout — open calendar/clock | date/time access | | system_tray_open — show hidden icons | tray access |

---

17. NOTIFICATIONS — Alert the User

HAVE

• show_notification — blocking MessageBox

NEXT

| Tool | Why | |——|—–| | toast_notify — Windows Toast notification (non-blocking) | modern notifications | | toast_dismiss — dismiss a notification | clean up |

FAR

| Tool | Why | |——|—–| | get_notification_history — read notification center | missed alerts |

---

18. USB & DEVICES — Peripheral Management

HAVE

— none

NEXT

| Tool | Why | |——|—–| | list_usb_devices — connected USB devices | peripheral awareness | | eject_usb — safely remove USB drive | safe removal |

FAR

| Tool | Why | |——|—–| | list_printers / get_default_printer | printing awareness | | get_connected_displays — external monitors | display config | | list_bluetooth_devices — paired BT devices | wireless peripherals |

---

19. TIME & DATE — System Clock

HAVE

— none

NEXT

| Tool | Why | |——|—–| | get_system_time — current date/time | time awareness | | get_timezone | timezone awareness |

FAR

| Tool | Why | |——|—–| | set_system_time | time synchronization | | get_uptime_display — human-readable uptime | user communication |

---

20. ACCESSIBILITY — Ease of Access

HAVE

— none

NEXT

| Tool | Why | |——|—–| | get_high_contrast — is high contrast mode on | accessibility awareness | | get_magnifier_state — Magnifier on/off + zoom level | support magnification | | get_narrator_state — is Narrator running | screen reader awareness |

FAR

| Tool | Why | |——|—–| | set_high_contrast | toggle accessibility | get_cursor_size / get_text_size | scaling awareness |

---

21. REMOTE SESSION — Terminal Services

HAVE

— none

NEXT

| Tool | Why | |——|—–| | get_session_type — console vs RDP | know if remote | | get_active_sessions — list terminal sessions | multi-user awareness |

FAR

| Tool | Why | |——|—–| | disconnect_session / logoff_session | session management |

---

22. SECURITY & IDENTITY

HAVE

• set_config — runtime config (training, priors, denylist, retention, abort, lock)

NEXT

| Tool | Why | |——|—–| | P0 list_tools — list all MCP tools with metadata | tool discovery | | enable_tool / disable_tool — per-tool allow/deny | done v0.2.37 — delivered as tool_denylist config field (server-level denylist) | | set_auth_token — configure bearer token for TCP transport | transport security (from Windows-MCP) | | set_tls — configure TLS cert for TCP transport | transport encryption (from Windows-MCP) |

FAR

| Tool | Why | |——|—–| | get_bitlocker_status — drive encryption | security status | | get_defender_status — antivirus status | security awareness | | set_cors_origin — restrict MCP clients by origin | remote access control | | set_ip_allowlist / set_ip_denylist — IP-based access control | network security |

---

23. SCREEN — Display Hardware

HAVE

• list_displays — name, res, position, primary
• get_screen_size — virtual width/height
• get_screen_dpi — per-monitor DPI + scale %
• get_display_modes — all available resolutions, refresh rates, color depths per monitor
• get_brightness — current display brightness level (0-100)
• set_brightness — set display brightness level (0-100)

NEXT

| Tool | Why | |——|—–| | get_display_orientation — landscape/portrait | rotation | | get_virtual_screen_bounds — total spanning rect | multi-monitor layout |

FAR

| Tool | Why | |——|—–| | set_display_resolution — change resolution | display config | | set_display_orientation | rotation | | identify_displays — flash display number | user communication |

---

24. WINDOWS SHELL — OS Integration

HAVE

• open_file_explorer
• open_file_location
• open_url

NEXT

| Tool | Why | |——|—–| | open_run_dialog — Win+R | quick command | | open_command_prompt / open_powershell | terminal access | | open_control_panel — specific applet | system settings | | open_settings_app — Windows Settings page | modern settings | | get_default_browser | URL handling awareness |

FAR

| Tool | Why | |——|—–| | open_recycle_bin | file recovery | | empty_recycle_bin | cleanup | | open_task_manager | process management | | get_association — file extension→app | default program |

---

25. CHAINED / COMPOSITE — Higher-Level Abstractions

HAVE

• find_text_and_click — OCR + click at text
• wait_for_text — poll OCR until text appears
• click_menu_item — within a window by title
• launch_and_wait — launch app + wait for window
• screenshot_element — screenshot a window
• hover — move + wait
• type_and_submit — type + Enter
• select_all_and_type — Ctrl+A + type
• dismiss_all_menus — press Escape + OCR before/after (v0.2.41)
• chain — server-side step sequencer with variable capture
• chain_abort — hotkey-triggered chain termination (v0.2.42)
• wait — pause for N milliseconds

NEXT

| Tool | Why | |——|—–| | find_text_and_right_click | right-click on OCR text | | find_text_and_double_click | double-click on OCR text | | find_all_text — return all matches | multi-target | | fill_form — map field labels→values | form filling | | drag_and_drop_file — drag file from Explorer | file operation | | download_and_open — download URL then open | remote file workflow | | scroll_until_text_visible — scroll + OCR poll | infinite scroll | Done — built into find_text_and_click and wait_for_text via max_scrolls/scroll_clicks/scroll_down params | | click_all_matching_text — click every occurrence | bulk dismissals | | dismiss_all_notifications — find+close | cleanup | | type_password — type from env var | secure input | | chain_template — save/load named chain recipes | reusable automation patterns | | chain_step_retry — retry individual steps on failure | resilient chains |

FAR

| Tool | Why | |——|—–| | login_to_website — URL + credentials → logged in page | web automation | | navigate_file_dialog — Open/Save dialog → select file | file dialog handling | | complete_wizard — button-by-button wizard completion | installer automation | | install_app — download + run installer + accept EULA | app installation | | ocr_and_chain — general OCR → tool call pipeline | agent-in-a-box |

---

26. VERIFICATION & FEEDBACK

HAVE (0 standalone tools — built-in features)

• Built-in features (not standalone MCP tools, but available on 11 action tools):
  • auto_verify — post-action OCR verification parameter
  • pre_expected — pre-action precondition check parameter
  • expected — configurable text/change/not_text criteria
  • chain verify step type — retry loop with OCR diff

NEXT

| Tool | Why | |——|—–| | verify_image_diff — screenshot comparison verification | visual change detection | | verify_with_timeout — poll verification until pass/timeout | async UI transitions |

---

27. DEBUGGING & DIAGNOSTICS

HAVE (4 tools + 2 built-in features)

• get_logs — read recent log entries with level/search/time filtering (v0.2.43)
• report_issue — generate GitHub issue with system info, logs, context; auto-submit via gh CLI (v0.2.43)
• reset_state — clear adaptive engine stats + bridge buffer (v0.2.41)
• bridge_debug — OCR→command bridge state inspection
• Built-in features: structured logging (via set_config log_file_enabled), panic recovery (safeHandler wrapper + global recover)

NEXT

| Tool | Why | |——|—–| | log_memory — current heap usage | memory monitoring | | benchmark_screenshot — capture timing | performance baseline | | debug_ocr — raw OCR output with confidence | OCR quality assessment |

FAR

| Tool | Why | |——|—–| | save_screenshot_to_file — save debug screenshot | examination | | save_all_state — windows + processes + disks | snapshot |

---

28. MEMORY & ML — Learn and Adapt

Note: ONNX models (YOLO, MobileNet) are temporary placeholders. Custom neural network planned in §34.

HAVE

• memory_set / memory_get / memory_search / memory_list / memory_forget — SQLite FTS5 persistent fact store with TTL
• agent_analyze / agent_suggest / agent_train — adaptive engine (timing stats, success rates, coordinate prediction)
• training_save_sample / training_list_samples / training_stats / training_mark_used / training_cleanup_noise / export_yolo_dataset — auto-collect + export pipeline
• datalog_query / datalog_status / datalog_export — action/OCR/chain/pair datalog
• priors_stats — learned element frequency + position per window
• find_ui_element — cascading locator: memory → prior → ONNX → OCR
• layout_validate — validate stored UI element layout against current screen
• template_store / template_find / template_list / template_forget — visual element templates
• onnx_status / onnx_detect / onnx_download — ONNX runtime + YOLO detection
• onnx_watch_start / onnx_watch_stop / onnx_watch_status / onnx_watch_cache — background ONNX watcher
• introspection_analyze / task_begin / task_end — post-task mining
• Go-native transformer engine (ml/ module) — 14K-param transformer via Gorgonia, self-improving action prediction from OCR context, persists to model.gob
• Web dashboard — live monitoring on random port (tool usage, commands, chains, sequences, training samples, model status). Configurable via dashboard_enabled. Auto-refreshes every 5s.

NEXT

| Tool | Why | |——|—–| | set_retention_policy — auto-prune training samples older than N days | done v0.2.37 — delivered as retention_days config field + background pruner every 6 hours | | P1 set_sensitive_content_filter — regex patterns to redact before saving | privacy guard (from Recall) | | P2 set_app_exclusion — exclude apps from auto-capture | source filtering (from Recall) | | P2 set_url_exclusion — exclude URLs from auto-capture | source filtering (from Recall) | | training_set_category_prompt — per-category task prompt | richer training context |

FAR

| Tool | Why | |——|—–| | in_context_rl — session-scoped outcome buffer for within-session learning | adaptive on-the-fly (from Agent-S) | | semantic_search_screenshots — vector embedding index over past screenshots | find-by-content (from Recall) | | rl_training_environment — simulated desktop for reinforcement learning | RL pipeline (from Cua) | | model_fine_tune — trigger local fine-tuning of ONNX model | self-improving vision |

---

29. TRANSPORT & SERVER — Connectivity

HAVE (0 tools — current transport state)

• stdio transport (not an MCP tool, just the current capability)
• HTTP dashboard on random port (monitoring only, not MCP transport)

NEXT

| Feature | Why | |———|—–| | SSE transport | run as standalone server, support remote MCP clients (from Windows-MCP) | | Streamable HTTP transport | modern MCP transport, bidirectional streaming | | Bearer token auth | authenticate MCP clients (from Windows-MCP) | | TLS support | encrypt transport layer (from Windows-MCP) | | CORS support | restrict browser-based MCP clients (from Windows-MCP) | | set_transport / get_transport_config — runtime transport switching | flexible deployment |

FAR

| Feature | Why | |———|—–| | OAuth 2.0 + PKCE | enterprise auth (from Windows-MCP) | | WebSocket transport | real-time bidirectional streaming | | Unix domain socket transport | local high-performance IPC |

---

30. BROWSER AUTOMATION — Web Tasks

HAVE

• open_url — open URL in default browser
• find_text_and_click — basic OCR-based browser interaction
• wait_for_text — poll OCR until text appears
• browser_focus_url_bar / browser_new_tab / browser_navigate / browser_search — keyboard-shortcut browser control in browseruse.go

NEXT

| Tool | Why | |——|—–| | browser_dom_connect — connect to browser via CDP (Chrome/Edge/Brave) or Playwright (Firefox) | enable DOM-level access; start in browseruse.go, extract to internal/browser/ if needed | | browser_dom_snapshot — capture DOM snapshot of current page | structured page data (from Windows-MCP) | | browser_dom_get — query DOM elements by CSS selector | fast element access (from Windows-MCP) | | browser_dom_get_text — get text content of DOM element | read page content (from Windows-MCP) | | browser_dom_invoke — click/submit DOM element | reliable browser interaction (from Windows-MCP) | | browser_dom_get_attributes — get element attributes | page structure analysis | | browser_execute_js — run JavaScript in page context | advanced page scripting |

FAR

| Tool | Why | |——|—–| | browser_new_tab / browser_close_tab — tab management | multi-page workflows | | browser_get_cookies / browser_set_cookies — session management | authenticated browsing | | browser_wait_for_page_load — wait until page fully loaded | reliable navigation | | browser_get_console_logs — capture console output | debugging | | browser_fill_form — auto-fill form fields by label | form automation (from Browser Use) | | browser_download_file — capture download from browser | file acquisition | | CAPTCHA solving | farm out to third-party service (from Browser Use) |

---

31. LINUX & CONTAINER — Cross-Platform Desktop

HAVE

— none

NEXT

| Tool | Why | |——|—–| | container_list — list running containers | container awareness | | container_exec — run command inside container | cross-platform execution (from Bytebot) |

FAR

| Tool | Why | |——|—–| | linux_x_desktop_control — control Linux X11/Wayland desktops | Linux support (from Bytebot) | | macos_a11y_control — control macOS via Accessibility API | macOS support | | container_desktop_launch — launch desktop app inside container | isolated Linux desktop (from Bytebot) | | cross_platform_interface — abstract Windows/Linux/macOS behind common API | portable agent tasks |

---

32. PERMISSIONS — Elevation & Access Control

HAVE

— none

NEXT

| Tool | Why | |——|—–| | P0 is_admin — check if running as admin | permission awareness | | P0 get_username — current user | identity | | get_user_sid — user security identifier | identity | | get_user_groups — group membership | permission awareness | | get_process_elevation — is a process elevated (admin) | UAC awareness | | check_file_permission — can current user read/write/execute a file | access control | | check_registry_permission — can current user read/write a registry key | access control |

FAR

| Tool | Why | |——|—–| | uac_prompt — trigger elevation dialog | admin escalation | | run_as_admin — re-launch process with elevation | privilege escalation | | get_integrity_level — process integrity level (low/medium/high/system) | security context | | impersonate_user — temporarily act as another user | multi-user scenarios |

---

33. UPGRADE — Self-Update System (v1.3.0)

Nice to have. Requires permission system (§32) and transport system (§29) first. Full spec: docs/meta/plan-self-update.md

HAVE

— none

NEXT

— none

FAR

| Tool | Why | |——|—–| | get_client_info — detect which AI client is using the MCP server | parent process detection, restart strategy | | update_server — check for updates and optionally apply | self-update with client-aware restart | | check_update_on_startup — config field to auto-check on start | proactive version awareness | | rollback_update — restore previous version from .bak | recovery from bad updates | | set_update_channel — stable/beta/nightly | pre-release access | | download_progress — track download progress for large binaries | UX feedback |

Client Detection

| Client | Type | Restart Strategy | |——–|——|—————–| | OpenCode Desktop | desktop | Kill parent, relaunch | | Claude Desktop | desktop | Kill parent, relaunch | | Cursor / Windsurf | desktop | Kill parent, relaunch | | opencode CLI | cli | Replace binary, marker file, user restarts | | claude CLI | cli | Replace binary, marker file, user restarts | | gemini / codex / copilot CLI | cli | Replace binary, marker file, user restarts | | pwsh/cmd (direct) | direct | Replace binary, exit | | Unknown | unknown | Download to safe location, tell user to replace manually |

---

34. CUSTOM NEURAL NETWORK — Self-Designing Model (v0.3.x)

Complete. v0.2.46-50 delivered the full Go-native transformer engine: 14K-param Gorgonia model, ml_bridge, tokenizer, spatial encoder, training pipeline, online learning, model export, smart cascade, sequence context, DPI auto-detect, data augmentation, model versioning, multi-task output head, transfer learning, model compression, NAS. 12/12 items in HAVE.

HAVE

See §28 Memory & ML — transformer engine, ml_bridge, tokenizer, spatial encoder, training pipeline, online learning, model export, smart cascade text finding, text_location_memory all delivered in v0.2.46-48.

• Sequence context (action history ring buffer → model.Forward)
• Auto-detect screen config (ScreenSize + GetDPIScaleForPoint)
• Improved tokenizer (OCR normalization, [NUM] token, bigram subword fallback)
• Online learning (ReplayBuffer 10K, 30s background retrain)
• Model versioning (versioned checkpoints, auto-rollback on >5% regression)
• Data augmentation (DPI scaling, OCR noise, coord jitter)
• Model presets (small/medium/large), DebugForward, ParamCount
• Multi-task output head (scroll direction + key category prediction)
• Transfer learning (FinetuneForApp, per-app models)
• Model compression (Prune, Quantize, IterativePrune)
• Neural architecture search (grid search over configs)

NEXT

All items complete.

FAR

All items complete.

---

35. ACTION PREDICTION EXPANSION — Drag, Sequences, Spatial (v0.4.x)

Next major milestone. §34 built the model; §35 makes it predict complex actions. Fixes: 1-coord limitation (can’t predict drag), 11-tool ceiling (chain has 51), no sequence prediction, no window targeting, no spatial relationships.

DONE (Multi-Coordinate Output)

| Component | Description | |———–|————-| | 4-coordinate output | Config.FromCoordDim, OutputDim=tool(11)+from_xy(2)+to_xy(2)+arg(10)=25. Predictor decodes 4 coords, trainer fills from_xy+to_xy targets | | Action template schema | decodeCoords() per-tool schema: drag→from_x/from_y+to_x/to_y, others→x/y as destination | | Drag prediction | Prediction.FromCoordX/Y fields, PredictedAction.FromCoord struct, bridge populates both coord pairs | | Drag-and-drop | drag tool already in mlDefaultTools with from_x/from_y/to_x/to_y. drag_and_drop alias in trainer decodeCoords | | Backward compat | Old models fail LoadParameters (param count mismatch) → recreated and retrained with new layout |

DONE (Sequence Prediction)

| Component | Description | |———–|————-| | Sequence head | Config.SequenceLen, output layout helpers PrimaryDim(), SeqSlotDim(), TotalOutputDim(). Slot = tool(numTools)+to_xy(2)+arg(argDim) | | Sequence training | LoadSequences(ctx, minLen) groups consecutive training_pairs by session_id. makeSequenceTargets() fills sequence section of target vector | | Sequence prediction | PredictSequence() returns SequencePrediction{Primary, Next[]}. decodePrimary(), decodeSlot() methods. Wired through MLEngine.PredictSequence() and AdaptiveEngine.PredictSequenceActions() | | Chain integration | chain_predict MCP tool — calls PredictSequenceActions(ocr_text), returns primary + future actions as JSON | | Confidence-gated sequences | SeqConfidenceThreshold=0.15. Steps below threshold truncate the sequence |

DONE (Window Context & Spatial Encoding)

| Component | Description | |———–|————-| | Window category output | Config.WindowDim=6, window category one-hot in primary head (browser/editor/terminal/file_manager/dialog/other). WindowCategories constants. Predictor decodes WindowCategory + WindowConf | | Window title as input | PredictWithWindow()/PredictSequenceWithWindow() prepend window title to OCR text. chain_predict accepts optional window_title param | | Spatial relationship encoding | Encoder expanded 7→12 features: added distFromCenter, isCenter, isEdge, windowAspect, isDialog. spatial.FeatureDim=12 | | Multi-window awareness | WindowInfo{Title,Category,Active} struct. PredictWithWindows()/PredictSequenceWithWindows() encode top-3 windows as [ACTIVE][cat]title prefix. Wired through MLEngine + AdaptiveEngine |

DONE (Tool Expansion, Batch Training, Full Attention)

| Component | Description | |———–|————-| | Expand tool set | 11→25 tools: click, move_mouse, scroll, drag, type, key_press, key_down, key_up, hover, focus_window, launch_and_wait, close_window, minimize_window, maximize_window, restore_window, move_window, find_window, set_clipboard, get_clipboard, type_and_submit, select_all_and_type, ocr, screenshot, click_menu_item, launch_app | | Batch training | TrainerConfig.BatchSize, mini-batch loop with ForwardBackward() + Step() + ResetGradients() per batch | | Full attention (Q/K/V/O) | Separate Q/K/V/O weight matrices per layer. Scaled dot-product via Q @ (Kᵀ @ V / √d) using MatMul-only path (avoids Gorgonia shape squeeze). Layer norm uses [1,d] matrices instead of [d] vectors to prevent HadamardProd squeezing. |

NEXT

| Component | Why | |———–|—–| | End-to-end chain test | OCR → predict sequence → auto-chain → verify. Accuracy measurement on common workflows |

FAR

| Component | Why | |———–|—–| | Conditional sequences | If/else branching in predicted sequences based on OCR state | | Multi-modal input | Screenshot pixels + OCR text combined (CNN feature extractor) | | Reinforcement learning | Reward-based training from action outcomes (success/failure) | | Cross-app transfer | Pre-trained model shipped, fine-tuned per-app on first run | | Real-time streaming | Predict while screen changes (video-like OCR feed) |

---

36. MULTI-MONITOR ML AWARENESS — Spatial Intelligence Across Displays

The MCP server already has display detection (list_displays, get_dpi_for_point, get_display_modes), but the ML model has no concept of which monitor an action targets. This section makes the model display-aware.

NEXT

| Component | Why | |———–|—–| | Display index as input feature | Model receives which monitor the target is on (0, 1, 2…). Combined with per-monitor DPI/resolution for coordinate normalization. | | Per-display coordinate normalization | Coordinates normalized relative to the target display’s bounds, not the virtual desktop. Prevents cross-monitor drift. | | Display-aware training targets | Training samples tagged with display index. Model learns display-specific click patterns. | | Multi-display chain prediction | chain_predict returns display context so chained actions stay on the correct monitor. |

FAR

| Component | Why | |———–|—–| | Cross-display drag prediction | Drag from monitor A → monitor B with coordinate transform across displays | | Display transition detection | Detect when user moves mouse across monitor boundaries | | Per-display OCR regions | OCR only the active monitor for faster, more relevant text extraction |

---

Summary

Domain	HAVE	NEXT	FAR	Total
Vision	13	3	9	25
Mouse	6	3	7	16
Keyboard	11	4	4	19
Windows	15	8	6	29
Virtual Desktops	0	6	2	8
Processes	3	8	8	19
File System	16	2	6	24
Clipboard	2	5	2	9
Audio	5	6	5	16
TTS / STT	0	4	4	8
Power & System	10	8	7	25
Network	2	7	5	14
Registry	0	2	1	3
Environment	0	3	2	5
UI Automation	7	2	4	13
Taskbar / Start	0	4	7	11
Notifications	1	2	1	4
USB & Devices	0	2	3	5
Time & Date	0	2	2	4
Accessibility	0	3	2	5
Remote Session	0	2	1	3
Security & Identity	1	3	4	8
Screen (HW)	6	2	3	11
Windows Shell	3	5	4	12
Chained	12	12	5	29
Verification	0	2	0	2
Debugging	4	3	2	9
Memory & ML	34	4	4	42
Transport & Server	0	7	3	10
Browser Automation	7	7	8	22
Linux & Container	0	2	4	6
Permissions	0	7	4	11
Upgrade (§33)	0	0	6	6
Custom NN (§34)	12	0	0	12
Action Prediction (§35)	17	1	5	23
Multi-Monitor ML (§36)	0	4	3	7
TOTAL	177	144	149	470

Strategy

1. Action prediction expansion (§35, v0.4.x) — All done except end-to-end chain test. Multi-coordinate output, sequence prediction, window context, spatial encoding, tool expansion, batch training, full attention all complete.
2. Multi-monitor ML awareness (§36) — MCP has display detection, ML model needs display index input, per-display coordinate normalization, display-aware training targets.
3. Build out NEXT items — straightforward and high value (~144 items remaining)
4. Error wrapping audit — remaining Slice 4 item for consistent error feedback
5. Security + Transport — TLS + SSE prerequisites for remote deployment
6. Browser DOM mode — CDP/Playwright integration for 10x faster web tasks
7. Background control — SendInput/UIA for non-disruptive clicks
8. Permissions — is_admin, get_username, UAC awareness (§32)
9. Cross-platform interface — Linux/macOS stubs + container support
10. Upgrade system (§33, v1.3.0) — self-update with client-aware restart (requires §32 + §29)