Skip to content
Merged

Dev #95

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
599affd
feat: update .gitignore to include local patched tauri-mcp source
pountzas Jul 31, 2026
3e98d0d
perf: cut IPC storm from keyboard poll, trackpad, and head tracking
pountzas Jul 31, 2026
70ecafe
fix: unblock Tokio/IPC for speak, macros, STT, and native dialogs
pountzas Jul 31, 2026
c6d6036
Harden CSP, asset scope, and IPC file/launch paths.
pountzas Jul 31, 2026
c8ae80d
Merge pull request #88 from pountzas/opt/security-hygiene
pountzas Jul 31, 2026
759bf4f
Merge pull request #89 from pountzas/opt/perf-ipc
pountzas Jul 31, 2026
a475d43
Merge pull request #90 from pountzas/opt/reliability-blocking
pountzas Jul 31, 2026
5a75928
Merge branch 'main' of https://github.com/pountzas/accessibility-keyb…
pountzas Jul 31, 2026
1ffa31d
chore: bump version to 0.6.0 in Cargo.toml
pountzas Jul 31, 2026
18a261f
fix: stop profile settings from resetting on startup and switch
pountzas Jul 31, 2026
9d84aa5
Merge pull request #91 from pountzas/fix/profile-persistence
pountzas Jul 31, 2026
22f44ef
feat: enhance music lesson panel with partiture view and song selection
pountzas Jul 31, 2026
53c6ba1
Merge pull request #92 from pountzas/feature/built-in-classical-songs
pountzas Jul 31, 2026
293b55a
feat: add internationalization support for German, Spanish, French, I…
pountzas Jul 31, 2026
f5a0e54
feat: expand language support and improve localization features
pountzas Jul 31, 2026
1b8e292
docs: update README and accessibility requirements for clarity and us…
pountzas Jul 31, 2026
5e813ea
docs: refine README for clarity and user focus
pountzas Jul 31, 2026
b16f43e
chore: ignore .worktrees directory
pountzas Jul 31, 2026
b6bb86b
feat: enhance predictive text functionality with offline word packs
pountzas Jul 31, 2026
fdcd3da
Merge pull request #93 from pountzas/feat/local-prediction-packs
pountzas Jul 31, 2026
7492fb4
fix: harden review findings for reliability and portable config
pountzas Aug 1, 2026
ae980b9
fix: improve database query consistency by removing unnecessary lower…
pountzas Aug 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .cursor/mcp.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"mcpServers": {
"tauri-mcp": {
"type": "stdio",
"command": "tauri-mcp",
"args": [
"--config",
"tauri-mcp.toml",
"--log-level",
"info",
"serve"
]
}
}
}
12 changes: 4 additions & 8 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,11 @@ body:
validations:
required: true

- type: checkboxes
id: personas
- type: textarea
id: who
attributes:
label: Affected personas
description: Who benefits most from this change?
options:
- label: Child (primary user)
- label: Adult
- label: Therapist / Caregiver
label: Who benefits
description: Which users or situations does this help (e.g. school laptop, secondary touchscreen, caregivers)?

- type: textarea
id: solution
Expand Down
4 changes: 2 additions & 2 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ Use a [Conventional Commits](https://www.conventionalcommits.org/) **PR title**

<!-- Link issue if applicable: Fixes #123 -->

## Persona / accessibility impact
## Accessibility impact

<!-- Child, Adult, Therapist — or N/A for internal changes -->
<!-- How this affects touch-first / motor-disability users — or N/A for internal changes -->

## Test plan

Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,9 @@ dist-ssr
# Rust / Tauri
src-tauri/target/
src-tauri/.tauri/

# Local patched tauri-mcp source (binary installed via cargo)
.cursor/tauri-mcp/

# Git worktrees
.worktrees/
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ After cloning or forking, maintainers should ensure:

## Accessibility changes

Changes that affect usability for people with motor disabilities should align with [docs/accessibility-requirements.md](docs/accessibility-requirements.md). Mention which persona (Child, Adult, Therapist) is affected in your PR description.
Changes that affect usability for people with motor disabilities should align with [docs/accessibility-requirements.md](docs/accessibility-requirements.md). Note accessibility impact in your PR description.

## Code style

Expand Down
82 changes: 57 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

Assistive virtual keyboard and mouse for Windows — built for users with severe motor disabilities who operate a wheelchair-mounted touchscreen while applications run on a laptop display.

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Platform: Windows](https://img.shields.io/badge/platform-Windows-lightgrey)]()
[![Build](https://github.com/pountzas/reach-Panel/actions/workflows/build.yml/badge.svg)](https://github.com/pountzas/reach-Panel/actions/workflows/build.yml)
[License: MIT](LICENSE)
[Platform: Windows](#requirements)
[Build](https://github.com/pountzas/reach-Panel/actions/workflows/build.yml)

Built with Tauri, React, Rust, and SQLite.

Expand All @@ -18,45 +18,59 @@ Pre-built installers are on the [Releases](https://github.com/pountzas/reach-Pan

> **macOS builds are experimental.** CI produces `.dmg` bundles, but input injection and TTS still require Windows APIs for full functionality. See [Roadmap](#roadmap).



## Screenshots

| Main layout | Virtual trackpad | Settings |
|-------------|------------------|----------|
| ![Main keyboard layout](docs/images/keyboard.png) | ![Virtual trackpad](docs/images/trackpad.png) | ![Settings panel](docs/images/settings.png) |

_Replace with captures from your release build when ready — see [docs/images/README.md](docs/images/README.md)._
| Main layout | Virtual trackpad | Settings |
| -------------------- | ---------------- | -------------- |
| Main keyboard layout | Virtual trackpad | Settings panel |


*Replace with captures from your release build when ready — see [docs/images/README.md](docs/images/README.md).*

## Features



### Keyboard

- System-wide keyboard injection with sticky modifiers
- Adjustable on-screen keyboard (size, spacing, colors, opacity)
- Fn key on the bottom row maps number keys 1–0 and `-` `=` to F1–F12
- Keyboard / synthesizer mode toggle with two-octave piano keyboard
- Language switch key with country flag icons (EN ↔ EL)
- Predictive text (English + Greek) with disable toggle
- Language switch key with country flag icons (follows installed Windows keyboards)
- Predictive text with offline word packs (English bundled; other languages downloadable), learns from typing, disable toggle
- Macro builder with JSON import/export



### Mouse

- Virtual trackpad with left/right/floating placement
- Move, click, scroll from the accessibility touchscreen
- Toggle between trackpad and on-screen numeric keypad



### Communication

- Quick Actions bar (launch apps and URLs)
- Communication phrases with TTS (Windows SAPI)
- Emergency phrase section with show/hide toggle



### Voice dictation (Windows)

- Mic control on the keyboard toolbar (enable **Show dictation (mic)** in Settings)
- Uses the **typing language** (EN ↔ EL), not the UI language
- **English:** Windows Speech Recognition (WinRT) when the speech language pack is installed and online speech is allowed
- Uses the **typing language**, not the UI language
- **English and Western European languages (DE, FR, IT, ES, PT):** Windows Speech Recognition (WinRT) when the speech language pack is installed and online speech is allowed
- **Greek and other languages without a Windows speech pack:** [Groq](https://groq.com) cloud Whisper (`whisper-large-v3-turbo`) over the internet



#### Set up Groq for Greek dictation

1. Create a free account at [console.groq.com](https://console.groq.com).
Expand All @@ -65,16 +79,18 @@ _Replace with captures from your release build when ready — see [docs/images/R
4. Under the keyboard / dictation options, paste the key into **Groq API key (cloud dictation)**.
5. Set **typing language** to Greek (EL), enable the mic control if needed, then start dictation.

You can also set the `GROQ_API_KEY` environment variable instead of (or as a fallback for) the Settings field. An internet connection is required for cloud dictation. English WinRT dictation does not need a Groq key.
You can also set the `GROQ_API_KEY` environment variable instead of (or as a fallback for) the Settings field. An internet connection is required for cloud dictation. WinRT dictation for English and Western European languages does not need a Groq key.

If Windows online speech is off, open **Settings → Privacy & security → Speech** and turn on **Online speech recognition**. For English speech packs, use **Settings → Time & language → Speech**.
If Windows online speech is off, open **Settings → Privacy & security → Speech** and turn on **Online speech recognition**. For speech packs, use **Settings → Time & language → Speech**.

### Profiles & accessibility

- Profiles (Child, Parent, Therapist)
- Multi-monitor detection and positioning
- Head tracking calibration wizard (webcam-based)
- English and Greek UI
- UI languages: English, Greek, German, French, Italian, Spanish, Portuguese



## Requirements

Expand All @@ -88,6 +104,8 @@ For development:
- Rust (via [rustup](https://rustup.rs/))
- Visual Studio Build Tools with the C++ workload



## Development

```bash
Expand All @@ -101,6 +119,8 @@ For Rust builds on Windows, ensure the MSVC environment is active. If `link.exe`
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
```



## Build from source

```bash
Expand All @@ -113,36 +133,48 @@ Installers and executables are written to:
src-tauri/target/release/bundle/
```



## Releases

Windows and macOS bundles are built in CI. Windows installers are published automatically when a [release-please](https://github.com/googleapis/release-please) Release PR is merged on `main`. Development happens on `dev`; merging `dev` → `main` opens the Release PR. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full branch and versioning workflow.

## Architecture

| Path | Responsibility |
|------|----------------|
| `src/` | React frontend (keyboard, mouse, phrases, settings, head tracking) |
| `src-tauri/src/input/` | Platform keyboard/mouse injection (Windows `SendInput`; macOS stubs) |
| `src-tauri/src/window/` | Monitor enumeration (Windows APIs; macOS stub) |
| `src-tauri/src/tts/` | Text-to-speech (Windows SAPI / WinRT; macOS stub) |
| `src-tauri/src/stt/` | Voice dictation (WinRT speech recognition; Groq cloud fallback) |
| `src-tauri/src/db/` | SQLite persistence (profiles, phrases, macros) |
| `src-tauri/src/prediction/` | Predictive text suggestions |
| `docs/accessibility-requirements.md` | Personas and acceptance criteria |

| Path | Responsibility |
| ------------------------------------ | -------------------------------------------------------------------- |
| `src/` | React frontend (keyboard, mouse, phrases, settings, head tracking) |
| `src-tauri/src/input/` | Platform keyboard/mouse injection (Windows `SendInput`; macOS stubs) |
| `src-tauri/src/window/` | Monitor enumeration (Windows APIs; macOS stub) |
| `src-tauri/src/tts/` | Text-to-speech (Windows SAPI / WinRT; macOS stub) |
| `src-tauri/src/stt/` | Voice dictation (WinRT speech recognition; Groq cloud fallback) |
| `src-tauri/src/db/` | SQLite persistence (profiles, phrases, macros) |
| `src-tauri/src/prediction/` | Predictive text + word-pack install/download |
| `src-tauri/resources/wordpacks/` | Bundled English dictionary |
| `wordpacks-dist/` | Downloadable packs for GitHub `wordpacks-v1` release |
| `docs/accessibility-requirements.md` | Target users and acceptance criteria |




## Roadmap

- Eye tracking support
- AutoHotkey macro import
- macOS port: CGEvent input, AVSpeechSynthesizer, native monitor APIs (experimental CI builds in progress)



## Known limitations

- Input injection does not work into elevated (admin) applications or UAC prompts
- macOS bundles lack system-wide input and TTS until the native backends land
- Voice dictation is Windows-only; Greek (and other WinRT-unsupported languages) need a Groq API key and internet
- Cloud dictation is utterance-based (speak, pause) rather than fully continuous streaming



## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) and [docs/accessibility-requirements.md](docs/accessibility-requirements.md) before opening a pull request.
Expand All @@ -159,4 +191,4 @@ MIT — see [LICENSE](LICENSE).

## Disclaimer

This is assistive software, not a medical device. Test thoroughly with your own hardware and accessibility setup before relying on it in daily use.
This is assistive software, not a medical device. Test thoroughly with your own hardware and accessibility setup before relying on it in daily use.
45 changes: 14 additions & 31 deletions docs/accessibility-requirements.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,20 @@
# Accessibility Requirements

## Personas
## Who it's for

### Child (primary)
- 7-year-old with complete tetraplegia
- Good head, eye, and pointing-finger movement
- Long attention span; comfortable with touchscreen tablets
- Needs large keys, simple navigation, emergency phrases
ReachPanel is for people who cannot use a physical keyboard and need touch-first control of a Windows PC. It was originally built for a child with complete tetraplegia so he can do schoolwork on a touch-capable laptop.

### Adult
- Full feature set, smaller keys acceptable, macro and layout customization
Typical needs:

### Therapist / Caregiver
- Profile management, emergency phrase visibility toggle, monitor selection, calibration
- Large keys, simple navigation, and emergency phrases
- Good use of head, eye, and pointing-finger movement on a touchscreen
- Single touch laptop at school, or a secondary touch display at distance when available

Caregivers can manage named profiles, emergency phrase visibility, monitor selection, and head-tracking calibration in Settings.

## Layout Wireframes

### Layout A (default)
### Layout A (default — mouse right)
```
+---------------------------------------------------+
| Quick Actions |
Expand All @@ -38,39 +36,24 @@
+---------------------------------------------------+
```

### Layout C (floating mouse)
```
+---------------------------------------------------+
| Quick Actions |
+---------------------------------------------------+
| Keyboard (full width) |
+---------------------------------------------------+
| Suggestions |
+---------------------------------------------------+

+------------------+
| Floating Mouse |
+------------------+
```

## Acceptance Criteria

| Feature | Criteria |
|---------|----------|
| Keyboard injection | Types into Notepad, Chrome, Word, Teams, Explorer |
| Sticky keys | Ctrl stays active until next key or toggle off |
| Mouse trackpad | Move, click, scroll from wheelchair touchscreen |
| Mouse placement | Right, left, floating; persisted per profile |
| Mouse trackpad | Move, click, scroll from the accessibility touchscreen |
| Mouse placement | Right or left next to the keyboard; persisted per profile |
| Quick actions | Launch exe and URL targets |
| Phrases | Type, speak, or both; favorites and emergency toggle |
| TTS | Offline via Windows SAPI installed voices |
| Prediction | Top suggestions per language; disable toggle |
| Prediction | Offline word packs (English bundled; other languages downloadable); learn on typed words; disable toggle |
| Macros | Multi-step sequences; JSON import/export |
| Head tracking | Calibration wizard; touch/head mode toggle |
| Multi-monitor | List displays; position app on accessibility screen |
| Profiles | Child, Parent, Therapist with separate settings |
| Profiles | Named saved profiles with separate settings |

## Known Limitations

- Input injection does not work into elevated (admin) applications or UAC prompts
- Eye tracking and AutoHotkey import are future phases
- Eye tracking, AutoHotkey import, and an Android tablet companion (connects to this Windows app to control the PC) are future phases
2 changes: 1 addition & 1 deletion docs/images/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ These images appear in the root [README.md](../../README.md).
2. Capture PNG screenshots at roughly 1200×700 (main window size).
3. Replace the files in this folder and commit.

Use Windows **Snipping Tool** or **Win+Shift+S** for captures. Prefer the Child profile with default layout for consistency.
Use Windows **Snipping Tool** or **Win+Shift+S** for captures. Prefer the default profile and layout for consistency.
Loading