diff --git a/desktop/.gitignore b/desktop/.gitignore index 15ee65bf..dd554f92 100644 --- a/desktop/.gitignore +++ b/desktop/.gitignore @@ -7,3 +7,7 @@ out .idea .vscode + +# compiled native capture helper (pnpm build:capture) +resources/nanokvm-capture +*.tsbuildinfo diff --git a/desktop/native/nanokvm-capture.swift b/desktop/native/nanokvm-capture.swift new file mode 100644 index 00000000..7eebc3c6 --- /dev/null +++ b/desktop/native/nanokvm-capture.swift @@ -0,0 +1,205 @@ +// nanokvm-capture — minimal AVFoundation capture helper for the desktop app's +// uncompressed video mode. Replaces ffmpeg on macOS: ffmpeg's avfoundation +// input freezes on large (>=1080p) uncompressed frames, native capture works. +// +// nanokvm-capture list +// stdout: one JSON object: {"devices":[{"name":...,"formats":[ +// {"pixfmt":"yuvs","width":...,"height":...,"fps":[...]}]}]} +// +// nanokvm-capture stream --device --width --height --fps +// stderr: one line "META {"width":W,"height":H}" with the ACTUAL delivered +// buffer size (the UVC stack may serve the signal-native mode +// regardless of the requested format), then raw yuvs (YUY2) frames +// on stdout, W*2 bytes per row, tightly packed. +// +// SIGINT/SIGTERM stop the session cleanly. Blocking stdout writes provide +// natural backpressure: late frames are discarded by AVFoundation. +import AVFoundation +import Foundation + +func fourCC(_ code: FourCharCode) -> String { + var s = "" + for shift in stride(from: 24, through: 0, by: -8) { + s.append(Character(UnicodeScalar(UInt8((code >> UInt32(shift)) & 0xff)))) + } + return s +} + +func discoverDevices() -> [AVCaptureDevice] { + AVCaptureDevice.DiscoverySession( + deviceTypes: [.external, .builtInWideAngleCamera], + mediaType: .video, + position: .unspecified + ).devices +} + +func listCommand() { + var devices: [[String: Any]] = [] + for d in discoverDevices() { + var formats: [[String: Any]] = [] + for f in d.formats { + let dim = CMVideoFormatDescriptionGetDimensions(f.formatDescription) + let sub = fourCC(CMFormatDescriptionGetMediaSubType(f.formatDescription)) + let fps = f.videoSupportedFrameRateRanges.map { $0.maxFrameRate } + formats.append(["pixfmt": sub, "width": Int(dim.width), "height": Int(dim.height), "fps": fps]) + } + devices.append(["name": d.localizedName, "formats": formats]) + } + let json = try! JSONSerialization.data(withJSONObject: ["devices": devices]) + FileHandle.standardOutput.write(json) + FileHandle.standardOutput.write("\n".data(using: .utf8)!) +} + +final class Writer: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate { + // The device may deliver a few transitional frames at a stale mode right + // after opening — only report META once dimensions are stable, and treat a + // later mode change (e.g. the HDMI source resolution changed) as a restart + // condition (exit 3) so every emitted frame matches META exactly. + var stableW = 0 + var stableH = 0 + var stableCount = 0 + var locked = false + func captureOutput(_: AVCaptureOutput, didOutput sb: CMSampleBuffer, from _: AVCaptureConnection) { + guard let pb = CMSampleBufferGetImageBuffer(sb) else { return } + CVPixelBufferLockBaseAddress(pb, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(pb, .readOnly) } + guard let base = CVPixelBufferGetBaseAddress(pb) else { return } + + let w = CVPixelBufferGetWidth(pb) + let h = CVPixelBufferGetHeight(pb) + let stride = CVPixelBufferGetBytesPerRow(pb) + let rowBytes = w * 2 // yuvs = 2 bytes/pixel + + if !locked { + if w == stableW && h == stableH { + stableCount += 1 + } else { + stableW = w + stableH = h + stableCount = 1 + } + if stableCount < 3 { return } // drop pre-stable frames + locked = true + FileHandle.standardError.write("META {\"width\":\(w),\"height\":\(h)}\n".data(using: .utf8)!) + } else if w != stableW || h != stableH { + FileHandle.standardError.write( + "ERROR video mode changed (\(stableW)x\(stableH) -> \(w)x\(h))\n".data(using: .utf8)!) + exit(3) + } + + if stride == rowBytes { + if fwrite(base, 1, rowBytes * h, stdout) != rowBytes * h { exit(0) } // EPIPE: consumer gone + } else { + var p = base + for _ in 0.. DispatchSourceSignal in + signal(sig, SIG_IGN) + let src = DispatchSource.makeSignalSource(signal: sig, queue: .main) + src.setEventHandler { + session.stopRunning() + exit(0) + } + src.resume() + return src + } + let sigint = stop(SIGINT) + let sigterm = stop(SIGTERM) + _ = (sigint, sigterm) + + RunLoop.main.run() +} + +// ---- arg parsing ---- +var args = Array(CommandLine.arguments.dropFirst()) +guard let cmd = args.first else { + FileHandle.standardError.write("usage: nanokvm-capture list | stream --device --width --height --fps \n".data(using: .utf8)!) + exit(2) +} +args.removeFirst() + +switch cmd { +case "list": + listCommand() +case "stream": + var device = "USB3 Video" + var width = 1920 + var height = 1080 + var fps = 60.0 + var i = 0 + while i < args.count - 1 { + switch args[i] { + case "--device": device = args[i + 1] + case "--width": width = Int(args[i + 1]) ?? width + case "--height": height = Int(args[i + 1]) ?? height + case "--fps": fps = Double(args[i + 1]) ?? fps + default: break + } + i += 2 + } + streamCommand(device: device, width: width, height: height, fps: fps) +default: + FileHandle.standardError.write("unknown command: \(cmd)\n".data(using: .utf8)!) + exit(2) +} diff --git a/desktop/notarize.js b/desktop/notarize.js index 30fbe0ca..9c37d55f 100644 --- a/desktop/notarize.js +++ b/desktop/notarize.js @@ -6,6 +6,11 @@ exports.default = async function notarizing(context) { return } + if (!process.env.APPLE_ID || !process.env.APPLE_APP_SPECIFIC_PASSWORD || !process.env.APPLE_TEAM_ID) { + console.log(' • skipped notarization: APPLE_ID / APPLE_APP_SPECIFIC_PASSWORD / APPLE_TEAM_ID not set') + return + } + const appName = context.packager.appInfo.productFilename return await notarize({ diff --git a/desktop/package.json b/desktop/package.json index e422c838..6c6ae99a 100644 --- a/desktop/package.json +++ b/desktop/package.json @@ -14,12 +14,13 @@ "start": "electron-vite preview", "dev": "electron-vite dev", "build": "npm run typecheck && electron-vite build", + "build:capture": "swiftc -O -o resources/nanokvm-capture native/nanokvm-capture.swift", "build:unpack": "npm run build && electron-builder --dir", "build:win": "npm run build && electron-builder --win", - "build:mac": "electron-vite build && electron-builder --mac", + "build:mac": "npm run build:capture && electron-vite build && electron-builder --mac", "build:linux": "electron-vite build && electron-builder --linux", "build:win-full": "npm run build && electron-builder --win --x64 --arm64", - "build:mac-full": "electron-vite build && electron-builder --mac --x64 --arm64", + "build:mac-full": "npm run build:capture && electron-vite build && electron-builder --mac --x64 --arm64", "build:linux-full": "electron-vite build && electron-builder --linux --x64 --arm64" }, "dependencies": { @@ -92,7 +93,7 @@ "minimatch@<9.0.6": ">=9.0.6", "minimatch@>=10.0.0 <10.2.1": ">=10.2.1", "tar@<7.5.8": ">=7.5.8", - "ajv@<6.14.0": ">=6.14.0", + "ajv@<6.14.0": ">=6.14.0 <7", "lodash@<4.17.23": ">=4.17.23", "@isaacs/brace-expansion@<=5.0.0": ">=5.0.1" } diff --git a/desktop/src/common/ipc-events.ts b/desktop/src/common/ipc-events.ts index b2b43e21..525b8006 100644 --- a/desktop/src/common/ipc-events.ts +++ b/desktop/src/common/ipc-events.ts @@ -6,6 +6,12 @@ export enum IpcEvents { REQUEST_MEDIA_PERMISSION = 'request-media-permission', SET_FULL_SCREEN = 'set-full-screen', + GET_CAPTURE_DEVICES = 'get-capture-devices', + START_CAPTURE = 'start-capture', + STOP_CAPTURE = 'stop-capture', + CAPTURE_PORT = 'capture-port', + CAPTURE_ERROR = 'capture-error', + GET_SERIAL_PORTS = 'get-serial-ports', OPEN_SERIAL_PORT = 'open-serial-port', OPEN_SERIAL_PORT_RSP = 'open-serial-port-rsp', diff --git a/desktop/src/main/capture/backends.ts b/desktop/src/main/capture/backends.ts new file mode 100644 index 00000000..9676e34c --- /dev/null +++ b/desktop/src/main/capture/backends.ts @@ -0,0 +1,248 @@ +// Platform backends for the uncompressed-capture engine. Each backend +// enumerates devices/modes and provides the command for a capture process +// that writes raw yuyv422 (YUY2) frames to stdout. +// +// - macOS: bundled nanokvm-capture Swift helper (native AVFoundation — +// ffmpeg's avfoundation input silently freezes on >=1080p uncompressed +// frames). Reports the actually-delivered dimensions via a stderr META +// line (awaitMeta). +// - Windows: system ffmpeg with DirectShow (devices are addressed by name). +// - Linux: system ffmpeg with V4L2 (devices are addressed by /dev/videoN). +import { execFile } from 'child_process' +import { existsSync, readdirSync, readFileSync } from 'fs' +import { join } from 'path' + +import type { Backend, Mode, ResolvedCapture, StreamSpec } from './engine' + +function ffmpegPath(): string { + const candidates = [ + process.env.FFMPEG_PATH, + '/opt/homebrew/bin/ffmpeg', + '/usr/local/bin/ffmpeg', + '/usr/bin/ffmpeg', + '/snap/bin/ffmpeg' + ].filter(Boolean) as string[] + return candidates.find((p) => existsSync(p)) || 'ffmpeg' +} + +function execStderr(command: string, args: string[]): Promise { + return new Promise((resolve) => { + execFile(command, args, (_err, _stdout, stderr) => resolve(stderr || '')) + }) +} + +const LOW_LATENCY_ARGS = [ + '-hide_banner', + '-loglevel', + 'error', + '-fflags', + 'nobuffer', + '-flags', + 'low_delay' +] +const RAW_OUT_ARGS = ['-pix_fmt', 'yuyv422', '-f', 'rawvideo', '-'] + +// ---------------------------------------------------------------- macOS ---- + +const HELPER_CANDIDATES = [ + process.env.NANOKVM_CAPTURE_PATH, + // packaged: resources/ is asarUnpacked (binaries can't execute from asar) + join(__dirname, '../../resources/nanokvm-capture').replace('app.asar', 'app.asar.unpacked') +].filter(Boolean) as string[] + +function helperPath(): string { + const p = HELPER_CANDIDATES.find((c) => existsSync(c)) + if (!p) throw new Error('nanokvm-capture helper not found (build it with pnpm build:capture)') + return p +} + +type HelperFormat = { pixfmt: string; width: number; height: number; fps: number[] } +type HelperDevice = { name: string; formats: HelperFormat[] } + +function helperList(): Promise { + return new Promise((resolve) => { + execFile(helperPath(), ['list'], (_err, stdout) => { + try { + resolve(JSON.parse(stdout).devices as HelperDevice[]) + } catch { + resolve([]) + } + }) + }) +} + +const darwin: Backend = { + async listDevices() { + return (await helperList()).map((d) => ({ id: d.name, name: d.name })) + }, + + async listModes(id: string) { + const device = (await helperList()).find((d) => d.name === id) + const modes = new Map() + for (const f of device?.formats ?? []) { + if (f.pixfmt !== 'yuvs') continue + const key = `${f.width}x${f.height}` + const mode = modes.get(key) || { width: f.width, height: f.height, fps: [] } + for (const v of f.fps) if (v && !mode.fps.includes(v)) mode.fps.push(v) + modes.set(key, mode) + } + return [...modes.values()] + }, + + streamSpec(r: ResolvedCapture): StreamSpec { + return { + command: helperPath(), + args: [ + 'stream', + '--device', + r.device, + '--width', + String(r.width), + '--height', + String(r.height), + '--fps', + String(r.fps) + ], + awaitMeta: true + } + } +} + +// -------------------------------------------------------------- Windows ---- + +const dshow: Backend = { + async listDevices() { + const stderr = await execStderr(ffmpegPath(), [ + '-hide_banner', + '-list_devices', + 'true', + '-f', + 'dshow', + '-i', + 'dummy' + ]) + const devices: { id: string; name: string }[] = [] + for (const m of stderr.matchAll(/"(.+?)"\s+\(video\)/g)) { + devices.push({ id: m[1], name: m[1] }) + } + return devices + }, + + async listModes(id: string) { + const stderr = await execStderr(ffmpegPath(), [ + '-hide_banner', + '-list_options', + 'true', + '-f', + 'dshow', + '-i', + `video=${id}` + ]) + const modes = new Map() + for (const m of stderr.matchAll(/pixel_format=yuyv422.*?max s=(\d+)x(\d+) fps=([\d.]+)/g)) { + const width = Number(m[1]) + const height = Number(m[2]) + const fps = Number(m[3]) + const key = `${width}x${height}` + const mode = modes.get(key) || { width, height, fps: [] } + if (fps && !mode.fps.includes(fps)) mode.fps.push(fps) + modes.set(key, mode) + } + return [...modes.values()] + }, + + streamSpec(r: ResolvedCapture): StreamSpec { + return { + command: ffmpegPath(), + args: [ + ...LOW_LATENCY_ARGS, + '-f', + 'dshow', + '-rtbufsize', + '128M', + '-pixel_format', + 'yuyv422', + '-video_size', + `${r.width}x${r.height}`, + '-framerate', + String(r.fps), + '-i', + `video=${r.device}`, + ...RAW_OUT_ARGS + ], + awaitMeta: false + } + } +} + +// ---------------------------------------------------------------- Linux ---- + +const v4l2: Backend = { + async listDevices() { + let nodes: string[] = [] + try { + nodes = readdirSync('/dev').filter((f) => /^video\d+$/.test(f)) + } catch { + /* no /dev access */ + } + nodes.sort((a, b) => Number(a.slice(5)) - Number(b.slice(5))) + return nodes.map((node) => { + let name = node + try { + name = readFileSync(`/sys/class/video4linux/${node}/name`, 'utf8').trim() + } catch { + /* sysfs name unavailable */ + } + return { id: `/dev/${node}`, name: `${name} (/dev/${node})` } + }) + }, + + async listModes(id: string) { + // "[video4linux2 ...] Raw : yuyv422 : YUYV 4:2:2 : 640x480 1280x720 ..." + const stderr = await execStderr(ffmpegPath(), [ + '-hide_banner', + '-f', + 'v4l2', + '-list_formats', + 'all', + '-i', + id + ]) + const modes: Mode[] = [] + const line = stderr.split('\n').find((l) => l.includes('yuyv422')) + if (line) { + for (const m of line.matchAll(/(\d+)x(\d+)/g)) { + // fps per size isn't listed here; the driver clamps to what it supports + modes.push({ width: Number(m[1]), height: Number(m[2]), fps: [] }) + } + } + return modes + }, + + streamSpec(r: ResolvedCapture): StreamSpec { + return { + command: ffmpegPath(), + args: [ + ...LOW_LATENCY_ARGS, + '-f', + 'v4l2', + '-input_format', + 'yuyv422', + '-video_size', + `${r.width}x${r.height}`, + '-framerate', + String(r.fps), + '-i', + r.device, + ...RAW_OUT_ARGS + ], + awaitMeta: false + } + } +} + +export function pickBackend(): Backend { + if (process.platform === 'darwin') return darwin + if (process.platform === 'win32') return dshow + return v4l2 +} diff --git a/desktop/src/main/capture/engine.ts b/desktop/src/main/capture/engine.ts new file mode 100644 index 00000000..c3230c03 --- /dev/null +++ b/desktop/src/main/capture/engine.ts @@ -0,0 +1,270 @@ +// Uncompressed (YUY2/yuyv422) capture engine. A platform backend (see +// backends.ts) enumerates devices and provides a capture process that writes +// raw frames to stdout; this module assembles exact-size frames (O(n) fill +// buffer — a naive Buffer.concat is O(n^2) and throttles the pipeline) and +// pushes each one to the renderer over a MessagePortMain. +import { ChildProcessWithoutNullStreams, spawn } from 'child_process' +import { performance } from 'perf_hooks' +import type { MessagePortMain } from 'electron' + +import { pickBackend } from './backends' + +export type CaptureDevice = { index: number; name: string } +export type CaptureOptions = { + deviceName?: string + deviceIndex?: number + width: number + height: number + fps: number + requestId?: number +} +export type Mode = { width: number; height: number; fps: number[] } +// device is the backend-specific id: AVFoundation/DirectShow device NAME +// (indices are not stable — e.g. Continuity cameras shift the macOS list), +// or the /dev/videoN path on Linux. +export type ResolvedCapture = { device: string; width: number; height: number; fps: number } +export type StreamSpec = { + command: string + args: string[] + // true: the process reports actually-delivered dimensions via a stderr + // "META {json}" line before frames flow (the capture stack may serve the + // signal-native mode regardless of the request). false: frames are exactly + // the requested size, and the first stdout data signals a healthy start. + awaitMeta: boolean +} +export type Backend = { + listDevices(): Promise<{ id: string; name: string }[]> + listModes(id: string): Promise + streamSpec(resolved: ResolvedCapture): StreamSpec +} + +const backend = pickBackend() + +const nowAbs = (): number => performance.timeOrigin + performance.now() + +/** Enumerate video capture devices. */ +export async function listDevices(): Promise { + return (await backend.listDevices()).map((d, index) => ({ index, name: d.name })) +} + +/** Pick the most likely capture device when the caller doesn't specify one. */ +function autoPick(devices: T[]): T | undefined { + return ( + devices.find( + (d) => + /usb|video|capture|hdmi|kvm/i.test(d.name) && + !/facetime|desk view|iphone|capture screen/i.test(d.name) + ) || devices[0] + ) +} + +/** + * Resolve a requested capture to a device + a mode the device advertises. + * If the requested resolution isn't offered, fall back to the largest one. + */ +export async function resolveCapture(opts: CaptureOptions): Promise { + const devices = await backend.listDevices() + const byName = opts.deviceName ? devices.find((d) => d.name === opts.deviceName) : undefined + const byIndex = opts.deviceIndex !== undefined ? devices[opts.deviceIndex] : undefined + const device = byName || byIndex || autoPick(devices) + if (!device) throw new Error('no capture device found') + + const modes = await backend.listModes(device.id) + + let { width, height } = opts + let mode = modes.find((m) => m.width === width && m.height === height) + if (!mode && modes.length) { + mode = modes.reduce((a, b) => (b.width * b.height > a.width * a.height ? b : a)) + width = mode.width + height = mode.height + } + + // Highest advertised fps <= requested (small tolerance), else the lowest. + // Backends that don't report per-mode rates (v4l2) leave fps empty and the + // driver clamps the requested rate itself. + let fps = opts.fps + if (mode && mode.fps.length) { + const candidates = [...mode.fps].sort((a, b) => a - b) + const atMost = candidates.filter((f) => f <= opts.fps + 0.5) + fps = atMost.length ? atMost[atMost.length - 1] : candidates[0] + } + + return { device: device.id, width, height, fps } +} + +export class CaptureSession { + private proc: ChildProcessWithoutNullStreams | null = null + private stopping = false + + /** + * Spawn the capture process and resolve with the frame dimensions once the + * stream is up (META line, or first data for exact-size backends). Frames + * then flow to `port`. + */ + start( + port: MessagePortMain, + opts: ResolvedCapture, + onError: (msg: string) => void + ): Promise<{ width: number; height: number }> { + this.stopping = false + + const spec = backend.streamSpec(opts) + const proc = spawn(spec.command, spec.args) + this.proc = proc + + return new Promise((resolve, reject) => { + let stderr = '' + let seq = 0 + let started = false + + const startupTimeout = setTimeout(() => { + if (!started) { + this.stop() + reject(new Error('capture start timed out')) + } + }, 10000) + + const begin = (width: number, height: number): void => { + started = true + clearTimeout(startupTimeout) + attachPump(width, height) + resolve({ width, height }) + } + + const attachPump = (width: number, height: number): void => { + const frameBytes = width * height * 2 // yuyv422 + const frameBuf = Buffer.allocUnsafe(frameBytes) + let filled = 0 + + // Ack-based backpressure: only one frame in flight; while the renderer + // is busy the freshest frame waits in pendingBuf (older ones drop). + // Flooding the port faster than the renderer draws backs up the message + // queue and starves its event loop (input goes dead). + let rendererReady = true + let hasPending = false + const pendingBuf = Buffer.allocUnsafe(frameBytes) + + const post = (buf: Buffer): void => { + const ab = new ArrayBuffer(frameBytes) + new Uint8Array(ab).set(buf) + rendererReady = false + try { + port.postMessage({ seq: seq++, emitAbs: nowAbs(), width, height, buf: ab }) + } catch { + this.stop() + } + } + + port.on('message', () => { + if (hasPending) { + hasPending = false + post(pendingBuf) + } else { + rendererReady = true + } + }) + + proc.stdout.on('data', (chunk: Buffer) => { + let off = 0 + while (off < chunk.length) { + const take = Math.min(frameBytes - filled, chunk.length - off) + chunk.copy(frameBuf, filled, off, off + take) + filled += take + off += take + if (filled < frameBytes) break + + filled = 0 + if (rendererReady) { + post(frameBuf) + } else { + frameBuf.copy(pendingBuf) + hasPending = true + } + } + }) + } + + if (spec.awaitMeta) { + proc.stderr.on('data', (d: Buffer) => { + const text = d.toString() + const meta = text.match(/^META\s+(\{.*\})/m) + if (meta && !started) { + try { + const { width, height } = JSON.parse(meta[1]) + begin(width, height) + } catch (e) { + this.stop() + clearTimeout(startupTimeout) + reject(e) + } + return + } + stderr += text + }) + } else { + proc.stderr.on('data', (d: Buffer) => { + stderr += d.toString() + }) + proc.stdout.once('data', () => { + if (!started) begin(opts.width, opts.height) + }) + } + + proc.on('close', () => { + const wasCurrent = this.proc === proc + if (wasCurrent) this.proc = null + clearTimeout(startupTimeout) + // Any unexpected end (device unplugged, signal loss) => notify the + // renderer so it doesn't keep showing a frozen frame. + if (wasCurrent && !this.stopping) { + const msg = stderr.trim() || 'capture ended (device disconnected?)' + if (started) onError(msg) + else reject(new Error(msg)) + } + }) + proc.on('error', (e) => { + clearTimeout(startupTimeout) + const err = e as NodeJS.ErrnoException + const msg = + err.code === 'ENOENT' + ? spec.awaitMeta + ? `capture helper not found: ${spec.command}` + : 'FFmpeg not found — install FFmpeg or set FFMPEG_PATH' + : e.message + if (started) onError(msg) + else reject(new Error(msg)) + }) + }) + } + + stop(): void { + this.stopping = true + if (this.proc) { + const proc = this.proc + this.proc = null + // Graceful first: SIGINT lets the process tear down the capture session + // cleanly (hard kills can wedge the device / camera daemon). On Windows + // there are no signals — ffmpeg quits on 'q' via stdin. + try { + if (process.platform === 'win32') { + proc.stdin.write('q') + } else { + proc.kill('SIGINT') + } + } catch { + /* already gone */ + } + setTimeout(() => { + try { + proc.kill('SIGKILL') + } catch { + /* already gone */ + } + }, 1500) + } + } + + get running(): boolean { + return this.proc !== null + } +} diff --git a/desktop/src/main/events/capture.ts b/desktop/src/main/events/capture.ts new file mode 100644 index 00000000..cad3d483 --- /dev/null +++ b/desktop/src/main/events/capture.ts @@ -0,0 +1,52 @@ +import { ipcMain, IpcMainInvokeEvent, MessageChannelMain } from 'electron' + +import { IpcEvents } from '../../common/ipc-events' +import { + CaptureDevice, + CaptureOptions, + CaptureSession, + listDevices, + resolveCapture +} from '../capture/engine' + +const session = new CaptureSession() + +export function registerCapture(): void { + ipcMain.handle(IpcEvents.GET_CAPTURE_DEVICES, getCaptureDevices) + ipcMain.handle(IpcEvents.START_CAPTURE, startCapture) + ipcMain.handle(IpcEvents.STOP_CAPTURE, stopCapture) +} + +async function getCaptureDevices(): Promise { + return listDevices() +} + +async function startCapture(e: IpcMainInvokeEvent, opts: CaptureOptions): Promise { + session.stop() + + // Resolve to a device (by name) + an advertised mode; the helper reports the + // ACTUAL delivered dimensions once the stream starts (the UVC stack may serve + // the signal-native mode regardless of the request). + const resolved = await resolveCapture(opts) + + const { port1, port2 } = new MessageChannelMain() + port1.start() + + const actual = await session.start(port1, resolved, (msg) => { + if (!e.sender.isDestroyed()) e.sender.send(IpcEvents.CAPTURE_ERROR, msg) + }) + + // Hand the renderer end of the channel to the page (preload forwards it on). + // requestId lets the renderer ignore ports from superseded start requests. + e.sender.postMessage( + IpcEvents.CAPTURE_PORT, + { width: actual.width, height: actual.height, requestId: opts.requestId }, + [port2] + ) + return true +} + +async function stopCapture(): Promise { + session.stop() + return true +} diff --git a/desktop/src/main/events/index.ts b/desktop/src/main/events/index.ts index 2ad88c16..9a967c2e 100644 --- a/desktop/src/main/events/index.ts +++ b/desktop/src/main/events/index.ts @@ -1,3 +1,4 @@ export * from './app' +export * from './capture' export * from './serial-port' export * from './updater' diff --git a/desktop/src/main/index.ts b/desktop/src/main/index.ts index d3469ee7..783ed202 100644 --- a/desktop/src/main/index.ts +++ b/desktop/src/main/index.ts @@ -19,7 +19,10 @@ function createWindow(): void { ...(process.platform === 'linux' ? { icon } : {}), webPreferences: { preload: join(__dirname, '../preload/index.js'), - sandbox: false + sandbox: false, + // Keep the capture canvas (rAF-driven) updating when the window is not + // focused — a KVM is often in the background while you use the target. + backgroundThrottling: false } }) @@ -54,6 +57,7 @@ app.whenReady().then(() => { events.registerApp() events.registerSerialPort() + events.registerCapture() createWindow() diff --git a/desktop/src/preload/index.ts b/desktop/src/preload/index.ts index 8fbf134e..887a747d 100644 --- a/desktop/src/preload/index.ts +++ b/desktop/src/preload/index.ts @@ -1,5 +1,13 @@ import { electronAPI } from '@electron-toolkit/preload' -import { contextBridge } from 'electron' +import { contextBridge, ipcRenderer } from 'electron' + +import { IpcEvents } from '../common/ipc-events' + +// MessagePorts cannot cross the contextBridge, so forward the capture channel's +// port directly into the page via window.postMessage (with transfer). +ipcRenderer.on(IpcEvents.CAPTURE_PORT, (e, meta) => { + window.postMessage({ type: 'capture-port', meta }, '*', e.ports) +}) // Custom APIs for renderer const api = {} diff --git a/desktop/src/renderer/src/App.tsx b/desktop/src/renderer/src/App.tsx index 85edcaf2..0668e220 100644 --- a/desktop/src/renderer/src/App.tsx +++ b/desktop/src/renderer/src/App.tsx @@ -1,7 +1,7 @@ -import { ReactElement, useEffect, useState } from 'react' -import { Result, Spin } from 'antd' +import { ReactElement, useEffect, useRef, useState } from 'react' +import { message, Result, Spin } from 'antd' import clsx from 'clsx' -import { useAtomValue, useSetAtom } from 'jotai' +import { useAtom, useAtomValue } from 'jotai' import { useTranslation } from 'react-i18next' import { useMediaQuery } from 'react-responsive' @@ -12,16 +12,20 @@ import { Menu } from '@renderer/components/menu' import { Mouse } from '@renderer/components/mouse' import { VirtualKeyboard } from '@renderer/components/virtual-keyboard' import { + captureDeviceAtom, + captureModeAtom, resolutionAtom, serialPortStateAtom, + sharpnessAtom, videoScaleAtom, videoStateAtom } from '@renderer/jotai/device' import { isKeyboardEnableAtom } from '@renderer/jotai/keyboard' import { mouseModeAtom, mouseStyleAtom } from '@renderer/jotai/mouse' +import { captureCamera } from '@renderer/libs/capture/capture-camera' import { camera } from '@renderer/libs/media/camera' import { requestCameraPermission } from '@renderer/libs/media/permission' -import { getVideoResolution } from '@renderer/libs/storage' +import * as storage from '@renderer/libs/storage' import type { Resolution } from '@renderer/types' type State = 'loading' | 'success' | 'failed' @@ -31,29 +35,112 @@ const App = (): ReactElement => { const isBigScreen = useMediaQuery({ minWidth: 850 }) const videoScale = useAtomValue(videoScaleAtom) - const videoState = useAtomValue(videoStateAtom) + const [videoState, setVideoState] = useAtom(videoStateAtom) + const [captureMode, setCaptureMode] = useAtom(captureModeAtom) + const [captureDevice, setCaptureDevice] = useAtom(captureDeviceAtom) + const [sharpness, setSharpness] = useAtom(sharpnessAtom) const serialPortState = useAtomValue(serialPortStateAtom) const mouseMode = useAtomValue(mouseModeAtom) const mouseStyle = useAtomValue(mouseStyleAtom) const isKeyboardEnable = useAtomValue(isKeyboardEnableAtom) - const setResolution = useSetAtom(resolutionAtom) + const [resolution, setResolution] = useAtom(resolutionAtom) const [state, setState] = useState('loading') + const [captureNonce, setCaptureNonce] = useState(0) + const prevCaptureMode = useRef(false) useEffect(() => { - const resolution = getVideoResolution() + const resolution = storage.getVideoResolution() if (resolution) { setResolution(resolution) } + setCaptureMode(storage.getCaptureMode()) + setCaptureDevice(storage.getCaptureDevice()) + setSharpness(storage.getSharpness()) requestMediaPermissions(resolution) return (): void => { camera.close() + captureCamera.close() window.electron.ipcRenderer.invoke(IpcEvents.CLOSE_SERIAL_PORT) } }, []) + // Drive the uncompressed-capture path when capture mode is on. + // Depends on `resolution` so a resolution change cleanly restarts the session. + useEffect(() => { + if (state !== 'success') return + + const wasCapture = prevCaptureMode.current + prevCaptureMode.current = captureMode + + if (captureMode) { + camera.close() + const canvas = document.getElementById('video') as HTMLCanvasElement | null + if (!canvas) return + captureCamera + .open({ + canvas, + width: resolution.width, + height: resolution.height, + fps: fpsFor(resolution.width, resolution.height), + deviceName: captureDevice || undefined, + sharpness, + onError: (msg) => { + if (/video mode changed/i.test(msg)) { + // HDMI source resolution changed — restart at the new mode + setTimeout(() => setCaptureNonce((n) => n + 1), 500) + } else { + failCapture(msg) + } + } + }) + .then(() => setVideoState('connected')) + .catch((err) => failCapture(err instanceof Error ? err.message : String(err))) + return (): void => captureCamera.close() + } + + captureCamera.close() + // Ticked -> unticked: restore the regular getUserMedia camera (the swapped-in + //