feat(document): adaptive contour-Canny for low-contrast backgrounds

[Barnabas A Nsoh (ayinloya)]

User description

Why

Document auto-capture on mobile only fires reliably when the document sits on a high-contrast surface (e.g. a card on metal) and stalls on general backgrounds (wood, matte desk, similar-toned paper).

Root cause: the contour gate — the stage that actually drives capture, since the distance/shape/fill checks all depend on finding the 4-corner outline — used a fixed cv.Canny(blurred, edges, 50, 150). That fixed pair needs a strong brightness gradient at the document border; on low-contrast backgrounds the gradient is too weak, the contour is never found, and capture never triggers.

What

Make the contour-stage Canny thresholds adaptive per frame, derived from the frame's own gradient-magnitude statistics:

• Compute Sobel gradient magnitude over the (full-res, guide-box-cropped) blurred image → mean, stddev.
• high = clamp(mean + autoCannySigma·stddev, 60, 150), low = max(15, 0.4·high).
• cv.Canny keeps its default gradient norm, so behaviour at the cap is unchanged.

The 150 ceiling is the previously-fixed value, so the high-contrast/metallic path that already works is byte-identical — this change only relaxes detection for low contrast, never tightens it. The 60 floor lets faint borders on plain backgrounds produce a contour so distance/shape gating can succeed.

Why gradient-stats, not pure-median auto-Canny

A raw-median anchor tracks brightness, so a bright-but-plain desk would push thresholds up and worsen low-contrast detection. Canny thresholds operate on gradient magnitude, so anchoring on the gradient distribution is what actually relaxes detection when the border is faint.

Tuning

• New autoCannySigma setting (default 1.0, mobile + desktop).
• TuningPanel: Edge Sensitivity (σ) slider + live Canny (lo/hi) metric (via the existing mergeDebugInfo helper) so the resolved thresholds are visible per surface.

Files

• hooks/useCardDetection.ts — CANNY_HIGH_MAX/MIN constants; adaptive Canny in the contour pass.
• DocumentAutoCapture.tsx — autoCannySigma default.
• components/TuningPanel.tsx — slider + metric.

Testing

• type-check, ESLint, Prettier all clean.
• ⚠️ Needs on-device validation (the real test): the [60,150] band and σ=1.0 are reasoned defaults, not measured. Open ⚙️ Settings on a phone, watch Canny (lo/hi) on plain vs. metallic surfaces, and tune σ so plain backgrounds capture without false-firing on busy ones. The winning σ should be locked into getOptimalDefaults before this leaves draft.

Base is document-capture-new-flow (not main) — this builds on the new capture flow.

🤖 Generated with Claude Code

---

PR Type

Enhancement, Bug fix

---

Description

• Adaptive per-frame Canny thresholds from gradient statistics for low-contrast backgrounds

• Chroma (Lab a/b) edge fusion to detect luminance-invisible card borders

• Tilt-robust shape gates using cv.minAreaRect instead of axis-aligned bbox

• Composite quality scoring, chroma-content gate, and transient-miss tolerance

---

Diagram Walkthrough

flowchart LR
  A["Frame input"] -- "Sobel gradient stats" --> B["Adaptive Canny thresholds"]
  B -- "Luminance edges" --> C["Edge map"]
  D["Lab a/b channels"] -- "Chroma Canny" --> E["Chroma edges"]
  E -- "bitwise_or" --> C
  C -- "findContours" --> F["minAreaRect shape gates"]
  F -- "Winner geometry" --> G["Chroma content gate"]
  G -- "Composite quality score" --> H["Best frame selection"]

Loading

File Walkthrough

	Relevant files
Enhancement	DocumentAutoCapture.tsx Refactor settings into shared defaults with device overrides packages/web-components/lib/components/document/src/document-auto-capture/DocumentAutoCapture.tsx Extracted shared settings (SHARED_DEFAULTS) and per-device overrides (MOBILE_OVERRIDES, DESKTOP_OVERRIDES) into named constants Added new settings: autoCannySigma, chromaEdgeFusion, chromaCannyLow/High, mobileRegionFallback, idAspectTolerance, bookDocAspectTolerance, minFillRatio, chromaContentGate, minChromaContent Simplified getOptimalDefaults() to spread shared + device-specific overrides +89/-38  TuningPanel.tsx Extend tuning panel with adaptive Canny and chroma controls packages/web-components/lib/components/document/src/document-auto-capture/components/TuningPanel.tsx Added new tuning interface fields for adaptive Canny, chroma fusion, aspect tolerance, fill ratio, and content gate Added debug info display for Canny thresholds, edge source, aspect, chroma, and quality score Added UI sliders for edge sensitivity (σ), aspect tolerances, min fill ratio, chroma Canny thresholds, and min chroma content Added toggle checkboxes for chroma edge fusion, mobile region fallback, and chroma content gate +182/-0  useCardDetection.ts Adaptive Canny, chroma fusion, tilt-robust gates, and quality scoring packages/web-components/lib/components/document/src/document-auto-capture/hooks/useCardDetection.ts Implemented adaptive Canny thresholds derived from per-frame Sobel gradient magnitude statistics (mean + σ·stddev, clamped to [60, 150]) Added chroma edge fusion: Lab a/b channel Canny edges OR'd into the luminance edge map for low-contrast borders Replaced axis-aligned boundingRect with cv.minAreaRect for tilt-invariant aspect ratio and fill ratio measurement Added chroma-content gate with rolling average to reject near-monochrome objects (keyboards, blank paper) Added mobile content-region fallback with stability frame counter Implemented composite per-frame quality score (sharpness, glare, fill, aspect, contour, chroma) replacing sharpness-only best-frame selection Added transient blur/glare miss tolerance (BEST_FRAME_MISS_TOLERANCE) to avoid discarding good captures on momentary bad frames Tightened aspect tolerance windows (id-card ±12%, passport ±10%) using tunable settings +536/-47
Formatting	DocumentCaptureInstructions.tsx Remove unused Ref import                                                                  packages/web-components/lib/components/document/src/document-capture-instructions/DocumentCaptureInstructions.tsx Removed unused Ref type import from preact +1/-1

---

---

Need help?

Type /help how to ... in the comments thread for any questions about PR-Agent usage.

Check out the documentation for more information.

[prfectionist]

Persistent review updated to latest commit c8c804d

[prfectionist]

Suggestion: If the catch block fires after some Mats were allocated (e.g. contourRgb succeeded but contourLab threw), those intermediate Mats leak because the catch only sets a flag. The shared finally cleanup only frees variables that are still non-null, but contourRgb is explicitly set to null after delete(). However, if the throw happens between allocation and the delete() call (e.g. cv.cvtColor for Lab throws after contourRgb was created), contourRgb is still non-null and will be cleaned up by finally — but contourFull was already deleted and set to null before the throw, so it's fine. The real risk is labPlanes: if cv.split succeeds but a later line throws, labPlanes is non-null and the finally block calls delete() on it — but aPlane/bPlane (borrowed views from labPlanes.get()) are never explicitly deleted in the catch path. Since labPlanes.delete() should free its children, this is likely OK. More critically, aAbs and bAbs are local variables inside the try block and are NOT in the shared finally cleanup list — if the throw happens after their allocation but before their delete() calls, they will leak permanently. [possible issue, importance: 6]

Suggested change

	if (chromaFusionOn) {
	try {
	contourRgb = new cv.Mat();
	cv.cvtColor(contourFull, contourRgb, cv.COLOR_RGBA2RGB, 0);
	contourFull.delete();
	contourFull = null;
	contourLab = new cv.Mat();
	cv.cvtColor(contourRgb, contourLab, cv.COLOR_RGB2Lab, 0);
	contourRgb.delete();
	contourRgb = null;
	labPlanes = new cv.MatVector();
	cv.split(contourLab, labPlanes); // [0]=L, [1]=a, [2]=b
	contourLab.delete();
	contourLab = null;
	// Borrowed views — owned by labPlanes, never deleted directly.
	const aPlane = labPlanes.get(1);
	const bPlane = labPlanes.get(2);
	const chromaK = new cv.Size(7, 7); // chroma is noisier than luma
	aBlur = new cv.Mat();
	bBlur = new cv.Mat();
	cv.GaussianBlur(aPlane, aBlur, chromaK, 0, 0, cv.BORDER_DEFAULT);
	cv.GaussianBlur(bPlane, bBlur, chromaK, 0, 0, cv.BORDER_DEFAULT);
	// Per-pixel chroma magnitude |a-128| + |b-128| (Lab neutral =
	// 128). Near 0 for a neutral white/gray object (keyboard, paper,
	// desk); high where a colour ID has a photo/printing. Kept alive
	// for the Level 2 content gate, measured per detected rectangle
	// below so background colour can't mask a monochrome object.
	const aAbs = new cv.Mat();
	const bAbs = new cv.Mat();
	cv.convertScaleAbs(aPlane, aAbs, 1, -128);
	cv.convertScaleAbs(bPlane, bAbs, 1, -128);
	chromaMag = new cv.Mat();
	cv.addWeighted(aAbs, 1, bAbs, 1, 0, chromaMag);
	aAbs.delete();
	bAbs.delete();
	const chromaLow = settingsRef.current.chromaCannyLow ?? 15;
	const chromaHigh = settingsRef.current.chromaCannyHigh ?? 40;
	aEdges = new cv.Mat();
	bEdges = new cv.Mat();
	cv.Canny(aBlur, aEdges, chromaLow, chromaHigh);
	cv.Canny(bBlur, bEdges, chromaLow, chromaHigh);
	cv.bitwise_or(edges, aEdges, edges);
	cv.bitwise_or(edges, bEdges, edges);
	edgeSource = 'lum+chroma';
	labPlanes.delete(); // frees L/a/b incl. borrowed aPlane/bPlane
	labPlanes = null;
	aBlur.delete();
	aBlur = null;
	bBlur.delete();
	bBlur = null;
	aEdges.delete();
	aEdges = null;
	bEdges.delete();
	bEdges = null;
	} catch {
	// Lab path failed on this device — disable for the session and
	// continue with the luminance edges already in `edges`.
	chromaUnavailableRef.current = true;
	edgeSource = 'lum';
	}
	if (chromaFusionOn) {
	let aAbs: any = null;
	let bAbs: any = null;
	try {
	contourRgb = new cv.Mat();
	cv.cvtColor(contourFull, contourRgb, cv.COLOR_RGBA2RGB, 0);
	contourFull.delete();
	contourFull = null;
	contourLab = new cv.Mat();
	cv.cvtColor(contourRgb, contourLab, cv.COLOR_RGB2Lab, 0);
	contourRgb.delete();
	contourRgb = null;
	labPlanes = new cv.MatVector();
	cv.split(contourLab, labPlanes); // [0]=L, [1]=a, [2]=b
	contourLab.delete();
	contourLab = null;
	const aPlane = labPlanes.get(1);
	const bPlane = labPlanes.get(2);
	const chromaK = new cv.Size(7, 7);
	aBlur = new cv.Mat();
	bBlur = new cv.Mat();
	cv.GaussianBlur(aPlane, aBlur, chromaK, 0, 0, cv.BORDER_DEFAULT);
	cv.GaussianBlur(bPlane, bBlur, chromaK, 0, 0, cv.BORDER_DEFAULT);
	aAbs = new cv.Mat();
	bAbs = new cv.Mat();
	cv.convertScaleAbs(aPlane, aAbs, 1, -128);
	cv.convertScaleAbs(bPlane, bAbs, 1, -128);
	chromaMag = new cv.Mat();
	cv.addWeighted(aAbs, 1, bAbs, 1, 0, chromaMag);
	aAbs.delete(); aAbs = null;
	bAbs.delete(); bAbs = null;
	const chromaLow = settingsRef.current.chromaCannyLow ?? 15;
	const chromaHigh = settingsRef.current.chromaCannyHigh ?? 40;
	aEdges = new cv.Mat();
	bEdges = new cv.Mat();
	cv.Canny(aBlur, aEdges, chromaLow, chromaHigh);
	cv.Canny(bBlur, bEdges, chromaLow, chromaHigh);
	cv.bitwise_or(edges, aEdges, edges);
	cv.bitwise_or(edges, bEdges, edges);
	edgeSource = 'lum+chroma';
	labPlanes.delete(); labPlanes = null;
	aBlur.delete(); aBlur = null;
	bBlur.delete(); bBlur = null;
	aEdges.delete(); aEdges = null;
	bEdges.delete(); bEdges = null;
	} catch {
	chromaUnavailableRef.current = true;
	edgeSource = 'lum';
	if (aAbs) { try { aAbs.delete(); } catch {} }
	if (bAbs) { try { bAbs.delete(); } catch {} }
	}
	}

[prfectionist]

Suggestion: The Sobel/magnitude computation allocates 5 temporary Mats (sobelX, sobelY, gradMag, gradMean, gradStdDev) every frame. If any OpenCV call between allocations throws, the already-allocated Mats leak because they are local variables not tracked in the shared finally cleanup. These should either be added to the outer cleanup list or wrapped in a local try/finally. Given this runs every frame, even a single leaked Mat per session accumulates significant WASM heap pressure. [general, importance: 5]

Suggested change

	const sobelX = new cv.Mat();
	const sobelY = new cv.Mat();
	cv.Sobel(blurred, sobelX, cv.CV_32F, 1, 0, 3);
	cv.Sobel(blurred, sobelY, cv.CV_32F, 0, 1, 3);
	const gradMag = new cv.Mat();
	cv.magnitude(sobelX, sobelY, gradMag);
	sobelX.delete();
	sobelY.delete();
	const gradMean = new cv.Mat();
	const gradStdDev = new cv.Mat();
	cv.meanStdDev(gradMag, gradMean, gradStdDev);
	const gMean = gradMean.doubleAt(0, 0);
	const gStd = gradStdDev.doubleAt(0, 0);
	gradMag.delete();
	gradMean.delete();
	gradStdDev.delete();
	// Declare at the outer scope alongside other cleanup-tracked Mats
	let sobelX: any = null;
	let sobelY: any = null;
	let gradMag: any = null;
	let gradMean: any = null;
	let gradStdDev: any = null;
	// ... then in the computation block:
	sobelX = new cv.Mat();
	sobelY = new cv.Mat();
	cv.Sobel(blurred, sobelX, cv.CV_32F, 1, 0, 3);
	cv.Sobel(blurred, sobelY, cv.CV_32F, 0, 1, 3);
	gradMag = new cv.Mat();
	cv.magnitude(sobelX, sobelY, gradMag);
	sobelX.delete(); sobelX = null;
	sobelY.delete(); sobelY = null;
	gradMean = new cv.Mat();
	gradStdDev = new cv.Mat();
	cv.meanStdDev(gradMag, gradMean, gradStdDev);
	const gMean = gradMean.doubleAt(0, 0);
	const gStd = gradStdDev.doubleAt(0, 0);
	gradMag.delete(); gradMag = null;
	gradMean.delete(); gradMean = null;
	gradStdDev.delete(); gradStdDev = null;

[prfectionist]

Suggestion: If cRect.x or cRect.y is negative (possible with contour bounding rects near edges), cx/cy become 0 but cw/ch are computed from chromaMag.cols - 0 and cRect.width — the width could exceed the actual available region. Also, if chromaMag.cols - cx is negative (shouldn't happen with the max(0) clamp, but cRect.width could be larger than chromaMag.cols), cw could be negative, causing cv.Rect to throw or produce undefined behavior. Clamp cw and ch to be non-negative before the cw > 0 && ch > 0 check. [possible issue, importance: 6]

Suggested change

	const cRect = cv.boundingRect(bestContour);
	const cx = Math.max(0, cRect.x);
	const cy = Math.max(0, cRect.y);
	const cw = Math.min(chromaMag.cols - cx, cRect.width);
	const ch = Math.min(chromaMag.rows - cy, cRect.height);
	const cRect = cv.boundingRect(bestContour);
	const cx = Math.max(0, Math.min(cRect.x, chromaMag.cols - 1));
	const cy = Math.max(0, Math.min(cRect.y, chromaMag.rows - 1));
	const cw = Math.max(0, Math.min(chromaMag.cols - cx, cRect.width));
	const ch = Math.max(0, Math.min(chromaMag.rows - cy, cRect.height));

[prfectionist]

Suggestion: When no candidate is found, the chroma window is cleared. However, winnerGeomRef.current retains stale geometry from the previous frame's winner. If a new candidate appears next frame and immediately reaches the quality-score section, it will use the stale winnerGeomRef values (from a different object) for one frame before being overwritten. Reset winnerGeomRef.current alongside the chroma window to avoid scoring a new candidate with the previous object's geometry. [general, importance: 5]

Suggested change

	chromaWindowRef.current = [];
	chromaWindowRef.current = [];
	winnerGeomRef.current = { aspect: 0, fillRatio: 0, synthetic: false };

[prfectionist]

Persistent review updated to latest commit 55f094d

[Barnabas A Nsoh (ayinloya)]

This branch has been deployed to s3 / cloudfront.

✅ Preview URL for Smart Camera Web:

https://cdn.smileidentity.com/js/preview-feat/mobile-capture-adaptive-canny/smart-camera-web.js

✅ Preview URL for Embed:

https://cdn.smileidentity.com/inline/preview-feat/mobile-capture-adaptive-canny/js/script.min.js

✅ Preview URL for Web Client (Sandbox):

https://d3qr3ogefp3sxy.cloudfront.net

✅ Preview URL for Web Client (Production):

https://d2zrugva4pgdqs.cloudfront.net