Repository navigation
Make power cone projection scale-invariant (fixes #491) - #493
Merged
Merged
Conversation
proj_power_cone used absolute tolerances (POW_CONE_TOL = 1e-9) for the membership checks and the Newton stopping rule. When the homogeneous, normalized iterates are small, Newton stopped with r far from the root and SCS reported SCS_SOLVED with the slack well outside the cone (issue #491: x = 0.1238 vs optimal 0.1). - Project v scaled to unit inf-norm and rescale the result, so all tolerances are relative to |v|. - Stop Newton on step size rather than |f|: near the polar boundary f is very flat in r, so a small |f| does not imply an accurate r. - Tighten POW_CONE_TOL to 1e-12 (now relative). - Add test_power_cone_small_scale regression test from the issue. Fixes #491 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI showed the regression test's dual cone check failing on Linux: for points just outside the polar cone, f(r) has a near-double root and the Newton iteration either crawled to the iteration cap or stopped on a spurious small step clamped to r = rh. - Evaluate x(r) in a cancellation-free form and drop the 1e-12 floor, which overrode the true value when a component was tiny and negative. - f(r) = phi(r) - r with phi concave, so the sign of f brackets the root. Keep [lo, hi] and fall back to bisection when a step leaves it. - On the right of the root also try the Newton step for f(r) / r, which converges fast near the double root. - Raise POW_CONE_MAX_ITERS to 50 (only hit in hard cases; mean ~7.5 on well-scaled inputs). - Use c = -100 in the regression test (still fails on master) so |y| is moderate for verify_solution_correct's absolute dual cone check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit changed the regression test's objective from -1000 to -100 to get past verify_solution_correct's absolute 1e-5 dual cone check. That masked a real inaccuracy rather than fixing it, so restore the issue's -1000. The inaccuracy: the Newton loop stopped once |r_new - r| <= 1e-12 in normalized units, but for inputs near the apex (e.g. -y for the issue's dual solution, |y| ~ 7500) the root r is ~1e-15 while x(r) is ~1000x more sensitive to r than r itself. Against a 60-digit reference that error reached 1.25e-9 relative (~9e-6 absolute here), right at the test's threshold. Stopping relative to r brings it to ~1.6e-15. Also use a 1e-6 tolerance under SFLOAT so it is reachable in float. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A converged Newton step lands on r, which is an end of the bracket, so the strict bracket test rejected it and replaced it with a bisection midpoint, wasting many iterations. Test convergence on the raw step (closed bracket) before the bisection fallback. - The step test relative to r cannot be met when the root is at or near 0 (near the apex, or p close to 0 or 1), so those calls ran to the iteration cap. Also stop once the bracket width, scaled by the sensitivity of x and y to r, is below POW_CONE_TOL. - Tone down the comment on the f(r) / r step: it only helps once r is close to the root. Against a 60-digit reference this keeps max error <= 1e-12 relative on typical, near-apex and extreme-p inputs, with no cap hits, and mean iterations on typical inputs drop from 8.1 to 5.9 (master: 4.9). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Converging iterates are typically just outside the cone, with the root near rh. Starting at rh / 2, Newton on the concave f overshoots past the root from the left and the bracket safeguard falls back to bisection. Start at rh instead when x(rh) = xh and y(rh) = yh are positive and phi(rh) >= rh / 2; the guard matters, since with a tiny xh the slope at rh is huge and the first step is spuriously small. Against the 60-digit reference: points just outside the primal boundary drop from 17.1 to 1.5 mean iterations (max error 8.7e-9 -> 1.1e-9); typical, near-apex, extreme-p and adversarial sets are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
63c0fa4 started the Newton solve at r = rh when phi(rh) >= rh / 2. For small p that does not exclude a tiny positive component (xh^p ~ 1), and there dx/dr = -a * rh / xh is so large that the first step rounds to 0, which the step test took for convergence: e.g. p = 0.001 and v = (1e-12, 1, 1.848) was returned unprojected (error 0.43). - Start at rh only when |dx/dr| and |dy/dr| at rh are at most POW_CONE_START_SLOPE (1e3; 10 under SFLOAT). - Converged step: also require |step| * max(1, |dx/dr|, |dy/dr|) <= POW_CONE_TOL, in addition to the relative-to-r test. (The sensitivity test alone, without the relative one, let tiny components come out inaccurate relative to themselves and made solves on such problems worse.) - Bracket stop: drop hi - lo <= POW_CONE_TOL * hi, which could fire with large sensitivity; stop when (hi - lo) * sens <= POW_CONE_TOL or when the bracket is a few ulps wide (new POW_CONE_EPS). - When a Newton step leaves the bracket, use an Illinois regula falsi step before bisecting: bisection alone needed ~50 iterations when the root is within ulps of rh (a point just outside the cone with a tiny component), and hit the iteration cap. - Add test_power_cone_proj_tiny_component, checking the projection of such points against 60-digit reference values; it fails on 63c0fa4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review nits: take POW_CONE_EPS from <float.h> instead of hard-coding it, note why POW_CONE_START_SLOPE is smaller under SFLOAT, and say what f_lo / f_hi are where they are declared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Profiling on projection inputs recorded from CVXPY power-cone models showed nearly every call runs the Newton loop, and that most of the extra cost over master was extra Newton evaluations from poor starting points: - With exactly one negative component (the common case for ADMM iterates), start at the root of the small-r approximation x(r) ~ a * rh * r / |xh|, i.e. r0 = (a * rh / |xh|)^(a / (1 - a)) * yh (or the mirror image), instead of rh / 2. The root is often tiny and Newton from rh / 2 took many shrinking steps to reach it. - Start at r = rh only when the point is close to the cone (phi(rh) >= rh / 2), as well as having moderate slopes there. - Take the Newton step on f(r) / r only when Newton on f would more than halve r (its slow regime near the apex); elsewhere it overshoots left of the root and alternates with the f step. - Compute phi(0) once for the membership check and the bracket. Accuracy against a 60-digit reference is unchanged or better on every point set, with no iteration-cap hits. On the CVXPY benchmark, cone time per iteration relative to master goes from +24% to +8% (geometric mean), and is below master on p-norm, portfolio and p = 0.95 utility models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
720f475 restricted the Newton step on f(r) / r to when Newton on f would more than halve r. Review showed that trigger is false in the near-double-root regime the step exists for (the f-Newton ratio there is above 1/2), so near-polar inputs, issue #491's class, lost it. A trigger on both components being negative fixed the projection-level cost, but whole solves were what mattered: on 390 jittered tiny-u solves (min -z s.t. (u, 1, z) in K_p), unsolved / total iterations were r_new < r / 2 (720f475): 18 / 2.77M xh < 0 && yh < 0: 26 / 3.48M unrestricted (this commit): 16 / 2.50M and the unrestricted step was also best on the reviewer's own sweep. The new starting points from 720f475 are kept. Accuracy against the 60-digit reference is unchanged on every point set; mean evaluations go from 30.0 to 25.3 near the issue's dual solution and 11.9 to 10.7 on the adversarial set, and 4.7 to 4.8 on typical points. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #491.
Problem
proj_power_coneused absolute tolerances (POW_CONE_TOL = 1e-9) for the membership checks and the Newton stopping rule|f| < tol. With normalization and Anderson acceleration, the iterates that reach the projection in the issue's problem are tiny (~1e-6 to 1e-9). Newton stopped atr = 2.69e-9,f = -5.2e-10, which passes the absolute test but is ~24% from the root. That is exactly the reportedx = 0.1238vs. optimal0.1. The residuals don't catch this because they assume the projection is exact, so SCS returnedSCS_SOLVED, and it does so at everyeps, not only1e-8.Fix (
src/cones.c,proj_power_cone)v / ||v||_infand rescale the result. Projection onto a cone is positively homogeneous, so the tolerances become relative to|v|.x(r): whenxh < 0,x(r)uses a form without cancellation, and the1e-12floor is gone. The floor overrode the true value when a component was tiny and negative.phi(r) = x^a y^(1-a)is concave, so the sign off = phi - rbrackets the root. When a Newton step leaves[lo, hi](from the left of a concavefit overshoots), an Illinois regula falsi step is used, then bisection. Bisection alone needed ~50 iterations when the root is within a few ulps ofrh.f(r)/ris tried (and used if it stays in the bracket). That form removes the trivial root at 0 and converges much faster than Newton onf, which is only linear near the double root. Restricting it to the near-apex regime saved a few projection evaluations on some inputs but made whole solves worse (see the solver sweep below), so it is taken wheneverf < 0.r = rh/2, with two better guesses where they apply.xh,yhnegative is the most common case for ADMM iterates. There, start at the root of the small-rapproximationx(r) ~ a * rh * r / |xh|:r0 = (a * rh / |xh|)^(a / (1 - a)) * yh, or the mirror image. The true root is often tiny, and Newton fromrh/2took many shrinking steps to reach it.phi(rh) >= rh/2), such as converging iterates, start atr = rh. This only applies when|dx/dr|and|dy/dr|atrhare at mostPOW_CONE_START_SLOPE; otherwise the first step is too small to represent and looks converged.|f|. The step must be small relative torand, scaled bysens = max(1, |dx/dr|, |dy/dr|), small relative to|v|. Near the apexxcan be ~1000x more sensitive torthanris. The relative part also keeps tiny components accurate relative to themselves when the root is near 0 (near the apex), which is the case that matters for the solves below. When the root is within a few ulps ofrhand a component is tiny, no stopping rule in thisrparametrization can do that; the error there is still small relative to|v|(the 2.3e-9 row below). The loop also stops when the bracket is that narrow ((hi - lo) * sens <= TOL) or only a few ulps wide.POW_CONE_TOL = 1e-12,POW_CONE_START_SLOPE = 1e3,POW_CONE_EPS = DBL_EPSILON(1e-6,10,FLT_EPSILONunderSFLOAT).POW_CONE_MAX_ITERS = 50.Testing
test_power_cone_small_scaleuses the issue's exact data. It fails on master and passes here.test_power_cone_proj_tiny_componentprojects points just outside the cone that have one tiny positive component andp = 0.001/0.999, and checks them against 60-digit reference values. It fails on an intermediate commit of this PR that returned them unprojected.run_tests_directandrun_tests_indirectpass. TheSFLOATlibrary builds with only a warning that is already on master.Issue reproducer:
x = 0.09999999974,dist(s, K) = 1e-17(wasx = 0.1238,dist = 2.9e-3).Projection accuracy vs. a 60-digit
decimalreference (KKT-verified on most sets; on the extreme-p/ adversarial sets some true roots are ~1e-3000, so the reference is exact only to ~1e-59, far below double precision). Max error relative to||v||_inf, mean Newton evaluations, and iteration-cap hits, 3000 points each:pnear 0/1p(0.001, 0.01, 0.99, 0.999)This PR never hits the 50-iteration cap in these sets.
CVXPY benchmark: 52 models built with real 3-d power cones (
approx=False), solved by scs-python built against master and against this PR on macOS arm64 / Accelerate, ateps = 1e-5and1e-8. The models are p-norm regression (p= 1.1, 1.5, 3), geometric-mean maximisation (uniform and skewed weights), power utility (p= 0.05, 0.3, 0.95), a power-penalty portfolio, CVXPY'spcp_1/2/3, and the issue's problem.sis at most 0.06x the primal tolerance; on master it is up to 0.19x, and 2.4e4x on the issue's problem.f(r)/rstep is taken): total solve time is dominated by iteration counts, which shift between builds in both directions because the ADMM trajectory is sensitive to last-bit differences in the projection. Per iteration, cone time is +8% vs. master (geometric mean over the 104 solves), and solve time is +8%. That ranges from 0.73-0.99x on p-norm (p= 1.1), portfolio andp = 0.95utility models, to 1.17-1.45x on geometric-mean,pcpandp= 0.05/0.3 utility models. Before the last commit (speed-ups from profiling recorded projection inputs) cone time per iteration was +24%.test_conic_solvers.py::TestSCS,test_pow_cone_nd.py,test_power_atom.py,test_power_tools.py,test_perspective.py,test_atoms.py) pass identically on both builds.Solver sweep, 390 problems: the 78 below with
ujittered by 0-4%,eps = 1e-8. Not solved / total iterations: master (78-problem version) 61 / 6.2M;f(r)/rstep only when Newton onfwould more than halver(720f475) 18 / 2.77M; only when both components are negative 26 / 3.48M; wheneverf < 0(final) 16 / 2.50M.Earlier solver sweep (measured on fd25f7a),
min -z s.t. (u, 1, z) in K_pwithuin 1e-12..1e-6 andpin {0.001, 0.01, 0.1, 0.9, 0.99, 0.999},eps = 1e-8(78 problems): master leaves 61 not solved (6.2M total iterations); this PR leaves 3 (476k). These problems are badly conditioned, so individual outcomes are sensitive to rounding-level changes in the projection. Forp = 0.9and smallu,z*is beloweps_abs, so errors inzthere reflect the solver tolerance, not the projection.Reviewed five times by a second model; the last review approved the final code with no substantive findings. Earlier findings that are addressed: a converged step being discarded for bisection, cap hits when the root is near 0, a start guard that let tiny components through for small
p, and the stopping-rule nits. One suggestion, stopping on the sensitivity-scaled step alone, made tiny-usolves worse, so it is combined with the relative test instead.🤖 Generated with Claude Code