planner: support basic usage of partial index (#65051)#68832
Open
ti-chi-bot wants to merge 1 commit into
Open
Conversation
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
added
component/statistics
do-not-merge/hold
13 tasks
Member
Author
|
@winoros This PR has conflicts, I have hold it. |
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
📝 WalkthroughWalkthroughThis PR implements partial index support for the TiDB query planner, enabling it to reason about index constraints and optimize plans accordingly. It introduces constraint validation, logical/physical operator updates, index pruning logic, and integrates these changes across the planning pipeline with comprehensive test coverage. ChangesPartial Index Support for Query Planning
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@ti-chi-bot: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 15
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/planner/core/casetest/index/index_test.go (1)
23-37:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winFix merge-conflict import block so pkg/planner/core/casetest/index/index_test.go builds
- Remove unresolved
<<<<<<</>>>>>>>markers in the import section (lines ~23-37); the file currently won’t compile.- Keep
github.com/pingcap/tidb/pkg/parser/modelsincemodel.NewCIStris still used (line ~295).- If both
github.com/pingcap/tidb/pkg/planner/utilandgithub.com/pingcap/tidb/pkg/utilare imported, one must be aliased because both packages declarepackage util.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/planner/core/casetest/index/index_test.go` around lines 23 - 37, Remove the merge markers and fix the import list so the file compiles: delete the <<<<<<< / ======= / >>>>>>> lines, keep github.com/pingcap/tidb/pkg/parser/model (because model.NewCIStr is used), and retain any needed imports from the other side (e.g., github.com/pingcap/tidb/pkg/parser/ast, github.com/pingcap/tidb/pkg/planner/util, github.com/pingcap/tidb/pkg/session/sessmgr) but avoid the package name collision with github.com/pingcap/tidb/pkg/util by aliasing one of them (for example alias github.com/pingcap/tidb/pkg/planner/util as plannerutil) and update any uses accordingly (referencing model.NewCIStr and any plannerutil symbols if renamed).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/planner/core/casetest/index/BUILD.bazel`:
- Around line 12-37: The BUILD.bazel hunk contains unresolved git conflict
markers (<<<<<<<, =======, >>>>>>>) around the shard_count and deps block;
remove the conflict markers and reconcile the two versions so the final block
sets shard_count = 11 and includes the merged deps list (keep "//pkg/domain",
"//pkg/domain/infosync", add "//pkg/parser/ast", "//pkg/planner/util",
"//pkg/session/sessmgr", and retain "//pkg/store/mockstore", "//pkg/testkit",
"//pkg/testkit/testdata", "//pkg/testkit/testfailpoint",
"//pkg/testkit/testmain", "//pkg/testkit/testsetup" and
"`@com_github_pingcap_failpoint//`:failpoint"), making sure commas and bracket
syntax are correct and there are no leftover conflict lines in the deps array.
In `@pkg/planner/core/casetest/index/index_test.go`:
- Around line 403-445: The test enables the failpoint named by fpName
("github.com/pingcap/tidb/pkg/planner/core/rule/InjectCheckForIndexPrune")
multiple times using failpoint.EnableCall but only calls failpoint.Disable at
the end, risking leakage if an assertion fails; immediately after each
failpoint.EnableCall (or right after the first EnableCall) register a guaranteed
cleanup via t.Cleanup(func(){ require.NoError(t, failpoint.Disable(fpName)) })
or pair each scenario's EnableCall with its own t.Cleanup to always disable the
failpoint; refer to fpName, failpoint.EnableCall, failpoint.Disable and
t.Cleanup to locate and fix the spots.
In `@pkg/planner/core/exhaust_physical_plans.go`:
- Around line 799-957: The file contains unresolved git conflict markers
(<<<<<<<, =======, >>>>>>>) around the implementations of
buildDataSource2IndexScanByIndexJoinProp,
buildDataSource2TableScanByIndexJoinProp and completeIndexJoinFeedBackInfo;
remove the conflict markers and reconcile the two variants so the final code
contains a single correct implementation for each of those functions (preserve
the intended logic such as indexValid, getBestIndexJoinPathResultByProp usage,
table-path handling, ranges/keyOff2IdxOff flow, and the call to
completeIndexJoinFeedBackInfo), ensure the function signatures and referenced
symbols (buildDataSource2IndexScanByIndexJoinProp,
buildDataSource2TableScanByIndexJoinProp, completeIndexJoinFeedBackInfo,
getBestIndexJoinPathResultByProp, constructDS2IndexScanTask,
constructDS2TableScanTask) remain consistent, and run build/tests to verify no
remaining conflict markers or compile errors.
In `@pkg/planner/core/indexmerge_path.go`:
- Around line 138-152: The file contains unresolved Git conflict markers inside
the function generateNormalIndexPartialPaths4DNF: you must remove the conflict
markers and reconcile the two variants into one consistent function signature
and body; decide whether the function should handle multiple dnfItems (dnfItems
[]expression.Expression, candidatePaths []*util.AccessPath, returning
([]*util.AccessPath, bool, []bool)) or the single-item variant (item
expression.Expression, candidatePath *util.AccessPath, returning
(*util.AccessPath, bool)), then update the function name/signature and its
internal logic accordingly (including the partial index rejection using
candidatePath.Index.HasCondition() if keeping the single-item form), remove all
<<<<<<<, =======, >>>>>>> markers, and ensure callers of
generateNormalIndexPartialPaths4DNF are adjusted to the chosen signature so the
file compiles.
In `@pkg/planner/core/operator/logicalop/logical_datasource.go`:
- Around line 32-36: Remove the leftover merge-conflict markers in
logical_datasource.go and restore both imports so the file imports
"github.com/pingcap/tidb/pkg/planner/core/constraint" and
"github.com/pingcap/tidb/pkg/planner/core/partidx" (used by PredicatePushDown ->
constraint.DeleteTrueExprs and CheckPartialIndexes -> partidx.CheckConstraints /
partidx.AlwaysMeetConstraints); then fix the DataSource struct so assignments in
CheckPartialIndexes compile by adding an AllPossibleAccessPaths field (matching
the type used where PossibleAccessPaths is defined) or alternatively update
CheckPartialIndexes to assign into the existing PossibleAccessPaths
consistently—prefer adding AllPossibleAccessPaths to DataSource to preserve
current logic.
- Around line 752-763: DataSource currently uses ds.AllPossibleAccessPaths but
the DataSource struct only defines PossibleAccessPaths; add a new field
AllPossibleAccessPaths []*util.AccessPath to the DataSource struct (same
visibility and type as PossibleAccessPaths) and ensure every place that
initializes or updates PossibleAccessPaths also sets/updates
AllPossibleAccessPaths (e.g., constructor/initializer and any methods that
modify access paths), so the slices.DeleteFunc calls in logical_datasource.go
and the iteration in stats.go use a populated AllPossibleAccessPaths;
alternatively, if you prefer the other approach, replace all references to
AllPossibleAccessPaths (in logical_datasource.go and pkg/planner/core/stats.go)
with PossibleAccessPaths consistently across the change set.
In `@pkg/planner/core/operator/physicalop/physical_index_scan.go`:
- Around line 482-499: Partition pruning currently drops the global-index
partition filter when all returned partition IDs are in pInfo.IDsInDDLToIgnore()
causing len(args)==1 and an early return of conditions; instead, detect this
case after building ignoreMap/args (using idxArr, pInfo.Definitions and
pInfo.IDsInDDLToIgnore()) and do not return the unmodified conditions—preserve
the global-index partition filter by adding a predicate that yields no matches
(e.g., a false/empty-partition predicate) or otherwise ensure the scan remains
restricted, so that an empty effective partition set does not become an
unfiltered global-index scan.
In `@pkg/planner/core/partidx/check_constraint.go`:
- Around line 181-220: The checkIsNullRejected function fails to recognize the
"IS NOT NULL" pattern because it never handles the unary NOT wrapper (e.g.,
NOT(ISNULL(a))), causing PartIdxCondNotAlwaysValid to flip incorrectly; update
checkIsNullRejected to detect a unary-not scalar function (ast.UnaryNot / NOT)
whose single argument is an IsNull scalar function on the target column and
treat that as not rejected (return false); ensure you reference
checkIsNullRejected and the ScalarFunction handling so the new branch checks for
filter.FuncName == ast.UnaryNot (or equivalent NOT token), extracts the inner
arg, confirms it's an IsNull on targetCol (using col.Equal with
sctx.GetExprCtx().GetEvalCtx()) and returns false.
In `@pkg/planner/core/plan_cacheable_checker.go`:
- Around line 585-609: The file contains unresolved git merge markers (<<<<<<<,
=======, >>>>>>>) around the plan-type switch; remove those markers and keep the
intended merged branch code: use x.PartialPlansRaw -> subPlans, handle
physicalop.PhysicalIndexReader (append x.IndexPlan),
physicalop.PhysicalIndexLookUpReader (append x.IndexPlan), and the checks on
physicalop.PhysicalIndexScan (use x.IsFullScan(), x.Index.HasCondition(),
x.NotAlwaysValid) and physicalop.PhysicalTableScan (use x.IsFullScan()); ensure
duplicate or old symbols like PartialPlans/PhysicalIndexScan casing are not left
behind so the file compiles.
In `@pkg/planner/core/planbuilder.go`:
- Around line 1454-1470: Remove the leftover merge conflict markers and
normalize the logic: keep the boolean flag allUndeterminedPath, iterate over the
available slice and call availablePath.IsUndetermined() (stop and set
allUndeterminedPath = false if any path is determined), and if
allUndeterminedPath append tablePath to available; delete any stray variables or
markers like allMVIIndexPath and the conflict tokens (<<<<<<<, =======, >>>>>>>)
so the path-selection branch in planbuilder.go compiles and behaves
unambiguously.
In `@pkg/planner/core/rule/rule_prune_indexes.go`:
- Around line 202-210: The second safety block in buildFinalResult that checks
"if len(result) == len(tablePaths)+len(mvIndexPaths) && len(preferredIndexes) ==
0 { return paths }" restores pruned zero-score regular indexes; remove that
condition (or its early return) so buildFinalResult does not undo pruning when
only mandatory table/MV paths remain — keep the existing first safety check "if
len(result) == 0 { return paths }" and delete the additional fallback
referencing result, tablePaths, mvIndexPaths, preferredIndexes, and paths.
In `@pkg/planner/core/stats.go`:
- Around line 138-145: Remove the leftover git conflict markers and restore the
intended logic: call ds.CheckPartialIndexes() and iterate over
ds.AllPossibleAccessPaths (not ds.PossibleAccessPaths). Replace the conflict
block (the <<<<<<</=======/>>>>>>> lines) with the two statements in the merged
change so partial-index initialization runs and you fill index paths for all
paths using ds.CheckPartialIndexes() followed by for _, path := range
ds.AllPossibleAccessPaths { ... } in the function (refer to symbols
ds.CheckPartialIndexes, ds.AllPossibleAccessPaths, and ds.PossibleAccessPaths to
locate the hunk).
In `@pkg/planner/util/path.go`:
- Around line 426-468: The file contains unresolved merge markers around the
AccessPath methods; remove the conflict markers so only a single implementation
remains (keep the IsFullScanRange, IsUndetermined and IsIndexJoinUnapplicable
methods as shown), and add the missing import for mysql so the call to
mysql.HasUnsignedFlag(...) in AccessPath.IsFullScanRange compiles; update the
import block to include "github.com/pingcap/tidb/parser/mysql" (or the correct
local module path used in the repo) and ensure there are no leftover <<<<<<<,
=======, >>>>>>> tokens in pkg/planner/util/path.go.
- Around line 431-441: The file uses mysql.HasUnsignedFlag inside
AccessPath.IsFullScanRange but the package is not imported; add an import for
"github.com/pingcap/tidb/pkg/parser/mysql" to the import block in
pkg/planner/util/path.go so mysql.HasUnsignedFlag(pkColInfo.GetFlag()) resolves;
locate the AccessPath.IsFullScanRange function and ensure the mysql package is
referenced by that import name.
In `@tests/integrationtest/r/planner/core/casetest/index/partialindex.result`:
- Around line 43-50: The fixture is asserting a Point_Get on a partial unique
index (Point_Get ... index:idx3(c)) but the planner change excludes partial
indexes from point-get plans; update the test expectation by regenerating the
integration test result for this case (partialindex) so it no longer asserts
Point_Get for queries involving the partial index (look for the Selection block
and the Point_Get operator info and replace the expected plan with the actual
plan produced after the merge-conflict resolution, or run the test generator to
refresh partialindex.result).
---
Outside diff comments:
In `@pkg/planner/core/casetest/index/index_test.go`:
- Around line 23-37: Remove the merge markers and fix the import list so the
file compiles: delete the <<<<<<< / ======= / >>>>>>> lines, keep
github.com/pingcap/tidb/pkg/parser/model (because model.NewCIStr is used), and
retain any needed imports from the other side (e.g.,
github.com/pingcap/tidb/pkg/parser/ast,
github.com/pingcap/tidb/pkg/planner/util,
github.com/pingcap/tidb/pkg/session/sessmgr) but avoid the package name
collision with github.com/pingcap/tidb/pkg/util by aliasing one of them (for
example alias github.com/pingcap/tidb/pkg/planner/util as plannerutil) and
update any uses accordingly (referencing model.NewCIStr and any plannerutil
symbols if renamed).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 03747b56-f2ff-4053-8862-8ac6b691a80d
📒 Files selected for processing (19)
pkg/expression/util.gopkg/planner/core/casetest/index/BUILD.bazelpkg/planner/core/casetest/index/index_test.gopkg/planner/core/exhaust_physical_plans.gopkg/planner/core/indexmerge_path.gopkg/planner/core/operator/logicalop/BUILD.bazelpkg/planner/core/operator/logicalop/logical_datasource.gopkg/planner/core/operator/physicalop/physical_index_scan.gopkg/planner/core/partidx/BUILD.bazelpkg/planner/core/partidx/check_constraint.gopkg/planner/core/plan_cacheable_checker.gopkg/planner/core/planbuilder.gopkg/planner/core/point_get_plan.gopkg/planner/core/rule/rule_prune_indexes.gopkg/planner/core/stats.gopkg/planner/util/path.gopkg/util/ranger/types.gotests/integrationtest/r/planner/core/casetest/index/partialindex.resulttests/integrationtest/t/planner/core/casetest/index/partialindex.test
Comment on lines
+12
to
+37
| <<<<<<< HEAD | ||
| shard_count = 8, | ||
| deps = [ | ||
| "//pkg/domain", | ||
| "//pkg/domain/infosync", | ||
| "//pkg/parser/model", | ||
| ======= | ||
| shard_count = 11, | ||
| deps = [ | ||
| "//pkg/domain", | ||
| "//pkg/domain/infosync", | ||
| "//pkg/parser/ast", | ||
| "//pkg/planner/util", | ||
| "//pkg/session/sessmgr", | ||
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) | ||
| "//pkg/store/mockstore", | ||
| "//pkg/testkit", | ||
| "//pkg/testkit/testdata", | ||
| "//pkg/testkit/testfailpoint", | ||
| "//pkg/testkit/testmain", | ||
| "//pkg/testkit/testsetup", | ||
| <<<<<<< HEAD | ||
| "//pkg/util", | ||
| ======= | ||
| "@com_github_pingcap_failpoint//:failpoint", | ||
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) |
There was a problem hiding this comment.
Resolve the cherry-pick conflict markers before merging.
This target is still in a merge-conflict state, so Bazel cannot parse it and none of the new test deps can be validated until this hunk is resolved.
#!/bin/bash
set -euo pipefail
rg -n '^(<<<<<<<|=======|>>>>>>>)' pkg/planner/core/casetest/index/BUILD.bazel pkg/planner/core/casetest/index/index_test.go🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/core/casetest/index/BUILD.bazel` around lines 12 - 37, The
BUILD.bazel hunk contains unresolved git conflict markers (<<<<<<<, =======,
>>>>>>>) around the shard_count and deps block; remove the conflict markers and
reconcile the two versions so the final block sets shard_count = 11 and includes
the merged deps list (keep "//pkg/domain", "//pkg/domain/infosync", add
"//pkg/parser/ast", "//pkg/planner/util", "//pkg/session/sessmgr", and retain
"//pkg/store/mockstore", "//pkg/testkit", "//pkg/testkit/testdata",
"//pkg/testkit/testfailpoint", "//pkg/testkit/testmain",
"//pkg/testkit/testsetup" and "`@com_github_pingcap_failpoint//`:failpoint"),
making sure commas and bracket syntax are correct and there are no leftover
conflict lines in the deps array.
Comment on lines
+403
to
+445
| // The failpoint will check whether all partial indexes are pruned. | ||
| fpName := "github.com/pingcap/tidb/pkg/planner/core/rule/InjectCheckForIndexPrune" | ||
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | ||
| for _, path := range paths { | ||
| if path != nil && path.Index != nil && path.Index.ConditionExprString != "" { | ||
| require.True(t, false, "Partial index should be pruned") | ||
| } | ||
| } | ||
| })) | ||
| tk.MustQuery("select * from t") | ||
|
|
||
| // idx1 is pruned because a is not referenced as interesting one. | ||
| // idx2 is kept though its constraint is not matched. | ||
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | ||
| idx2Found := false | ||
| for _, path := range paths { | ||
| if path != nil && path.Index != nil && path.Index.Name.L == "idx1" { | ||
| require.True(t, false, "Partial index idx1 should be pruned") | ||
| } | ||
| if path != nil && path.Index != nil && path.Index.Name.L == "idx2" { | ||
| idx2Found = true | ||
| } | ||
| } | ||
| require.True(t, idx2Found, "Partial index idx2 should not be pruned") | ||
| })) | ||
| tk.MustQuery("explain select * from t order by b").CheckNotContain("idx2") | ||
|
|
||
| // idx2 is pruned because b is not referenced as interesting one. | ||
| // idx1 is kept though its constraint is not matched. | ||
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | ||
| idx1Found := false | ||
| for _, path := range paths { | ||
| if path != nil && path.Index != nil && path.Index.Name.L == "idx2" { | ||
| require.True(t, false, "Partial index idx2 should be pruned") | ||
| } | ||
| if path != nil && path.Index != nil && path.Index.Name.L == "idx1" { | ||
| idx1Found = true | ||
| } | ||
| } | ||
| require.True(t, idx1Found, "Partial index idx1 should not be pruned") | ||
| })) | ||
| tk.MustQuery("explain select * from t where a is null").CheckNotContain("idx1") | ||
| require.NoError(t, failpoint.Disable(fpName)) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C2 'failpoint\.EnableCall\(|failpoint\.Disable\(' pkg/planner/core/casetest/index/index_test.goRepository: pingcap/tidb
Length of output: 1161
Always disable InjectCheckForIndexPrune via t.Cleanup to prevent failpoint leakage
InjectCheckForIndexPrune is enabled multiple times (around lines 405/416/432) but only disabled on the success path at line 445. If any assertion/query fails before then, the failpoint can leak into later tests. Add a t.Cleanup immediately after the first EnableCall (or ensure each scenario is paired with its own guaranteed disable).
Suggested cleanup pattern
fpName := "github.com/pingcap/tidb/pkg/planner/core/rule/InjectCheckForIndexPrune"
+ t.Cleanup(func() {
+ require.NoError(t, failpoint.Disable(fpName))
+ })
require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) {
for _, path := range paths {
if path != nil && path.Index != nil && path.Index.ConditionExprString != "" {
require.True(t, false, "Partial index should be pruned")
}
@@
- require.NoError(t, failpoint.Disable(fpName))
})
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // The failpoint will check whether all partial indexes are pruned. | |
| fpName := "github.com/pingcap/tidb/pkg/planner/core/rule/InjectCheckForIndexPrune" | |
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | |
| for _, path := range paths { | |
| if path != nil && path.Index != nil && path.Index.ConditionExprString != "" { | |
| require.True(t, false, "Partial index should be pruned") | |
| } | |
| } | |
| })) | |
| tk.MustQuery("select * from t") | |
| // idx1 is pruned because a is not referenced as interesting one. | |
| // idx2 is kept though its constraint is not matched. | |
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | |
| idx2Found := false | |
| for _, path := range paths { | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx1" { | |
| require.True(t, false, "Partial index idx1 should be pruned") | |
| } | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx2" { | |
| idx2Found = true | |
| } | |
| } | |
| require.True(t, idx2Found, "Partial index idx2 should not be pruned") | |
| })) | |
| tk.MustQuery("explain select * from t order by b").CheckNotContain("idx2") | |
| // idx2 is pruned because b is not referenced as interesting one. | |
| // idx1 is kept though its constraint is not matched. | |
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | |
| idx1Found := false | |
| for _, path := range paths { | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx2" { | |
| require.True(t, false, "Partial index idx2 should be pruned") | |
| } | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx1" { | |
| idx1Found = true | |
| } | |
| } | |
| require.True(t, idx1Found, "Partial index idx1 should not be pruned") | |
| })) | |
| tk.MustQuery("explain select * from t where a is null").CheckNotContain("idx1") | |
| require.NoError(t, failpoint.Disable(fpName)) | |
| // The failpoint will check whether all partial indexes are pruned. | |
| fpName := "github.com/pingcap/tidb/pkg/planner/core/rule/InjectCheckForIndexPrune" | |
| t.Cleanup(func() { | |
| require.NoError(t, failpoint.Disable(fpName)) | |
| }) | |
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | |
| for _, path := range paths { | |
| if path != nil && path.Index != nil && path.Index.ConditionExprString != "" { | |
| require.True(t, false, "Partial index should be pruned") | |
| } | |
| } | |
| })) | |
| tk.MustQuery("select * from t") | |
| // idx1 is pruned because a is not referenced as interesting one. | |
| // idx2 is kept though its constraint is not matched. | |
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | |
| idx2Found := false | |
| for _, path := range paths { | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx1" { | |
| require.True(t, false, "Partial index idx1 should be pruned") | |
| } | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx2" { | |
| idx2Found = true | |
| } | |
| } | |
| require.True(t, idx2Found, "Partial index idx2 should not be pruned") | |
| })) | |
| tk.MustQuery("explain select * from t order by b").CheckNotContain("idx2") | |
| // idx2 is pruned because b is not referenced as interesting one. | |
| // idx1 is kept though its constraint is not matched. | |
| require.NoError(t, failpoint.EnableCall(fpName, func(paths []*util.AccessPath) { | |
| idx1Found := false | |
| for _, path := range paths { | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx2" { | |
| require.True(t, false, "Partial index idx2 should be pruned") | |
| } | |
| if path != nil && path.Index != nil && path.Index.Name.L == "idx1" { | |
| idx1Found = true | |
| } | |
| } | |
| require.True(t, idx1Found, "Partial index idx1 should not be pruned") | |
| })) | |
| tk.MustQuery("explain select * from t where a is null").CheckNotContain("idx1") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/core/casetest/index/index_test.go` around lines 403 - 445, The
test enables the failpoint named by fpName
("github.com/pingcap/tidb/pkg/planner/core/rule/InjectCheckForIndexPrune")
multiple times using failpoint.EnableCall but only calls failpoint.Disable at
the end, risking leakage if an assertion fails; immediately after each
failpoint.EnableCall (or right after the first EnableCall) register a guaranteed
cleanup via t.Cleanup(func(){ require.NoError(t, failpoint.Disable(fpName)) })
or pair each scenario's EnableCall with its own t.Cleanup to always disable the
failpoint; refer to fpName, failpoint.EnableCall, failpoint.Disable and
t.Cleanup to locate and fix the spots.
Comment on lines
+799
to
+957
| <<<<<<< HEAD | ||
| ======= | ||
| // buildDataSource2IndexScanByIndexJoinProp builds an IndexScan as the inner child for an | ||
| // IndexJoin based on IndexJoinProp included in prop if possible. | ||
| // | ||
| // buildDataSource2IndexScanByIndexJoinProp differs with buildIndexJoinInner2IndexScan in that | ||
| // the first one is try to build a single table scan as the inner child of an index join then return | ||
| // this inner task(raw table scan) bottom-up, which will be attached with other inner parents of an | ||
| // index join in attach2Task when bottom-up of enumerating the physical plans; | ||
| // | ||
| // while the second is try to build a table scan as the inner child of an index join, then build | ||
| // entire inner subtree of a index join out as innerTask instantly according those validated and | ||
| // zipped inner patterns with calling constructInnerIndexScanTask. That's not done yet, it also | ||
| // tries to enumerate kinds of index join operators based on the finished innerTask and un-decided | ||
| // outer child which will be physical-ed in the future. | ||
| func buildDataSource2IndexScanByIndexJoinProp( | ||
| ds *logicalop.DataSource, | ||
| prop *property.PhysicalProperty) base.Task { | ||
| indexValid := func(path *util.AccessPath) bool { | ||
| if path.IsTablePath() { | ||
| return false | ||
| } | ||
| // if path is index path. index path currently include two kind of, one is normal, and the other is mv index. | ||
| // for mv index like mvi(a, json, b), if driving condition is a=1, and we build a prefix scan with range [1,1] | ||
| // on mvi, it will return many index rows which breaks handle-unique attribute here. | ||
| // | ||
| // the basic rule is that: mv index can be and can only be accessed by indexMerge operator. (embedded handle duplication) | ||
| if !path.IsIndexJoinUnapplicable() { | ||
| return true // not a MVIndex path, it can successfully be index join probe side. | ||
| } | ||
| return false | ||
| } | ||
| indexJoinResult, keyOff2IdxOff := getBestIndexJoinPathResultByProp(ds, prop.IndexJoinProp, indexValid) | ||
| if indexJoinResult == nil { | ||
| return base.InvalidTask | ||
| } | ||
| rangeInfo, maxOneRow := indexJoinPathGetRangeInfoAndMaxOneRow(ds.SCtx(), prop.IndexJoinProp.OuterJoinKeys, indexJoinResult) | ||
| var innerTask base.Task | ||
| if !prop.IsSortItemEmpty() && matchProperty(ds, indexJoinResult.chosenPath, prop) == property.PropMatched { | ||
| innerTask = constructDS2IndexScanTask(ds, indexJoinResult.chosenPath, indexJoinResult.chosenRanges.Range(), indexJoinResult.chosenRemained, indexJoinResult.idxOff2KeyOff, rangeInfo, true, prop.SortItems[0].Desc, prop.IndexJoinProp.AvgInnerRowCnt, maxOneRow) | ||
| } else { | ||
| innerTask = constructDS2IndexScanTask(ds, indexJoinResult.chosenPath, indexJoinResult.chosenRanges.Range(), indexJoinResult.chosenRemained, indexJoinResult.idxOff2KeyOff, rangeInfo, false, false, prop.IndexJoinProp.AvgInnerRowCnt, maxOneRow) | ||
| } | ||
| // since there is a possibility that inner task can't be built and the returned value is nil, we just return base.InvalidTask. | ||
| if innerTask == nil { | ||
| return base.InvalidTask | ||
| } | ||
| // prepare the index path chosen information and wrap them as IndexJoinInfo and fill back to CopTask. | ||
| // here we don't need to construct physical index join here anymore, because we will encapsulate it bottom-up. | ||
| // chosenPath and lastColManager of indexJoinResult should be returned to the caller (seen by index join to keep | ||
| // index join aware of indexColLens and compareFilters). | ||
| completeIndexJoinFeedBackInfo(innerTask.(*physicalop.CopTask), indexJoinResult, indexJoinResult.chosenRanges, keyOff2IdxOff) | ||
| return innerTask | ||
| } | ||
|
|
||
| // buildDataSource2TableScanByIndexJoinProp builds a TableScan as the inner child for an | ||
| // IndexJoin if possible. | ||
| // If the inner side of an index join is a TableScan, only one tuple will be | ||
| // fetched from the inner side for every tuple from the outer side. This will be | ||
| // promised to be no worse than building IndexScan as the inner child. | ||
| func buildDataSource2TableScanByIndexJoinProp( | ||
| ds *logicalop.DataSource, | ||
| prop *property.PhysicalProperty) base.Task { | ||
| var tblPath *util.AccessPath | ||
| for _, path := range ds.PossibleAccessPaths { | ||
| if path.IsTablePath() && path.StoreType == kv.TiKV { // old logic | ||
| tblPath = path | ||
| break | ||
| } | ||
| } | ||
| if tblPath == nil { | ||
| return base.InvalidTask | ||
| } | ||
| var keyOff2IdxOff []int | ||
| var ranges ranger.MutableRanges = ranger.Ranges{} | ||
| var innerTask base.Task | ||
| var indexJoinResult *indexJoinPathResult | ||
| if ds.TableInfo.IsCommonHandle { | ||
| // for the leaf datasource, we use old logic to get the indexJoinResult, which contain the chosen path and ranges. | ||
| indexJoinResult, keyOff2IdxOff = getBestIndexJoinPathResultByProp(ds, prop.IndexJoinProp, func(path *util.AccessPath) bool { return path.IsCommonHandlePath }) | ||
| // if there is no chosen info, it means the leaf datasource couldn't even leverage this indexJoinProp, return InvalidTask. | ||
| if indexJoinResult == nil { | ||
| return base.InvalidTask | ||
| } | ||
| // prepare the range info with outer join keys, it shows like: [xxx] decided by: | ||
| rangeInfo, maxOneRow := indexJoinPathGetRangeInfoAndMaxOneRow(ds.SCtx(), prop.IndexJoinProp.OuterJoinKeys, indexJoinResult) | ||
| // construct the inner task with chosen path and ranges, note: it only for this leaf datasource. | ||
| // like the normal way, we need to check whether the chosen path is matched with the prop, if so, we will set the `keepOrder` to true. | ||
| if matchProperty(ds, indexJoinResult.chosenPath, prop) == property.PropMatched { | ||
| innerTask = constructDS2TableScanTask(ds, indexJoinResult.chosenRanges.Range(), rangeInfo, true, !prop.IsSortItemEmpty() && prop.SortItems[0].Desc, prop.IndexJoinProp.AvgInnerRowCnt, maxOneRow) | ||
| } else { | ||
| innerTask = constructDS2TableScanTask(ds, indexJoinResult.chosenRanges.Range(), rangeInfo, false, false, prop.IndexJoinProp.AvgInnerRowCnt, maxOneRow) | ||
| } | ||
| ranges = indexJoinResult.chosenRanges | ||
| } else { | ||
| var ( | ||
| ok bool | ||
| chosenPath *util.AccessPath | ||
| newOuterJoinKeys []*expression.Column | ||
| // note: pk col doesn't have mutableRanges, the global var(ranges) which will be handled as empty range in constructIndexJoin. | ||
| localRanges ranger.Ranges | ||
| ) | ||
| keyOff2IdxOff, newOuterJoinKeys, localRanges, chosenPath, ok = getIndexJoinIntPKPathInfo(ds, prop.IndexJoinProp.InnerJoinKeys, prop.IndexJoinProp.OuterJoinKeys, func(path *util.AccessPath) bool { return path.IsIntHandlePath }) | ||
| if !ok { | ||
| return base.InvalidTask | ||
| } | ||
| // For IntHandle (integer primary key), it's always a unique match. | ||
| maxOneRow := true | ||
| rangeInfo := indexJoinIntPKRangeInfo(ds.SCtx().GetExprCtx().GetEvalCtx(), newOuterJoinKeys) | ||
| if !prop.IsSortItemEmpty() && matchProperty(ds, chosenPath, prop) == property.PropMatched { | ||
| innerTask = constructDS2TableScanTask(ds, localRanges, rangeInfo, true, prop.SortItems[0].Desc, prop.IndexJoinProp.AvgInnerRowCnt, maxOneRow) | ||
| } else { | ||
| innerTask = constructDS2TableScanTask(ds, localRanges, rangeInfo, false, false, prop.IndexJoinProp.AvgInnerRowCnt, maxOneRow) | ||
| } | ||
| } | ||
| // since there is a possibility that inner task can't be built and the returned value is nil, we just return base.InvalidTask. | ||
| if innerTask == nil { | ||
| return base.InvalidTask | ||
| } | ||
| // prepare the index path chosen information and wrap them as IndexJoinInfo and fill back to CopTask. | ||
| // here we don't need to construct physical index join here anymore, because we will encapsulate it bottom-up. | ||
| // chosenPath and lastColManager of indexJoinResult should be returned to the caller (seen by index join to keep | ||
| // index join aware of indexColLens and compareFilters). | ||
| completeIndexJoinFeedBackInfo(innerTask.(*physicalop.CopTask), indexJoinResult, ranges, keyOff2IdxOff) | ||
| return innerTask | ||
| } | ||
|
|
||
| // completeIndexJoinFeedBackInfo completes the IndexJoinInfo for the innerTask. | ||
| // indexJoin | ||
| // | ||
| // +--- outer child | ||
| // +--- inner child (say: projection ------------> unionScan -------------> ds) | ||
| // <-------RootTask(IndexJoinInfo) <--RootTask(IndexJoinInfo) <--copTask(IndexJoinInfo) | ||
| // | ||
| // when we build the underlying datasource as table-scan, we will return wrap it and | ||
| // return as a CopTask, inside which the index join contains some index path chosen | ||
| // information which will be used in indexJoin execution runtime: ref IndexJoinInfo | ||
| // declaration for more information. | ||
| // the indexJoinInfo will be filled back to the innerTask, passed upward to RootTask | ||
| // once this copTask is converted to RootTask type, and finally end up usage in the | ||
| // indexJoin's attach2Task with calling completePhysicalIndexJoin. | ||
| func completeIndexJoinFeedBackInfo(innerTask *physicalop.CopTask, indexJoinResult *indexJoinPathResult, ranges ranger.MutableRanges, keyOff2IdxOff []int) { | ||
| info := innerTask.IndexJoinInfo | ||
| if info == nil { | ||
| info = &physicalop.IndexJoinInfo{} | ||
| } | ||
| if indexJoinResult != nil { | ||
| if indexJoinResult.chosenPath != nil { | ||
| info.IdxColLens = indexJoinResult.chosenPath.IdxColLens | ||
| } | ||
| info.CompareFilters = indexJoinResult.lastColManager | ||
| } | ||
| info.Ranges = ranges | ||
| info.KeyOff2IdxOff = keyOff2IdxOff | ||
| // fill it back to the bottom-up Task. | ||
| innerTask.IndexJoinInfo = info | ||
| } | ||
|
|
||
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) |
There was a problem hiding this comment.
Finish resolving this cherry-pick conflict before merge.
Lines 799-957 still include merge-conflict markers, which leaves the file in an unbuildable state.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/core/exhaust_physical_plans.go` around lines 799 - 957, The file
contains unresolved git conflict markers (<<<<<<<, =======, >>>>>>>) around the
implementations of buildDataSource2IndexScanByIndexJoinProp,
buildDataSource2TableScanByIndexJoinProp and completeIndexJoinFeedBackInfo;
remove the conflict markers and reconcile the two variants so the final code
contains a single correct implementation for each of those functions (preserve
the intended logic such as indexValid, getBestIndexJoinPathResultByProp usage,
table-path handling, ranges/keyOff2IdxOff flow, and the call to
completeIndexJoinFeedBackInfo), ensure the function signatures and referenced
symbols (buildDataSource2IndexScanByIndexJoinProp,
buildDataSource2TableScanByIndexJoinProp, completeIndexJoinFeedBackInfo,
getBestIndexJoinPathResultByProp, constructDS2IndexScanTask,
constructDS2TableScanTask) remain consistent, and run build/tests to verify no
remaining conflict markers or compile errors.
Comment on lines
+138
to
+152
| <<<<<<< HEAD | ||
| dnfItems []expression.Expression, | ||
| candidatePaths []*util.AccessPath, | ||
| ) (paths []*util.AccessPath, needSelection bool, usedMap []bool) { | ||
| paths = make([]*util.AccessPath, 0, len(dnfItems)) | ||
| usedMap = make([]bool, len(dnfItems)) | ||
| ======= | ||
| item expression.Expression, | ||
| candidatePath *util.AccessPath, | ||
| ) (paths *util.AccessPath, needSelection bool) { | ||
| // Reject partial index first. | ||
| if candidatePath.Index != nil && candidatePath.Index.HasCondition() { | ||
| return nil, false | ||
| } | ||
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# 1) Detect unresolved merge-conflict markers in Go files
rg -n --type=go '^(<<<<<<<|=======|>>>>>>>)'
# 2) Inspect this function definition and nearby call sites for signature consistency
rg -n --type=go -C3 '\bgenerateNormalIndexPartialPaths4DNF\s*\('Repository: pingcap/tidb
Length of output: 3692
Fix unresolved Git merge conflict markers in pkg/planner/core/indexmerge_path.go (lines 138–152) — the generateNormalIndexPartialPaths4DNF hunk still contains <<<<<<<, =======, >>>>>>> markers, making this Go file invalid until the conflict is resolved (and the resulting function signature/body are made consistent).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/core/indexmerge_path.go` around lines 138 - 152, The file
contains unresolved Git conflict markers inside the function
generateNormalIndexPartialPaths4DNF: you must remove the conflict markers and
reconcile the two variants into one consistent function signature and body;
decide whether the function should handle multiple dnfItems (dnfItems
[]expression.Expression, candidatePaths []*util.AccessPath, returning
([]*util.AccessPath, bool, []bool)) or the single-item variant (item
expression.Expression, candidatePath *util.AccessPath, returning
(*util.AccessPath, bool)), then update the function name/signature and its
internal logic accordingly (including the partial index rejection using
candidatePath.Index.HasCondition() if keeping the single-item form), remove all
<<<<<<<, =======, >>>>>>> markers, and ensure callers of
generateNormalIndexPartialPaths4DNF are adjusted to the chosen signature so the
file compiles.
Comment on lines
+32
to
+36
| <<<<<<< HEAD | ||
| "github.com/pingcap/tidb/pkg/planner/core/constraint" | ||
| ======= | ||
| "github.com/pingcap/tidb/pkg/planner/core/partidx" | ||
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="pkg/planner/core/operator/logicalop/logical_datasource.go"
# Basic sanity: does file still contain merge conflict markers?
echo "== conflict markers =="
rg -n "<<<<<<<|=======|>>>>>>>" "$FILE" || true
echo
echo "== import block =="
# Print imports section with line numbers
start=$(rg -n "^import \\(" "$FILE" | head -n1 | cut -d: -f1)
end=$(tail -n +"$start" "$FILE" | rg -n "^\\)" | head -n1 | cut -d: -f1)
# end is relative; compute absolute
end_abs=$((start + end - 1))
sed -n "${start},${end_abs}p" "$FILE" | cat -n | sed "s/^/$(basename "$FILE"):/" | head -n 120
echo
echo "== DataSource fields (PossibleAccessPaths / AllPossibleAccessPaths) =="
rg -n "type DataSource|PossibleAccessPaths|AllPossibleAccessPaths" "$FILE" -n || true
# If struct is present, dump around its definition
struct_line=$(rg -n "^type DataSource" "$FILE" | head -n1 | cut -d: -f1 || true)
if [ -n "${struct_line:-}" ]; then
# print next ~120 lines
sed -n "$struct_line,$((struct_line+160))p" "$FILE" | cat -n | head -n 220
fi
echo
echo "== CheckPartialIndexes implementation usage =="
rg -n "CheckPartialIndexes\\(" "$FILE" || true
chk_line=$(rg -n "CheckPartialIndexes\\(" "$FILE" | head -n1 | cut -d: -f1 || true)
if [ -n "${chk_line:-}" ]; then
sed -n "$chk_line,$((chk_line+220))p" "$FILE" | cat -n | head -n 260
fi
echo
echo "== PredicatePushDown uses constraint? =="
rg -n "PredicatePushDown\\(" "$FILE" || true
pd_line=$(rg -n "PredicatePushDown\\(" "$FILE" | head -n1 | cut -d: -f1 || true)
if [ -n "${pd_line:-}" ]; then
sed -n "$pd_line,$((pd_line+160))p" "$FILE" | cat -n | head -n 220
fiRepository: pingcap/tidb
Length of output: 22510
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Search where AllPossibleAccessPaths is defined (struct fields, methods, etc.)
rg -n "AllPossibleAccessPaths" -S --hidden --glob '!.git/*' .
# Also locate definitions of embedded types referenced by DataSource (LogicalSchemaProducer) to see if it provides the field.
rg -n "type LogicalSchemaProducer" -S --hidden --glob '!.git/*' pkg/planner/core/operator/logicalop/logical_datasource.go pkg/planner/core/operator/logicalop -S || trueRepository: pingcap/tidb
Length of output: 588
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="pkg/planner/core/operator/logicalop/logical_schema_producer.go"
echo "== File: $FILE =="
wc -l "$FILE"
sed -n '1,220p' "$FILE" | cat -n
echo
echo "== Fields containing PossibleAccessPaths / AllPossibleAccessPaths =="
rg -n "AllPossibleAccessPaths|PossibleAccessPaths" "$FILE" || trueRepository: pingcap/tidb
Length of output: 5306
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find definition of BaseLogicalPlan
rg -n "type BaseLogicalPlan struct" -S pkg/planner/core/operator/logicalop pkg/planner/core pkg/planner | head -n 20
# Then inspect the file around the struct definition
BASE_FILE=$(rg -n "type BaseLogicalPlan struct" -S pkg/planner | head -n1 | cut -d: -f1)
echo "== BaseLogicalPlan file: $BASE_FILE =="
sed -n '1,220p' "$BASE_FILE" | cat -n
echo
echo "== Search for AllPossibleAccessPaths in BaseLogicalPlan file =="
rg -n "AllPossibleAccessPaths|PossibleAccessPaths" "$BASE_FILE" || trueRepository: pingcap/tidb
Length of output: 10055
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== AllPossibleAccessPaths occurrences (all files) =="
rg -n "AllPossibleAccessPaths" -S --hidden --glob '!.git/*' . | head -n 50
echo
echo "== Count occurrences =="
rg -n "AllPossibleAccessPaths" -S --hidden --glob '!.git/*' . | wc -l
echo
echo "== Look for struct field declarations involving AllPossibleAccessPaths in Go =="
rg -n "AllPossibleAccessPaths\\s" -S --hidden --glob '!.git/*' -g'*.go' . | head -n 50
echo
echo "== Look for 'type .*struct' lines that mention AllPossibleAccessPaths =="
rg -n "type .*struct[^{]*\\{[^}]*AllPossibleAccessPaths" -S --hidden --glob '!.git/*' -g'*.go' . || trueRepository: pingcap/tidb
Length of output: 1151
Fix unresolved merge conflict + missing AllPossibleAccessPaths field in logical_datasource.go
pkg/planner/core/operator/logicalop/logical_datasource.go:32-36: the import block still contains merge-conflict markers betweencore/constraintandcore/partidx; keep both sincePredicatePushDown()usesconstraint.DeleteTrueExprsandCheckPartialIndexes()usespartidx.CheckConstraints/partidx.AlwaysMeetConstraints.pkg/planner/core/operator/logicalop/logical_datasource.go:699-766:CheckPartialIndexes()assigns tods.AllPossibleAccessPaths, butDataSourceonly definesPossibleAccessPathsand there is noAllPossibleAccessPathsfield declared anywhere in the repo—add the field or switch to the correct existing field.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/core/operator/logicalop/logical_datasource.go` around lines 32 -
36, Remove the leftover merge-conflict markers in logical_datasource.go and
restore both imports so the file imports
"github.com/pingcap/tidb/pkg/planner/core/constraint" and
"github.com/pingcap/tidb/pkg/planner/core/partidx" (used by PredicatePushDown ->
constraint.DeleteTrueExprs and CheckPartialIndexes -> partidx.CheckConstraints /
partidx.AlwaysMeetConstraints); then fix the DataSource struct so assignments in
CheckPartialIndexes compile by adding an AllPossibleAccessPaths field (matching
the type used where PossibleAccessPaths is defined) or alternatively update
CheckPartialIndexes to assign into the existing PossibleAccessPaths
consistently—prefer adding AllPossibleAccessPaths to DataSource to preserve
current logic.
Comment on lines
+202
to
+210
| // Safety check: if we ended up with nothing, return the original paths | ||
| if len(result) == 0 { | ||
| return paths | ||
| } | ||
|
|
||
| // Additional safety: if we only have table paths and MVIndex paths and no regular indexes, keep original | ||
| if len(result) == len(tablePaths)+len(mvIndexPaths) && len(preferredIndexes) == 0 { | ||
| return paths | ||
| } |
There was a problem hiding this comment.
This fallback restores zero-score indexes that were just pruned.
When all regular indexes score 0, buildFinalResult intentionally keeps only the mandatory table/MV/hinted paths. This block then returns the original paths, so pruning becomes a no-op for queries with a table path plus unrelated indexes.
💡 Suggested fix
- // Additional safety: if we only have table paths and MVIndex paths and no regular indexes, keep original
- if len(result) == len(tablePaths)+len(mvIndexPaths) && len(preferredIndexes) == 0 {
+ // Only preserve the original slice when there were never any regular index paths to prune.
+ if len(paths) == len(tablePaths)+len(mvIndexPaths)+len(indexMergeIndexPaths) {
return paths
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // Safety check: if we ended up with nothing, return the original paths | |
| if len(result) == 0 { | |
| return paths | |
| } | |
| // Additional safety: if we only have table paths and MVIndex paths and no regular indexes, keep original | |
| if len(result) == len(tablePaths)+len(mvIndexPaths) && len(preferredIndexes) == 0 { | |
| return paths | |
| } | |
| // Safety check: if we ended up with nothing, return the original paths | |
| if len(result) == 0 { | |
| return paths | |
| } | |
| // Only preserve the original slice when there were never any regular index paths to prune. | |
| if len(paths) == len(tablePaths)+len(mvIndexPaths)+len(indexMergeIndexPaths) { | |
| return paths | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/core/rule/rule_prune_indexes.go` around lines 202 - 210, The
second safety block in buildFinalResult that checks "if len(result) ==
len(tablePaths)+len(mvIndexPaths) && len(preferredIndexes) == 0 { return paths
}" restores pruned zero-score regular indexes; remove that condition (or its
early return) so buildFinalResult does not undo pruning when only mandatory
table/MV paths remain — keep the existing first safety check "if len(result) ==
0 { return paths }" and delete the additional fallback referencing result,
tablePaths, mvIndexPaths, preferredIndexes, and paths.
Comment on lines
+138
to
+145
| <<<<<<< HEAD | ||
| for _, path := range ds.PossibleAccessPaths { | ||
| ======= | ||
| // Index pruning is now done earlier in CollectPredicateColumnsPoint to avoid loading stats for pruned indexes. | ||
| // Fill index paths for all paths | ||
| ds.CheckPartialIndexes() | ||
| for _, path := range ds.AllPossibleAccessPaths { | ||
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) |
There was a problem hiding this comment.
Resolve the leftover cherry-pick conflict in this hunk.
The <<<<<<< / ======= / >>>>>>> markers make stats.go uncompilable, and they leave the partial-index path initialization logic undefined.
Suggested resolution
-<<<<<<< HEAD
- for _, path := range ds.PossibleAccessPaths {
-=======
// Index pruning is now done earlier in CollectPredicateColumnsPoint to avoid loading stats for pruned indexes.
- // Fill index paths for all paths
ds.CheckPartialIndexes()
for _, path := range ds.AllPossibleAccessPaths {
->>>>>>> 959bf330874 (planner: support basic usage of partial index (`#65051`))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| <<<<<<< HEAD | |
| for _, path := range ds.PossibleAccessPaths { | |
| ======= | |
| // Index pruning is now done earlier in CollectPredicateColumnsPoint to avoid loading stats for pruned indexes. | |
| // Fill index paths for all paths | |
| ds.CheckPartialIndexes() | |
| for _, path := range ds.AllPossibleAccessPaths { | |
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) | |
| // Index pruning is now done earlier in CollectPredicateColumnsPoint to avoid loading stats for pruned indexes. | |
| ds.CheckPartialIndexes() | |
| for _, path := range ds.AllPossibleAccessPaths { |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/core/stats.go` around lines 138 - 145, Remove the leftover git
conflict markers and restore the intended logic: call ds.CheckPartialIndexes()
and iterate over ds.AllPossibleAccessPaths (not ds.PossibleAccessPaths). Replace
the conflict block (the <<<<<<</=======/>>>>>>> lines) with the two statements
in the merged change so partial-index initialization runs and you fill index
paths for all paths using ds.CheckPartialIndexes() followed by for _, path :=
range ds.AllPossibleAccessPaths { ... } in the function (refer to symbols
ds.CheckPartialIndexes, ds.AllPossibleAccessPaths, and ds.PossibleAccessPaths to
locate the hunk).
Comment on lines
+426
to
+468
| <<<<<<< HEAD | ||
| ======= | ||
|
|
||
| // IsFullScanRange checks that a table scan does not have any filtering such that it can limit the range of | ||
| // the table scan. | ||
| func (path *AccessPath) IsFullScanRange(tableInfo *model.TableInfo) bool { | ||
| var unsignedIntHandle bool | ||
| if path.IsIntHandlePath && tableInfo.PKIsHandle { | ||
| if pkColInfo := tableInfo.GetPkColInfo(); pkColInfo != nil { | ||
| unsignedIntHandle = mysql.HasUnsignedFlag(pkColInfo.GetFlag()) | ||
| } | ||
| } | ||
| if ranger.HasFullRange(path.Ranges, unsignedIntHandle) { | ||
| return true | ||
| } | ||
| return false | ||
| } | ||
|
|
||
| // IsUndetermined checks if the path is undetermined. | ||
| // The undetermined path is the one that may not be always valid. | ||
| // e.g. The multi value index for JSON is not always valid, because the index must be used with JSON functions. | ||
| func (path *AccessPath) IsUndetermined() bool { | ||
| if path.IsTablePath() || path.Index == nil { | ||
| return false | ||
| } | ||
| if path.Index.MVIndex || path.Index.ConditionExprString != "" { | ||
| return true | ||
| } | ||
| return false | ||
| } | ||
|
|
||
| // IsIndexJoinUnapplicable checks if the path is unapplicable for index join. | ||
| // If path is mv index path: | ||
| // for mv index like mvi(a, json, b), if driving condition is a=1, and we build a prefix scan with range [1,1] | ||
| // on mvi, it will return many index rows which breaks handle-unique attribute here. | ||
| // So we cannot use mv index path for index join. | ||
| // If path is partial index path: | ||
| // We need to first determine whether we already meet the partial index condition. | ||
| // Currently we don't support that, so we conservatively return true here. | ||
| func (path *AccessPath) IsIndexJoinUnapplicable() bool { | ||
| return path.IsUndetermined() | ||
| } | ||
| >>>>>>> 959bf330874 (planner: support basic usage of partial index (#65051)) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="pkg/planner/util/path.go"
echo "== conflict markers in $FILE =="
rg -n "<<<<<<<|>>>>>>>|=======" "$FILE" || true
echo
echo "== snippet around target lines =="
nl -ba "$FILE" | sed -n '380,520p'
echo
echo "== imports section =="
# show top of file through imports
nl -ba "$FILE" | sed -n '1,120p'
echo
echo "== mysql.HasUnsignedFlag usage =="
rg -n "mysql\.HasUnsignedFlag" "$FILE" || trueRepository: pingcap/tidb
Length of output: 314
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="pkg/planner/util/path.go"
echo "== imports block (approx) =="
# Print from package declaration through end of import block
awk '
BEGIN{p=0}
/^(package |package\t)/{p=1}
{print}
p==1 && $0 ~ /^import[[:space:]]*\(/ {inimp=1}
inimp==1 && $0 ~ /^\)[[:space:]]*$/ {exit}
' "$FILE" || true
echo
echo "== mysql.HasUnsignedFlag usage =="
rg -n "mysql\.HasUnsignedFlag" "$FILE" || true
echo
echo "== mysql import candidates =="
rg -n "pkg/parser/mysql|parser/mysql|\"mysql\"" "$FILE" || true
echo
echo "== snippet around conflict markers =="
# show just the region with markers using awk line range
awk 'NR>=410 && NR<=485 {printf "%4d:%s\n", NR, $0}' "$FILE"Repository: pingcap/tidb
Length of output: 3719
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "HasUnsignedFlag" pkg | head -n 20
rg -n "pkg/parser/mysql" pkg/planner/util/path.go pkg | head -n 20Repository: pingcap/tidb
Length of output: 3371
Resolve merge conflict markers and fix missing mysql import in pkg/planner/util/path.go
pkg/planner/util/path.gostill contains<<<<<<< HEAD,=======,>>>>>>> ...aroundAccessPath.IsFullScanRange/IsUndetermined/IsIndexJoinUnapplicable; fully resolve the conflict so only one branch remains.AccessPath.IsFullScanRangecallsmysql.HasUnsignedFlag(...), butgithub.com/pingcap/tidb/pkg/parser/mysqlis not imported in this file—add the import after resolving the conflict.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/util/path.go` around lines 426 - 468, The file contains
unresolved merge markers around the AccessPath methods; remove the conflict
markers so only a single implementation remains (keep the IsFullScanRange,
IsUndetermined and IsIndexJoinUnapplicable methods as shown), and add the
missing import for mysql so the call to mysql.HasUnsignedFlag(...) in
AccessPath.IsFullScanRange compiles; update the import block to include
"github.com/pingcap/tidb/parser/mysql" (or the correct local module path used in
the repo) and ensure there are no leftover <<<<<<<, =======, >>>>>>> tokens in
pkg/planner/util/path.go.
Comment on lines
+431
to
+441
| func (path *AccessPath) IsFullScanRange(tableInfo *model.TableInfo) bool { | ||
| var unsignedIntHandle bool | ||
| if path.IsIntHandlePath && tableInfo.PKIsHandle { | ||
| if pkColInfo := tableInfo.GetPkColInfo(); pkColInfo != nil { | ||
| unsignedIntHandle = mysql.HasUnsignedFlag(pkColInfo.GetFlag()) | ||
| } | ||
| } | ||
| if ranger.HasFullRange(path.Ranges, unsignedIntHandle) { | ||
| return true | ||
| } | ||
| return false |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the relevant section around the function and the import block
sed -n '1,120p' pkg/planner/util/path.go
# Print imports with line numbers
rg -n "import \\(" -n pkg/planner/util/path.go || true
nl -ba pkg/planner/util/path.go | sed -n '120,220p'
# Show the target lines with context
nl -ba pkg/planner/util/path.go | sed -n '380,470p'
# Check whether mysql is imported anywhere in this file
rg -n "parser/mysql" pkg/planner/util/path.go || true
rg -n "\\bmysql\\." pkg/planner/util/path.go || trueRepository: pingcap/tidb
Length of output: 6074
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="pkg/planner/util/path.go"
# Show top-of-file imports area
sed -n '1,80p' "$FILE"
# Locate the function
rg -n "func \\(path \\*AccessPath\\) IsFullScanRange" "$FILE"
# Show the function body with some surrounding context
rg -n -C 5 "func \\(path \\*AccessPath\\) IsFullScanRange" "$FILE"
# Check for mysql references and whether mysql/parser is imported
rg -n "mysql\\.HasUnsignedFlag" "$FILE" || true
rg -n "\\bmysql\\." "$FILE" || true
rg -n "pkg/parser/mysql|parser/mysql" "$FILE" || true
# Also show the full import block if it exists later (defensive)
rg -n "^[[:space:]]*import[[:space:]]*\\(" "$FILE" || trueRepository: pingcap/tidb
Length of output: 4653
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="pkg/planner/util/path.go"
# Confirm whether mysql is imported anywhere in this file
rg -n "github\\.com/pingcap/tidb/pkg/parser/mysql" "$FILE" || true
rg -n "pkg/parser/mysql" "$FILE" || true
# Confirm whether mysql.HasUnsignedFlag is referenced (and how many times)
rg -n "mysql\\.HasUnsignedFlag" "$FILE" || true
# Dump the full import block (in case it spans further than the first 80 lines)
rg -n "^[[:space:]]*import[[:space:]]*\\(" "$FILE" || true
start=$(rg -n "^[[:space:]]*import[[:space:]]*\\(" "$FILE" | head -n1 | cut -d: -f1)
# Print imports block until closing paren
if [ -n "${start:-}" ]; then
end=$((start+120))
sed -n "${start},${end}p" "$FILE"
fi
# Show exact region around the conflict markers / function definition area
sed -n '400,460p' "$FILE"Repository: pingcap/tidb
Length of output: 8507
Fix missing pkg/parser/mysql import in pkg/planner/util/path.go
IsFullScanRange() calls mysql.HasUnsignedFlag(...) but pkg/planner/util/path.go does not import github.com/pingcap/tidb/pkg/parser/mysql, so it won’t compile. Add the import near the existing imports.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/planner/util/path.go` around lines 431 - 441, The file uses
mysql.HasUnsignedFlag inside AccessPath.IsFullScanRange but the package is not
imported; add an import for "github.com/pingcap/tidb/pkg/parser/mysql" to the
import block in pkg/planner/util/path.go so
mysql.HasUnsignedFlag(pkColInfo.GetFlag()) resolves; locate the
AccessPath.IsFullScanRange function and ensure the mysql package is referenced
by that import name.
Comment on lines
+43
to
+50
| explain format='brief' select * from t where c = 10 and b > 30; | ||
| id estRows task access object operator info | ||
| Selection 1.00 root gt(planner__core__casetest__index__partialindex.t.b, 30) | ||
| └─Point_Get 1.00 root table:t, index:idx3(c) | ||
| explain format='brief' select * from t where c = 10 and b > 40; | ||
| id estRows task access object operator info | ||
| Selection 1.00 root gt(planner__core__casetest__index__partialindex.t.b, 40) | ||
| └─Point_Get 1.00 root table:t, index:idx3(c) |
There was a problem hiding this comment.
The expected Point_Get here looks backwards for a partial index.
These two cases currently lock in Point_Get ... index:idx3(c) on a partial unique index. The planner changes in this PR stack explicitly exclude partial indexes from point-get plans, so this fixture should be regenerated after the merge-conflict resolution instead of asserting Point_Get here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/integrationtest/r/planner/core/casetest/index/partialindex.result`
around lines 43 - 50, The fixture is asserting a Point_Get on a partial unique
index (Point_Get ... index:idx3(c)) but the planner change excludes partial
indexes from point-get plans; update the test expectation by regenerating the
integration test result for this case (partialindex) so it no longer asserts
Point_Get for queries involving the partial index (look for the Selection block
and the Point_Get operator info and replace the expected plan with the actual
plan produced after the merge-conflict resolution, or run the test generator to
refresh partialindex.result).
|
@ti-chi-bot: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
ti-chi-bot
Bot
added
cherry-pick-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automated cherry-pick of #65051
What problem does this PR solve?
Issue Number: close #64344
Problem Summary:
What changed and how does it work?
This pr adds the basic support for the partial index.
Please feel free to review the unit test for its usage.
We are using the existing extracting range logic to support the partial index's flexibility: we can use the partial index defined as
idx(b) where b > 2when the given SQL iswhere b > 3.For the complex functions like
idx(b) where sin(b) > 0.1, we check whether there's an exact one in the WHERE clause.Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.
Summary by CodeRabbit
Release Notes
New Features
Bug Fixes
Tests