ddl,tables: only write the index when it meets partial index condition (#65700)

[YangKeao]

This PR is a cherry-pick of #65700

This is a cherry-pick for partial index and some related bug fixes:

• ddl,tables: only write the index when it meets partial index condition #62762
• ddl: enhance checkpoint management with chunk tracking #64607 is needed to avoid the issue caused by 0 scan count.
• ddl: maintain the affect columns for partial index affect columns #64476 is also needed to fix partial index wrong schema after some DDLs.
• dml: fix the data/index inconsistency when updating partial index or using on dup key update #65808 is needed to fix several partial index issues.

What problem does this PR solve?

Issue Number: close #62761, close #62758

Problem Summary:

What changed and how does it work?

1. Modify the tables.index to parse/store the expression of partialConditionExpression.
2. Add a MeetPartialCondition to check whether it'll meet the condition.
3. Add some tests about the DML during the DDL on the same table.
4. Modify the row count of admin show ddl job from the added count to the scanned count.
5. Support admin check table for partial index.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be LGTMed and approved by the reviewers firstly.
2. For pull requests to TiDB-x branches, it must have no failed tests.
3. AFTER it has lgtm and approved labels, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gmhdbjd, hawkingrei, yujuncen for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• br/OWNERS
• pkg/ddl/OWNERS
• pkg/infoschema/OWNERS
• pkg/meta/OWNERS
• pkg/planner/OWNERS
• pkg/table/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR implements comprehensive support for partial indexes in TiDB—indexes that only contain rows matching a WHERE condition. The implementation spans metadata schema updates, condition evaluation, DML/DDL integration, backfilling pipelines, admin operations, and extensive test coverage across unit and integration test suites.

Changes

Partial Index Support

Layer / File(s)	Summary
Metadata and Condition Definitions pkg/meta/model/index.go, pkg/meta/model/job_args.go, pkg/errno/errcode.go, pkg/errno/errname.go, errors.toml, pkg/infoschema/tables.go	IndexInfo gains ConditionExprString and AffectColumn fields; IndexArg stores condition string in DDL job args; error codes 8272/8273 defined for column/partial-index validation; TIDB_INDEXES schema extended with PREDICATE column.
Index Condition Evaluation pkg/table/tables/index.go, pkg/table/index.go	NewIndex signature changed to return (Index, error); adds condition parsing and cached evaluators; implements MeetPartialCondition and MeetPartialConditionWithChunk methods; exports ExtractColumnsFromCondition and DedupIndexColumns helpers.
DDL Validation and Index Creation pkg/ddl/index.go, pkg/ddl/create_table.go, pkg/ddl/partition.go, pkg/ddl/column.go, pkg/ddl/modify_column.go, pkg/ddl/executor.go	Validates supported condition predicates via CheckAndBuildIndexConditionString; rejects partial indexes on partitioned tables and primary keys; prevents modification/dropping of columns referenced by partial conditions; enforces fast-reorg requirement; validates non-existent condition columns.
Table Mutation and DML pkg/table/tables/tables.go, pkg/executor/batch_checker.go	Updates addIndices, rebuildUpdateRecordIndices, removeRowIndices to evaluate partial conditions before writing/deleting index entries; respects AffectColumn for untouched-index detection; filters unique-index key generation by partial predicate.
Index Backfilling Pipeline pkg/ddl/backfilling_operators.go, pkg/ddl/index.go	Extracts conditions from cop context; builds per-worker condition checkers; tracks row-count deltas from runtime stats instead of chunk row counts; conditionally skips TiDB-side evaluation when conditions are pushed down; updates IndexRecordChunk with tableScanRowCount and conditionPushed.
Cop Context and Table Scan Planning pkg/ddl/copr/copr_ctx.go, pkg/ddl/index_cop.go, pkg/ddl/backfilling_txn_executor.go	Removes DistSQLCtx from cop context constructors; adds GetCondition() method for condition extraction; implements condition pushdown via buildDAGPB and constructSelectionPB; uses DDLReorgMeta-based context construction; records condition-pushdown status per chunk.
Admin Operations pkg/executor/admin.go, pkg/executor/check_table_index.go, pkg/executor/builder.go	Validates partial-index conditions during admin check table/index (requires tidb_enable_fast_table_check=ON); skips non-matching rows during admin recover index; includes condition predicates in checksum SQL for FastCheckTableExec; renames internal flag from containsGenedCol to containsGenedColOrPartialIndex.
Information Schema and Display pkg/executor/infoschema_reader.go, pkg/executor/show.go	Populates PREDICATE column in information_schema.tidb_indexes queries; outputs partial index conditions in SHOW CREATE TABLE DDL statements.
Transaction Executor Refactoring pkg/ddl/backfilling_txn_executor.go, pkg/ddl/export_test.go, pkg/ddl/bench_test.go, pkg/ddl/index_cop_test.go	Removes kv.Storage parameter from NewReorgCopContext; uses DDLReorgMeta-based constructor; sets explicit RuntimeStatsColl in default context; propagates errors from worker initialization.
Comprehensive Testing pkg/ddl/integration_test.go, tests/integrationtest/t/ddl/integration.test, tests/integrationtest/t/executor/admin.test, tests/integrationtest/t/executor/show.test, tests/integrationtest/t/executor/infoschema_reader.test, tests/integrationtest/t/table/tables.test, tests/realtikvtest/addindextest4/partial_index_test.go, pkg/table/tables/index_test.go, br/tests/br_partial_index/run.sh	Validates partial-index DDL constraints (nonexistent columns, primary keys, partitioned tables); verifies column offset maintenance through schema changes; tests condition evaluation for multiple handle types; covers DML with partial conditions and partial unique indexes; validates admin operations under tidb_enable_fast_table_check settings; includes BR backup/restore for partial indexes.
Build and Supporting Changes pkg/table/tables/BUILD.bazel, pkg/table/tables/testutil/BUILD.bazel, pkg/table/tables/testutil/indexcheck.go, pkg/meta/model/BUILD.bazel, pkg/ddl/copr/BUILD.bazel, pkg/ddl/ingest/checkpoint.go, pkg/executor/test/admintest/admin_test.go, pkg/planner/core/logical_plan_builder.go, tools/tazel/util.go	Updates Bazel dependencies and test sharding; adds CheckIndexKVCount test utility for index KV validation; removes unused distsql/context dependency; adds chunk-level progress tracking; handles error returns from tables.NewIndex across 30+ test files; disables column pruning when table contains partial indexes.

---

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

This is a large, multi-system feature touching core DDL, DML, backfilling, and admin operations with dense logic throughout the index subsystem, complemented by extensive test coverage that must be validated for correctness.

---

Suggested labels

sig/planner, size/XXL, ok-to-test, cherry-pick-approved, type/cherry-pick-for-release-8.5, approved, lgtm

---

Suggested reviewers

• wjhuang2016
• D3Hunter
• yudongusa
• winoros
• bb7133

---

Poem

🐰 Conditions dance through indexes now,
Where clauses whisper row by row,
Partial dreams in metadata stored,
DML respects what rows afford,
Admin checks what meets the call—
Selective indexing conquers all!

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be LGTMed and approved by the reviewers firstly.
2. For pull requests to TiDB-x branches, it must have no failed tests.
3. AFTER it has lgtm and approved labels, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov]

Codecov Report

❌ Patch coverage is 64.41281% with 300 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-8.5@2431361). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff                @@
##             release-8.5     #68946   +/-   ##
================================================
  Coverage               ?   35.4076%           
================================================
  Files                  ?       1731           
  Lines                  ?     641136           
  Branches               ?          0           
================================================
  Hits                   ?     227011           
  Misses                 ?     392399           
  Partials               ?      21726           

Flag	Coverage Δ
integration	35.4076% <64.4128%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	∅ <0.0000%> (?)
parser	∅ <0.0000%> (?)
br	3.7272% <0.0000%> (?)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.