[Kallyan01]

What

Moves Algolia indexing off the request thread and onto Action Scheduler. A re-index now runs as a parent job that queues one batch job per 100 posts, and a post save queues a sync job instead of writing to Algolia inline. Jobs are stored in a new wp_onesearch_index_jobs table. The admin gets a live progress modal, cancel, retry and a paginated job history, on the governing site and on its brand sites.

Why

POST /re-index used to index the whole site inside the REST request. On a governing site, that request also waited over HTTP for each brand site's re-index. On larger sites the request times out partway through and leaves a half-built index. There was no way to see what finished, retry what failed or stop the run. Post saves had the same problem on a smaller scale: every publish or update waited for an Algolia write before the editor got its response.

Related Issue(s):

• Closes https://github.com/rtCamp/OnePress/issues/72
• Previous Closed PR feat: batch sync content using action scheduler #247

How

Job stack on each site

flowchart TB
    Start(["POST /re-index"])
    Save(["Post saved"])
    Ctl(["/jobs/{id} requests"])

    SC["Search_Controller<br/>token check · re-index lock"]
    PT["Indexable post types<br/>governing: own settings<br/>brand: GET /brand-config, cached"]
    RJ["Reindex_Job<br/>parent"]
    W["Watcher"]
    JRC["Job_REST_Controller"]
    JS["Job_Scheduler"]
    DB[("wp_onesearch_index_jobs<br/>+ transient cache")]
    AS[["Action Scheduler"]]
    SJ["Sync_Job × N<br/>100 posts each"]
    IDX["Indexer"]
    ALG[("Algolia index")]

    Start --> SC
    SC --> PT
    SC -->|"governing: runs it now<br/>brand: schedules it on AS"| RJ
    RJ -->|"1 · clear site records"| IDX
    RJ -->|"2 · store + enqueue batches"| JS
    Save --> W
    W -->|"queue Sync_Job"| JS
    Ctl --> JRC
    JRC -->|"status · retry · cancel"| JS
    JS <--> DB
    JS --> AS
    AS --> SJ
    SJ -->|"save / delete posts"| IDX
    SJ -.->|"notify_parent()"| JS
    IDX -->|"batched writes"| ALG

Loading

Re-index: starting a run

sequenceDiagram
    autonumber
    actor Admin
    participant UI as Admin UI
    participant G as Governing site
    participant GQ as Governing jobs
    participant B as Brand site
    participant BQ as Brand jobs
    participant A as Algolia

    Admin->>UI: Click Re-index
    UI->>G: POST /re-index
    G->>G: Take re-index lock, 409 if a run is live

    loop Each connected brand site, one at a time
        G->>B: POST /re-index with X-OneSearch-Token, 45s timeout
        B->>B: Check token, take the brand's re-index lock
        opt Brand config not cached
            B->>G: GET /brand-config with token
            G-->>B: indexable_entities, algolia_credentials
        end
        B->>B: Count batches for the brand's post types
        B->>BQ: Store Reindex_Job, schedule it, start the runner
        B->>B: Save re-index state, release lock
        B-->>G: job_id, batch_count
    end
    Note over G,B: A brand that errors or returns no job_id goes into errors and is left out of the run

    G->>A: Clear governing records
    G->>GQ: Store one Sync_Job per 100 posts and enqueue them
    G->>G: Store sites and total_batches on the parent, save re-index state, release lock
    G-->>UI: job_id, jobs[] for every site

Loading

Feature list

• Batched async re-index: Clears existing Algolia records and processes posts in batches of 100 (max 500) using Action Scheduler, with atomic progress tracking.
• Async post sync: Queues post updates through Action Scheduler, with inline sync as a fallback if scheduling fails.
• Governing and brand sites: Triggers re-indexing across connected brand sites, tracks their progress, and handles unreachable sites.
• Live progress modal: Displays real-time batch progress, updates every 2 seconds, and restores after a page refresh.
• Retry from history: Retries only failed batches where possible, or restarts the run under the same job ID.
• Job history: Provides a paginated history of completed runs with status, duration, and batch details.
• One run at a time: Prevents concurrent re-indexing using a DB lock, with a 15-minute stale-job timeout.

AI Disclosure

Architected the overall job workflow, implemented key components, and use Claude 5.5 for migration, testing, feedback, and feature enhancements.

Added API

All routes are under /wp-json/onesearch/v1.

Permission key

• Read: manage_options, or an X-OneSearch-Token header that matches this site's API key. The governing site uses the token to read its brand sites' jobs.
• API: check_api_permissions(), the same check the existing OneSearch endpoints use.
• Admin: manage_options only.

New endpoints

Method	Route	Permission	Description
GET	/re-index/status	API	The live run's jobs for each site, used to restore the progress UI after a refresh. Returns active: false once every job is finished or stale.
GET	/jobs/history	Read	Finished top-level runs, paginated. Query: page (default 1), per_page (default 5, max 100). Returns jobs, total, page, per_page, total_pages.
GET	/jobs/{id}	Read	One job's full state (see Job object).
GET	/jobs/{id}/children	Read	The batch jobs of a parent run.
POST	/jobs/{id}/retry	API	Retries a failed job. For a parent run, it retries only the failed batches, or restarts the run if nothing indexed yet.
DELETE	/jobs/{id}	API	Cancels a job and its pending batches, and clears the active re-index state.
GET	/jobs/remote-status	Admin	Calls GET /jobs/{id} on a brand site from the governing site. Query: site_url, job_id.
POST	/jobs/remote-retry	Admin	Calls POST /jobs/{id}/retry on a brand site from the governing site. Body: site_url, job_id.
POST	/jobs/remote-cancel	Admin	Calls DELETE /jobs/{id} on a brand site from the governing site. Returns success even if the brand site is unreachable. Body: site_url, job_id.

Changed endpoints

POST /re-index now schedules jobs and returns right away:

{
  "success": true,
  "message": "Re-indexing scheduled successfully.",
  "errors": [],
  "job_id": "3f2c…",
  "batch_count": 12,
  "jobs": [
    { "site_name": "Governing Site", "site_url": "https://gov.example", "job_id": "3f2c…", "batch_count": 12 },
    { "site_name": "Brand A", "site_url": "https://brand-a.example", "job_id": "9a1e…", "batch_count": 8 }
  ]
}

• Returns 409 onesearch_reindex_active while another run is live.
• On a governing site, rejects requests that carry a brand token (rest_forbidden).
• On failure, message now contains each failing site's own error messages instead of the generic "Re-indexing was unsuccessful".

POST /algolia-credentials: on a governing site, rejects requests that carry a brand token (rest_forbidden).

Testing Instructions

OneSearch Async Re-indexing – Testing Instructions

Prerequisites

• Set up a governing site with at least one connected brand site.
• Configure valid Algolia credentials on both sites.
• Ensure Action Scheduler is available by running composer install.
• Build the frontend assets using npm run build.
• Have access to WP-CLI, Algolia Dashboard, and Tools → Scheduled Actions.

Phase 1: Basic Functionality

1. Database Table Creation

1. Activate or update the OneSearch plugin.
2. Verify that the jobs table exists using:
   wp db query "SHOW TABLES LIKE '%onesearch_index_jobs'"
3. Expected: The wp_onesearch_index_jobs table exists.

2. Re-indexing

1. Navigate to OneSearch → Search on the governing site.
2. Click Re-index.
3. Observe the progress modal for the governing and connected brand sites.
4. Wait for the re-indexing to finish.
5. Expected: All sites complete successfully, batch progress reaches the total, and the run appears in Sync Job History.

3. Progress Persistence

1. Start a re-index with enough posts to keep it running.
2. Refresh the page while indexing is in progress.
3. Expected: The progress modal reappears with the existing job's progress.

4. Post Sync

1. Create or update a published post.
2. Check Tools → Scheduled Actions, under the onesearch_watcher group.
3. Verify the corresponding onesearch_execute_job action completes.
4. Confirm the changes appear in Algolia.
5. Move the post to Draft or Trash.
6. Expected: The post's Algolia records are removed.

Phase 2: Job Management

5. Concurrent Re-index Prevention

1. Start a re-index on the governing site.
2. Open another tab and attempt to start another re-index.
3. Expected: The second request is rejected with HTTP 409 and the message "A re-index is already in progress."

6. Job Cancellation

1. Start a re-index across the governing and brand sites.
2. Click Cancel Job while it is running.
3. Check the job status on both sites.
4. Inspect Tools → Scheduled Actions.
5. Expected: All associated jobs show Cancelled, with no pending onesearch_execute_job actions remaining for that run.

7. Job History

1. Complete a successful re-index.
2. Generate a failed or cancelled run.
3. Navigate to Sync Job History.
4. Expected: The history displays job ID, type, creation time, duration, status, and batch counts, with pagination working correctly.

Phase 3: Failure Recovery and Retries

8. Retry – Full Restart

1. Configure an invalid Algolia admin key on a brand site.
2. Start a re-index from the governing site.
3. Wait for the brand site's clear operation to fail and the run to be marked Failed.
4. Restore the valid Algolia key.
5. Click Retry in Sync Job History.
6. Expected: The run restarts under the same job ID and completes successfully.

9. Retry – Failed Batches Only

1. Start a re-index with multiple batches.
2. After batches are queued and some complete, invalidate the Algolia key.
3. Wait for the remaining batches to fail.
4. Restore the valid key and click Retry.
5. Expected: Only failed batches are processed again; successfully completed batches are skipped.

Final acceptance criteria: Re-indexing, post synchronization, governing-to-brand orchestration, cancellation, retries, history, and access control work correctly without leaving orphaned jobs or pending scheduled actions.

Screenshots

Checklist

• I have read the Contribution Guidelines.
• I have read the Development Guidelines.
• I have added necessary tests to cover my changes.
• I have updated the project documentation as needed.
• My code has detailed inline documentation.
• My code is tested to the best of my abilities.
• My code passes all lints, tests, and checks.

[justlevine]

Superseded by #365 (@Kallyan01 if you're curious, I went with a simpler approach)

[Kallyan01]

@justlevine, I think it would be beneficial to introduce a plan or design approval step before starting implementation of a new feature and raising a PR. This would help align everyone on the proposed approach early and reduce the chances of disagreements during code review and rework. Thoughts?

[justlevine]

@Kallyan01 👍
lets move this discussion to DM or to the internal repo. https://github.com/rtCamp/OnePress/issues/72 . Tl;dr

• yes the scope should be clarified before a PR is picked up, but I want to avoid granular bike shedding
• the scope exceeding https://github.com/rtCamp/OnePress/issues/72 was only one part of the problem here, the implementation details being the bigger issue and prevent a partial cherry-pick like the previous PR. Part of that could be opus compensation for a lack of a plan.