diff --git a/.changeset/subquery-index.md b/.changeset/subquery-index.md new file mode 100644 index 0000000000..711e4f75eb --- /dev/null +++ b/.changeset/subquery-index.md @@ -0,0 +1,5 @@ +--- +'@core/sync-service': patch +--- + +New subquery index that reduces memory footprint and solves lag issues due to slow shape removal. diff --git a/.changeset/subquery-memory-and-lag-fix.md b/.changeset/subquery-memory-and-lag-fix.md new file mode 100644 index 0000000000..bb0bdd3a00 --- /dev/null +++ b/.changeset/subquery-memory-and-lag-fix.md @@ -0,0 +1,5 @@ +--- +'@core/sync-service': patch +--- + +Reduced memory consumption of subqueries and fix lag issue caused by subquery removal. diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md new file mode 100644 index 0000000000..cd422d1067 --- /dev/null +++ b/docs/rfcs/subquery-index.md @@ -0,0 +1,1619 @@ +--- +title: Shared Subquery Indexes with Logical-Time Views +version: "0.2" +status: draft +owner: robacourt +contributors: [] +created: 2026-05-18 +last_updated: 2026-05-18 +prd: "N/A - based on https://github.com/electric-sql/electric/issues/4279" +prd_version: "N/A" +--- + +# Shared Subquery Indexes with Logical-Time Views + +## Summary + +Electric v1.6 added per-shape subquery indexing so shapes with boolean subquery +filters can stay live while dependency rows move across `WHERE` boundaries. +That solved correctness, but it stores the same dependency view repeatedly in +the filter index and in consumer event handlers. This RFC proposes one shared, +logical-time view per subquery. Consumers register the subqueries they read, +keep only the logical time they are reading, and call +`SubqueryProgressMonitor.notify_processed_up_to(time, subquery_id)` when they no +longer need older times. The filter index routes conservatively across retained +times and verifies exact membership by asking the shared view at the consumer's +logical time. + +## Background + +Issue: https://github.com/electric-sql/electric/issues/4279 + +Related work: + +- PR #4051 introduced the v1.6 subquery move correctness work: + https://github.com/electric-sql/electric/pull/4051 +- PR #4280 proposed a narrower SubqueryIndex memory design using shared base + views with sparse XOR exceptions: + https://github.com/electric-sql/electric/pull/4280 +- Current `SubqueryIndex`: + `packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex` +- Current consumer view setup: + `packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex` +- Current move buffering: + `packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex` +- Current SQL move-in query construction: + `packages/sync-service/lib/electric/shapes/querying.ex` + +The v1.6 subquery work allowed shapes with boolean combinations around +subqueries to stay live when dependency rows move. Without that, Electric would +invalidate the outer shape and require a full resync. + +The current implementation achieves correctness by letting each consumer own a +local dependency view. `EventHandlerBuilder` reads each dependency +materializer's values into a per-consumer `MapSet`. During an active move, +`ActiveMove` stores `views_before_move` and `views_after_move`. Separately, +`SubqueryIndex` stores per-shape routing rows and exact membership rows keyed +by `shape_handle`, `subquery_ref`, and value. + +That model is correct because consumers can temporarily disagree about the same +subquery. One consumer may have processed a dependency move while another has +not. The current implementation represents that by copying the dependency view +per consumer. + +## Problem + +For a popular subquery, memory currently scales roughly with: + +```text +number_of_outer_consumers * number_of_values_in_subquery +``` + +There are two large duplicated pools: + +- `SubqueryIndex` stores value membership and routing rows per outer shape. +- Consumer event handlers store dependency views per outer shape, and store + both before and after views while a move-in is active. + +Shape removal is also expensive because current value-keyed membership rows do +not have a cheap reverse path from a shape to all of the rows it owns. Adding a +reverse index such as `shape_handle -> all values` would improve removal, but +it would add another copy of the full per-shape dependency view. + +The wider design problem is that the current system optimizes for the +exceptional case, where every consumer has a distinct subquery view, by paying +that memory cost in the common case where many consumers share the same view +and only diverge briefly during moves. + +**Link to PRD hypothesis:** There is no PRD for this RFC. The working +hypothesis comes from issue #4279: + +> Redesigning the SubqueryIndex so it does not store full per-shape dependency +> views will make shape add/remove scalable and reduce memory consumption, +> while preserving v1.6 subquery move correctness. + +## Goals & Non-Goals + +### Goals + +- Store one shared materialized view per subquery. +- Allow consumers to read exact subquery membership at separate logical times. +- Remove long-lived per-consumer `MapSet` views from event handlers. +- Remove per-shape exact membership rows from `SubqueryIndex`. +- Keep routing conservative while consumers are at different logical times. +- Keep first-time child creation correct by synchronously seeding routing before + the child is considered indexed. +- Keep shape removal proportional to the shape's subquery participants and + routing edges, not to the full dependency view. +- Preserve correctness for positive subqueries, negated subqueries, `AND`, + `OR`, and `NOT`. +- Avoid changing the client wire protocol. + +### Non-Goals + +- Do not change Electric's HTTP protocol. +- Do not change supported subquery semantics. +- Do not redesign DNF planning, tags, or `active_conditions`. +- Do not remove the need to materialize SQL array parameters for move-in + queries in the first implementation. The goal is to avoid long-lived copies; + transient query-local arrays may remain. +- Do not make negated-subquery routing better than + `O(number_of_affected_shapes)`. If a value is absent from a large negated + group, all of those shapes are genuinely affected. +- Do not intern equivalent SQL subqueries that have different dependency shape + handles. A `subquery_id` is the dependency shape handle for v1. + +## Proposal + +### Core Idea + +Move subquery membership out of per-shape state and into one versioned view per +subquery: + +```text +MultiTimeView[{subquery_id, value}] -> membership_history +consumer[{shape_handle, subquery_ref}] -> {subquery_id, logical_time} +``` + +Consumers no longer copy the subquery view. They register each subquery they +read, store the logical time returned by the materializer, and ask +`MultiTimeView.member?(subquery_id, value, time)` when they need exact +membership. + +The filter index no longer stores exact per-shape membership rows. It stores +compact routing topology: + +```text +subquery_group_id +child_node_id per {subquery_group_id, subquery_id} +shape participant rows +fallback rows while initial indexing is incomplete +``` + +Positive routing is value-keyed for values that are members at some retained +logical time. Negated routing is group-keyed and then filtered by shared +membership history. + +### Architecture + +```text +Dependency materializer + -> writes MultiTimeView at monotonically increasing logical times + -> emits dependency move events with from_time and to_time + +Consumer event handler + -> registers subqueries through the materializer + -> stores subquery_id and logical times, not MapSet views + -> calls notify_processed_up_to/2 after old times are no longer needed + +SubqueryIndex + -> stores subquery groups, child nodes, and participant routing + -> asks MultiTimeView for membership at some/all retained times for routing + -> over-routes when consumers diverge; exact split is the consumer's job +``` + +Routing is intentionally conservative: when consumers diverge across logical +times — including the common case of a single consumer that is mid-move and +effectively reading at two times at once — the filter cannot encode that with +a per-shape logical-time pin. Exact membership is therefore checked at +`Shape.convert_change`/`WhereClause.includes_record?/3`, using a +`subquery_member?` callback that the consumer builds from its own logical +time(s) against `MultiTimeView`. + +### Definitions + +#### Subquery + +A subquery is represented by its dependency shape. The `subquery_id` is the +dependency shape handle. + +Different `SELECT` statements are different subqueries, even if they differ +only by constants. For example: + +```sql +SELECT id FROM users WHERE company_id = 7 +SELECT id FROM users WHERE company_id = 8 +``` + +These get different `subquery_id` values. + +#### Subquery Group + +A subquery group is a set of subquery occurrences with the same filter tree +node, field key, and polarity. + +For example, these outer shapes use different subqueries but can share the same +subquery group if the occurrence is at the same filter node: + +```sql +WHERE user_id IN (SELECT id FROM users WHERE company_id = 7) +WHERE user_id IN (SELECT id FROM users WHERE company_id = 8) +``` + +The field key is `user_id`, and the polarity is positive. + +#### Child Node + +A `child_node_id` is created per `{subquery_group_id, subquery_id}` pair. + +The child node owns a child `WhereCondition` containing all outer shapes using +that subquery in that group. Many outer shapes can therefore share one child +node. + +#### Logical Time + +Logical time is a monotonically increasing integer per subquery. + +Time `0` represents the materializer's initial view. Each committed dependency +move that changes subquery membership increments the logical time and records +the transition at the new time. + +Use normal BEAM integers. Wrapping is unnecessary and would make comparison and +compaction harder to reason about. + +#### Processed-Up-To Time + +The public progress API is: + +```elixir +SubqueryProgressMonitor.notify_processed_up_to(time, subquery_id) +``` + +Consumers call this after they no longer need to read the subquery at `time` or +earlier. For a move from logical time `a` to logical time `b`, once the +consumer has finished processing that move and is steady at `b`, it notifies +that it has processed up to `a`. + +Internally, the monitor tracks `required_time`: the earliest logical time a +live consumer may still read. `notify_processed_up_to(a, subquery_id)` advances +that consumer's `required_time` to `a + 1`. + +The compaction lower bound is: + +```text +min(required_time_for_live_consumers) +``` + +Consumers register at the logical time they are starting from. If a consumer +starts from current logical time `t`, its initial `required_time` is `t` +because it may read time `t`. + +`required_time` is a retention bound. It is separate from the consumer's current +logical time for a specific subquery. During an active move, a consumer may need +the old time for buffered conversion or move-in query work while its current +logical time for that subquery has already advanced to the new time. The +implementation must keep `required_time` and per-subquery `logical_time` +explicit. + +### MultiTimeView + +`Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView` stores one +shared view per subquery in ETS, with one table per stack. + +The logical key is: + +```text +{subquery_id, value} -> membership_history +``` + +Absence means the value is not a member at any retained logical time. + +The common case is a value that is always present for the retained window. That +is represented as an empty history: + +```elixir +[] +``` + +Values that moved use compact flat histories: + +```elixir +[:out, 9] +[:out, 9, 11] +[:in, 9] +[:in, 9, 11] +``` + +The first list item is membership before the first transition. Each integer +after it is a logical time where membership toggles from that time onwards. + +Examples: + +```elixir +# Out before 9, in from 9 onwards. +[:out, 9] + +# Out before 9, in from 9 to 10, out from 11 onwards. +[:out, 9, 11] + +# In before 9, out from 9 to 10, in from 11 onwards. +[:in, 9, 11] +``` + +Use `[]` rather than `true` for the always-present case for consistency with +other histories. On BEAM, both `[]` and `true` are immediate terms, so neither +is more compact as an ETS value. + +Use flat lists such as `[:out, 9]` rather than tuples containing lists such as +`{:out, [9]}` because the flat list is smaller and is enough for the common +short-history case. + +The API should support: + +```elixir +member?(subquery_id, value, time) +member_at_some_time?(subquery_id, value) +member_at_all_times?(subquery_id, value) +values(subquery_id) +values(subquery_id, time) +mark_ready(subquery_id) +ready?(subquery_id) +set_min_required_time(subquery_id, time) +remove_subquery(subquery_id) +``` + +`member_at_some_time?/2` and `member_at_all_times?/2` operate over the retained +time window for that subquery. + +### Compaction + +`SubqueryProgressMonitor` provides the minimum required logical time for each +subquery. `MultiTimeView` can compact entries by evaluating membership at that +time and removing transitions at or before it. + +Compaction must preserve membership at all retained times. For example: + +```elixir +[:out, 9, 11] +``` + +If `min_required_time = 10`, membership at time `10` is `true`, and the +compacted history becomes: + +```elixir +[:in, 11] +``` + +If `min_required_time = 12`, the value is out for the whole retained window, so +the row can be deleted. + +Compaction should run: + +- when a value is read +- when a value is written +- in a periodic asynchronous pass +- when a consumer unregisters and releases the minimum pinned time + +### SubqueryIndex Data Model + +The hot ETS rows should use compact integer IDs for groups, children, and +subqueries where practical. Full shape handles and dependency handles can be +stored in metadata rows and interned at boundaries. + +Suggested logical rows: + +```text +{:group, group_key} -> group_id +{:child, group_id, subquery_id} -> child_node_id +{:child_meta, child_node_id} -> {group_id, subquery_id, polarity, next_condition_id} +{:subquery_child, subquery_id} -> child_node_id +{:child_shape, child_node_id} -> {shape_handle, branch_key} +{:shape_child, shape_handle} -> child_node_id +{:shape_subquery, shape_handle, subquery_ref} -> {subquery_id, logical_time} +{:fallback, shape_handle} -> true +``` + +Positive routing keeps value-keyed entries: + +```text +{:positive, group_id, value} -> child_node_id +``` + +Negated routing keeps group-keyed entries: + +```text +{:negated, group_id} -> child_node_id +``` + +This replaces per-shape value membership rows with per-child routing rows and a +shared membership view. + +### First-Time Child Creation + +First-time child creation must seed synchronously. + +When `SubqueryIndex` creates a new `child_node_id` for +`{subquery_group_id, subquery_id}`, it must: + +1. Ensure the dependency materializer has populated `MultiTimeView` and marked + the subquery ready. +2. Create the child `WhereCondition`. +3. Insert the outer shapes into the child condition. +4. Seed positive routing for every value in + `MultiTimeView.values(subquery_id, current_time)`. +5. Add negated group routing if the group is negated. +6. Remove fallback only after the child is fully routable. + +This is `O(number_of_values_in_subquery)` for the first child of a +`{group, subquery_id}` pair. That cost is acceptable because it happens on +child creation, not on every consumer using the same child. + +### Routing + +Positive routing should route a root-table value to a child if the value is a +member of the child subquery at any retained logical time: + +```elixir +MultiTimeView.member_at_some_time?(subquery_id, value) +``` + +This is conservative. If some consumers still read an old time and others read +a new time, both old and new members remain routable until compaction proves no +consumer can read the old time. + +Negated routing should enumerate the negated children for the group and keep +children where the value is not a member at all retained times: + +```elixir +not MultiTimeView.member_at_all_times?(subquery_id, value) +``` + +This is `O(number_of_affected_shapes)` for large negated groups. That is +acceptable because a value absent from a large negated group genuinely affects +all of those shapes. + +Exact membership verification is not done by the filter. The filter cannot +correctly perform a per-shape split at routing time because, during a buffered +move-in, a consumer is effectively reading at *both* `from_time` and `to_time` +for the same subquery — splice-plan evaluates buffered transactions at +`MTV(from_time)` for pre-ops and `MTV(to_time)` for post-ops. A single +per-shape logical-time pin in the filter cannot represent that, and the filter +does not know how a given record relates to a given consumer's move window +without duplicating consumer state. + +Therefore the boundary at which exact membership is evaluated is the consumer, +via the `subquery_member?` callback passed into `WhereClause.includes_record?/3` +from `Shape.convert_change`. The consumer constructs that callback from its own +per-subquery logical time(s) — one callback for the steady case, two callbacks +(`old_member?` and `new_member?`) during a buffered move — and calls +`MultiTimeView.member?(subquery_id, typed_value, logical_time)` against the +shared view. + +The filter does maintain `{shape_handle, subquery_ref} -> {subquery_id, +logical_time}` rows so it can answer exact membership for sublink refs that +survive in the residual `and_where` (i.e. sublinks at *other* positions in the +shape's WHERE that were not the routed position). For those, the filter's +`subquery_member_from_index` callback is sufficient because the shape is not +mid-move on that other position at this routing step. + +### Operation Examples And Costs + +Use this concrete setup for the examples: + +```sql +-- subquery_id = s7, current logical time 0 +SELECT id FROM users WHERE company_id = 7 +-- current values: 10, 20 + +-- subquery_id = s8, current logical time 0 +SELECT id FROM users WHERE company_id = 8 +-- current values: 30 +``` + +Outer shapes: + +```sql +-- shape_a and shape_b share the same positive group and subquery. +WHERE user_id IN (SELECT id FROM users WHERE company_id = 7) + +-- shape_c uses the same positive group but a different subquery. +WHERE user_id IN (SELECT id FROM users WHERE company_id = 8) + +-- shape_n uses a negated group for s7. +WHERE user_id NOT IN (SELECT id FROM users WHERE company_id = 7) +``` + +#### Initial `MultiTimeView` State + +The initial materializer state for `s7` stores one row per dependency value, +not one row per outer shape: + +```text +{s7, 10} -> [] +{s7, 20} -> [] +{:current_time, s7} -> 0 +{:min_required_time, s7} -> 0 +{:ready, s7} -> true +``` + +The empty history means the value is present for the whole retained window. + +Memory is `O(number_of_values_in_subquery_retained_window)` for the shared +view. In this example, `shape_a` and `shape_b` do not duplicate `{10, 20}`. + +#### `register_subquery_consumer` + +Before an outer consumer can read `s7`, it registers through the materializer: + +```elixir +{:ok, 0} = + Materializer.register_subquery_consumer( + s7, + shape_a, + consumer_pid_a + ) +``` + +Progress monitor rows are conceptually: + +```text +{s7, shape_a} -> required_time 0 +{s7, 0, shape_a} -> true +``` + +The shape's subquery reference is then recorded by the indexing/setup path: + +```text +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 0} +``` + +If registration is called from `add_shape`, this row is inserted once as part +of that setup path; it is shown here to make the registration result explicit. + +What is evaluated: + +1. Wait until `s7` is ready. +2. Read `{:current_time, s7}`. +3. Insert progress monitor rows for `shape_a`. +4. Return `0` to the consumer. + +Cost: + +```text +O(wait_until_ready + progress_index_insert) +``` + +No dependency values are copied. Memory added is +`O(number_of_subqueries_read_by_shape)`. + +#### `add_shape`: First Positive Shape For `{group, subquery}` + +Adding `shape_a` creates a positive group `g_user_pos` and a child `c_s7_pos` +for `{g_user_pos, s7}`. + +Rows stored: + +```text +{:group, {:node_1, :user_id, :positive}} -> g_user_pos +{:child, g_user_pos, s7} -> c_s7_pos +{:child_meta, c_s7_pos} -> {g_user_pos, s7, :positive, wc_s7_pos} +{:subquery_child, s7} -> c_s7_pos + +{:child_shape, c_s7_pos} -> {shape_a, branch_a} +{:shape_child, shape_a} -> c_s7_pos +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 0} + +{:positive, g_user_pos, 10} -> c_s7_pos +{:positive, g_user_pos, 20} -> c_s7_pos +``` + +The child `WhereCondition` `wc_s7_pos` also stores `shape_a` with the residual +non-subquery predicates for the branch. + +What is evaluated: + +1. Compile or reuse the DNF subquery group key. +2. Register the consumer with the dependency materializer and get logical time + `0`. +3. Create the child `WhereCondition`. +4. Insert `shape_a` into the child condition. +5. Synchronously seed positive routing from `MultiTimeView.values(s7, 0)`. +6. Remove fallback for `shape_a`. + +Cost: + +```text +O( + number_of_subquery_occurrences_in_shape + + number_of_values_in_s7_retained_window + + child_where_insert +) +``` + +The value-count term only applies because this is the first child for +`{g_user_pos, s7}`. Memory added is +`O(number_of_values_in_s7_retained_window)` positive routing rows for the child +plus `O(number_of_subquery_occurrences_in_shape)` participant rows. + +#### `add_shape`: Additional Shape Sharing An Existing Child + +Adding `shape_b` finds the existing child `c_s7_pos`. + +Rows added: + +```text +{:child_shape, c_s7_pos} -> {shape_b, branch_b} +{:shape_child, shape_b} -> c_s7_pos +{:shape_subquery, shape_b, ["$sublink", "0"]} -> {s7, 0} +``` + +No new rows are added for values `10` or `20`. + +What is evaluated: + +1. Resolve `{g_user_pos, s7}` to `c_s7_pos`. +2. Register the consumer and get logical time `0`. +3. Insert `shape_b` into the child condition. +4. Remove fallback for `shape_b`. + +Cost: + +```text +O(number_of_subquery_occurrences_in_shape + child_where_insert) +``` + +Memory added is per-shape metadata only, not +`O(number_of_values_in_s7_retained_window)`. + +#### `add_shape`: Same Group, Different Subquery + +Adding `shape_c` reuses group `g_user_pos`, but creates child `c_s8_pos` for +`{g_user_pos, s8}`. + +Rows added include: + +```text +{:child, g_user_pos, s8} -> c_s8_pos +{:child_meta, c_s8_pos} -> {g_user_pos, s8, :positive, wc_s8_pos} +{:subquery_child, s8} -> c_s8_pos +{:positive, g_user_pos, 30} -> c_s8_pos +{:shape_subquery, shape_c, ["$sublink", "0"]} -> {s8, 0} +``` + +Cost is `O(number_of_values_in_s8_retained_window)` for the first `s8` child in +this group. This is expected: `s8` has different dependency values from `s7`. + +#### `add_shape`: Negated Shape + +Adding `shape_n` creates or reuses a negated group `g_user_neg` and child +`c_s7_neg`. + +Rows stored: + +```text +{:group, {:node_2, :user_id, :negated}} -> g_user_neg +{:child, g_user_neg, s7} -> c_s7_neg +{:child_meta, c_s7_neg} -> {g_user_neg, s7, :negated, wc_s7_neg} +{:subquery_child, s7} -> c_s7_neg +{:negated, g_user_neg} -> c_s7_neg + +{:child_shape, c_s7_neg} -> {shape_n, branch_n} +{:shape_child, shape_n} -> c_s7_neg +{:shape_subquery, shape_n, ["$sublink", "0"]} -> {s7, 0} +``` + +No per-value negated routing rows are stored. + +Cost: + +```text +O(number_of_subquery_occurrences_in_shape + child_where_insert) +``` + +Memory added for negated routing is `O(1)` per child, not +`O(number_of_values_in_s7_retained_window)`. + +#### `affected_shapes`: Positive Group + +For a root-table record: + +```text +%{"user_id" => 10} +``` + +Routing does: + +1. Evaluate the left-hand side `user_id` to `10`. +2. Look up `{:positive, g_user_pos, 10}` and get `[c_s7_pos]`. +3. Evaluate child condition `wc_s7_pos`, which considers `shape_a` and + `shape_b`. +4. Return both as candidates. No exact membership check happens in the filter + at the routed position — that's the consumer's job in `convert_change`. + +In this case the consumers in both shapes will confirm `10 ∈ s7` at their +logical time and the record is delivered: + +```text +MultiTimeView.member?(s7, 10, 0) -> true # shape_a +MultiTimeView.member?(s7, 10, 0) -> true # shape_b +``` + +Both shapes are affected. + +Cost at the filter: + +```text +O(children_for_value + child_where_eval) +``` + +For this example, `children_for_value = 1`. There is no scan of all shapes and +no scan of all values in `s7`. The per-consumer exact check is paid downstream +in `convert_change`, at `O(transition_history_length_for_value)` per shape. + +#### `affected_shapes`: Positive Group With Divergent Consumer Times + +Suppose the materializer adds value `30` to `s7` at logical time `1`: + +```text +{s7, 30} -> [:out, 1] +{:current_time, s7} -> 1 +{:positive, g_user_pos, 30} -> c_s7_pos +``` + +Now `shape_a` has advanced to logical time `1`, but `shape_b` still reads +logical time `0`: + +```text +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 1} +{:shape_subquery, shape_b, ["$sublink", "0"]} -> {s7, 0} +``` + +For: + +```text +%{"user_id" => 30} +``` + +routing finds `c_s7_pos` because `30` is a member at some retained time. The +filter returns both `shape_a` and `shape_b` as candidates — it does *not* +attempt to split them at this point. The downstream exact check then drops the +false positive at each consumer: + +```text +# In shape_a's consumer, evaluating includes_record? at logical_time 1: +MultiTimeView.member?(s7, 30, 1) -> true # shape_a keeps the record + +# In shape_b's consumer, evaluating includes_record? at logical_time 0: +MultiTimeView.member?(s7, 30, 0) -> false # shape_b drops the record +``` + +End-to-end, only `shape_a` emits the change. `shape_b` over-routes briefly but +filters the record in `Shape.convert_change`. + +Cost at the filter remains: + +```text +O(children_for_value + child_where_eval) +``` + +The per-consumer exact check is paid downstream, in `convert_change`, at: + +```text +O(transition_history_length_for_value) per shape +``` + +The extra memory for the move is one history row for `{s7, 30}` plus one +positive routing row per positive child for `s7` in that group. + +#### `affected_shapes`: Negated Group + +For: + +```text +%{"user_id" => 30} +``` + +while `{s7, 30} -> [:out, 1]` is retained, `30` is absent at time `0` and +present at time `1`. Negated routing does: + +1. Look up `{:negated, g_user_neg}` and get `[c_s7_neg]`. +2. Keep `c_s7_neg` because: + +```elixir +not MultiTimeView.member_at_all_times?(s7, 30) +``` + +3. Evaluate `wc_s7_neg` and return the attached negated shapes as candidates. + +Per-shape correctness for the negated case is again paid in +`Shape.convert_change`: for `shape_n` at logical time `0`, `NOT IN s7` is true +for `30`; if it later advances to logical time `1`, `NOT IN s7` is false for +`30`. That distinction is made by the consumer, not the filter. + +Cost at the filter: + +```text +O( + number_of_negated_children_in_group * transition_history_length_for_value + + child_where_eval +) +``` + +This is intentionally proportional to the number of negated children kept by +routing. No complement index is stored. The per-consumer exact check is again +paid downstream in `convert_change`, at +`O(transition_history_length_for_value)` per shape. + +#### Dependency Move: Add Or Remove Values + +For a move that adds `30` to `s7`: + +```text +from_time = 0 +to_time = 1 +changed_values = [30] +``` + +Rows written: + +```text +{s7, 30} -> [:out, 1] +{:current_time, s7} -> 1 +{:positive, g_user_pos, 30} -> c_s7_pos +``` + +Rows not written: + +```text +{:membership, shape_a, ["$sublink", "0"], 30} +{:membership, shape_b, ["$sublink", "0"], 30} +``` + +What is evaluated: + +1. Update the `MultiTimeView` history for each changed value. +2. Find children from `{:subquery_child, s7}`. +3. For each positive child, insert a positive routing row if the value changed + from not routable to routable for the retained window. +4. Emit a move event containing `from_time`, `to_time`, `subquery_id`, and + changed values. + +Cost: + +```text +O(number_of_changed_values * (history_update + child_nodes_for_subquery)) +``` + +For a remove of `20` from `s7` at time `2`, the history becomes: + +```text +{s7, 20} -> [:in, 2] +``` + +The positive routing row for `20` stays while any retained time still contains +`20`. It is removed later when compaction proves `member_at_some_time?(s7, 20)` +is false. + +#### Consumer Move Handling + +When `shape_a` receives the `s7` move from `0` to `1`, `ActiveMove` stores: + +```elixir +%ActiveMove{ + subquery_id: s7, + from_time: 0, + to_time: 1, + move_in_values: [{30, "30"}], + move_out_values: [] +} +``` + +It does not store: + +```text +views_before_move: MapSet.new([10, 20]) +views_after_move: MapSet.new([10, 20, 30]) +``` + +Buffered row conversion evaluates exact membership by calling +`MultiTimeView.member?/3` at `from_time` or `to_time`. Move-in SQL may +materialise `values(s7, 1)` as a query-local parameter array, but that memory +belongs to the query task and is released after the query. + +If additional materializer payloads for `s7` queue up during `shape_a`'s +buffering — say a move-in of `{40, "40"}` at time `2` and a move-out of +`{20, "20"}` at time `3` — they reduce into the *next* combined batch the +consumer pops after splicing this move: + +```elixir +%ActiveMove{ + subquery_id: s7, + from_time: 1, + to_time: 3, + move_in_values: [{40, "40"}], + move_out_values: [{20, "20"}] +} +``` + +`from_time` here is the consumer's processed time when the first follow-up +payload arrived (i.e. `shape_a`'s previous `ActiveMove.to_time`); `to_time` +is the max of the contributing `to_time`s; the value lists are the reduced +net effect. By construction +`MTV(s7, 3) = MTV(s7, 1) + {40} - {20}`. + +Steady memory added per active move is: + +```text +O(number_of_changed_values + number_of_subquery_refs) +``` + +not `O(number_of_values_in_s7_retained_window)`. + +#### `notify_processed_up_to` And Compaction + +After `shape_a` no longer needs time `0`, it calls: + +```elixir +SubqueryProgressMonitor.notify_processed_up_to(0, s7) +``` + +Progress monitor rows conceptually change from: + +```text +{s7, shape_a} -> required_time 0 +{s7, shape_b} -> required_time 0 +``` + +to: + +```text +{s7, shape_a} -> required_time 1 +{s7, shape_b} -> required_time 0 +``` + +The minimum is still `0`, so `MultiTimeView` cannot compact away time `0`. +After `shape_b` also notifies up to `0`, the minimum becomes `1`. Then: + +```text +{s7, 30} -> [:out, 1] +``` + +can compact to: + +```text +{s7, 30} -> [] +``` + +For a removed value: + +```text +{s7, 20} -> [:in, 2] +``` + +if the minimum required time later advances past `2`, compaction can delete the +`MultiTimeView` row and remove stale positive routes: + +```text +delete {s7, 20} +delete {:positive, g_user_pos, 20} -> c_s7_pos +``` + +Cost for notification: + +```text +O(progress_index_update + min_recompute_for_subquery) +``` + +With an index keyed by `{subquery_id, required_time, consumer_id}`, reading the +minimum is `O(1)` or `O(log consumers_for_subquery)` depending on the ETS +layout chosen. Compaction cost is paid separately and can be incremental. For +one compacted value it is: + +```text +O(transition_history_length_for_value + positive_children_for_subquery) +``` + +If compaction is batched, total work is proportional to the histories visited +and the stale route rows removed. + +#### Move-In Query Construction + +For the `s7` move from time `0` to `1`, existing SQL generation may still need +arrays for the before and after views. The new design builds them from +`MultiTimeView` inside the query task: + +```elixir +values_for.(["$sublink", "0"], 0) -> [10, 20] +values_for.(["$sublink", "0"], 1) -> [10, 20, 30] +``` + +What is stored persistently: + +```text +nothing beyond the ActiveMove times and changed values +``` + +What is allocated transiently: + +```text +query-local arrays for values(s7, 0) and values(s7, 1) +move-in snapshot rows returned by Postgres +``` + +Cost for the compatibility implementation: + +```text +O(number_of_values_in_s7_retained_window + root_rows_returned_by_move_in_query) +``` + +This does not yet minimize move-in query memory, but it moves full-view arrays +out of steady consumer state and into short-lived query tasks. + +#### `remove_shape` + +Removing `shape_a` reads: + +```text +{:shape_child, shape_a} -> c_s7_pos +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 1} +``` + +Rows removed: + +```text +{:child_shape, c_s7_pos} -> {shape_a, branch_a} +{:shape_child, shape_a} -> c_s7_pos +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 1} +``` + +The monitor registration for `{shape_a, s7}` is removed. `shape_a` is removed +from the child `WhereCondition`. + +If `shape_b` still uses `c_s7_pos`, no value routing rows are touched. Cost is: + +```text +O(children_for_shape + subqueries_for_shape + child_where_remove) +``` + +If this removes the last shape from `c_s7_pos`, the child is deleted too: + +```text +{:child, g_user_pos, s7} +{:child_meta, c_s7_pos} +{:subquery_child, s7} -> c_s7_pos +{:positive, g_user_pos, value} -> c_s7_pos for each retained value +``` + +The positive route cleanup iterates `MultiTimeView.values(s7)` and deletes the +specific `{group, value, child}` route rows. That last-child case costs: + +```text +O(number_of_values_in_s7_retained_window + child_metadata) +``` + +It does not scan unrelated subqueries or unrelated shapes. + +#### `remove_subquery` + +Removing dependency subquery `s7` reads: + +```text +{:subquery_child, s7} -> c_s7_pos +{:subquery_child, s7} -> c_s7_neg +``` + +Then it removes: + +```text +child metadata for c_s7_pos and c_s7_neg +participant rows for shapes attached to those children +positive routing rows for s7 values +negated group rows for s7 negated children +MultiTimeView rows with key prefix s7 +progress monitor rows for s7 +``` + +Cost: + +```text +O( + child_nodes_for_subquery + + sum(shapes_attached_to_each_child) + + number_of_values_in_s7_retained_window +) +``` + +This is proportional to the removed subquery's children, participants, and +values. It should not scan the whole `SubqueryIndex` or all shapes in the +stack. + +### Memory Savings Prototype + +The prototype script is: + +```text +packages/sync-service/scripts/subquery_logical_time_memory.exs +``` + +Run it directly with Elixir so it does not start the sync-service application: + +```sh +elixir scripts/subquery_logical_time_memory.exs +``` + +There is also a focused test file: + +```text +packages/sync-service/test/electric/shapes/filter/subquery_logical_time_memory_bench_test.exs +``` + +The prototype compares: + +- the current model: current `SubqueryIndex`-style ETS rows, per-consumer + `MapSet` views, and active-move before/after views; +- the logical-time model: shared `MultiTimeView` rows, shared child routing and + metadata rows, progress-monitor rows, compact per-consumer subquery + references, and active moves that store changed values plus logical times. + +The model intentionally uses small integer dependency values. That is +conservative for workloads with large text, UUID, or composite values because +the current model duplicates those values per shape, while the logical-time +model stores them once per retained subquery value plus routing rows. + +The local run below was generated on: + +```text +OTP: 28 +Elixir: 1.19.5 +Architecture: aarch64-apple-darwin24.5.0 +Word size: 8 bytes +``` + +#### Local Measured Scenarios + +| Scenario | Current total | Current index | Current consumers | Logical total | Logical ETS | Logical consumers | Savings | +|----------|---------------|---------------|-------------------|---------------|-------------|-------------------|---------| +| 1 shape, 1k values, steady | 331.6 KiB | 302.4 KiB | 29.3 KiB | 222.9 KiB | 222.6 KiB | 256 B | 32.8% | +| 10 shapes, 1k values, steady | 3.2 MiB | 2.91 MiB | 292.5 KiB | 229.1 KiB | 226.6 KiB | 2.5 KiB | 93.0% | +| 100 shapes, 1k values, steady | 31.92 MiB | 29.06 MiB | 2.86 MiB | 290.9 KiB | 265.9 KiB | 25.0 KiB | 99.1% | +| 100 shapes, 10k values, steady | 318.9 MiB | 290.02 MiB | 28.87 MiB | 1.78 MiB | 1.76 MiB | 25.0 KiB | 99.4% | +| 100 shapes, 1k base, 100 added x 10 advanced | 32.24 MiB | 29.35 MiB | 2.88 MiB | 309.6 KiB | 284.6 KiB | 25.0 KiB | 99.1% | +| 100 shapes, 1k base, 100 added x 99 advanced | 35.07 MiB | 31.94 MiB | 3.13 MiB | 309.6 KiB | 284.6 KiB | 25.0 KiB | 99.1% | +| 100 shapes, 1k base, 100 added x 10 active move | 32.87 MiB | 29.35 MiB | 3.52 MiB | 349.8 KiB | 284.6 KiB | 65.2 KiB | 99.0% | +| 100 shapes, 1k base, 1k added x 99 active move | 75.51 MiB | 57.77 MiB | 17.75 MiB | 4.25 MiB | 453.4 KiB | 3.81 MiB | 94.4% | + +Interpretation: + +- Subqueries used by one shape still save memory, but only by a constant + factor. There is no sharing benefit when a subquery has one participant. +- Shared steady-state subqueries get the largest win because the current model + stores value membership and consumer views once per shape. +- Active moves remain materially smaller because the logical-time model stores + changed values and times, not before and after full dependency views. +- The harsh `1k added x 99 active move` case still grows because every active + move stores the changed values. It is still much smaller than the current + model because it avoids duplicating the 1k base view twice per active move. + +#### Customer-Shaped Estimates + +These estimates use the same script. They extrapolate from measured row costs +and use the customer workload ratios from PR #4280: + +- HumanLayer: 75 observed `WHERE` clauses, 134 subquery occurrences, 13 + distinct literal subqueries. +- AutoArc: 611 observed `WHERE` clauses, 291 subquery occurrences, 209 + distinct literal subqueries. +- Hazel: 13 observed shape handles, 4 subquery occurrences, 4 distinct literal + subqueries. + +The extrapolation is for 100k shapes and preserves each workload's observed +ratio of subquery occurrences to distinct literal subqueries. A distinct +literal subquery here means a distinct dependency subquery, not a subquery +group. + +| Customer | Observed occurrences -> distinct subqueries | Shared occurrences | Participants @100k | Distinct subqueries @100k | Rows/subquery | Current | Logical-time | Savings | +|----------|----------------------------------------------|--------------------|--------------------|---------------------------|---------------|---------|--------------|---------| +| HumanLayer | 134 -> 13 | 90.3% | 178,667 | 17,334 | 1,000 | 55.77 GiB | 4.2 GiB | 92.5% | +| HumanLayer | 134 -> 13 | 90.3% | 178,667 | 17,334 | 10,000 | 556.19 GiB | 40.59 GiB | 92.7% | +| AutoArc | 291 -> 209 | 28.2% | 47,627 | 34,207 | 1,000 | 14.87 GiB | 8.04 GiB | 45.9% | +| AutoArc | 291 -> 209 | 28.2% | 47,627 | 34,207 | 10,000 | 148.26 GiB | 79.86 GiB | 46.1% | +| Hazel | 4 -> 4 | 0.0% | 30,770 | 30,770 | 1,000 | 9.61 GiB | 7.23 GiB | 24.8% | +| Hazel | 4 -> 4 | 0.0% | 30,770 | 30,770 | 10,000 | 95.79 GiB | 71.83 GiB | 25.0% | + +Interpretation: + +- HumanLayer benefits most because the captured workload has high literal + subquery sharing. +- AutoArc still benefits, but many literal subqueries are not shared, so the + logical-time model stores more per-subquery shared views. +- Hazel has no observed literal sharing. The estimate still shows a constant + factor reduction because the current model stores both index membership rows + and consumer `MapSet` views per shape, while the logical-time model stores + one shared view per one-participant subquery and compact consumer references. +- If a production workload has one-off subqueries with large dependency views, + the logical-time design is still better than current state, but it is not the + main win. The main win comes when multiple shapes share a subquery. + +### Materializer Integration + +The materializer owns the source of truth for a dependency subquery. It should +populate `MultiTimeView` during initial materialization and mark the subquery +ready only after the full initial view is visible. + +When a committed dependency change alters membership, the materializer should: + +1. Read the current logical time `a`. +2. Increment to logical time `b`. +3. Write the transition into `MultiTimeView` at `b`. +4. Update positive routing before emitting the move if the value is newly + routable at some retained time. +5. Emit the dependency move with `from_time: a`, `to_time: b`, `subquery_id`, + changed values, and move kind. + +Consumers must not observe a move event whose target time is absent from +`MultiTimeView`. + +### Consumer Registration + +Consumers register for each subquery they read. Registration should be +serialized through the dependency materializer so the returned time and the +shared view are consistent: + +```elixir +{:ok, current_time} = + Materializer.register_subquery_consumer( + subquery_id, + outer_shape_handle, + self() + ) +``` + +The registration side effects are: + +- wait until the dependency materializer has finished initial population +- register the consumer with `SubqueryProgressMonitor` +- set the consumer's initial `required_time` to `current_time` +- **atomically add the consumer to the materializer's subscribers list** + before returning `current_time` +- return `current_time` to the caller + +The atomic-subscribe step is required for correctness. A two-call shape +("register, then subscribe") opens a race window where the materializer +commits between the calls: those commits go to the *old* subscribers list +and the new consumer never sees them. Its `current_time` stays at the value +returned by `register`, but the materializer's logical time has advanced +past it, so the consumer's first observable `materializer_changes` event +arrives with a `from_time` strictly greater than `current_time` — and +`MTV(current_time)` no longer reflects "what the consumer has processed." +That breaks the times-as-views invariant the rest of the design depends on +(see *Consumer Move Handling* below). + +This replaces the current `Materializer.get_link_values/1` setup path for +subquery event handlers. The handler should keep compact references such as: + +```elixir +%{ + ["$sublink", "0"] => %{subquery_id: dep_handle, time: current_time} +} +``` + +not `MapSet` views. + +The monitor should track consumers by process monitor plus registered +subqueries so dead consumers automatically release pinned times. An explicit +unregister path can be added for normal shutdown, but correctness must not +depend on it. + +### Consumer Move Handling + +For a move from time `a` to time `b`, `ActiveMove` should store times and +the values whose membership changed between those times. It does *not* store +view snapshots: + +```elixir +%ActiveMove{ + subquery_id: subquery_id, + dep_index: dep_index, + subquery_ref: subquery_ref, + from_time: a, + to_time: b, + move_in_values: ins, # values entering the dep view in [a, b] + move_out_values: outs, # values leaving the dep view in [a, b] + txids: [...], # source PG xids + ... # snapshot/query state for the move-in query +} +``` + +Elixir-side evaluation of buffered transactions uses callbacks into +`MultiTimeView`: + +```elixir +before_member? = fn ref, value -> member?(ref, value, a) end +after_member? = fn ref, value -> member?(ref, value, b) end +``` + +For SQL move-in queries, the first implementation can still materialise +query-local arrays by calling `MultiTimeView.values(subquery_id, time)`. The +important change is that these arrays are transient query parameters, not +long-lived per-consumer state. + +After the move is spliced and the consumer no longer needs time `a`, it calls: + +```elixir +SubqueryProgressMonitor.notify_processed_up_to(a, subquery_id) +``` + +The consumer's current logical time for that subquery is separate from this +retention notification. It should advance to `b` at the same point the current +implementation would update per-shape membership rows for subsequent routing. +The important invariant is that live routing must not under-route, while +`required_time` continues to pin `a` until the consumer no longer needs the old +view. + +#### Combined Move Batches (Times vs Views) + +The pre-RFC implementation stores frozen `MapSet` `views_before_move` and +`views_after_move`. Those snapshots are *path-dependent*: they reflect what +this consumer has processed, in the order it chose to process it. The +consumer's MoveQueue freely reorders move-outs ahead of move-ins because set +operations on disjoint values commute, so the consumer's final view is +unchanged. + +`MultiTimeView` doesn't have that freedom. `MTV(t)` is the materializer's +canonical view at logical time `t` — and the materializer applies moves in +PG commit order. Times are points in a totally ordered history, not view +deltas. So the consumer cannot pop the move-out batch first and "advance to +some intermediate time where only the outs have been applied" — no such +time exists if a move-in committed between them. + +To preserve the times-as-views invariant +(`MTV(consumer.time) = what the consumer has processed so far`), an +`ActiveMove` covers a *single contiguous window* `[a, b]` per dep and carries +*both* move-in and move-out values for that window. `MoveQueue.pop_next/1` +returns one combined batch per dep at a time: + +```elixir +{:ok, batch} = MoveQueue.pop_next(queue) +# batch = %{ +# dep_index: 0, +# move_in_values: [{V2, "V2"}, {V3, "V3"}], +# move_out_values: [{V1, "V1"}], +# from_time: a, +# to_time: b, +# txids: [...] +# } +``` + +The batch's `from_time` is the consumer's processed time when the *first* +payload in the window arrived (preserved across subsequent enqueues for the +same dep). The `to_time` is the max of the contributing payload `to_time`s. +By construction of the queue's per-dep reduce, +`MTV(b) = MTV(a) + move_in_values - move_out_values`. + +The splice plan for a combined batch emits effects in this order: + +``` +pre_ops — buffered txns before move-in snapshot, evaluated at MTV(a) +move_out_broadcast — for outer move-out values (may be empty) +move_in_broadcast — for outer move-in values (may be empty) +snapshot — records loaded by the move-in query +post_ops — buffered txns after snapshot, evaluated at MTV(b) +``` + +`pre_ops` first means a buffered txn that references a value about to be +moved out is stored at the *pre-batch* view (consistent with `MTV(a)`), and +the subsequent move-out broadcast cleans it up — the client sees `UPDATE +then DELETE` for that row, never `DELETE then UPDATE` (which would surface +as "update for row that does not exist"). + +For pure move-out batches (no move-in values, hence no PG query needed) the +consumer skips Buffering and broadcasts the move-out inline, advancing time +to `b` and recursing on the queue. + +#### MoveQueue Compaction Rules + +Per dep, within one contiguous window `[a, b]`, the queue may compact +arbitrary sequences of moves into a single `(move_in_values, move_out_values)` +pair as long as the net effect preserves +`MTV(b) = MTV(a) + ins - outs`. Specifically: + +- multiple adds of the same value collapse to one (idempotent) +- multiple removes of the same value collapse to one +- `add V` then `remove V` cancel (net zero) +- `remove V` then `add V` cancel (net zero) +- adds and removes for disjoint values keep both + +This is the same reduce the pre-RFC queue uses; the only change is that the +result is now expressed as two value lists carried by one `ActiveMove`, +rather than two batches popped separately. + +Cross-dep compaction is not safe — each dep has its own `subquery_id` and +its own MTV history. Each dep gets its own `ActiveMove`. + +### Querying Changes + +`Querying.move_in_where_clause/5` currently receives +`views_before_move` and `views_after_move` maps. Replace those maps with a view +resolver that can provide values for a subquery ref at a logical time: + +```elixir +values_for.(subquery_ref, time) +``` + +Initial implementation can adapt this resolver back into arrays at the SQL +boundary, preserving existing SQL generation behavior. A later optimization can +special-case the triggering subquery position and use only the changed values +for candidate selection when the DNF plan makes that safe. + +This keeps the first implementation smaller while still removing long-lived +view copies. + +### Failure Modes + +If `MultiTimeView` is not ready for a subquery, shapes using that subquery must +stay in fallback routing. They must not be marked ready. + +If a consumer dies while it pins an old time, `SubqueryProgressMonitor` must +release its registration via the process monitor. Otherwise compaction can be +blocked indefinitely. + +If a dependency materializer is removed, `MultiTimeView.remove_subquery/1` must +remove the view and `SubqueryIndex` must remove the children and participants +associated with that subquery without scanning unrelated shapes. + +If compaction falls behind, correctness is preserved but routing becomes more +conservative and histories grow. Add telemetry so this is visible. + +### Telemetry + +Add enough telemetry to prove or disprove the design: + +- number of values per subquery +- number of retained histories per subquery +- max and average history length +- min/current logical time gap per subquery +- number of registered consumers per subquery +- number of child nodes per subquery group +- first-child synchronous seed duration +- shape removal duration +- transient SQL move-in array size + +### Complexity Check + +- **Is this the simplest approach?** No. The simplest immediate fix is adding a + reverse index for shape-owned values or using tombstones. Those approaches do + less architectural work, but they keep or increase the duplicated full-view + memory that caused the problem. This proposal is more complex because it + crosses the materializer, event handler, querying, and filter index + boundaries, but it removes both major long-lived duplicate view pools. +- **What could we cut?** The first implementation can keep existing SQL array + generation, materializing arrays only at query time. It can also postpone + aggressive history encoding, background compaction tuning, and cross-handle + subquery interning. +- **What's the 90/10 solution?** Implement `MultiTimeView`, serialized + registration, per-consumer logical times, and shared child routing. Keep + move-in SQL generation structurally the same by resolving values from the + shared view at the SQL boundary. Add telemetry before optimizing the query + format further. + +## Open Questions + +Unresolved questions that need further discussion or will be determined during +implementation: + +| Question | Options | Resolution Path | +|----------|---------|-----------------| +| **How should `values(subquery_id, time)` expose large views?** | Materialized `MapSet`, stream, both | Start with query-local materialization for compatibility, then prototype streaming or chunked array construction if telemetry shows spikes. | +| **Where should per-subquery logical times live?** | In `SubqueryIndex` participant rows, in `SubqueryProgressMonitor`, or in consumer-owned state with callbacks | Decide during implementation. Exact membership checks need fast `shape_handle + subquery_ref -> {subquery_id, logical_time}` lookup, so `SubqueryIndex` is the likely owner. | +| **When should positive routing rows be removed after compaction?** | Opportunistically on read/write, periodic cleanup, immediate cleanup when min time advances | Implement opportunistic plus periodic cleanup first. Add immediate cleanup only if stale positive routes are expensive. | +| **Should long histories switch representation?** | Keep flat lists, switch to tuples/arrays after a threshold, or compact eagerly | Keep flat lists for v1 and add telemetry for max history length before adding another representation. | + +## Definition of Success + +### Primary Hypothesis + +> We believe that implementing shared subquery logical-time views will enable +> the issue #4279 hypothesis: subquery indexing can become scalable for shape +> add/remove and memory use while preserving v1.6 subquery move correctness. +> +> We'll know we're right if shared subqueries no longer allocate full +> per-consumer dependency views in steady state, shape removal no longer scans +> value-keyed membership rows owned by unrelated shapes, and existing subquery +> move correctness tests continue to pass. +> +> We'll know we're wrong if retained histories grow without bound under normal +> consumer lag, move-in query memory still dominates production incidents, or +> the cross-subsystem complexity creates correctness regressions compared with +> the current per-consumer view model. + +### Functional Requirements + +| Requirement | Acceptance Criteria | +|-------------|---------------------| +| Shared subquery view | One `MultiTimeView` view exists per `subquery_id`, and steady-state consumers do not store full `MapSet` views. | +| Per-consumer per-subquery logical time | Each consumer can evaluate each subquery at that subquery's own logical time. | +| Correct registration | Consumer registration is serialized with the materializer and returns a current logical time whose view is ready. | +| Progress notification | Consumers call `notify_processed_up_to(time, subquery_id)` after finishing moves, and compaction uses the minimum required time. | +| Synchronous first child seed | First-time child creation seeds routing for the current view before removing fallback. | +| Positive routing correctness | Values that are members at any retained time route to the relevant child node. | +| Negated routing correctness | Negated groups route conservatively and filter with `member_at_all_times?/2`. | +| Shape removal scalability | Removing a shape follows participant and child rows, not all subquery values for unrelated shapes. | +| Move-in compatibility | Existing move-in SQL behavior can be produced from logical-time views without long-lived before/after `MapSet` copies. | +| Observability | Telemetry reports retained time gaps, history sizes, seed duration, and removal duration. | + +### Learning Goals + +1. Measure how large retained logical-time windows become under realistic + consumer lag. +2. Measure whether transient move-in SQL arrays remain a material memory cost + after removing long-lived view copies. +3. Determine whether flat list histories are sufficient or whether a threshold + representation is needed. +4. Determine whether conservative positive routing creates measurable extra + filter work before compaction catches up. + +## Alternatives Considered + +These alternatives are based on the discussion and rejected approaches in +PR #4280. + +### Alternative 1: Add `shape_handle -> all values` + +**Description:** Add a reverse index from each shape to the full set of values +it has inserted into `SubqueryIndex`. + +**Why not:** This improves shape removal, but it adds another full per-shape +dependency view. It makes the removal path easier by increasing the same memory +duplication this RFC is trying to remove. + +### Alternative 2: Tombstone Removed Shapes And Clean Later + +**Description:** Mark removed shapes as tombstoned and clean their value-keyed +membership rows asynchronously. + +**Why not:** This is useful as an emergency mitigation, but it is not a +structural memory fix. It leaves stale rows in the hot routing path and +requires liveness checks or cleanup debt elsewhere. + +### Alternative 3: One Global Widened Filter + +**Description:** Store one widened filter for each subquery and route every +value that might match any participant, relying on downstream exact filtering. + +**Why not:** A slow or stalled consumer can keep the shared filter broad and +over-route work for every other participant. This preserves correctness, but it +can move cost from memory to sustained routing and filtering work. + +### Alternative 4: Intern Full Dependency Views + +**Description:** Deduplicate identical full dependency views by interning +`MapSet` values or equivalent view structures. + +**Why not:** This handles exact equality at a point in time, but one-value +moves immediately create new views or require a second delta representation. +At that point the design becomes a versioned or sparse-delta view. Logical time +models that state directly. + +### Alternative 5: Versioned Lazy Exception Clearing + +**Description:** Keep sparse exceptions and clear or promote them lazily with +versions instead of doing eager cleanup. + +**Why not:** This can reduce some hot-path work, but it adds versioning and +cleanup complexity while retaining a separate exception model. This is better +as a follow-up optimization if measurements show cleanup cost is high. + +### Alternative 6: Shared Base View With Sparse XOR Exceptions + +**Description:** The design in PR #4280 stores one base dependency view per +grouped subquery index entry and stores sparse per-participant XOR exceptions +for values where a participant temporarily differs from the base. + +**Why not:** This is a lower-risk, index-focused approach and may still be the +right short-term fix if this RFC is too broad. However, it leaves consumer-held +before/after views in place and represents temporary divergence as +per-participant exceptions instead of as consumers reading different logical +times. +The logical-time design is a broader refactor, but it addresses the duplicated +state in both `SubqueryIndex` and consumer event handlers. + +## Revision History + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 0.2 | 2026-05-18 | robacourt | Added operation examples, a memory prototype script, measured local memory scenarios, and customer-shaped estimates based on PR #4280 ratios. | +| 0.1 | 2026-05-18 | robacourt | Initial draft using the Stratovolt RFC template and alternatives from PR #4280. | + +--- + +## RFC Quality Checklist + +Before submitting for review, verify: + +**Alignment** +- [x] RFC implements the working issue hypothesis, with no separate PRD. +- [x] API naming matches ElectricSQL conventions. +- [x] Success criteria link back to the issue hypothesis. + +**Calibration for Level 1-2 PMF** +- [x] This is the smallest version of the logical-time design that validates + the memory hypothesis. +- [x] Non-goals explicitly defer protocol changes, DNF redesign, and deeper + query optimization. +- [x] Complexity Check section is filled out honestly. +- [x] An engineer could start implementing tomorrow. + +**Completeness** +- [x] Happy path is clear. +- [x] Critical failure modes are addressed. +- [x] Open questions are acknowledged, not glossed over. diff --git a/packages/sync-service/bench/subquery_index_bench.exs b/packages/sync-service/bench/subquery_index_bench.exs new file mode 100644 index 0000000000..cb71fc7e12 --- /dev/null +++ b/packages/sync-service/bench/subquery_index_bench.exs @@ -0,0 +1,822 @@ +# SubqueryIndex latency benchmarks. +# +# Run from packages/sync-service: +# +# mix run --no-start bench/subquery_index_bench.exs +# +# `--no-start` skips the sync-service application (no replication client, +# admission control, etc.) which keeps the bench output focused. +# +# Each benchmark group sweeps a single size dimension (values, history +# length, children, participants) so the Benchee output makes the scaling +# behaviour directly visible. The RFC at docs/rfcs/subquery-index.md states +# expected complexity for every operation here — see the comment above each +# `Benchee.run/2` call for the expected curve. +# +# The "add_shape: new subquery → triggers materializer" case from the task +# brief is intentionally not benchmarked here. That cost is paid in the +# materializer's initial population, which lives outside SubqueryIndex's +# responsibility and which the RFC explicitly acknowledges cannot be O(1). + +alias Electric.Replication.Eval.Parser.{Func, Ref} +alias Electric.Shapes.DnfPlan +alias Electric.Shapes.Filter +alias Electric.Shapes.Filter.Indexes.SubqueryIndex +alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView +alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor +alias Electric.Shapes.Filter.WhereCondition + +defmodule Bench.Pop do + @moduledoc false + + @field "par_id" + @subquery_ref ["$sublink", "0"] + + def field, do: @field + def subquery_ref, do: @subquery_ref + + def make_plan(opts \\ []) do + polarity = Keyword.get(opts, :polarity, :positive) + dep_index = Keyword.get(opts, :dep_index, 0) + subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) + field = Keyword.get(opts, :field, @field) + + testexpr = %Ref{path: [field], type: :int8} + ref = %Ref{path: subquery_ref, type: {:array, :int8}} + + ast = %Func{ + name: "sublink_membership_check", + args: [testexpr, ref], + type: :bool + } + + %DnfPlan{ + disjuncts: [], + disjuncts_positions: [], + position_count: 1, + positions: %{ + 0 => %{ + ast: ast, + sql: "fake", + is_subquery: true, + negated: polarity == :negated, + dependency_index: dep_index, + subquery_ref: subquery_ref, + tag_columns: [field] + } + }, + dependency_positions: %{dep_index => [0]}, + dependency_disjuncts: %{}, + dependency_polarities: %{dep_index => polarity} + } + end + + def subquery_optimisation(opts \\ []) do + field = Keyword.get(opts, :field, @field) + + %{ + operation: "subquery", + field: field, + testexpr: %Ref{path: [field], type: :int8}, + subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), + dep_index: Keyword.get(opts, :dep_index, 0), + polarity: Keyword.get(opts, :polarity, :positive), + and_where: Keyword.get(opts, :and_where) + } + end + + @doc """ + Build a fresh Filter with `subquery_count` subqueries, `values_per_subquery` + values each, attached at one shared condition_id. Returns + `{filter, condition_id, dep_handles, shape_ids}`. + """ + def build(opts) do + subquery_count = Keyword.get(opts, :subquery_count, 1) + values_per_subquery = Keyword.get(opts, :values_per_subquery, 1) + shapes_per_subquery = Keyword.get(opts, :shapes_per_subquery, 1) + polarity = Keyword.get(opts, :polarity, :positive) + + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + index = filter.subquery_index + + dep_handles = + for i <- 0..(subquery_count - 1) do + dep = "dep_#{i}" + values = for v <- 0..(values_per_subquery - 1), do: i * 10_000_000 + v + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + dep + end + + shape_ids = + for i <- 0..(subquery_count - 1), + dep = Enum.at(dep_handles, i), + j <- 0..(shapes_per_subquery - 1) do + shape_id = "shape_#{i}_#{j}" + + SubqueryIndex.register_shape( + index, + shape_id, + make_plan(polarity: polarity), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + subquery_optimisation(polarity: polarity), + [] + ) + + SubqueryIndex.mark_ready(index, shape_id) + shape_id + end + + {filter, condition_id, dep_handles, shape_ids} + end + + @doc """ + Build a filter where one subquery has `child_count` positive children, each + in a distinct group (distinct condition_id), with one shape per child. Used + for benchmarks that sweep "positive_children_for_subquery". + """ + def build_n_positive_children(child_count, opts \\ []) do + values = Keyword.get(opts, :values, [1, 2]) + polarity = Keyword.get(opts, :polarity, :positive) + filter = Filter.new() + index = filter.subquery_index + dep = "dep_shared" + + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + + condition_ids = + for i <- 0..(child_count - 1) do + cid = make_ref() + WhereCondition.init(filter, cid) + shape_id = "child_#{i}" + + SubqueryIndex.register_shape( + index, + shape_id, + make_plan(polarity: polarity), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + cid, + shape_id, + subquery_optimisation(polarity: polarity), + [] + ) + + SubqueryIndex.mark_ready(index, shape_id) + cid + end + + {filter, condition_ids, dep} + end + + @doc """ + Build a filter where one *negated* group has `child_count` children, each + on a distinct subquery (distinct dep_handle). Used to sweep + "negated_children_in_group". + """ + def build_n_negated_children_same_group(child_count) do + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + index = filter.subquery_index + + deps = + for i <- 0..(child_count - 1) do + dep = "dep_neg_#{i}" + # Each subquery contains one member at all times (so the negated + # routing path still has to consult MTV.member_at_all_times? on + # the query value; we use a different query value below). + MultiTimeView.init_subquery(index.multi_time_view, dep, [42]) + MultiTimeView.mark_ready(index.multi_time_view, dep) + + shape_id = "neg_#{i}" + + SubqueryIndex.register_shape( + index, + shape_id, + make_plan(polarity: :negated, dep_index: 0), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + subquery_optimisation(polarity: :negated), + [] + ) + + SubqueryIndex.mark_ready(index, shape_id) + dep + end + + {filter, condition_id, deps} + end + + @doc """ + Build a `History.t/0` of length `n` toggling per logical time. Returns + the produced history list and the final logical time. + """ + def build_history(view, dep, value, n) do + for t <- 1..n do + if rem(t, 2) == 0 do + MultiTimeView.mark_out(view, dep, value, t) + else + MultiTimeView.mark_in(view, dep, value, t) + end + end + + {:ok, n} + end +end + +# ============================================================================ +# Routing hot path +# ============================================================================ + +IO.puts("\n\n# ========== Routing hot path ==========\n") + +# affected_shapes/4 — positive group +# Sweep: values_in_subquery. Expected: ~O(1) per call (value-keyed lookup). +Benchee.run( + %{ + "affected_shapes (positive)" => fn {filter, condition_id, _deps, _shapes} -> + SubqueryIndex.affected_shapes(filter, condition_id, Bench.Pop.field(), %{ + "par_id" => "42" + }) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values", n} + end, + before_scenario: fn n -> + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# affected_shapes/4 — negated group +# Sweep: negated_children_in_group. Expected: O(N) (RFC explicitly accepts). +Benchee.run( + %{ + "affected_shapes (negated)" => fn {filter, condition_id, _deps} -> + SubqueryIndex.affected_shapes(filter, condition_id, Bench.Pop.field(), %{ + "par_id" => "999" + }) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} negated children", n} + end, + before_scenario: &Bench.Pop.build_n_negated_children_same_group/1, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# MultiTimeView.member?/4 +# Sweep: history_length_for_value. Expected: O(history) — list walk. +Benchee.run( + %{ + "MultiTimeView.member?" => fn {view, dep, value, t} -> + MultiTimeView.member?(view, dep, value, t) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"history length #{n}", n} + end, + before_scenario: fn n -> + view = MultiTimeView.new() + dep = "dep" + MultiTimeView.init_subquery(view, dep, []) + Bench.Pop.build_history(view, dep, 42, n) + # Ask at the middle of the history so we don't always short-circuit at + # the head. + {view, dep, 42, div(n, 2)} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Subquery lifecycle +# ============================================================================ + +IO.puts("\n\n# ========== Subquery lifecycle ==========\n") + +# MultiTimeView.mark_ready/2 — expected O(1). +Benchee.run( + %{ + "MultiTimeView.mark_ready" => fn {view, dep} -> + MultiTimeView.mark_ready(view, dep) + end + }, + inputs: %{"single" => :only}, + before_each: fn :only -> + view = MultiTimeView.new() + dep = "dep_#{System.unique_integer([:positive])}" + MultiTimeView.init_subquery(view, dep, [1]) + {view, dep} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# SubqueryIndex.remove_subquery/2 — sweep values × 1 child. +# Expected: O(values + children) for *this* subquery, no scan of unrelated. +Benchee.run( + %{ + "remove_subquery" => fn {index, dep} -> + SubqueryIndex.remove_subquery(index, dep) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values", n} + end, + before_each: fn n -> + {filter, _cid, [dep], _shapes} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + {filter.subquery_index, dep} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Shape lifecycle +# ============================================================================ + +IO.puts("\n\n# ========== Shape lifecycle ==========\n") + +# add_shape — existing child in existing group (additional shape sharing). +# Expected: O(1) per call. +Benchee.run( + %{ + "add_shape (existing child)" => fn {filter, condition_id, dep, n} -> + shape_id = "extra_#{n}" + + SubqueryIndex.register_shape( + filter.subquery_index, + shape_id, + Bench.Pop.make_plan(), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + {filter, condition_id, [dep], _shapes} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + {filter, condition_id, dep, n} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# add_shape — new child, MTV ready: must seed positive routing from MTV. +# Expected: O(values_in_subquery) one-off. +Benchee.run( + %{ + "add_shape (new child, MTV ready, seeds routing)" => fn {filter, dep, n} -> + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + + shape_id = "first_#{n}" + + SubqueryIndex.register_shape( + filter.subquery_index, + shape_id, + Bench.Pop.make_plan(), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + filter = Filter.new() + index = filter.subquery_index + dep = "dep" + values = for v <- 0..(n - 1), do: v + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + {filter, dep, n} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# add_shape — new child, MTV NOT ready (fallback path, no seeding). +# Expected: O(1) — no value walk. +Benchee.run( + %{ + "add_shape (new child, MTV not ready, fallback)" => fn {filter, dep, n} -> + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + + shape_id = "fb_#{n}" + + SubqueryIndex.register_shape( + filter.subquery_index, + shape_id, + Bench.Pop.make_plan(), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery (no MTV current_time)", n} + end, + before_each: fn n -> + filter = Filter.new() + dep = "dep" + # Deliberately do NOT call init_subquery — current_time(view, dep) == nil + # forces the fallback path in seed_child_routing. + _ = n + {filter, dep, n} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# remove_shape — other shapes remain on the child. +# Expected: O(participants_for_shape) — does NOT walk subquery values. +Benchee.run( + %{ + "remove_shape (other shapes remain)" => fn {filter, condition_id, shape_to_remove} -> + SubqueryIndex.remove_shape( + filter, + condition_id, + shape_to_remove, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + # Two shapes on the same child, so removing one keeps the child alive. + {filter, condition_id, _deps, [s1, _s2]} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 2) + + {filter, condition_id, s1} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# remove_shape — last shape on the child (collapses child, drops routing). +# Expected: O(values_in_subquery) — must remove every positive route row. +Benchee.run( + %{ + "remove_shape (last on child, drops routes)" => fn {filter, condition_id, shape_to_remove} -> + SubqueryIndex.remove_shape( + filter, + condition_id, + shape_to_remove, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + {filter, condition_id, _deps, [s1]} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + {filter, condition_id, s1} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Value changes (materializer-driven) +# ============================================================================ + +IO.puts("\n\n# ========== Value changes ==========\n") + +# MultiTimeView.mark_in/4 — first time vs extending history. +Benchee.run( + %{ + "MultiTimeView.mark_in (extend existing history)" => fn {view, dep, value, next_t} -> + MultiTimeView.mark_in(view, dep, value, next_t) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"history length #{n}", n} + end, + before_each: fn n -> + view = MultiTimeView.new() + dep = "dep_#{System.unique_integer([:positive])}" + MultiTimeView.init_subquery(view, dep, []) + Bench.Pop.build_history(view, dep, 42, n) + # Next history toggle. If n is odd, value is currently :in, so the next + # mark_in is a no-op; bias to even so mark_in actually appends. + next_t = n + if(rem(n, 2) == 0, do: 1, else: 2) + {view, dep, 42, next_t} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +Benchee.run( + %{ + "MultiTimeView.mark_out (extend existing history)" => fn {view, dep, value, next_t} -> + MultiTimeView.mark_out(view, dep, value, next_t) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"history length #{n}", n} + end, + before_each: fn n -> + view = MultiTimeView.new() + dep = "dep_#{System.unique_integer([:positive])}" + MultiTimeView.init_subquery(view, dep, []) + Bench.Pop.build_history(view, dep, 42, n) + next_t = n + if(rem(n, 2) == 1, do: 1, else: 2) + {view, dep, 42, next_t} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# add_positive_route / remove_positive_route — sweep positive_children_for_subquery. +# Expected: O(positive_children). +Benchee.run( + %{ + "add_positive_route" => fn {filter, dep, value} -> + SubqueryIndex.add_positive_route(filter.subquery_index, dep, value) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"#{n} positive children", n} + end, + before_each: fn n -> + {filter, _cids, dep} = Bench.Pop.build_n_positive_children(n, values: []) + # Pick a fresh value each iteration to avoid no-op writes when the bag + # already contains the row. + value = System.unique_integer([:positive]) + {filter, dep, value} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +Benchee.run( + %{ + "remove_positive_route" => fn {filter, dep, value} -> + SubqueryIndex.remove_positive_route(filter.subquery_index, dep, value) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"#{n} positive children", n} + end, + before_each: fn n -> + {filter, _cids, dep} = Bench.Pop.build_n_positive_children(n, values: []) + value = System.unique_integer([:positive]) + # Seed the route on every child first, so remove has actual work to do. + SubqueryIndex.add_positive_route(filter.subquery_index, dep, value) + {filter, dep, value} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Progress / compaction +# ============================================================================ + +IO.puts("\n\n# ========== Progress / compaction ==========\n") + +stack_for_pm = "bench-stack-#{System.unique_integer([:positive])}" +{:ok, pm_pid} = ProgressMonitor.start_link(stack_id: stack_for_pm) + +# notify_processed_up_to/3 — expected O(1) when no consumer change makes +# the minimum recompute walk many rows. Use a single consumer for one +# subquery; only the consumer's own row is touched. +Benchee.run( + %{ + "ProgressMonitor.notify_processed_up_to" => fn {dep, shape_handle, t} -> + :ok = ProgressMonitor.notify_processed_up_to(stack_for_pm, t, dep, shape_handle) + end + }, + inputs: %{"single consumer" => :only}, + before_each: fn :only -> + dep = "dep_#{System.unique_integer([:positive])}" + shape_handle = "shape_#{System.unique_integer([:positive])}" + :ok = ProgressMonitor.register_consumer(stack_for_pm, dep, shape_handle, self(), 0) + t = System.unique_integer([:positive]) + {dep, shape_handle, t} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# Compactor pass — sweep "values touched" via dirty histories that have to +# compact. Each iteration sets a min_required_time past the toggle and +# measures `MultiTimeView.set_min_required_time/3` plus the route cleanup +# loop in `Compactor.compact_subquery/4` directly, since the GenServer tick +# is just a wrapper around those. +Benchee.run( + %{ + "Compactor pass (set_min_required_time + route cleanup)" => + fn {filter, dep, min_time} -> + removed = MultiTimeView.set_min_required_time(filter.subquery_index.multi_time_view, dep, min_time) + + for value <- removed do + SubqueryIndex.remove_positive_route(filter.subquery_index, dep, value) + end + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} dirty values", n} + end, + before_each: fn n -> + {filter, _cid, [dep], _shapes} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + # Mark every value out at time 1, so a min_time of 2 compacts every + # history to empty and triggers the full deletion + route cleanup path. + for v <- 0..(n - 1) do + MultiTimeView.mark_out(filter.subquery_index.multi_time_view, dep, v, 1) + end + + {filter, dep, 2} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# Dead-consumer release: a consumer monitored by ProgressMonitor dies while +# it had pinned times on N subqueries. The DOWN handler must process each +# of those subqueries. +Benchee.run( + %{ + "ProgressMonitor consumer DOWN release" => fn {pid, _deps} -> + ref = Process.monitor(pid) + send(pid, :stop) + + receive do + {:DOWN, ^ref, :process, ^pid, _} -> :ok + after + 1_000 -> raise "timeout waiting for consumer DOWN" + end + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"#{n} pinned subqueries", n} + end, + before_each: fn n -> + {consumer_pid, _ref} = + spawn_monitor(fn -> + receive do + :stop -> :ok + end + end) + + deps = + for i <- 0..(n - 1) do + dep = "dep_down_#{System.unique_integer([:positive])}_#{i}" + :ok = ProgressMonitor.register_consumer(stack_for_pm, dep, "shape", consumer_pid, 0) + dep + end + + {consumer_pid, deps} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# Tear the GenServer down so we exit cleanly. +GenServer.stop(pm_pid) + +IO.puts("\n\nDone.\n") diff --git a/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex b/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex index 886b418761..e5f4b646ab 100644 --- a/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex +++ b/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex @@ -10,6 +10,8 @@ defmodule Electric.Replication.ShapeLogCollector.Supervisor do use Supervisor alias Electric.Replication.ShapeLogCollector + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.Compactor + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor def name(stack_id) do Electric.ProcessRegistry.name(stack_id, __MODULE__) @@ -26,6 +28,8 @@ defmodule Electric.Replication.ShapeLogCollector.Supervisor do Electric.Telemetry.Sentry.set_tags_context(stack_id: stack_id) children = [ + {ProgressMonitor, stack_id: stack_id}, + {Compactor, stack_id: stack_id}, {ShapeLogCollector, opts}, {ShapeLogCollector.RequestBatcher, stack_id: stack_id} ] diff --git a/packages/sync-service/lib/electric/shapes/consumer/effects.ex b/packages/sync-service/lib/electric/shapes/consumer/effects.ex index 05d4f9a783..172442d769 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/effects.ex @@ -6,7 +6,6 @@ defmodule Electric.Shapes.Consumer.Effects do alias Electric.Connection.Manager alias Electric.Postgres.SnapshotQuery - alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.ShapeCache.Storage alias Electric.LogItems alias Electric.Replication.LogOffset @@ -39,14 +38,24 @@ defmodule Electric.Shapes.Consumer.Effects do defmodule StartMoveInQuery do @moduledoc false - defstruct [:dnf_plan, :trigger_dep_index, :values, :views_before_move, :views_after_move] + defstruct [ + :dnf_plan, + :trigger_dep_index, + :values, + :subquery_id, + :subquery_ref, + :from_time, + :to_time + ] @type t() :: %__MODULE__{ dnf_plan: Electric.Shapes.DnfPlan.t(), trigger_dep_index: non_neg_integer(), values: list(), - views_before_move: Electric.Shapes.Consumer.Subqueries.Views.t(), - views_after_move: Electric.Shapes.Consumer.Subqueries.Views.t() + subquery_id: term(), + subquery_ref: [String.t()], + from_time: non_neg_integer(), + to_time: non_neg_integer() } end @@ -219,25 +228,16 @@ defmodule Electric.Shapes.Consumer.Effects do acc end - defp execute_effect(%AddToSubqueryIndex{} = effect, acc) do - update_subquery_index(acc, effect.dep_index, effect.subquery_ref, effect.values, :add) - end - - defp execute_effect(%RemoveFromSubqueryIndex{} = effect, acc) do - update_subquery_index(acc, effect.dep_index, effect.subquery_ref, effect.values, :remove) - end - - defp update_subquery_index(acc, dep_index, subquery_ref, values, op) do - state = acc.state - index = SubqueryIndex.for_stack(state.stack_id) - fun = if op == :add, do: &SubqueryIndex.add_value/5, else: &SubqueryIndex.remove_value/5 - - for {value, _original} <- values do - fun.(index, state.shape_handle, subquery_ref, dep_index, value) - end - - acc - end + # TODO phase 2 (subquery-index RFC): re-wire AddToSubqueryIndex / RemoveFromSubqueryIndex + # against the new shared MultiTimeView + grouped SubqueryIndex routing. With + # the rewrite, per-shape value membership is no longer stored — the + # materializer writes transitions into MultiTimeView and triggers routing + # updates via SubqueryIndex.add_positive_route / remove_positive_route at + # the *subquery* level, not per consumer. The consumer effect path becomes a + # progress notification (SubqueryProgressMonitor.notify_processed_up_to/4) + # rather than a per-shape index update. + defp execute_effect(%AddToSubqueryIndex{} = _effect, acc), do: acc + defp execute_effect(%RemoveFromSubqueryIndex{} = _effect, acc), do: acc @spec query_move_in_async(pid() | atom(), map(), StartMoveInQuery.t(), pid()) :: :ok def query_move_in_async( @@ -246,19 +246,17 @@ defmodule Electric.Shapes.Consumer.Effects do %StartMoveInQuery{} = request, consumer_pid ) do - {where, params} = - Querying.move_in_where_clause( - request.dnf_plan, - request.trigger_dep_index, - request.views_before_move, - request.views_after_move, - consumer_state.shape.where.used_refs - ) + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView - pool = Manager.pool_name(consumer_state.stack_id, :snapshot) + # Extract the specific fields the task needs so the closure doesn't + # capture the whole consumer_state. stack_id = consumer_state.stack_id shape = consumer_state.shape shape_handle = consumer_state.shape_handle + storage = consumer_state.storage + subquery_refs = consumer_state.event_handler.subquery_refs + used_refs = consumer_state.shape.where.used_refs + pool = Manager.pool_name(stack_id, :snapshot) :telemetry.execute([:electric, :subqueries, :move_in_triggered], %{count: 1}, %{ stack_id: stack_id @@ -272,6 +270,37 @@ defmodule Electric.Shapes.Consumer.Effects do Task.Supervisor.start_child(supervisor, fn -> OpenTelemetry.set_current_context(trace_context) + # Build the move-in SQL inside the task so the materialised value + # arrays from `MultiTimeView.values/3` live on the task heap and die + # with it, not on the long-lived consumer process. + # + # `values_for.(ref, :before | :after)` — for the trigger ref `:before` + # uses `request.from_time` and `:after` uses `request.to_time`; other + # refs read at the consumer's currently-pinned time so the move-in + # query sees a consistent view across all dependencies. + mtv = MultiTimeView.for_stack(stack_id) + + values_for = fn ref, when_ -> + %{subquery_id: id, time: pinned_time} = Map.fetch!(subquery_refs, ref) + + time = + cond do + ref != request.subquery_ref -> pinned_time + when_ == :before -> request.from_time + when_ == :after -> request.to_time + end + + MultiTimeView.values(mtv, id, time) + end + + {where, params} = + Querying.move_in_where_clause( + request.dnf_plan, + request.trigger_dep_index, + values_for, + used_refs + ) + snapshot_name = Electric.Utils.uuid4() try do @@ -286,7 +315,7 @@ defmodule Electric.Shapes.Consumer.Effects do Querying.query_move_in(conn, stack_id, shape_handle, shape, {where, params}, dnf_plan: request.dnf_plan, - views: request.views_after_move + values_for_ref: fn ref -> values_for.(ref, :after) end ) |> Stream.transform( fn -> {0, 0} end, @@ -297,7 +326,7 @@ defmodule Electric.Shapes.Consumer.Effects do send(task_pid, {:move_in_snapshot_stats, row_count, row_bytes}) end ) - |> Storage.write_move_in_snapshot!(snapshot_name, consumer_state.storage) + |> Storage.write_move_in_snapshot!(snapshot_name, storage) {row_count, row_bytes} = receive do diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex index 430e81ec21..8cff173ae8 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex @@ -14,40 +14,59 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do alias Electric.Shapes.Consumer.Subqueries.RefResolver alias Electric.Shapes.Consumer.Subqueries.ShapeInfo alias Electric.Shapes.Consumer.Subqueries.SplicePlan - alias Electric.Shapes.Consumer.Subqueries.Views + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor - @enforce_keys [:shape_info, :queue, :active_move] - defstruct [:shape_info, :queue, :active_move] + @enforce_keys [:shape_info, :queue, :active_move, :subquery_refs] + defstruct [:shape_info, :queue, :active_move, :subquery_refs] @type t() :: %__MODULE__{ shape_info: ShapeInfo.t(), queue: MoveQueue.t(), - active_move: ActiveMove.t() + active_move: ActiveMove.t(), + subquery_refs: Steady.subquery_refs() } @spec start( ShapeInfo.t(), - Views.t(), + Steady.subquery_refs(), MoveQueue.t(), - IndexChanges.move(), + MoveQueue.combined_batch(), [String.t()], keyword() ) :: {:ok, t(), [Effects.t()]} def start( %ShapeInfo{} = shape_info, - views, + subquery_refs, %MoveQueue{} = queue, - {dep_move_kind, dep_index, values, txids} = move, + %{ + dep_index: dep_index, + move_in_values: move_in_values, + move_out_values: move_out_values, + from_time: from_time, + to_time: to_time, + txids: txids + }, subquery_ref, opts \\ [] - ) - when is_map(views) do + ) do + %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) + state = %__MODULE__{ shape_info: shape_info, queue: queue, + subquery_refs: subquery_refs, active_move: - views - |> ActiveMove.start(dep_index, dep_move_kind, subquery_ref, values, txids) + ActiveMove.start( + subquery_id, + dep_index, + subquery_ref, + move_in_values, + move_out_values, + from_time, + to_time, + txids + ) |> ActiveMove.carry_latest_seen_lsn(Keyword.get(opts, :latest_seen_lsn)) } @@ -55,7 +74,10 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do EffectList.new() |> maybe_subscribe_global_lsn(Keyword.get(opts, :subscribe_global_lsn?, true)) |> EffectList.append_all( - IndexChanges.effects_for_buffering(state.shape_info.dnf_plan, move, subquery_ref) + IndexChanges.effects_for_buffering_active_move( + state.shape_info.dnf_plan, + state.active_move + ) ) |> EffectList.append(start_move_in_query_effect(state)) |> EffectList.to_list() @@ -87,9 +109,10 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do def handle_event(%__MODULE__{} = state, {:materializer_changes, dep_handle, payload}) do subquery_ref = RefResolver.ref_from_dep_handle!(state.shape_info.ref_resolver, dep_handle) dep_index = subquery_ref |> List.last() |> String.to_integer() - dep_view = Views.current(state.active_move.views_after_move, subquery_ref) + mtv = MultiTimeView.for_stack(state.shape_info.stack_id) + member? = member_after_active_move(mtv, state.active_move, state.subquery_refs, subquery_ref) - {:ok, %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)}, []} + {:ok, %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, member?)}, []} end def handle_event(%__MODULE__{} = state, {:pg_snapshot_known, snapshot}) do @@ -125,18 +148,23 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do end defp splice(%{active_move: active_move} = state) do - with {:ok, splice_plan} <- SplicePlan.build(active_move, state.shape_info) do + with {:ok, splice_plan} <- + SplicePlan.build(active_move, state.shape_info, state.subquery_refs) do index_effects = - IndexChanges.effects_for_complete( - state.shape_info.dnf_plan, - {active_move.dep_move_kind, active_move.dep_index, active_move.values, - active_move.txids}, - active_move.subquery_ref + IndexChanges.effects_for_complete_active_move(state.shape_info.dnf_plan, active_move) + + advance_consumer_to_after_move(state, active_move) + + next_subquery_refs = + Steady.advance_subquery_time( + state.subquery_refs, + active_move.subquery_ref, + active_move.to_time ) steady_state = %Steady{ shape_info: state.shape_info, - views: active_move.views_after_move, + subquery_refs: next_subquery_refs, queue: state.queue } @@ -173,12 +201,23 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do %Effects.StartMoveInQuery{ dnf_plan: shape_info.dnf_plan, trigger_dep_index: active_move.dep_index, - values: active_move.values, - views_before_move: active_move.views_before_move, - views_after_move: active_move.views_after_move + values: active_move.move_in_values, + subquery_id: active_move.subquery_id, + subquery_ref: active_move.subquery_ref, + from_time: active_move.from_time, + to_time: active_move.to_time } end + defp advance_consumer_to_after_move(%__MODULE__{shape_info: shape_info}, active_move) do + ProgressMonitor.notify_processed_up_to( + shape_info.stack_id, + active_move.from_time, + active_move.subquery_id, + shape_info.shape_handle + ) + end + defp maybe_subscribe_global_lsn(effects, true) do EffectList.append(effects, %Effects.SubscribeGlobalLsn{}) end @@ -190,4 +229,23 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do defp maybe_notify_flushed(effects, log_offset) do EffectList.append(effects, %Effects.NotifyFlushed{log_offset: log_offset}) end + + # The base view for reducing buffered move queue entries is the consumer's + # view *as if* the in-flight active move had already spliced. For the + # trigger ref that means MTV at `active_move.to_time`; for every other ref + # the consumer is still pinned at its currently-tracked time. Returns a + # `(value) -> boolean()` callback so MoveQueue can ask membership per + # value without materialising the whole view. + defp member_after_active_move(mtv, active_move, subquery_refs, subquery_ref) do + %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) + + time = + if subquery_ref == active_move.subquery_ref do + active_move.to_time + else + subquery_refs[subquery_ref].time + end + + fn value -> MultiTimeView.member?(mtv, subquery_id, value, time) end + end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex index 053ba678e5..20aa4a964f 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex @@ -13,26 +13,29 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do alias Electric.Shapes.Consumer.Subqueries.MoveQueue alias Electric.Shapes.Consumer.Subqueries.RefResolver alias Electric.Shapes.Consumer.Subqueries.ShapeInfo - alias Electric.Shapes.Consumer.Subqueries.Views + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor - @enforce_keys [:shape_info, :views] - defstruct [:shape_info, :views, queue: MoveQueue.new()] + @enforce_keys [:shape_info, :subquery_refs] + defstruct [:shape_info, :subquery_refs, queue: MoveQueue.new()] + + @type subquery_ref_meta() :: %{subquery_id: term(), time: non_neg_integer()} + @type subquery_refs() :: %{[String.t()] => subquery_ref_meta()} @type t() :: %__MODULE__{ shape_info: ShapeInfo.t(), - views: Views.t(), + subquery_refs: subquery_refs(), queue: MoveQueue.t() } @impl true def handle_event(%__MODULE__{} = state, %Transaction{} = txn) do - with {:ok, effects} <- append_txn_effects(txn, state.shape_info, state.views) do + with {:ok, effects} <- append_txn_effects(txn, state) do {:ok, state, effects} end end def handle_event(%__MODULE__{} = state, {:global_last_seen_lsn, _lsn}) do - # Straggler message after unsubscribe; ignore. {:ok, state, []} end @@ -48,8 +51,17 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do def handle_event(%__MODULE__{} = state, {:materializer_changes, dep_handle, payload}) do subquery_ref = RefResolver.ref_from_dep_handle!(state.shape_info.ref_resolver, dep_handle) dep_index = subquery_ref |> List.last() |> String.to_integer() - dep_view = Views.current(state.views, subquery_ref) - next_state = %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)} + mtv = MultiTimeView.for_stack(state.shape_info.stack_id) + %{subquery_id: subquery_id, time: pinned_time} = Map.fetch!(state.subquery_refs, subquery_ref) + member? = fn value -> MultiTimeView.member?(mtv, subquery_id, value, pinned_time) end + + payload_with_default_from_time = Map.put_new(Map.new(payload), :from_time, pinned_time) + + next_state = %{ + state + | queue: + MoveQueue.enqueue(state.queue, dep_index, payload_with_default_from_time, member?) + } with {:ok, next_state, effects} <- drain_queue(next_state, EffectList.new()) do {:ok, next_state, EffectList.to_list(effects)} @@ -75,74 +87,125 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do nil -> {:ok, state, effects} - {{dep_move_kind, dep_index, values, txids} = move, queue} -> - subquery_ref = RefResolver.ref_from_dep_index!(state.shape_info.ref_resolver, dep_index) - subscription_active? = Keyword.get(opts, :subscription_active?, false) - latest_seen_lsn = Keyword.get(opts, :latest_seen_lsn) - - case outer_move_kind(state.shape_info, dep_index, dep_move_kind) do - :move_in -> - with {:ok, next_state, start_effects} <- - Buffering.start( - state.shape_info, - state.views, - queue, - move, - subquery_ref, - subscribe_global_lsn?: not subscription_active?, - latest_seen_lsn: latest_seen_lsn - ) do - {:ok, next_state, EffectList.append_all(effects, start_effects)} - end - - :move_out -> - next_state = %{ - state - | queue: queue, - views: Views.apply_move(state.views, subquery_ref, values, dep_move_kind) - } - - index_effects = - IndexChanges.effects_for_complete(state.shape_info.dnf_plan, move, subquery_ref) - - effects = - effects - |> EffectList.append( - MoveBroadcast.effect_for_move_out(dep_index, values, txids, state.shape_info) - ) - |> EffectList.append_all(index_effects) - - drain_queue( - next_state, - effects, - opts - ) + {batch, queue} -> + subquery_ref = + RefResolver.ref_from_dep_index!(state.shape_info.ref_resolver, batch.dep_index) + + polarity = + Map.fetch!(state.shape_info.dnf_plan.dependency_polarities, batch.dep_index) + + if buffering_required?(polarity, batch) do + start_buffering(state, queue, batch, subquery_ref, effects, opts) + else + process_inline(state, queue, batch, subquery_ref, polarity, effects, opts) end end end - defp outer_move_kind( - %ShapeInfo{dnf_plan: %{dependency_polarities: polarities}}, - dep_index, - move_kind - ) do - case {Map.fetch!(polarities, dep_index), move_kind} do - {:positive, effect} -> effect - {:negated, :move_in} -> :move_out - {:negated, :move_out} -> :move_in + defp buffering_required?(:positive, %{move_in_values: [_ | _]}), do: true + defp buffering_required?(:negated, %{move_out_values: [_ | _]}), do: true + defp buffering_required?(_, _), do: false + + defp start_buffering(state, queue, batch, subquery_ref, effects, opts) do + subscription_active? = Keyword.get(opts, :subscription_active?, false) + latest_seen_lsn = Keyword.get(opts, :latest_seen_lsn) + + with {:ok, next_state, start_effects} <- + Buffering.start( + state.shape_info, + state.subquery_refs, + queue, + batch, + subquery_ref, + subscribe_global_lsn?: not subscription_active?, + latest_seen_lsn: latest_seen_lsn + ) do + {:ok, next_state, EffectList.append_all(effects, start_effects)} end end - defp append_txn_effects(%Transaction{} = txn, %ShapeInfo{} = shape_info, views) - when is_map(views) do + # Inline path: the batch carries only non-Buffering-kind moves (positive + # polarity move-out, or negated polarity move-in). No PG query needed — + # we broadcast the outer move-out, update routing, and advance time. + defp process_inline(state, queue, batch, subquery_ref, polarity, effects, opts) do + %{subquery_id: subquery_id, time: from_time} = + Map.fetch!(state.subquery_refs, subquery_ref) + + to_time = batch.to_time || from_time + + {outer_move_out_values, outer_move_out_kind} = + case polarity do + :positive -> {batch.move_out_values, :move_out} + :negated -> {batch.move_in_values, :move_in} + end + + advance_subquery_index_time( + state.shape_info, + subquery_ref, + subquery_id, + from_time, + to_time + ) + + next_subquery_refs = advance_subquery_time(state.subquery_refs, subquery_ref, to_time) + next_state = %{state | queue: queue, subquery_refs: next_subquery_refs} + + move = {outer_move_out_kind, batch.dep_index, outer_move_out_values, batch.txids} + + index_effects = + IndexChanges.effects_for_complete(state.shape_info.dnf_plan, move, subquery_ref) + + effects = + effects + |> EffectList.append( + MoveBroadcast.effect_for_move_out( + batch.dep_index, + outer_move_out_values, + batch.txids, + state.shape_info + ) + ) + |> EffectList.append_all(index_effects) + + drain_queue(next_state, effects, opts) + end + + defp advance_subquery_index_time( + %ShapeInfo{} = shape_info, + _subquery_ref, + subquery_id, + from_time, + _to_time + ) do + ProgressMonitor.notify_processed_up_to( + shape_info.stack_id, + from_time, + subquery_id, + shape_info.shape_handle + ) + end + + @doc false + def advance_subquery_time(subquery_refs, _subquery_ref, nil), do: subquery_refs + + def advance_subquery_time(subquery_refs, subquery_ref, to_time) do + Map.update!(subquery_refs, subquery_ref, fn meta -> %{meta | time: to_time} end) + end + + defp append_txn_effects(%Transaction{} = txn, %__MODULE__{} = state) do + mtv = MultiTimeView.for_stack(state.shape_info.stack_id) + + member? = + Electric.Shapes.WhereClause.subquery_member_from_mtv(mtv, state.subquery_refs) + with {:ok, effects} <- TransactionConverter.transaction_to_effects( txn, - shape_info.shape, - stack_id: shape_info.stack_id, - shape_handle: shape_info.shape_handle, - extra_refs: {views, views}, - dnf_plan: shape_info.dnf_plan + state.shape_info.shape, + stack_id: state.shape_info.stack_id, + shape_handle: state.shape_info.shape_handle, + extra_refs: {member?, member?}, + dnf_plan: state.shape_info.dnf_plan ) do effects = effects diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex index feb251e914..d8684fe796 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex @@ -15,18 +15,20 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do {:ok, dnf_plan} = DnfPlan.compile(state.shape) dependency_move_policy = dependency_move_policy(state.stack_id, state.shape) - {views, dep_handle_to_ref, dep_index_to_ref} = + {subquery_refs, dep_handle_to_ref, dep_index_to_ref} = dep_handles |> Enum.with_index() |> Enum.reduce({%{}, %{}, %{}}, fn {handle, index}, - {views, handle_mapping, index_mapping} -> + {subquery_refs, handle_mapping, index_mapping} -> materializer_opts = %{stack_id: state.stack_id, shape_handle: handle} - :ok = Materializer.wait_until_ready(materializer_opts) - view = Materializer.get_link_values(materializer_opts) + + {:ok, time} = + Materializer.register_subquery_consumer(materializer_opts, state.shape_handle, self()) + ref = ["$sublink", Integer.to_string(index)] - {Map.put(views, ref, view), Map.put(handle_mapping, handle, {index, ref}), - Map.put(index_mapping, index, ref)} + {Map.put(subquery_refs, ref, %{subquery_id: handle, time: time}), + Map.put(handle_mapping, handle, {index, ref}), Map.put(index_mapping, index, ref)} end) buffer_max_transactions = @@ -47,7 +49,7 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do buffer_max_transactions: buffer_max_transactions, dependency_move_policy: dependency_move_policy }, - views: views + subquery_refs: subquery_refs } {:ok, handler, diff --git a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex index c98dc4c1f0..1b075380cd 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex @@ -15,11 +15,12 @@ defmodule Electric.Shapes.Consumer.Materializer do alias Electric.ShapeCache.Storage alias Electric.Replication.LogOffset alias Electric.Replication.Eval - alias Electric.Shapes.Shape + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor import Electric.Replication.LogOffset import Electric, only: [is_stack_id: 1, is_shape_handle: 1] - import Shape, only: :macros def name(stack_id, shape_handle) when is_stack_id(stack_id) and is_shape_handle(shape_handle) do Electric.ProcessRegistry.name(stack_id, __MODULE__, shape_handle) @@ -50,57 +51,27 @@ defmodule Electric.Shapes.Consumer.Materializer do end @doc """ - Creates the per-stack ETS table that caches link values for all materializers - in a stack. Called by `ConsumerRegistry` during stack initialization. Idempotent — - safe to call when the table already exists. + Register `pid` as a consumer of this materializer's dependency subquery + on behalf of `outer_shape_handle`. Blocks until the materializer has + finished its initial materialization, then registers the consumer with + `SubqueryProgressMonitor` at the materializer's current logical time and + returns that time. + + Replaces the pre-RFC pattern of `wait_until_ready/1 + get_link_values/1` + for indexing-path setup; consumers no longer copy the dependency view. """ - @spec init_link_values_table(stack_id :: term()) :: :ets.table() | :undefined - def init_link_values_table(stack_id) do - :ets.new(link_values_table_name(stack_id), [ - :named_table, - :public, - :set, - read_concurrency: true, - write_concurrency: true - ]) - rescue - ArgumentError -> :ets.whereis(link_values_table_name(stack_id)) - end - - @doc """ - Returns the current set of materialized link values for a shape. - Checks the shared ETS cache first (written after each committed transaction); - falls back to a synchronous GenServer call if the cache has no entry yet. - """ - def get_link_values(%{stack_id: stack_id, shape_handle: shape_handle} = opts) do - table = link_values_table_name(stack_id) - - case :ets.lookup(table, shape_handle) do - [{^shape_handle, values}] -> values - _ -> genserver_get_link_values(opts) - end - rescue - ArgumentError -> genserver_get_link_values(opts) - end - - defp genserver_get_link_values(opts) do - GenServer.call(name(opts), :get_link_values) - catch - :exit, reason -> - raise "Materializer for stack #{inspect(opts.stack_id)} and handle " <> - "#{inspect(opts.shape_handle)} is not available: #{inspect(reason)}" - end - - def get_all_as_refs(shape, stack_id) when are_deps_filled(shape) do - shape.shape_dependencies_handles - |> Enum.with_index() - |> Map.new(fn {shape_handle, index} -> - {["$sublink", Integer.to_string(index)], - get_link_values(%{ - shape_handle: shape_handle, - stack_id: stack_id - })} - end) + @spec register_subquery_consumer( + %{stack_id: Electric.stack_id(), shape_handle: Electric.shape_handle()}, + outer_shape_handle :: term(), + pid() + ) :: {:ok, non_neg_integer()} + def register_subquery_consumer(opts, outer_shape_handle, pid) + when is_pid(pid) do + GenServer.call( + name(opts), + {:register_subquery_consumer, outer_shape_handle, pid}, + :infinity + ) end def subscribe(pid) when is_pid(pid), do: GenServer.call(pid, :subscribe) @@ -131,7 +102,8 @@ defmodule Electric.Shapes.Consumer.Materializer do offset: LogOffset.before_all(), subscribed_offset: nil, ref: nil, - subscribers: MapSet.new() + subscribers: MapSet.new(), + logical_time: 0 }) {:ok, state, {:continue, :start_materializer}} @@ -176,11 +148,17 @@ defmodule Electric.Shapes.Consumer.Materializer do |> decode_json_stream() |> apply_changes(state) - write_link_values(state) + publish_initial_view_to_mtv(state) {:noreply, %{state | offset: offset}} end + defp publish_initial_view_to_mtv(%{stack_id: stack_id, shape_handle: shape_handle} = state) do + mtv = MultiTimeView.new(stack_id: stack_id) + MultiTimeView.init_subquery(mtv, shape_handle, Map.keys(state.value_counts)) + MultiTimeView.mark_ready(mtv, shape_handle) + end + @doc """ Get a stream of log entries from storage, bounded by the subscribed offset. @@ -198,10 +176,6 @@ defmodule Electric.Shapes.Consumer.Materializer do end end - def handle_call(:get_link_values, _from, %{value_counts: value_counts} = state) do - {:reply, link_values_from_counts(value_counts), state} - end - def handle_call(:wait_until_ready, _from, state) do {:reply, :ok, state} end @@ -234,6 +208,37 @@ defmodule Electric.Shapes.Consumer.Materializer do {:reply, :ok, %{state | subscribers: MapSet.put(state.subscribers, pid)}} end + def handle_call({:register_subquery_consumer, outer_shape_handle, pid}, _from, state) do + %{stack_id: stack_id, shape_handle: shape_handle, logical_time: time} = state + + :ok = + ProgressMonitor.register_consumer( + stack_id, + shape_handle, + outer_shape_handle, + pid, + time + ) + + # Atomically subscribe `pid` so the returned `time` is the materializer's + # logical time at the consumer's first observable commit. Without this + # there's a race window between this call returning and the consumer's + # follow-up `subscribe/1` — any commits in that window go to other + # subscribers only, advancing the materializer's logical time past + # `time` while the new consumer never sees those events. The result is + # `MTV(from_time)` on the consumer's first batch reflecting moves the + # consumer never processed, which breaks the times-as-views invariant. + state = + if MapSet.member?(state.subscribers, pid) do + state + else + Process.monitor(pid) + %{state | subscribers: MapSet.put(state.subscribers, pid)} + end + + {:reply, {:ok, time}, state} + end + # if the supervisor is going down then this process will also be taken down # but let's state the dependency explictly. def handle_info({{:consumer_down, _}, _ref, :process, _pid, :shutdown}, state) do @@ -259,52 +264,6 @@ defmodule Electric.Shapes.Consumer.Materializer do {:noreply, %{state | subscribers: MapSet.delete(state.subscribers, pid)}} end - @spec link_values_table_name(Electric.stack_id()) :: atom() - def link_values_table_name(stack_id) do - :"Electric.Materializer.LinkValues:#{stack_id}" - end - - @doc """ - Removes the cached link values for `shape_handle` from the shared ETS table. - Safe to call even if the table does not exist (e.g. after a stack shutdown). - """ - @spec delete_link_values(Electric.stack_id(), Electric.shape_handle()) :: :ok - def delete_link_values(stack_id, shape_handle) do - :ets.delete(link_values_table_name(stack_id), shape_handle) - :ok - rescue - ArgumentError -> - Logger.debug(fn -> - "delete_link_values: link-values table for stack #{inspect(stack_id)} " <> - "not found when deleting handle #{inspect(shape_handle)}" - end) - - :ok - end - - defp link_values_from_counts(value_counts) do - MapSet.new(Map.keys(value_counts)) - end - - defp write_link_values(%{ - stack_id: stack_id, - shape_handle: shape_handle, - value_counts: value_counts - }) do - :ets.insert( - link_values_table_name(stack_id), - {shape_handle, link_values_from_counts(value_counts)} - ) - rescue - ArgumentError -> - Logger.warning( - "write_link_values: link-values ETS table missing for stack #{inspect(stack_id)} " <> - "— cache will fall back to GenServer calls for handle #{inspect(shape_handle)}" - ) - - :ok - end - defp decode_json_stream(stream) do stream |> Stream.map(&Jason.decode!/1) @@ -410,21 +369,44 @@ defmodule Electric.Shapes.Consumer.Materializer do events = cancel_matching_move_events(state.pending_events) - if events != %{} do - events = finalize_txids(events) + state = + if events != %{} do + events = finalize_txids(events) + from_time = state.logical_time + to_time = from_time + 1 + apply_events_to_mtv(state, events, to_time) + envelope = Map.merge(events, %{from_time: from_time, to_time: to_time}) + + for pid <- state.subscribers do + send(pid, {:materializer_changes, state.shape_handle, envelope}) + end - for pid <- state.subscribers do - send(pid, {:materializer_changes, state.shape_handle, events}) + %{state | logical_time: to_time} + else + state end - end - - write_link_values(state) %{state | pending_events: %{}} end defp maybe_flush_pending_events(state, _commit?), do: state + defp apply_events_to_mtv(%{stack_id: stack_id, shape_handle: shape_handle}, events, to_time) do + mtv = MultiTimeView.new(stack_id: stack_id) + index = SubqueryIndex.for_stack(stack_id) + + for {value, _original} <- Map.get(events, :move_in, []) do + MultiTimeView.mark_in(mtv, shape_handle, value, to_time) + if index, do: SubqueryIndex.add_positive_route(index, shape_handle, value) + end + + for {value, _original} <- Map.get(events, :move_out, []) do + MultiTimeView.mark_out(mtv, shape_handle, value, to_time) + end + + :ok + end + defp finalize_txids(events) do Map.update(events, :txids, [], &Enum.sort(&1)) end diff --git a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex index 61177f707f..cf490c230e 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex @@ -43,28 +43,14 @@ defmodule Electric.Shapes.Consumer.SetupEffects do end end - defp execute_effect(%SeedSubqueryIndex{}, %State{event_handler: %{views: views}} = state) do + defp execute_effect(%SeedSubqueryIndex{}, %State{} = state) do case SubqueryIndex.for_stack(state.stack_id) do nil -> {:ok, state} index -> - for {ref, view} <- views do - dep_index = ref |> List.last() |> String.to_integer() - - SubqueryIndex.seed_membership( - index, - state.shape_handle, - ref, - dep_index, - view - ) - end - - SubqueryIndex.mark_ready(index, state.shape_handle) + :ok = SubqueryIndex.mark_ready(index, state.shape_handle) {:ok, state} end end - - defp execute_effect(%SeedSubqueryIndex{}, %State{} = state), do: {:ok, state} end diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex index 56ae4aac91..7edb329597 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex @@ -1,27 +1,33 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do # Tracks a single buffered move-in while we wait to splice it into the log. + # + # Holds the logical-time window of the move (`from_time` and `to_time`) + # against the shared `MultiTimeView` — not a per-consumer copy of the + # dependency view. The splice path materialises views from MTV at + # `from_time` / `to_time` on demand. alias Electric.Postgres.Lsn alias Electric.Replication.Changes.Transaction - alias Electric.Shapes.Consumer.Subqueries.Views @type move_value() :: {term(), term()} @enforce_keys [ + :subquery_id, :dep_index, - :dep_move_kind, :subquery_ref, - :values, - :views_before_move, - :views_after_move + :move_in_values, + :move_out_values, + :from_time, + :to_time ] defstruct [ + :subquery_id, :dep_index, - :dep_move_kind, :subquery_ref, - :values, - :views_before_move, - :views_after_move, + :move_in_values, + :move_out_values, + :from_time, + :to_time, txids: [], snapshot: nil, move_in_snapshot_name: nil, @@ -35,12 +41,13 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do ] @type t() :: %__MODULE__{ + subquery_id: term(), dep_index: non_neg_integer(), - dep_move_kind: :move_in | :move_out, subquery_ref: [String.t()], - values: [move_value()], - views_before_move: Views.t(), - views_after_move: Views.t(), + move_in_values: [move_value()], + move_out_values: [move_value()], + from_time: non_neg_integer(), + to_time: non_neg_integer(), txids: [non_neg_integer()], snapshot: {term(), term(), [term()]} | nil, move_in_snapshot_name: String.t() | nil, @@ -54,26 +61,52 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do } @spec start( - Views.t(), + subquery_id :: term(), non_neg_integer(), - :move_in | :move_out, [String.t()], - [move_value()], + move_in_values :: [move_value()], + move_out_values :: [move_value()], + from_time :: non_neg_integer(), + to_time :: non_neg_integer(), [non_neg_integer()] ) :: t() - def start(views, dep_index, dep_move_kind, subquery_ref, values, txids \\ []) - when is_map(views) do + def start( + subquery_id, + dep_index, + subquery_ref, + move_in_values, + move_out_values, + from_time, + to_time, + txids \\ [] + ) do %__MODULE__{ + subquery_id: subquery_id, dep_index: dep_index, - dep_move_kind: dep_move_kind, subquery_ref: subquery_ref, - values: values, - txids: txids, - views_before_move: views, - views_after_move: Views.apply_move(views, subquery_ref, values, dep_move_kind) + move_in_values: move_in_values, + move_out_values: move_out_values, + from_time: from_time, + to_time: to_time, + txids: txids } end + @doc """ + Returns true if the active move carries any move-in values that need a + PG query to load records. + """ + @spec has_move_in?(t()) :: boolean() + def has_move_in?(%__MODULE__{move_in_values: []}), do: false + def has_move_in?(%__MODULE__{move_in_values: _}), do: true + + @doc """ + Returns true if the active move carries any move-out values to broadcast. + """ + @spec has_move_out?(t()) :: boolean() + def has_move_out?(%__MODULE__{move_out_values: []}), do: false + def has_move_out?(%__MODULE__{move_out_values: _}), do: true + @spec buffer_txn(t(), Transaction.t()) :: t() def buffer_txn(%__MODULE__{} = active_move, %Transaction{} = txn) do active_move diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex index 65cd4e1b6f..58b82429e7 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex @@ -133,4 +133,89 @@ defmodule Electric.Shapes.Consumer.Subqueries.IndexChanges do [] end end + + @doc """ + Effects to broaden the filter when entering Buffering for a combined + ActiveMove that may carry both `move_in_values` and `move_out_values`. + + - Positive polarity: add `move_in_values` to the index (broadens). + - Negated polarity: remove `move_out_values` from the index (broadens). + """ + @spec effects_for_buffering_active_move( + DnfPlan.t(), + Electric.Shapes.Consumer.Subqueries.ActiveMove.t() + ) :: [Effects.AddToSubqueryIndex.t() | Effects.RemoveFromSubqueryIndex.t()] + def effects_for_buffering_active_move(%DnfPlan{dependency_polarities: polarities}, active_move) do + polarity = Map.get(polarities, active_move.dep_index, :positive) + + case polarity do + :positive -> + if active_move.move_in_values == [] do + [] + else + [ + %Effects.AddToSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_in_values + } + ] + end + + :negated -> + if active_move.move_out_values == [] do + [] + else + [ + %Effects.RemoveFromSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_out_values + } + ] + end + end + end + + @doc """ + Effects to narrow the filter when a combined ActiveMove splices. + + - Positive polarity: remove `move_out_values` from the index. + - Negated polarity: add `move_in_values` to the index. + """ + @spec effects_for_complete_active_move( + DnfPlan.t(), + Electric.Shapes.Consumer.Subqueries.ActiveMove.t() + ) :: [Effects.AddToSubqueryIndex.t() | Effects.RemoveFromSubqueryIndex.t()] + def effects_for_complete_active_move(%DnfPlan{dependency_polarities: polarities}, active_move) do + polarity = Map.get(polarities, active_move.dep_index, :positive) + + case polarity do + :positive -> + if active_move.move_out_values == [] do + [] + else + [ + %Effects.RemoveFromSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_out_values + } + ] + end + + :negated -> + if active_move.move_in_values == [] do + [] + else + [ + %Effects.AddToSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_in_values + } + ] + end + end + end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex index 61f69b3324..9db453e1cf 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex @@ -12,12 +12,24 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveBroadcast do @spec effect_for_move_in(move(), ShapeInfo.t()) :: %Effects.AppendControl{} def effect_for_move_in(active_move, %ShapeInfo{} = shape_info) do + polarity = + Map.get(shape_info.dnf_plan.dependency_polarities, active_move.dep_index, :positive) + + # The active move's "outer move-in" values are the dep values whose + # entry into (positive) or exit from (negated) the dep view promotes + # rows into the outer shape. + outer_move_in_values = + case polarity do + :positive -> active_move.move_in_values + :negated -> active_move.move_out_values + end + %Effects.AppendControl{ message: make( shape_info.dnf_plan, active_move.dep_index, - active_move.values, + outer_move_in_values, active_move.txids, "move-in", shape_info.stack_id, diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex index fbc35c6e6c..aff30a372e 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex @@ -1,29 +1,44 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do @moduledoc """ - Multi-dependency move queue. Tracks move_in/move_out operations per dependency index, - with deduplication and redundancy elimination scoped per dependency. - - Move-outs from any dependency are drained before move-ins from any dependency. - - Each per-dep batch also accumulates the upstream Postgres transaction ids that - contributed to it, so move-in/move-out broadcasts can carry `txids` for client - attribution (mirroring `Electric.LogItems.from_change/4`). + Multi-dependency move queue. Tracks move_in/move_out operations per + dependency index, with deduplication and redundancy elimination scoped + per dependency via `reduce/2`. + + Each `pop_next/1` call returns ONE combined entry for a single dep that + carries both `move_in_values` and `move_out_values`, along with the + `from_time` (the materializer logical time the consumer was at when the + first payload in this batch was enqueued) and `to_time` (the max of all + payload `to_time`s queued for this dep). The combined shape lets the + splice plan handle a dep's full transition window as one atomic + ActiveMove — `MTV(from_time)` and `MTV(to_time)` are well-defined + endpoints, and `MTV(to_time) = MTV(from_time) + move_in_values + - move_out_values` by construction of the reduce. + + Per-dep txids are accumulated across the contributing payloads so the + broadcasts can carry `txids` for client attribution. """ @type move_value() :: {term(), term()} @type txid() :: pos_integer() @type entry() :: {[move_value()], MapSet.t(txid())} - # move_out/move_in are maps from dep_index to {[move_value], MapSet} - defstruct move_out: %{}, move_in: %{} + defstruct move_out: %{}, move_in: %{}, from_times: %{}, to_times: %{} @type t() :: %__MODULE__{ move_out: %{non_neg_integer() => entry()}, - move_in: %{non_neg_integer() => entry()} + move_in: %{non_neg_integer() => entry()}, + from_times: %{non_neg_integer() => non_neg_integer()}, + to_times: %{non_neg_integer() => non_neg_integer()} } - @type batch_kind() :: :move_out | :move_in - @type batch() :: {batch_kind(), non_neg_integer(), [move_value()], [txid()]} + @type combined_batch() :: %{ + dep_index: non_neg_integer(), + move_in_values: [move_value()], + move_out_values: [move_value()], + from_time: non_neg_integer() | nil, + to_time: non_neg_integer() | nil, + txids: [txid()] + } @spec new() :: t() def new, do: %__MODULE__{} @@ -39,14 +54,23 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do @doc """ Enqueue a materializer payload for a specific dependency. - `dep_view` is the current view for this dependency, used for redundancy elimination. - The payload may include a `:txids` key listing the upstream xids that produced - the moves. Those xids are unioned with any already accumulated for this dep. + `member?` is a callback `(value) -> boolean()` that answers "is `value` + currently a member of the dep view at the consumer's pinned time for + this dep (or, during Buffering, the view-after-active-move for the + trigger ref)?". It's used by `reduce/2` to drop redundant ops. A + callback rather than a `MapSet` lets the caller skip materialising the + full dependency view on the consumer's heap — production passes a + closure that consults `MultiTimeView.member?/4` per value. + + The payload may include `:from_time`, `:to_time`, and `:txids` keys. + The first enqueue for a dep records `from_time` (subsequent payloads + leave it untouched). `to_time` is updated to `max(current, new)`. Txids + accumulate. """ - @spec enqueue(t(), non_neg_integer(), map() | keyword(), MapSet.t()) :: t() - def enqueue(%__MODULE__{} = queue, dep_index, payload, %MapSet{} = dep_view) - when is_map(payload) or is_list(payload) do + @spec enqueue(t(), non_neg_integer(), map() | keyword(), (term() -> boolean())) :: t() + def enqueue(%__MODULE__{} = queue, dep_index, payload, member?) + when (is_map(payload) or is_list(payload)) and is_function(member?, 1) do payload = Map.new(payload) new_txids = payload |> Map.get(:txids, []) |> MapSet.new() @@ -58,7 +82,19 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do Enum.map(existing_ins, &{:move_in, &1}) ++ payload_to_ops(payload) - {new_outs, new_ins} = reduce(ops, dep_view) + {new_outs, new_ins} = reduce(ops, member?) + + from_times = + case Map.get(payload, :from_time) do + nil -> queue.from_times + new_from_time -> Map.put_new(queue.from_times, dep_index, new_from_time) + end + + to_times = + case Map.get(payload, :to_time) do + nil -> queue.to_times + new_to_time -> Map.update(queue.to_times, dep_index, new_to_time, &max(&1, new_to_time)) + end %__MODULE__{ move_out: @@ -74,40 +110,60 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do dep_index, new_ins, MapSet.union(existing_in_txids, new_txids) - ) + ), + from_times: from_times, + to_times: to_times } end @doc """ - Pop the next batch of operations. Returns move-out batches (any dep) before move-in batches. - Returns `{batch, updated_queue}` or `nil` if the queue is empty. + Pop the next combined entry for one dep. Returns `{batch, updated_queue}` + where `batch` carries both `move_in_values` and `move_out_values` (either + may be empty, but not both — empty entries are never enqueued). Returns + `nil` if the queue is empty. """ - @spec pop_next(t()) :: {batch(), t()} | nil - def pop_next(%__MODULE__{move_out: move_out} = queue) when move_out != %{} do - {dep_index, {values, txids}} = Enum.min_by(move_out, &elem(&1, 0)) - - {{:move_out, dep_index, values, sorted_txids(txids)}, - %{queue | move_out: Map.delete(move_out, dep_index)}} - end + @spec pop_next(t()) :: {combined_batch(), t()} | nil + def pop_next(%__MODULE__{move_in: move_in, move_out: move_out}) + when move_in == %{} and move_out == %{}, + do: nil + + def pop_next(%__MODULE__{} = queue) do + dep_index = pick_dep_index(queue) + + {move_in_values, in_txids} = Map.get(queue.move_in, dep_index, {[], MapSet.new()}) + {move_out_values, out_txids} = Map.get(queue.move_out, dep_index, {[], MapSet.new()}) + txids = MapSet.union(in_txids, out_txids) |> Enum.sort() + + batch = %{ + dep_index: dep_index, + move_in_values: move_in_values, + move_out_values: move_out_values, + from_time: Map.get(queue.from_times, dep_index), + to_time: Map.get(queue.to_times, dep_index), + txids: txids + } - def pop_next(%__MODULE__{move_out: move_out, move_in: move_in} = queue) - when move_out == %{} and move_in != %{} do - {dep_index, {values, txids}} = Enum.min_by(move_in, &elem(&1, 0)) + next_queue = %__MODULE__{ + move_in: Map.delete(queue.move_in, dep_index), + move_out: Map.delete(queue.move_out, dep_index), + from_times: Map.delete(queue.from_times, dep_index), + to_times: Map.delete(queue.to_times, dep_index) + } - {{:move_in, dep_index, values, sorted_txids(txids)}, - %{queue | move_in: Map.delete(move_in, dep_index)}} + {batch, next_queue} end - def pop_next(%__MODULE__{}), do: nil - - defp sorted_txids(%MapSet{} = txids), do: Enum.sort(txids) + defp pick_dep_index(%__MODULE__{move_in: move_in, move_out: move_out}) do + candidates = Map.keys(move_in) ++ Map.keys(move_out) + Enum.min(candidates) + end defp payload_to_ops(payload) do Enum.map(Map.get(payload, :move_out, []), &{:move_out, &1}) ++ Enum.map(Map.get(payload, :move_in, []), &{:move_in, &1}) end - defp reduce(ops, base_view) do + defp reduce(ops, member?) do terminal_ops = ops |> Enum.with_index() @@ -115,7 +171,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do Map.put(acc, elem(move_value, 0), %{kind: kind, move_value: move_value, index: index}) end) |> Map.values() - |> Enum.reject(&redundant?(&1, base_view)) + |> Enum.reject(&redundant?(&1, member?)) |> Enum.sort_by(& &1.index) { @@ -124,12 +180,12 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do } end - defp redundant?(%{kind: :move_in, move_value: {value, _}}, base_view) do - MapSet.member?(base_view, value) + defp redundant?(%{kind: :move_in, move_value: {value, _}}, member?) do + member?.(value) end - defp redundant?(%{kind: :move_out, move_value: {value, _}}, base_view) do - not MapSet.member?(base_view, value) + defp redundant?(%{kind: :move_out, move_value: {value, _}}, member?) do + not member?.(value) end defp put_or_delete(map, key, [], _txids), do: Map.delete(map, key) diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex index 76d2b99c8e..1e59c52a18 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex @@ -8,6 +8,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do alias Electric.Shapes.Consumer.Subqueries.MoveBroadcast alias Electric.Shapes.Consumer.Subqueries.ShapeInfo alias Electric.Shapes.Consumer.TransactionConverter + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.WhereClause @enforce_keys [:effects] defstruct [:effects, :flushed_log_offset] @@ -17,17 +19,36 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do flushed_log_offset: LogOffset.t() | nil } - @spec build(ActiveMove.t(), ShapeInfo.t()) :: {:ok, t()} | {:error, term()} - def build(%ActiveMove{} = active_move, %ShapeInfo{} = shape_info) do + @spec build(ActiveMove.t(), ShapeInfo.t(), map()) :: {:ok, t()} | {:error, term()} + def build(%ActiveMove{} = active_move, %ShapeInfo{} = shape_info, subquery_refs) do {pre_txns, post_txns} = ActiveMove.split_buffer(active_move) + mtv = MultiTimeView.for_stack(shape_info.stack_id) - with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, active_move.views_before_move), - {:ok, post_ops} <- convert_txns(post_txns, shape_info, active_move.views_after_move) do + before_member? = + WhereClause.subquery_member_from_mtv( + mtv, + subquery_refs, + {active_move.subquery_ref, active_move.from_time} + ) + + after_member? = + WhereClause.subquery_member_from_mtv( + mtv, + subquery_refs, + {active_move.subquery_ref, active_move.to_time} + ) + + polarity = + Map.get(shape_info.dnf_plan.dependency_polarities, active_move.dep_index, :positive) + + with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, before_member?), + {:ok, post_ops} <- convert_txns(post_txns, shape_info, after_member?) do effects = EffectList.new() |> EffectList.append_all(pre_ops) - |> EffectList.append(MoveBroadcast.effect_for_move_in(active_move, shape_info)) - |> EffectList.append(move_in_snapshot_effect(active_move)) + |> maybe_append_move_out_broadcast(active_move, shape_info, polarity) + |> maybe_append_move_in_broadcast(active_move, shape_info, polarity) + |> maybe_append_move_in_snapshot(active_move) |> EffectList.append_all(post_ops) |> EffectList.to_list() @@ -39,23 +60,66 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do end end - defp convert_txns(txns, %ShapeInfo{} = shape_info, views) when is_map(views) do + defp convert_txns(txns, %ShapeInfo{} = shape_info, member?) when is_function(member?, 2) do TransactionConverter.transactions_to_effects( txns, shape_info.shape, stack_id: shape_info.stack_id, shape_handle: shape_info.shape_handle, - extra_refs: {views, views}, + extra_refs: {member?, member?}, dnf_plan: shape_info.dnf_plan ) end - defp move_in_snapshot_effect(%ActiveMove{} = active_move) do - %Effects.AppendMoveInSnapshot{ + # Outer-perspective move-out broadcast — values whose exit from the dep + # view (positive) or entry into it (negated) drops rows out of the outer + # shape. + defp maybe_append_move_out_broadcast(effects, %ActiveMove{} = active_move, shape_info, polarity) do + outer_move_out_values = + case polarity do + :positive -> active_move.move_out_values + :negated -> active_move.move_in_values + end + + case outer_move_out_values do + [] -> + effects + + values -> + EffectList.append( + effects, + MoveBroadcast.effect_for_move_out( + active_move.dep_index, + values, + active_move.txids, + shape_info + ) + ) + end + end + + defp maybe_append_move_in_broadcast(effects, %ActiveMove{} = active_move, shape_info, polarity) do + outer_move_in_values = + case polarity do + :positive -> active_move.move_in_values + :negated -> active_move.move_out_values + end + + if outer_move_in_values == [] do + effects + else + EffectList.append(effects, MoveBroadcast.effect_for_move_in(active_move, shape_info)) + end + end + + defp maybe_append_move_in_snapshot(effects, %ActiveMove{move_in_snapshot_name: nil}), do: effects + + defp maybe_append_move_in_snapshot(effects, %ActiveMove{} = active_move) do + EffectList.append(effects, %Effects.AppendMoveInSnapshot{ snapshot_name: active_move.move_in_snapshot_name, row_count: active_move.move_in_row_count, row_bytes: active_move.move_in_row_bytes, snapshot: active_move.snapshot - } + }) end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/views.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/views.ex deleted file mode 100644 index 5191d8ab7a..0000000000 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/views.ex +++ /dev/null @@ -1,26 +0,0 @@ -defmodule Electric.Shapes.Consumer.Subqueries.Views do - # Applies dependency move operations against the current subquery view map. - - @type ref() :: [String.t()] - @type t() :: %{ref() => MapSet.t()} - - @spec current(t(), ref()) :: MapSet.t() - def current(views, subquery_ref), do: Map.get(views, subquery_ref, MapSet.new()) - - @spec apply_move(t(), ref(), list(), :move_in | :move_out) :: t() - def apply_move(views, subquery_ref, values, :move_in) do - Map.update!(views, subquery_ref, fn view -> - Enum.reduce(values, view, fn {value, _original_value}, view -> - MapSet.put(view, value) - end) - end) - end - - def apply_move(views, subquery_ref, values, :move_out) do - Map.update!(views, subquery_ref, fn view -> - Enum.reduce(values, view, fn {value, _original_value}, view -> - MapSet.delete(view, value) - end) - end) - end -end diff --git a/packages/sync-service/lib/electric/shapes/consumer_registry.ex b/packages/sync-service/lib/electric/shapes/consumer_registry.ex index 935420b2d0..5a23cb4edc 100644 --- a/packages/sync-service/lib/electric/shapes/consumer_registry.ex +++ b/packages/sync-service/lib/electric/shapes/consumer_registry.ex @@ -296,7 +296,6 @@ defmodule Electric.Shapes.ConsumerRegistry do def new(stack_id, opts \\ []) when is_binary(stack_id) do table = registry_table(stack_id) - Electric.Shapes.Consumer.Materializer.init_link_values_table(stack_id) state = struct(__MODULE__, Keyword.merge(opts, stack_id: stack_id, table: table)) diff --git a/packages/sync-service/lib/electric/shapes/filter.ex b/packages/sync-service/lib/electric/shapes/filter.ex index 2117bef4ed..04e6ce4c89 100644 --- a/packages/sync-service/lib/electric/shapes/filter.ex +++ b/packages/sync-service/lib/electric/shapes/filter.ex @@ -84,8 +84,8 @@ defmodule Electric.Shapes.Filter do where_cond_id = get_or_create_table_condition(filter, shape.root_table) - WhereCondition.add_shape(filter, where_cond_id, shape_id, shape.where) maybe_register_subquery_shape(filter, shape_id, shape) + WhereCondition.add_shape(filter, where_cond_id, shape_id, shape.where) filter end @@ -96,7 +96,7 @@ defmodule Electric.Shapes.Filter do %Shape{shape_dependencies: [_ | _]} = shape ) do {:ok, plan} = DnfPlan.compile(shape) - SubqueryIndex.register_shape(index, shape_id, plan) + SubqueryIndex.register_shape(index, shape_id, plan, shape.shape_dependencies_handles) end defp maybe_register_subquery_shape(_filter, _shape_id, _shape), do: :ok diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index 47632a854d..823fcdde82 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -1,27 +1,17 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do - # Index for subquery routing and exact membership. - - # Each subquery predicate in the filter tree registers a node identified by - # `{condition_id, field_key}`. For each node, this table acts as a reverse - # index from the value seen on the root-table record to the shapes whose - # current subquery view makes that value relevant at that node. - - # Each shape consumer maintains its own entries in the index. On startup it - # seeds the node memberships for its current dependency views, then updates - # only those memberships as its subquery views change. This keeps the filter's - # materialized view of subquery membership aligned with the view that shape - # currently needs, without re-evaluating subqueries globally. - - # The same table also stores exact `shape_handle + subquery_ref + typed_value` - # membership rows used by `WhereClause.includes_record?/3` when the filter - # needs to verify subquery membership for a specific shape. - - # Shapes begin in a fallback set until their consumer has loaded and seeded - # that local state. Fallback routing is needed for restored or lazily started - # consumers: before their subquery view is available we still need to route - # root-table changes conservatively so the shape can be started and brought up - # to date. `mark_ready/2` removes the shape from fallback once its index - # entries reflect the consumer's current view. + # Shared subquery routing index. + # + # The hot rows describe the *topology* shared across all outer shapes that + # reference the same subquery: groups (per filter node + polarity), child + # nodes (per group + dependency subquery), value-keyed positive routing, + # group-keyed negated routing, and child participants. Per-shape value + # membership is *not* stored here, and the filter is intentionally + # time-unaware: it routes conservatively across all retained logical times + # and lets each consumer's `Shape.convert_change` do the exact check at its + # own per-subquery logical time(s). See `docs/rfcs/subquery-index.md` + # §Routing for why a per-shape logical-time pin in the filter cannot + # represent a consumer that is mid-move on a subquery (it reads at both + # `from_time` and `to_time` simultaneously). @moduledoc false import Electric, only: [is_stack_id: 1] @@ -30,479 +20,635 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do alias Electric.Replication.Eval.Runner alias Electric.Shapes.DnfPlan alias Electric.Shapes.Filter + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.WhereCondition - @type t :: :ets.tid() | atom() - @type node_id :: {reference(), term()} + defstruct [:table, :counters, :multi_time_view] + + @type t :: %SubqueryIndex{ + table: :ets.tid() | atom(), + counters: :ets.tid() | atom(), + multi_time_view: MultiTimeView.t() | nil + } defp table_name(stack_id) when is_stack_id(stack_id), do: :"subquery_index:#{stack_id}" - @doc """ - Create a new SubqueryIndex ETS table. + defp counters_table_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_index_counters:#{stack_id}" - The table is `:public` so consumer processes can seed and update membership - while the filter reads candidates during routing. - """ @spec new(keyword()) :: t() def new(opts \\ []) do - case Keyword.get(opts, :stack_id) do - nil -> - :ets.new(:subquery_index, [:bag, :public]) + {table, counters} = + case Keyword.get(opts, :stack_id) do + nil -> + { + :ets.new(:subquery_index, [:bag, :public]), + :ets.new(:subquery_index_counters, [:set, :public]) + } + + stack_id -> + { + try do + :ets.new(table_name(stack_id), [:bag, :public, :named_table]) + rescue + ArgumentError -> table_name(stack_id) + end, + try do + :ets.new(counters_table_name(stack_id), [:set, :public, :named_table]) + rescue + ArgumentError -> counters_table_name(stack_id) + end + } + end - stack_id -> - :ets.new(table_name(stack_id), [:bag, :public, :named_table]) - end + multi_time_view = MultiTimeView.new(Keyword.take(opts, [:stack_id])) + %SubqueryIndex{table: table, counters: counters, multi_time_view: multi_time_view} end - @doc """ - Look up the SubqueryIndex table for a stack. - """ @spec for_stack(String.t()) :: t() | nil def for_stack(stack_id) when is_stack_id(stack_id) do case :ets.whereis(table_name(stack_id)) do - :undefined -> nil - _tid -> table_name(stack_id) + :undefined -> + nil + + _tid -> + %SubqueryIndex{ + table: table_name(stack_id), + counters: counters_table_name(stack_id), + multi_time_view: MultiTimeView.for_stack(stack_id) + } end end @doc """ - Register per-shape exact membership metadata from a compiled DnfPlan. + Register per-shape metadata: per-dep-index dependency handle (a.k.a. + `subquery_id`) and the fallback flag. - Node-local routing metadata is registered by `add_shape/4` when the filter - adds the shape to a concrete subquery node. + `dep_handles` is the outer shape's `shape_dependencies_handles` list, + indexed by `dep_index`. """ - @spec register_shape(t(), term(), DnfPlan.t()) :: :ok - def register_shape(table, shape_handle, %DnfPlan{} = plan) do - polarities = - plan.positions - |> Enum.filter(fn {_pos, info} -> info.is_subquery end) - |> Map.new(fn {_pos, info} -> - {info.subquery_ref, if(info.negated, do: :negated, else: :positive)} - end) - - for {subquery_ref, polarity} <- polarities do - :ets.insert(table, {{:polarity, shape_handle, subquery_ref}, polarity}) + @spec register_shape(t(), term(), DnfPlan.t(), [term()]) :: :ok + def register_shape(%SubqueryIndex{table: table}, shape_handle, %DnfPlan{} = plan, dep_handles) do + for {dep_index, polarity} <- plan.dependency_polarities do + dep_handle = Enum.at(dep_handles, dep_index) || {shape_handle, dep_index} + :ets.insert(table, {{:dep_handle, shape_handle, dep_index}, {dep_handle, polarity}}) end :ets.insert(table, {{:fallback, shape_handle}, true}) - :ok end @doc """ - Remove all exact membership metadata for a shape. + Look up `{dep_handle, polarity}` for `{shape_handle, dep_index}`. + + Used by the filter's residual-`and_where` evaluator to ask + `MultiTimeView` a polarity-aware conservative membership question per + sublink. Raises `ArgumentError` if no dep was registered. """ + @spec lookup_dep!(t(), term(), non_neg_integer()) :: {term(), :positive | :negated} + def lookup_dep!(%SubqueryIndex{table: table}, shape_handle, dep_index) do + case :ets.lookup(table, {:dep_handle, shape_handle, dep_index}) do + [{_, {dep_handle, polarity}}] -> + {dep_handle, polarity} + + [] -> + raise ArgumentError, + "no dep_handle registered for shape #{inspect(shape_handle)} dep_index " <> + inspect(dep_index) + end + end + + @doc "Remove all metadata for `shape_handle`." @spec unregister_shape(t(), term()) :: :ok - def unregister_shape(table, shape_handle) do - :ets.match_delete(table, {{:membership, shape_handle, :_, :_}, true}) - :ets.match_delete(table, {{:polarity, shape_handle, :_}, :_}) - :ets.match_delete(table, {{:shape_node, shape_handle}, :_}) - :ets.match_delete(table, {{:shape_dep_node, shape_handle, :_}, :_}) + def unregister_shape(%SubqueryIndex{table: table}, shape_handle) do + :ets.match_delete(table, {{:dep_handle, shape_handle, :_}, :_}) :ets.delete(table, {:fallback, shape_handle}) :ok end @doc """ - Register a shape on a concrete subquery filter node. + Attach a shape to the indexed subquery node identified by + `{condition_id, optimisation.field, optimisation.polarity}`. Creates the + group and child node lazily. + + When the child is fresh, seeds positive (or negated) routing from the + shared `MultiTimeView` at the subquery's current logical time. If the + view is not yet ready for the subquery, the child is left unseeded — + shapes still attached to that child are routed conservatively via the + per-shape fallback rows until the seed happens (phase 2 of the RFC). """ @spec add_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :ok def add_shape( - %Filter{subquery_index: table} = filter, + %Filter{subquery_index: %SubqueryIndex{table: table, counters: counters} = index} = filter, condition_id, - shape_id, + shape_handle, optimisation, branch_key ) do - node_id = {condition_id, optimisation.field} - next_condition_id = make_ref() + ensure_node_meta(table, condition_id, optimisation.field, optimisation.testexpr) + + group_id = + ensure_group(table, counters, condition_id, optimisation.field, optimisation.polarity) + + subquery_id = lookup_dep_handle!(table, shape_handle, optimisation.dep_index) - WhereCondition.init(filter, next_condition_id) + {child_node_id, next_condition_id} = + ensure_child( + filter, + index, + group_id, + subquery_id, + optimisation.polarity, + condition_id, + optimisation.field + ) WhereCondition.add_shape( filter, next_condition_id, - shape_id, + shape_handle, optimisation.and_where, branch_key ) - ensure_node_meta(table, node_id, optimisation.testexpr) + :ets.insert(table, {{:child_shape, child_node_id}, {shape_handle, branch_key}}) + :ets.insert(table, {{:shape_child, shape_handle}, {child_node_id, branch_key}}) - :ets.insert( - table, - {{:node_shape, node_id}, - {shape_id, optimisation.dep_index, optimisation.polarity, next_condition_id, branch_key}} - ) - - if optimisation.polarity == :negated do - :ets.insert(table, {{:node_negated_shape, node_id}, {shape_id, next_condition_id}}) + if shape_in_fallback?(table, shape_handle) do + :ets.insert( + table, + {{:node_fallback, condition_id, optimisation.field}, {child_node_id, shape_handle}} + ) end - :ets.insert( - table, - {{:shape_node, shape_id}, - {node_id, optimisation.dep_index, optimisation.polarity, next_condition_id, branch_key}} - ) - - :ets.insert( - table, - {{:shape_dep_node, shape_id, optimisation.dep_index}, - {node_id, optimisation.polarity, next_condition_id, branch_key}} - ) - - :ets.insert(table, {{:node_fallback, node_id}, {shape_id, next_condition_id}}) :ok end @doc """ - Remove a shape from a concrete subquery filter node. + Detach a shape from an indexed subquery node. Returns `:deleted` when the + node has no remaining children (so the parent `WhereCondition` can drop + its index key), else `:ok`. """ @spec remove_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :deleted | :ok def remove_shape( - %Filter{subquery_index: table} = filter, + %Filter{ + subquery_index: %SubqueryIndex{table: table, counters: counters, multi_time_view: mtv} + } = filter, condition_id, - shape_id, + shape_handle, optimisation, branch_key ) do - node_id = {condition_id, optimisation.field} - - case node_shape_entry_for_shape(table, shape_id, node_id, branch_key) do + case lookup_child_for_shape( + table, + condition_id, + optimisation.field, + optimisation.polarity, + shape_handle, + branch_key + ) do nil -> - :deleted + node_status(counters, condition_id, optimisation.field) - {dep_index, polarity, next_condition_id} -> + {child_node_id, next_condition_id} -> _ = WhereCondition.remove_shape( filter, next_condition_id, - shape_id, + shape_handle, optimisation.and_where, branch_key ) - :ets.match_delete( - table, - {{:node_shape, node_id}, {shape_id, dep_index, polarity, next_condition_id, branch_key}} - ) - - if polarity == :negated do - :ets.match_delete( - table, - {{:node_negated_shape, node_id}, {shape_id, next_condition_id}} - ) - end - - :ets.match_delete( - table, - {{:shape_node, shape_id}, {node_id, dep_index, polarity, next_condition_id, branch_key}} - ) + :ets.delete_object(table, {{:child_shape, child_node_id}, {shape_handle, branch_key}}) + :ets.delete_object(table, {{:shape_child, shape_handle}, {child_node_id, branch_key}}) - :ets.match_delete( + :ets.delete_object( table, - {{:shape_dep_node, shape_id, dep_index}, - {node_id, polarity, next_condition_id, branch_key}} + {{:node_fallback, condition_id, optimisation.field}, {child_node_id, shape_handle}} ) - :ets.match_delete(table, {{:node_fallback, node_id}, {shape_id, next_condition_id}}) - delete_node_members(table, node_id, shape_id, polarity, next_condition_id) - - if node_empty?(table, node_id) do - :ets.delete(table, {:node_meta, node_id}) - :deleted - else - :ok + if child_empty?(table, child_node_id) do + delete_child(table, counters, mtv, child_node_id) end - end - end - @doc """ - Seed membership entries from a dependency view. - """ - @spec seed_membership(t(), term(), [String.t()], non_neg_integer(), MapSet.t()) :: :ok - def seed_membership(table, shape_handle, subquery_ref, dep_index, view) do - for value <- view do - add_value(table, shape_handle, subquery_ref, dep_index, value) + node_status(counters, condition_id, optimisation.field) end + end + @doc "Mark a shape as routable (clear fallback rows)." + @spec mark_ready(t(), term()) :: :ok + def mark_ready(%SubqueryIndex{table: table}, shape_handle) do + :ets.delete(table, {:fallback, shape_handle}) + :ets.match_delete(table, {{:node_fallback, :_, :_}, {:_, shape_handle}}) :ok end + @spec fallback?(t(), term()) :: boolean() + def fallback?(%SubqueryIndex{table: table}, shape_handle), + do: shape_in_fallback?(table, shape_handle) + + defp shape_in_fallback?(table, shape_handle), + do: :ets.member(table, {:fallback, shape_handle}) + + @doc "Whether `shape_handle` is attached to at least one indexed subquery node." + @spec has_positions?(t(), term()) :: boolean() + def has_positions?(%SubqueryIndex{table: table}, shape_handle) do + :ets.member(table, {:shape_child, shape_handle}) + end + @doc """ - Mark a shape as ready for indexed routing. + Add a positive route for `value` to every existing positive child of + `subquery_id`. Called by the materializer when a value enters the + retained window for that subquery. """ - @spec mark_ready(t(), term()) :: :ok - def mark_ready(table, shape_handle) do - :ets.delete(table, {:fallback, shape_handle}) - - for {node_id, _dep_index, _polarity, _next_condition_id, _branch_key} <- - nodes_for_shape(table, shape_handle) do - :ets.match_delete(table, {{:node_fallback, node_id}, {shape_handle, :_}}) + @spec add_positive_route(t(), term(), term()) :: :ok + def add_positive_route(%SubqueryIndex{table: table}, subquery_id, value) do + for child_node_id <- children_for_subquery(table, subquery_id) do + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{polarity: :positive, group_id: group_id}}] -> + :ets.insert(table, {{:positive, group_id, value}, child_node_id}) + + _ -> + :ok + end end :ok end @doc """ - Add a value to both the node-local routing index and the exact membership set. + Drop the positive route for `value` from every positive child of + `subquery_id`. Called by compaction when `value` is out for the whole + retained window. """ - @spec add_value(t(), term(), [String.t()], non_neg_integer(), term()) :: :ok - def add_value(table, shape_handle, subquery_ref, dep_index, value) do - for {node_id, polarity, next_condition_id, _branch_key} <- - nodes_for_shape_dependency(table, shape_handle, dep_index) do - case polarity do - :positive -> - :ets.insert( - table, - {{:node_positive_member, node_id, value}, {shape_handle, next_condition_id}} - ) - - :negated -> - :ets.insert( - table, - {{:node_negated_member, node_id, value}, {shape_handle, next_condition_id}} - ) + @spec remove_positive_route(t(), term(), term()) :: :ok + def remove_positive_route(%SubqueryIndex{table: table}, subquery_id, value) do + for child_node_id <- children_for_subquery(table, subquery_id) do + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{polarity: :positive, group_id: group_id}}] -> + :ets.delete_object(table, {{:positive, group_id, value}, child_node_id}) + + _ -> + :ok end end - :ets.insert(table, {{:membership, shape_handle, subquery_ref, value}, true}) :ok end @doc """ - Remove a value from both the node-local routing index and the exact membership set. + Cascade removal of `subquery_id`: drop every child node, participant + row, and routing row tied to that subquery. The bundled MultiTimeView is + also cleared so values for the subquery are gone everywhere. """ - @spec remove_value(t(), term(), [String.t()], non_neg_integer(), term()) :: :ok - def remove_value(table, shape_handle, subquery_ref, dep_index, value) do - for {node_id, polarity, next_condition_id, _branch_key} <- - nodes_for_shape_dependency(table, shape_handle, dep_index) do - case polarity do - :positive -> - :ets.match_delete( - table, - {{:node_positive_member, node_id, value}, {shape_handle, next_condition_id}} - ) - - :negated -> - :ets.match_delete( - table, - {{:node_negated_member, node_id, value}, {shape_handle, next_condition_id}} - ) - end + @spec remove_subquery(t(), term()) :: :ok + def remove_subquery( + %SubqueryIndex{table: table, counters: counters, multi_time_view: mtv}, + subquery_id + ) do + for child_node_id <- children_for_subquery(table, subquery_id) do + cleanup_child_shapes(table, child_node_id) + delete_child(table, counters, mtv, child_node_id) end - :ets.delete(table, {:membership, shape_handle, subquery_ref, value}) + if mtv, do: MultiTimeView.remove_subquery(mtv, subquery_id) :ok end @doc """ - Get affected shape handles for a specific subquery node. + Shape candidates for a record entering the node `{condition_id, + field_key}`. Combines value-keyed positive children with + conservatively-kept negated children and any fallback children, then + recurses through each child's `WhereCondition`. """ @spec affected_shapes(Filter.t(), reference(), term(), map()) :: MapSet.t() - def affected_shapes(%Filter{subquery_index: table} = filter, condition_id, field_key, record) do - node_id = {condition_id, field_key} - + def affected_shapes( + %Filter{subquery_index: %SubqueryIndex{table: table, multi_time_view: mtv}} = filter, + condition_id, + field_key, + record + ) do candidates = - case evaluate_node_lhs(table, node_id, record) do + case evaluate_node_lhs(table, condition_id, field_key, record) do {:ok, typed_value} -> - positive = - values_for_key(table, {:node_positive_member, node_id, typed_value}) |> MapSet.new() - - negated = - MapSet.difference( - values_for_key(table, {:node_negated_shape, node_id}) |> MapSet.new(), - values_for_key(table, {:node_negated_member, node_id, typed_value}) |> MapSet.new() - ) - - fallback = values_for_key(table, {:node_fallback, node_id}) |> MapSet.new() - - positive - |> MapSet.union(negated) - |> MapSet.union(fallback) + positive_children(table, condition_id, field_key, typed_value) + |> MapSet.union(negated_children(table, mtv, condition_id, field_key, typed_value)) + |> MapSet.union(fallback_children(table, condition_id, field_key)) :error -> - all_node_shapes(table, node_id) + all_children(table, condition_id, field_key) end - Enum.reduce(candidates, MapSet.new(), fn {_shape_id, next_condition_id}, acc -> - MapSet.union( - acc, - WhereCondition.affected_shapes(filter, next_condition_id, record) - ) + Enum.reduce(candidates, MapSet.new(), fn child_node_id, acc -> + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{next_condition_id: next_condition_id}}] -> + MapSet.union(acc, WhereCondition.affected_shapes(filter, next_condition_id, record)) + + [] -> + acc + end end) end - @doc """ - Get all shape ids registered on a specific subquery node. - """ @spec all_shape_ids(Filter.t(), reference(), term()) :: MapSet.t() - def all_shape_ids(%Filter{subquery_index: table} = filter, condition_id, field_key) do + def all_shape_ids( + %Filter{subquery_index: %SubqueryIndex{table: table}} = filter, + condition_id, + field_key + ) do table - |> all_node_shapes({condition_id, field_key}) - |> Enum.reduce(MapSet.new(), fn {_shape_id, next_condition_id}, acc -> - MapSet.union(acc, WhereCondition.all_shape_ids(filter, next_condition_id)) + |> all_children(condition_id, field_key) + |> Enum.reduce(MapSet.new(), fn child_node_id, acc -> + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{next_condition_id: next_condition_id}}] -> + MapSet.union(acc, WhereCondition.all_shape_ids(filter, next_condition_id)) + + [] -> + acc + end end) end - @doc """ - Check if a specific shape has a value in its current dependency view - for a canonical subquery ref. - """ - @spec member?(t(), term(), [String.t()], term()) :: boolean() - def member?(table, shape_handle, subquery_ref, typed_value) do - :ets.member(table, {:membership, shape_handle, subquery_ref, typed_value}) + defp ensure_node_meta(table, condition_id, field_key, testexpr) do + case :ets.lookup(table, {:node_testexpr, condition_id, field_key}) do + [] -> :ets.insert(table, {{:node_testexpr, condition_id, field_key}, testexpr}) + _ -> :ok + end end - @doc """ - Check subquery membership for exact evaluation, falling back to the shape's - dependency polarity while the shape is still unseeded. - """ - @spec membership_or_fallback?(t(), term(), [String.t()], term()) :: boolean() - def membership_or_fallback?(table, shape_handle, subquery_ref, typed_value) do - if shape_ready?(table, shape_handle) do - member?(table, shape_handle, subquery_ref, typed_value) - else - case polarity_for_shape_ref(table, shape_handle, subquery_ref) do - :positive -> true - :negated -> false - end + defp ensure_group(table, counters, condition_id, field_key, polarity) do + key = {:group, condition_id, field_key, polarity} + + case :ets.lookup(table, key) do + [{_, group_id}] -> + group_id + + [] -> + group_id = make_ref() + :ets.insert(table, {key, group_id}) + + :ets.update_counter( + counters, + {:node_groups, condition_id, field_key}, + 1, + {{:node_groups, condition_id, field_key}, 0} + ) + + group_id end end - @doc """ - Check if a shape is in the fallback set. - """ - @spec fallback?(t(), term()) :: boolean() - def fallback?(table, shape_handle) do - :ets.member(table, {:fallback, shape_handle}) - end + defp ensure_child(filter, index, group_id, subquery_id, polarity, condition_id, field_key) do + %SubqueryIndex{table: table, counters: counters, multi_time_view: mtv} = index - @doc """ - Check if a shape has any registered subquery nodes. - """ - @spec has_positions?(t(), term()) :: boolean() - def has_positions?(table, shape_handle) do - nodes_for_shape(table, shape_handle) != [] + case :ets.lookup(table, {:child, group_id, subquery_id}) do + [{_, child_node_id}] -> + [{_, meta}] = :ets.lookup(table, {:child_meta, child_node_id}) + {child_node_id, meta.next_condition_id} + + [] -> + child_node_id = make_ref() + next_condition_id = make_ref() + + WhereCondition.init(filter, next_condition_id) + + meta = %{ + group_id: group_id, + subquery_id: subquery_id, + polarity: polarity, + next_condition_id: next_condition_id, + field_key: field_key, + condition_id: condition_id + } + + :ets.insert(table, {{:child, group_id, subquery_id}, child_node_id}) + :ets.insert(table, {{:child_meta, child_node_id}, meta}) + :ets.insert(table, {{:subquery_child, subquery_id}, child_node_id}) + + :ets.update_counter( + counters, + {:group_children, group_id}, + 1, + {{:group_children, group_id}, 0} + ) + + seed_child_routing(table, mtv, child_node_id, meta) + {child_node_id, next_condition_id} + end end - @doc """ - Return the registered node ids for a shape. - """ - @spec positions_for_shape(t(), term()) :: [node_id()] - def positions_for_shape(table, shape_handle) do - table - |> nodes_for_shape(shape_handle) - |> Enum.map(fn {node_id, _dep_index, _polarity, _next_condition_id, _branch_key} -> - node_id - end) + defp children_for_subquery(table, subquery_id) do + for {_, cnid} <- :ets.lookup(table, {:subquery_child, subquery_id}), do: cnid end - defp ensure_node_meta(table, node_id, testexpr) do - case :ets.lookup(table, {:node_meta, node_id}) do - [] -> - :ets.insert(table, {{:node_meta, node_id}, %{testexpr: testexpr}}) + defp seed_child_routing(_table, nil, _child_node_id, _meta), do: :ok + + defp seed_child_routing(table, mtv, child_node_id, %{ + polarity: :positive, + group_id: group_id, + subquery_id: subquery_id + }) do + case MultiTimeView.current_time(mtv, subquery_id) do + nil -> + :ok + + time -> + for value <- MultiTimeView.values(mtv, subquery_id, time) do + :ets.insert(table, {{:positive, group_id, value}, child_node_id}) + end - _ -> :ok end end - defp delete_node_members(table, node_id, shape_id, polarity, next_condition_id) do - case polarity do - :positive -> - :ets.match_delete( - table, - {{:node_positive_member, node_id, :_}, {shape_id, next_condition_id}} - ) + defp seed_child_routing(table, _mtv, child_node_id, %{ + polarity: :negated, + group_id: group_id + }) do + :ets.insert(table, {{:negated, group_id}, child_node_id}) + :ok + end - :negated -> - :ets.match_delete( - table, - {{:node_negated_member, node_id, :_}, {shape_id, next_condition_id}} - ) + defp lookup_dep_handle!(table, shape_handle, dep_index) do + case :ets.lookup(table, {:dep_handle, shape_handle, dep_index}) do + [{_, {dep_handle, _polarity}}] -> + dep_handle + + [] -> + raise ArgumentError, + "no dep_handle registered for shape #{inspect(shape_handle)} dep_index " <> + inspect(dep_index) end end - defp nodes_for_shape(table, shape_handle) do - table - |> :ets.lookup({:shape_node, shape_handle}) - |> Enum.map(&elem(&1, 1)) + defp lookup_child_for_shape(table, condition_id, field_key, polarity, shape_handle, branch_key) do + with [{_, group_id}] <- :ets.lookup(table, {:group, condition_id, field_key, polarity}), + {cnid, next} when not is_nil(cnid) <- + Enum.find_value( + :ets.lookup(table, {:shape_child, shape_handle}), + {nil, nil}, + fn + {_, {cnid, ^branch_key}} -> + case :ets.lookup(table, {:child_meta, cnid}) do + [{_, %{group_id: ^group_id, next_condition_id: next}}] -> {cnid, next} + _ -> nil + end + + _ -> + nil + end + ) do + {cnid, next} + else + _ -> nil + end end - defp nodes_for_shape_dependency(table, shape_handle, dep_index) do - table - |> :ets.lookup({:shape_dep_node, shape_handle, dep_index}) - |> Enum.map(&elem(&1, 1)) + defp child_empty?(table, child_node_id) do + not :ets.member(table, {:child_shape, child_node_id}) end - defp node_shape_entry_for_shape(table, shape_id, node_id, branch_key) do - table - |> nodes_for_shape(shape_id) - |> Enum.find_value(fn - {^node_id, dep_index, polarity, next_condition_id, ^branch_key} -> - {dep_index, polarity, next_condition_id} + defp delete_child(table, counters, mtv, child_node_id) do + case :ets.lookup(table, {:child_meta, child_node_id}) do + [] -> + :ok - _ -> - nil - end) - end + [{_, meta}] -> + case meta.polarity do + :positive -> + if mtv != nil do + for value <- MultiTimeView.values(mtv, meta.subquery_id) do + :ets.delete_object(table, {{:positive, meta.group_id, value}, child_node_id}) + end + end - defp node_empty?(table, node_id) do - :ets.lookup(table, {:node_shape, node_id}) == [] - end + :negated -> + :ets.delete_object(table, {{:negated, meta.group_id}, child_node_id}) + end - defp all_node_shapes(table, node_id) do - table - |> :ets.lookup({:node_shape, node_id}) - |> Enum.reduce(MapSet.new(), fn - {{:node_shape, ^node_id}, {shape_id, _dep_index, _polarity, next_condition_id, _branch_key}}, - acc -> - MapSet.put(acc, {shape_id, next_condition_id}) - - _, acc -> - acc - end) - end + :ets.match_delete(table, {{:node_fallback, :_, :_}, {child_node_id, :_}}) + :ets.delete(table, {:child, meta.group_id, meta.subquery_id}) + :ets.delete_object(table, {{:subquery_child, meta.subquery_id}, child_node_id}) + :ets.delete(table, {:child_meta, child_node_id}) - defp evaluate_node_lhs(table, node_id, record) do - case :ets.lookup(table, {:node_meta, node_id}) do - [{_, %{testexpr: testexpr}}] -> - expr = Expr.wrap_parser_part(testexpr) + case :ets.update_counter(counters, {:group_children, meta.group_id}, -1) do + 0 -> + :ets.delete(counters, {:group_children, meta.group_id}) + + :ets.delete( + table, + {:group, meta.condition_id, meta.field_key, meta.polarity} + ) - case Runner.record_to_ref_values(expr.used_refs, record) do - {:ok, ref_values} -> - case Runner.execute(expr, ref_values) do - {:ok, value} -> {:ok, value} - _ -> :error + case :ets.update_counter( + counters, + {:node_groups, meta.condition_id, meta.field_key}, + -1 + ) do + 0 -> + :ets.delete(counters, {:node_groups, meta.condition_id, meta.field_key}) + :ets.delete(table, {:node_testexpr, meta.condition_id, meta.field_key}) + + _ -> + :ok end _ -> - :error + :ok end + :ok + end + end + + defp cleanup_child_shapes(table, child_node_id) do + for {_, {shape_handle, branch_key}} <- :ets.lookup(table, {:child_shape, child_node_id}) do + :ets.delete_object(table, {{:shape_child, shape_handle}, {child_node_id, branch_key}}) + :ets.delete_object(table, {{:child_shape, child_node_id}, {shape_handle, branch_key}}) + end + end + + defp node_empty?(counters, condition_id, field_key) do + case :ets.lookup(counters, {:node_groups, condition_id, field_key}) do + [{_, c}] when c > 0 -> false + _ -> true + end + end + + defp positive_children(table, condition_id, field_key, value) do + case :ets.lookup(table, {:group, condition_id, field_key, :positive}) do [] -> - :error + MapSet.new() + + [{_, group_id}] -> + for {_, cnid} <- :ets.lookup(table, {:positive, group_id, value}), + into: MapSet.new(), + do: cnid + end + end + + defp negated_children(table, mtv, condition_id, field_key, value) do + case :ets.lookup(table, {:group, condition_id, field_key, :negated}) do + [] -> + MapSet.new() + + [{_, group_id}] -> + for {_, cnid} <- :ets.lookup(table, {:negated, group_id}), + keep_negated_child?(table, mtv, cnid, value), + into: MapSet.new(), + do: cnid + end + end + + defp keep_negated_child?(_table, nil, _cnid, _value), do: true + + defp keep_negated_child?(table, mtv, cnid, value) do + case :ets.lookup(table, {:child_meta, cnid}) do + [{_, %{subquery_id: subquery_id}}] -> + not MultiTimeView.member_at_all_times?(mtv, subquery_id, value) + + [] -> + false end end - defp values_for_key(table, key) do + defp fallback_children(table, condition_id, field_key) do + for {_, {cnid, _shape}} <- :ets.lookup(table, {:node_fallback, condition_id, field_key}), + into: MapSet.new(), + do: cnid + end + + defp all_children(table, condition_id, field_key) do table - |> :ets.lookup(key) - |> Enum.map(&elem(&1, 1)) + |> :ets.match({{:group, condition_id, field_key, :"$1"}, :"$2"}) + |> Enum.flat_map(fn [_polarity, group_id] -> + table + |> :ets.match({{:child, group_id, :_}, :"$1"}) + |> Enum.map(fn [cnid] -> cnid end) + end) + |> MapSet.new() end - defp shape_ready?(table, shape_handle) do - not fallback?(table, shape_handle) + defp node_status(counters, condition_id, field_key) do + if node_empty?(counters, condition_id, field_key), do: :deleted, else: :ok end - defp polarity_for_shape_ref(table, shape_handle, subquery_ref) do - case :ets.lookup(table, {:polarity, shape_handle, subquery_ref}) do - [{_, polarity}] -> - polarity + defp evaluate_node_lhs(table, condition_id, field_key, record) do + case :ets.lookup(table, {:node_testexpr, condition_id, field_key}) do + [{_, testexpr}] -> + expr = Expr.wrap_parser_part(testexpr) + + with {:ok, ref_values} <- Runner.record_to_ref_values(expr.used_refs, record), + {:ok, value} <- Runner.execute(expr, ref_values) do + {:ok, value} + else + _ -> :error + end [] -> - raise ArgumentError, - "missing polarity for shape #{inspect(shape_handle)} and ref #{inspect(subquery_ref)}" + :error end end end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/compactor.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/compactor.ex new file mode 100644 index 0000000000..2194a59b24 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/compactor.ex @@ -0,0 +1,90 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.Compactor do + @moduledoc """ + Periodic compactor for `MultiTimeView` retained histories. + + Every `interval_ms` the compactor walks every subquery known to the stack's + `MultiTimeView` and advances its `min_required_time` to the minimum required + by any registered consumer (read from `SubqueryProgressMonitor`). Histories + that compact to empty (values no longer a member at any retained time) have + their positive-routing rows removed from `SubqueryIndex` as well, so the + routing path doesn't grow without bound. + + See RFC §*Compaction*. + """ + + use GenServer + + require Logger + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor + + import Electric, only: [is_stack_id: 1] + + @default_interval_ms 10_000 + + def name(stack_id) when is_stack_id(stack_id) do + Electric.ProcessRegistry.name(stack_id, __MODULE__) + end + + def start_link(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + GenServer.start_link(__MODULE__, opts, name: name(stack_id)) + end + + @doc """ + Run one compaction pass synchronously. Intended for tests; production + compaction runs from the periodic tick. + """ + def compact_now(stack_id) when is_stack_id(stack_id) do + GenServer.call(name(stack_id), :compact_now) + end + + @impl true + def init(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + interval_ms = Keyword.get(opts, :interval_ms, @default_interval_ms) + Process.set_label({:subquery_compactor, stack_id}) + schedule_tick(interval_ms) + + {:ok, %{stack_id: stack_id, interval_ms: interval_ms}} + end + + @impl true + def handle_call(:compact_now, _from, state) do + run_compaction(state.stack_id) + {:reply, :ok, state} + end + + @impl true + def handle_info(:tick, state) do + run_compaction(state.stack_id) + schedule_tick(state.interval_ms) + {:noreply, state} + end + + defp schedule_tick(interval_ms), do: Process.send_after(self(), :tick, interval_ms) + + defp run_compaction(stack_id) do + with mtv when not is_nil(mtv) <- MultiTimeView.for_stack(stack_id) do + index = SubqueryIndex.for_stack(stack_id) + + for subquery_id <- MultiTimeView.subquery_ids(mtv), + min_time = ProgressMonitor.min_required_time(stack_id, subquery_id), + is_integer(min_time) do + compact_subquery(mtv, index, subquery_id, min_time) + end + end + end + + defp compact_subquery(mtv, index, subquery_id, min_time) do + removed = MultiTimeView.set_min_required_time(mtv, subquery_id, min_time) + + if index do + for value <- removed do + SubqueryIndex.remove_positive_route(index, subquery_id, value) + end + end + end +end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex new file mode 100644 index 0000000000..b270bfd554 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex @@ -0,0 +1,117 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.History do + @moduledoc """ + Purely functional representation of a single value's membership history in a + subquery's shared multi-time view. + + A history records the times at which a value's membership of a subquery + toggled. It always has one of these shapes: + + [] # in at every retained logical time + [:in | :out, t1, t2, ...] # initial state, then toggle times + + The first element is the membership at the start of the retained window. Each + subsequent integer is a logical time at which membership flipped; toggle + times are strictly increasing. + + ## Examples + + [] # member at all retained times + [:out, 9] # out before 9, in from 9 onwards + [:out, 9, 11] # out before 9, in from 9..10, out from 11 onwards + [:in, 9] # in before 9, out from 9 onwards + [:in, 9, 11] # in before 9, out from 9..10, in from 11 onwards + + ## Absence + + `nil` represents a value that is not a member at any retained logical time — + no row exists for it in the shared view. This module treats `nil` as a + first-class history for all queries. Constructors and `compact/2` return + `nil` when the value is entirely out for the retained window. + """ + + @type time :: non_neg_integer() + @type t :: [] | nonempty_list(:in | :out | time()) + @type history :: t() | nil + + @doc "A history for a value that is a member at every retained logical time." + @spec new() :: t() + def new(), do: [] + + @doc """ + Is the value a member at `time`? + """ + @spec member?(history(), time()) :: boolean() + def member?(nil, _time), do: false + def member?([], _time), do: true + + def member?([initial, t | _], time) when time < t, do: initial == :in + def member?([initial, _t | rest], time), do: member?([flip(initial) | rest], time) + def member?([initial], _time), do: initial == :in + + @doc "Is the value a member at any retained logical time?" + @spec member_at_some_time?(history()) :: boolean() + def member_at_some_time?(nil), do: false + def member_at_some_time?(_history), do: true + + @doc "Is the value a member at every retained logical time?" + @spec member_at_all_times?(history()) :: boolean() + def member_at_all_times?([]), do: true + def member_at_all_times?(_history), do: false + + @doc """ + Record that the value becomes a member from `time` onwards. + + A no-op when the latest tracked state is already `:in`. `time` must be + strictly greater than any previously recorded toggle. + """ + @spec mark_in(history(), time()) :: t() + def mark_in(history, time) do + if member?(history, time) do + history + else + append_time(history, time) + end + end + + @doc """ + Record that the value stops being a member from `time` onwards. + + A no-op when the latest tracked state is already `:out`. `time` must be + strictly greater than any previously recorded toggle. + """ + @spec mark_out(history(), time()) :: history() + def mark_out(history, time) do + if member?(history, time) do + append_time(history, time) + else + history + end + end + + @doc """ + Drop toggles at or before `min_required_time`, folding their effect into the + initial state. + + Returns `nil` if, after compaction, the value is out for the entire retained + window — the row can be deleted from the shared view. + """ + @spec compact(history(), time()) :: history() + def compact(nil, _min_required_time), do: nil + def compact([], _min_required_time), do: [] + + def compact([_initial, t | _] = history, min_required_time) when t > min_required_time, + do: history + + def compact([initial, _t | rest], min_required_time), + do: compact([flip(initial) | rest], min_required_time) + + def compact([:in], _min_required_time), do: [] + def compact([:out], _min_required_time), do: nil + + defp flip(:in), do: :out + defp flip(:out), do: :in + + defp append_time(nil, time), do: [:out, time] + defp append_time([], time), do: [:in, time] + defp append_time(history, time), do: history ++ [time] +end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex new file mode 100644 index 0000000000..305b0958d1 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex @@ -0,0 +1,237 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView do + @moduledoc """ + Shared, logical-time view of subquery membership. + + Stores one membership history per `{subquery_id, value}` pair, plus + per-subquery metadata (current logical time, min required time, ready flag), + all in a single ETS table per stack. Multiple consumer processes can read + the same view at different logical times without copying it into their own + state. + + See `docs/rfcs/subquery-index.md` for the broader design. + """ + + import Electric, only: [is_stack_id: 1] + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.History + + @type t :: :ets.tid() | atom() + @type subquery_id :: term() + @type value :: term() + @type time :: non_neg_integer() + + defp table_name(stack_id) when is_stack_id(stack_id), + do: :"multi_time_view:#{stack_id}" + + @doc """ + Create a new MultiTimeView ETS table. + + The table is `:public` so the materializer can write transitions while + consumer processes read membership. + """ + @spec new(keyword()) :: t() + def new(opts \\ []) do + case Keyword.get(opts, :stack_id) do + nil -> + :ets.new(:multi_time_view, [:set, :public]) + + stack_id -> + try do + :ets.new(table_name(stack_id), [:set, :public, :named_table]) + rescue + ArgumentError -> table_name(stack_id) + end + end + end + + @doc "Look up the MultiTimeView table for a stack, or `nil` if none exists." + @spec for_stack(String.t()) :: t() | nil + def for_stack(stack_id) when is_stack_id(stack_id) do + case :ets.whereis(table_name(stack_id)) do + :undefined -> nil + _tid -> table_name(stack_id) + end + end + + @doc """ + Initialise a subquery at logical time `0` with the given initial member + values. The subquery is left not-ready; call `mark_ready/2` once initial + population is finished. + """ + @spec init_subquery(t(), subquery_id(), Enumerable.t()) :: :ok + def init_subquery(view, subquery_id, initial_values) do + :ets.insert(view, {{:current_time, subquery_id}, 0}) + :ets.insert(view, {{:min_required_time, subquery_id}, 0}) + + for value <- initial_values do + :ets.insert(view, {{:value, subquery_id, value}, History.new()}) + end + + :ok + end + + @doc "Mark a subquery as ready for consumers to read." + @spec mark_ready(t(), subquery_id()) :: :ok + def mark_ready(view, subquery_id) do + :ets.insert(view, {{:ready, subquery_id}, true}) + :ok + end + + @doc "Is the subquery ready for consumers to read?" + @spec ready?(t(), subquery_id()) :: boolean() + def ready?(view, subquery_id) do + :ets.member(view, {:ready, subquery_id}) + end + + @doc """ + Record that `value` becomes a member of `subquery_id` from `time` onwards. + Advances the subquery's current logical time to `time`. + """ + @spec mark_in(t(), subquery_id(), value(), time()) :: :ok + def mark_in(view, subquery_id, value, time) do + update_history(view, subquery_id, value, &History.mark_in(&1, time)) + advance_current_time(view, subquery_id, time) + :ok + end + + @doc """ + Record that `value` stops being a member of `subquery_id` from `time` + onwards. Advances the subquery's current logical time to `time`. + """ + @spec mark_out(t(), subquery_id(), value(), time()) :: :ok + def mark_out(view, subquery_id, value, time) do + update_history(view, subquery_id, value, &History.mark_out(&1, time)) + advance_current_time(view, subquery_id, time) + :ok + end + + @doc "Is `value` a member of `subquery_id` at logical `time`?" + @spec member?(t(), subquery_id(), value(), time()) :: boolean() + def member?(view, subquery_id, value, time) do + view |> lookup_history(subquery_id, value) |> History.member?(time) + end + + @doc "Is `value` a member of `subquery_id` at any retained logical time?" + @spec member_at_some_time?(t(), subquery_id(), value()) :: boolean() + def member_at_some_time?(view, subquery_id, value) do + view |> lookup_history(subquery_id, value) |> History.member_at_some_time?() + end + + @doc "Is `value` a member of `subquery_id` at every retained logical time?" + @spec member_at_all_times?(t(), subquery_id(), value()) :: boolean() + def member_at_all_times?(view, subquery_id, value) do + view |> lookup_history(subquery_id, value) |> History.member_at_all_times?() + end + + @doc "All values retained for `subquery_id` (members at some retained time)." + @spec values(t(), subquery_id()) :: [value()] + def values(view, subquery_id) do + view + |> :ets.match({{:value, subquery_id, :"$1"}, :_}) + |> Enum.map(fn [value] -> value end) + end + + @doc "All values that are members of `subquery_id` at logical `time`." + @spec values(t(), subquery_id(), time()) :: [value()] + def values(view, subquery_id, time) do + view + |> :ets.match({{:value, subquery_id, :"$1"}, :"$2"}) + |> Enum.flat_map(fn [value, history] -> + if History.member?(history, time), do: [value], else: [] + end) + end + + @doc "Current logical time for `subquery_id`, or `nil` if unknown." + @spec current_time(t(), subquery_id()) :: time() | nil + def current_time(view, subquery_id) do + case :ets.lookup(view, {:current_time, subquery_id}) do + [{_, time}] -> time + [] -> nil + end + end + + @doc """ + Advance the minimum required logical time for `subquery_id` and compact all + retained histories. Returns the list of values whose history compacted to + empty (and were therefore deleted) — useful for cascading routing cleanup. + """ + @spec set_min_required_time(t(), subquery_id(), time()) :: [value()] + def set_min_required_time(view, subquery_id, time) do + :ets.insert(view, {{:min_required_time, subquery_id}, time}) + + view + |> :ets.match({{:value, subquery_id, :"$1"}, :"$2"}) + |> Enum.flat_map(fn [value, history] -> + case compact_history(view, subquery_id, value, history, time) do + :deleted -> [value] + :ok -> [] + end + end) + end + + @doc """ + All `subquery_id`s currently tracked by this view (every subquery that + has been initialised and not yet `remove_subquery`'d). + """ + @spec subquery_ids(t()) :: [subquery_id()] + def subquery_ids(view) do + view + |> :ets.match({{:current_time, :"$1"}, :_}) + |> Enum.map(fn [id] -> id end) + end + + @doc "Delete every row for `subquery_id`." + @spec remove_subquery(t(), subquery_id()) :: :ok + def remove_subquery(view, subquery_id) do + :ets.match_delete(view, {{:value, subquery_id, :_}, :_}) + :ets.delete(view, {:current_time, subquery_id}) + :ets.delete(view, {:min_required_time, subquery_id}) + :ets.delete(view, {:ready, subquery_id}) + :ok + end + + defp lookup_history(view, subquery_id, value) do + case :ets.lookup(view, {:value, subquery_id, value}) do + [{_, history}] -> history + [] -> nil + end + end + + defp update_history(view, subquery_id, value, fun) do + history = lookup_history(view, subquery_id, value) + + case fun.(history) do + ^history -> :ok + nil -> :ets.delete(view, {:value, subquery_id, value}) + new -> :ets.insert(view, {{:value, subquery_id, value}, new}) + end + end + + defp advance_current_time(view, subquery_id, time) do + case current_time(view, subquery_id) do + nil -> + :ets.insert(view, {{:current_time, subquery_id}, time}) + + current when time > current -> + :ets.insert(view, {{:current_time, subquery_id}, time}) + + _ -> + :ok + end + end + + defp compact_history(view, subquery_id, value, history, min_required_time) do + case History.compact(history, min_required_time) do + ^history -> + :ok + + nil -> + :ets.delete(view, {:value, subquery_id, value}) + :deleted + + new -> + :ets.insert(view, {{:value, subquery_id, value}, new}) + :ok + end + end +end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex new file mode 100644 index 0000000000..cffefe06a1 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex @@ -0,0 +1,236 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do + @moduledoc """ + Tracks, per subquery, the earliest logical time any registered consumer may + still need to read. The minimum across live consumers is the compaction + lower bound for `MultiTimeView`. + + One GenServer + two ETS tables per stack: + + - A `:set` "positions" table holding one row per registered consumer + (`{:consumer, subquery_id, shape_handle} -> {time, monitor_ref}`) plus a + denormalised `{:min_required_time, subquery_id} -> time` row for + lock-free external reads from the routing/compactor path. + - An `:ordered_set` "times" table keyed by + `{subquery_id, time, shape_handle}`. The min for a subquery is the + first entry whose key starts with that `subquery_id`, found in + `O(log N)` via `:ets.next/2` against a sentinel key. + + Together they keep register/notify/unregister/DOWN at `O(log N)` instead + of `O(total_consumers)`. + + `min_required_time/2` and `registered?/3` read the positions table + directly without touching the GenServer. + + See `docs/rfcs/subquery-index.md`, section *Processed-Up-To Time*. + """ + + use GenServer, restart: :temporary + + import Electric, only: [is_stack_id: 1] + + @type subquery_id :: term() + @type shape_handle :: term() + @type time :: non_neg_integer() + @type stack_id :: String.t() + + defp registered_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_progress_monitor:#{stack_id}" + + defp table_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_progress_monitor_table:#{stack_id}" + + defp times_table_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_progress_monitor_times:#{stack_id}" + + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + GenServer.start_link(__MODULE__, opts, name: registered_name(stack_id)) + end + + @doc "Look up the ETS table for a stack, or `nil` if none exists." + @spec for_stack(stack_id()) :: atom() | nil + def for_stack(stack_id) when is_stack_id(stack_id) do + case :ets.whereis(table_name(stack_id)) do + :undefined -> nil + _tid -> table_name(stack_id) + end + end + + @doc """ + Register `pid` as a consumer of `subquery_id` for `shape_handle`. The + consumer's initial required time is `time` — the materializer's current + logical time when registration succeeds. The consumer process is + monitored; if it dies the pinned time is released automatically. + + Re-registering an existing `{subquery_id, shape_handle}` replaces the + previous registration. + """ + @spec register_consumer(stack_id(), subquery_id(), shape_handle(), pid(), time()) :: :ok + def register_consumer(stack_id, subquery_id, shape_handle, pid, time) do + GenServer.call( + registered_name(stack_id), + {:register, subquery_id, shape_handle, pid, time} + ) + end + + @doc """ + Remove the registration for `{subquery_id, shape_handle}`. Idempotent. + """ + @spec unregister_consumer(stack_id(), subquery_id(), shape_handle()) :: :ok + def unregister_consumer(stack_id, subquery_id, shape_handle) do + GenServer.call( + registered_name(stack_id), + {:unregister, subquery_id, shape_handle} + ) + end + + @doc """ + Advance the consumer's required time past `time`. After this call the + consumer asserts it no longer needs to read `subquery_id` at any time + `<= time`. + """ + @spec notify_processed_up_to(stack_id(), time(), subquery_id(), shape_handle()) :: :ok + def notify_processed_up_to(stack_id, time, subquery_id, shape_handle) do + GenServer.call( + registered_name(stack_id), + {:notify, time, subquery_id, shape_handle} + ) + end + + @doc """ + Earliest logical time any live consumer may still need to read for + `subquery_id`. `nil` when no consumer is registered — callers may + compact freely. + """ + @spec min_required_time(stack_id() | atom(), subquery_id()) :: time() | nil + def min_required_time(stack_id, subquery_id) when is_stack_id(stack_id), + do: min_required_time(table_name(stack_id), subquery_id) + + def min_required_time(table, subquery_id) when is_atom(table) do + case :ets.lookup(table, {:min_required_time, subquery_id}) do + [{_, time}] -> time + [] -> nil + end + end + + @spec registered?(stack_id() | atom(), subquery_id(), shape_handle()) :: boolean() + def registered?(stack_id, subquery_id, shape_handle) when is_stack_id(stack_id), + do: registered?(table_name(stack_id), subquery_id, shape_handle) + + def registered?(table, subquery_id, shape_handle) when is_atom(table) do + :ets.member(table, {:consumer, subquery_id, shape_handle}) + end + + @impl true + def init(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + + positions = + :ets.new(table_name(stack_id), [ + :set, + :public, + :named_table, + read_concurrency: true + ]) + + times = + :ets.new(times_table_name(stack_id), [ + :ordered_set, + :public, + :named_table + ]) + + {:ok, %{stack_id: stack_id, positions: positions, times: times, monitors: %{}}} + end + + @impl true + def handle_call({:register, subquery_id, shape_handle, pid, time}, _from, state) do + state = remove_registration(state, subquery_id, shape_handle) + + monitor_ref = Process.monitor(pid) + + :ets.insert( + state.positions, + {{:consumer, subquery_id, shape_handle}, time, monitor_ref} + ) + + :ets.insert(state.times, {{subquery_id, time, shape_handle}}) + + state = put_in(state.monitors[monitor_ref], {subquery_id, shape_handle}) + update_min(state.positions, state.times, subquery_id) + {:reply, :ok, state} + end + + def handle_call({:unregister, subquery_id, shape_handle}, _from, state) do + state = remove_registration(state, subquery_id, shape_handle) + update_min(state.positions, state.times, subquery_id) + {:reply, :ok, state} + end + + def handle_call({:notify, time, subquery_id, shape_handle}, _from, state) do + case :ets.lookup(state.positions, {:consumer, subquery_id, shape_handle}) do + [{key, current, monitor_ref}] -> + new_required = max(current, time + 1) + + if new_required != current do + :ets.delete(state.times, {subquery_id, current, shape_handle}) + :ets.insert(state.times, {{subquery_id, new_required, shape_handle}}) + :ets.insert(state.positions, {key, new_required, monitor_ref}) + update_min(state.positions, state.times, subquery_id) + end + + {:reply, :ok, state} + + [] -> + {:reply, :ok, state} + end + end + + @impl true + def handle_info({:DOWN, monitor_ref, :process, _pid, _reason}, state) do + case Map.pop(state.monitors, monitor_ref) do + {nil, _} -> + {:noreply, state} + + {{subquery_id, shape_handle}, monitors} -> + case :ets.lookup(state.positions, {:consumer, subquery_id, shape_handle}) do + [{_key, time, _ref}] -> + :ets.delete(state.positions, {:consumer, subquery_id, shape_handle}) + :ets.delete(state.times, {subquery_id, time, shape_handle}) + + [] -> + :ok + end + + update_min(state.positions, state.times, subquery_id) + {:noreply, %{state | monitors: monitors}} + end + end + + defp remove_registration(state, subquery_id, shape_handle) do + case :ets.lookup(state.positions, {:consumer, subquery_id, shape_handle}) do + [{key, time, monitor_ref}] -> + Process.demonitor(monitor_ref, [:flush]) + :ets.delete(state.positions, key) + :ets.delete(state.times, {subquery_id, time, shape_handle}) + %{state | monitors: Map.delete(state.monitors, monitor_ref)} + + [] -> + state + end + end + + # The min for `subquery_id` is the smallest entry in the ordered_set + # whose key starts with `{subquery_id, ...}`. `:ets.next/2` against a + # sentinel below any real time gives that entry in O(log N). + defp update_min(positions, times, subquery_id) do + case :ets.next(times, {subquery_id, -1, nil}) do + {^subquery_id, time, _shape_handle} -> + :ets.insert(positions, {{:min_required_time, subquery_id}, time}) + + _ -> + :ets.delete(positions, {:min_required_time, subquery_id}) + end + end +end diff --git a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex index 5e34b4831f..57d0203c7c 100644 --- a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex +++ b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex @@ -350,7 +350,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do [shape_count: map_size(other_shapes)], fn -> for {{shape_id, _branch_key}, where} <- other_shapes, - other_shape_matches?(index, shape_id, where, record), + other_shape_matches?(where, record, index, shape_id), into: MapSet.new() do shape_id end @@ -358,11 +358,19 @@ defmodule Electric.Shapes.Filter.WhereCondition do ) end - defp other_shape_matches?(index, shape_id, where, record) do + # Residual `and_where` evaluation. Non-subquery conjuncts evaluate + # precisely; sublinks consult `MultiTimeView` via + # `WhereClause.subquery_member_conservative_from_index/2` so that values + # provably outside (positive) or always inside (negated) the dep view are + # excluded here instead of over-routing. Anything ambiguous still routes + # through and is refined by the consumer's exact check in + # `Shape.convert_change`. If the runner can't compute a result we treat + # it as "include" — same fallback as before this MTV-based path landed. + defp other_shape_matches?(where, record, index, shape_id) do case WhereClause.includes_record_result( where, record, - WhereClause.subquery_member_from_index(index, shape_id) + WhereClause.subquery_member_conservative_from_index(index, shape_id) ) do {:ok, included?} -> included? :error -> true diff --git a/packages/sync-service/lib/electric/shapes/querying.ex b/packages/sync-service/lib/electric/shapes/querying.ex index 460c1cae26..2bb63380a0 100644 --- a/packages/sync-service/lib/electric/shapes/querying.ex +++ b/packages/sync-service/lib/electric/shapes/querying.ex @@ -307,15 +307,15 @@ defmodule Electric.Shapes.Querying do tags_sqls = tags_sql(plan, stack_id, shape_handle) {active_conditions_sqls, params} = - case Keyword.get(opts, :views) do + case Keyword.get(opts, :values_for_ref) do nil -> {active_conditions_sql(plan), []} - views -> + resolver when is_function(resolver, 1) -> {sqls, params, _next_idx} = - active_conditions_sql_for_views( + active_conditions_sql_for_resolver( plan, - views, + resolver, shape.where.used_refs, Keyword.get(opts, :start_param_idx, 1) ) @@ -349,37 +349,41 @@ defmodule Electric.Shapes.Querying do end end - def move_in_where_clause(plan, dep_index, views_before_move, views_after_move, used_refs) do + @typedoc """ + Resolver function that returns the subquery membership for a `subquery_ref` + at a given logical point (`:before` or `:after` the move). Replaces the + pre-RFC `%{subquery_ref => MapSet}` view maps; the closure can read from a + shared `MultiTimeView` at query build time instead of consumers carrying + long-lived MapSet copies. + + The returned value may be a `MapSet`, list, or any enumerable; consumers in + this module materialise it as a list at the SQL boundary. + """ + @type values_for() :: ([String.t()], :before | :after -> Enumerable.t()) + + @spec move_in_where_clause( + plan :: term(), + dep_index :: non_neg_integer(), + values_for(), + used_refs :: map() + ) :: {String.t(), [term()]} + def move_in_where_clause(plan, dep_index, values_for, used_refs) + when is_function(values_for, 2) do impacted = Map.get(plan.dependency_disjuncts, dep_index, []) all_idxs = Enum.to_list(0..(length(plan.disjuncts) - 1)) unaffected = all_idxs -- impacted + after_resolver = fn ref -> values_for.(ref, :after) end + before_resolver = fn ref -> values_for.(ref, :before) end + {candidate_sql, candidate_params, next_param} = - build_disjuncts_sql( - plan, - impacted, - views_after_move, - used_refs, - 1 - ) + build_disjuncts_sql(plan, impacted, after_resolver, used_refs, 1) {impacted_before_sql, impacted_before_params, next_param} = - build_disjuncts_sql( - plan, - impacted, - views_before_move, - used_refs, - next_param - ) + build_disjuncts_sql(plan, impacted, before_resolver, used_refs, next_param) {unaffected_sql, unaffected_params, _} = - build_disjuncts_sql( - plan, - unaffected, - views_before_move, - used_refs, - next_param - ) + build_disjuncts_sql(plan, unaffected, before_resolver, used_refs, next_param) where = case join_sql(" OR ", [impacted_before_sql, unaffected_sql]) do @@ -403,7 +407,8 @@ defmodule Electric.Shapes.Querying do end) end - def active_conditions_sql_for_views(plan, views, used_refs, start_param_idx \\ 1) do + def active_conditions_sql_for_resolver(plan, resolver, used_refs, start_param_idx \\ 1) + when is_function(resolver, 1) do {sqls, params, next_param_idx} = Enum.reduce(0..(plan.position_count - 1), {[], [], start_param_idx}, fn pos, {sqls, params, @@ -411,7 +416,7 @@ defmodule Electric.Shapes.Querying do info = Map.fetch!(plan.positions, pos) {base_sql, sql_params, next_param_idx} = - position_to_sql(info, views, used_refs, param_idx) + position_to_sql(info, resolver, used_refs, param_idx) sql = if info.negated do @@ -497,13 +502,14 @@ defmodule Electric.Shapes.Querying do defp position_to_sql( %{is_subquery: true} = info, - views, + resolver, used_refs, pidx - ) do + ) + when is_function(resolver, 1) do lhs_sql = lhs_sql_from_ast(info.ast) ref_type = Map.get(used_refs, info.subquery_ref) - values = Map.get(views, info.subquery_ref, MapSet.new()) |> MapSet.to_list() + values = info.subquery_ref |> resolver.() |> Enum.to_list() case ref_type do {:array, {:row, col_types}} -> diff --git a/packages/sync-service/lib/electric/shapes/shape.ex b/packages/sync-service/lib/electric/shapes/shape.ex index f146a9b038..7f1ab5aee3 100644 --- a/packages/sync-service/lib/electric/shapes/shape.ex +++ b/packages/sync-service/lib/electric/shapes/shape.ex @@ -651,9 +651,9 @@ defmodule Electric.Shapes.Shape do %Changes.NewRecord{record: record} = change, opts ) do - {_old_refs, new_refs} = opts[:extra_refs] || {%{}, %{}} + {_old_member?, new_member?} = normalise_extra_refs(opts[:extra_refs]) - case project_row_metadata(shape, record, new_refs, opts) do + case project_row_metadata(shape, record, new_member?, opts) do {:ok, true, metadata} -> [change |> put_row_metadata(metadata) |> filter_change_columns(selected_columns)] @@ -667,9 +667,9 @@ defmodule Electric.Shapes.Shape do %Changes.DeletedRecord{old_record: record} = change, opts ) do - {old_refs, _new_refs} = opts[:extra_refs] || {%{}, %{}} + {old_member?, _new_member?} = normalise_extra_refs(opts[:extra_refs]) - case project_row_metadata(shape, record, old_refs, opts) do + case project_row_metadata(shape, record, old_member?, opts) do {:ok, true, metadata} -> [change |> put_row_metadata(metadata) |> filter_change_columns(selected_columns)] @@ -683,10 +683,12 @@ defmodule Electric.Shapes.Shape do %Changes.UpdatedRecord{old_record: old_record, record: record} = change, opts ) do - {old_refs, new_refs} = opts[:extra_refs] || {%{}, %{}} + {old_member?, new_member?} = normalise_extra_refs(opts[:extra_refs]) - {:ok, old_included?, old_metadata} = project_row_metadata(shape, old_record, old_refs, opts) - {:ok, new_included?, new_metadata} = project_row_metadata(shape, record, new_refs, opts) + {:ok, old_included?, old_metadata} = + project_row_metadata(shape, old_record, old_member?, opts) + + {:ok, new_included?, new_metadata} = project_row_metadata(shape, record, new_member?, opts) converted_changes = case {old_included?, new_included?} do @@ -721,10 +723,11 @@ defmodule Electric.Shapes.Shape do defp project_row_metadata( %__MODULE__{where: where}, record, - refs, + subquery_member?, %{dnf_plan: %DnfPlan{} = dnf_plan, stack_id: stack_id, shape_handle: shape_handle} - ) do - case get_row_metadata(dnf_plan, record, refs, where, stack_id, shape_handle) do + ) + when is_function(subquery_member?, 2) do + case get_row_metadata(dnf_plan, record, subquery_member?, where, stack_id, shape_handle) do {:ok, included?, move_tags, active_conditions} -> {:ok, included?, %{move_tags: move_tags, active_conditions: active_conditions}} end @@ -733,11 +736,11 @@ defmodule Electric.Shapes.Shape do defp project_row_metadata( %__MODULE__{where: where, tag_structure: tag_structure}, record, - refs, + subquery_member?, opts - ) do - {:ok, - WhereClause.includes_record?(where, record, WhereClause.subquery_member_from_refs(refs)), + ) + when is_function(subquery_member?, 2) do + {:ok, WhereClause.includes_record?(where, record, subquery_member?), %{ move_tags: make_tags_from_pattern(tag_structure, record, opts[:stack_id], opts[:shape_handle]), @@ -745,6 +748,26 @@ defmodule Electric.Shapes.Shape do }} end + # `extra_refs` is `{old_subquery_member?, new_subquery_member?}` — two + # 2-arity callbacks of shape `(subquery_ref, value) -> boolean`. Older + # callers (notably tests) pass `{old_views_map, new_views_map}` where + # each side is `%{subquery_ref => MapSet}`; we convert those to the + # callback shape on the fly so the underlying DNF evaluator only ever + # sees callbacks. + defp normalise_extra_refs(nil), do: {default_member_callback(), default_member_callback()} + + defp normalise_extra_refs({old, new}) do + {coerce_member_callback(old), coerce_member_callback(new)} + end + + defp coerce_member_callback(callback) when is_function(callback, 2), do: callback + + defp coerce_member_callback(views) when is_map(views) do + WhereClause.subquery_member_from_refs(views) + end + + defp default_member_callback, do: fn _, _ -> false end + defp filter_change_columns(change, nil), do: change defp filter_change_columns(change, selected_columns) do @@ -826,23 +849,35 @@ defmodule Electric.Shapes.Shape do } end - def get_row_metadata(dnf_plan, record, views, where_expr, stack_id, shape_handle) do + def get_row_metadata(dnf_plan, record, views, where_expr, stack_id, shape_handle) + when is_map(views) do + get_row_metadata( + dnf_plan, + record, + WhereClause.subquery_member_from_refs(views), + where_expr, + stack_id, + shape_handle + ) + end + + def get_row_metadata(dnf_plan, record, subquery_member?, where_expr, stack_id, shape_handle) + when is_function(subquery_member?, 2) do with {:ok, ref_values} <- Runner.record_to_ref_values(where_expr.used_refs, record) do - refs = Map.merge(ref_values, views) - active_conditions = compute_active_conditions(dnf_plan, refs) + active_conditions = compute_active_conditions(dnf_plan, ref_values, subquery_member?) tags = compute_tags(dnf_plan, record, stack_id, shape_handle) included? = compute_inclusion(dnf_plan, active_conditions) {:ok, included?, tags, active_conditions} end end - defp compute_active_conditions(dnf_plan, refs) do + defp compute_active_conditions(dnf_plan, ref_values, subquery_member?) do Enum.map(0..(dnf_plan.position_count - 1), fn pos -> info = dnf_plan.positions[pos] pos_expr = Expr.wrap_parser_part(info.ast) base_result = - case Runner.execute(pos_expr, refs) do + case Runner.execute(pos_expr, ref_values, subquery_member?: subquery_member?) do {:ok, value} when value not in [nil, false] -> true _ -> false end diff --git a/packages/sync-service/lib/electric/shapes/where_clause.ex b/packages/sync-service/lib/electric/shapes/where_clause.ex index 98d7cc8e8c..6510659e75 100644 --- a/packages/sync-service/lib/electric/shapes/where_clause.ex +++ b/packages/sync-service/lib/electric/shapes/where_clause.ex @@ -2,6 +2,7 @@ defmodule Electric.Shapes.WhereClause do alias PgInterop.Sublink alias Electric.Replication.Eval.Runner alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @spec includes_record_result( Electric.Replication.Eval.Expr.t() | nil, @@ -45,16 +46,96 @@ defmodule Electric.Shapes.WhereClause do end @doc """ - Build a subquery_member? callback that queries the SubqueryIndex. + Build a `subquery_member?` callback the filter can use while evaluating + a residual `and_where` for one outer shape, answering each sublink with + the conservative MTV-based test the routing path uses (see RFC + §Routing). - Used for filter-side exact verification: checks whether a specific - shape currently contains a typed value for a canonical subquery ref. + - Positive sublink: returns `true` iff the value is a member at *some* + retained logical time (`MultiTimeView.member_at_some_time?/3`). + Surrounding `IN` evaluates `true` ⇒ include. Records whose value is + provably absent at every retained time get excluded here instead of + over-routing to every consumer for the exact check. + - Negated sublink: returns `true` iff the value is a member at *every* + retained logical time (`MultiTimeView.member_at_all_times?/3`). + Surrounding `NOT IN` evaluates `false` ⇒ exclude. Records whose value + is provably a member at every retained time get excluded. + - Anything in between (member at some retained times, not others) yields + a conservative-include in both polarities; the consumer's exact check + in `Shape.convert_change` refines further. + + The shape's polarity per `dep_index` is read from the SubqueryIndex's + `{:dep_handle, …}` rows, which `SubqueryIndex.register_shape/4` populates + with `{dep_handle, polarity}` at filter-add time. + + Raises if `shape_handle` has no row for the referenced `dep_index` or if + `subquery_ref` doesn't parse. The runner catches the raise and the + caller (`other_shape_matches?`) treats it as "include" — matching the + pre-existing failure mode for unknown sublinks. """ - @spec subquery_member_from_index(SubqueryIndex.t(), term()) :: + @spec subquery_member_conservative_from_index(SubqueryIndex.t(), term()) :: ([String.t()], term() -> boolean()) - def subquery_member_from_index(index, shape_handle) do + def subquery_member_conservative_from_index(%SubqueryIndex{} = index, shape_handle) do + mtv = index.multi_time_view + fn subquery_ref, typed_value -> - SubqueryIndex.membership_or_fallback?(index, shape_handle, subquery_ref, typed_value) + {:ok, dep_index} = dep_index_from_ref(subquery_ref) + {dep_handle, polarity} = SubqueryIndex.lookup_dep!(index, shape_handle, dep_index) + + case polarity do + :positive -> MultiTimeView.member_at_some_time?(mtv, dep_handle, typed_value) + :negated -> MultiTimeView.member_at_all_times?(mtv, dep_handle, typed_value) + end + end + end + + defp dep_index_from_ref([_prefix, idx]) when is_binary(idx) do + case Integer.parse(idx) do + {n, ""} -> {:ok, n} + _ -> :error + end + end + + defp dep_index_from_ref(_), do: :error + + @doc """ + Build a subquery_member? callback that reads `MultiTimeView` at the + per-ref logical time given by `subquery_refs`. The optional + `time_override` lets a single ref read at a different time — used by + splice-plan to read the trigger ref's `from_time`/`to_time` while + every other ref stays at the consumer's currently-pinned time. + + Replaces the pre-RFC pattern of materialising a full MapSet per ref + up front; membership is now checked lazily as the DNF evaluator walks + each record's sublinks. + """ + @spec subquery_member_from_mtv( + MultiTimeView.t() | nil, + map(), + {term(), non_neg_integer()} | nil + ) :: + ([String.t()], term() -> boolean()) + def subquery_member_from_mtv(mtv, subquery_refs, time_override \\ nil) + + def subquery_member_from_mtv(nil, _subquery_refs, _time_override) do + fn _, _ -> false end + end + + def subquery_member_from_mtv(mtv, subquery_refs, time_override) do + fn subquery_ref, typed_value -> + case Map.get(subquery_refs, subquery_ref) do + nil -> + false + + %{subquery_id: id, time: pinned_time} -> + time = + case time_override do + {^subquery_ref, override_time} -> override_time + _ -> pinned_time + end + + MultiTimeView.member?(mtv, id, typed_value, time) + end end end end diff --git a/packages/sync-service/mix.exs b/packages/sync-service/mix.exs index c06759cb94..a79f02873f 100644 --- a/packages/sync-service/mix.exs +++ b/packages/sync-service/mix.exs @@ -115,6 +115,7 @@ defmodule Electric.MixProject do defp dev_and_test_deps do [ + {:benchee, "~> 1.3", only: [:dev, :test], runtime: false}, {:dialyxir, "~> 1.4", only: [:test], runtime: false}, {:excoveralls, "~> 0.18", only: [:test], runtime: false}, {:junit_formatter, "~> 3.4", only: [:test], runtime: false}, diff --git a/packages/sync-service/mix.lock b/packages/sync-service/mix.lock index acb8cb92f2..3a4d27acbf 100644 --- a/packages/sync-service/mix.lock +++ b/packages/sync-service/mix.lock @@ -2,12 +2,14 @@ "acceptor_pool": {:hex, :acceptor_pool, "1.0.1", "d88c2e8a0be9216cf513fbcd3e5a4beb36bee3ff4168e85d6152c6f899359cdb", [:rebar3], [], "hexpm", "f172f3d74513e8edd445c257d596fc84dbdd56d2c6fa287434269648ae5a421e"}, "backoff": {:hex, :backoff, "1.1.6", "83b72ed2108ba1ee8f7d1c22e0b4a00cfe3593a67dbc792799e8cce9f42f796b", [:rebar3], [], "hexpm", "cf0cfff8995fb20562f822e5cc47d8ccf664c5ecdc26a684cbe85c225f9d7c39"}, "bandit": {:hex, :bandit, "1.11.0", "dbdd9c9963f146ee9da9860d1ee5b0ffd65cea51fe2aab3f3273df84329d133a", [:mix], [{:hpax, "~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}, {:plug, "~> 1.18", [hex: :plug, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:thousand_island, "~> 1.0", [hex: :thousand_island, repo: "hexpm", optional: false]}, {:websock, "~> 0.5", [hex: :websock, repo: "hexpm", optional: false]}], "hexpm", "c949d93a325a28da2333dde5a9ab61986ad2c2b7226347db6a28303b9139865e"}, + "benchee": {:hex, :benchee, "1.5.0", "4d812c31d54b0ec0167e91278e7de3f596324a78a096fd3d0bea68bb0c513b10", [:mix], [{:deep_merge, "~> 1.0", [hex: :deep_merge, repo: "hexpm", optional: false]}, {:statistex, "~> 1.1", [hex: :statistex, repo: "hexpm", optional: false]}, {:table, "~> 0.1.0", [hex: :table, repo: "hexpm", optional: true]}], "hexpm", "5b075393aea81b8ae74eadd1c28b1d87e8a63696c649d8293db7c4df3eb67535"}, "cc_precompiler": {:hex, :cc_precompiler, "0.1.11", "8c844d0b9fb98a3edea067f94f616b3f6b29b959b6b3bf25fee94ffe34364768", [:mix], [{:elixir_make, "~> 0.7", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "3427232caf0835f94680e5bcf082408a70b48ad68a5f5c0b02a3bea9f3a075b9"}, "chatterbox": {:hex, :ts_chatterbox, "0.15.1", "5cac4d15dd7ad61fc3c4415ce4826fc563d4643dee897a558ec4ea0b1c835c9c", [:rebar3], [{:hpack, "~> 0.3.0", [hex: :hpack_erl, repo: "hexpm", optional: false]}], "hexpm", "4f75b91451338bc0da5f52f3480fa6ef6e3a2aeecfc33686d6b3d0a0948f31aa"}, "combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm", "1b1dbc1790073076580d0d1d64e42eae2366583e7aecd455d1215b0d16f2451b"}, "ctx": {:hex, :ctx, "0.6.0", "8ff88b70e6400c4df90142e7f130625b82086077a45364a78d208ed3ed53c7fe", [:rebar3], [], "hexpm", "a14ed2d1b67723dbebbe423b28d7615eb0bdcba6ff28f2d1f1b0a7e1d4aa5fc2"}, "db_connection": {:hex, :db_connection, "2.10.0", "8ff756471e41765bd5563b633f73e9a94bbc138816e8644bb17d0d91bf260a95", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "02cdd01b45efb1b550e68edbbea41be32de9b24bb07e1ea0e9cbc522ac377e54"}, "decimal": {:hex, :decimal, "2.3.0", "3ad6255aa77b4a3c4f818171b12d237500e63525c2fd056699967a3e7ea20f62", [:mix], [], "hexpm", "a4d66355cb29cb47c3cf30e71329e58361cfcb37c34235ef3bf1d7bf3773aeac"}, + "deep_merge": {:hex, :deep_merge, "1.0.0", "b4aa1a0d1acac393bdf38b2291af38cb1d4a52806cf7a4906f718e1feb5ee961", [:mix], [], "hexpm", "ce708e5f094b9cd4e8f2be4f00d2f4250c4095be93f8cd6d018c753894885430"}, "dialyxir": {:hex, :dialyxir, "1.4.7", "dda948fcee52962e4b6c5b4b16b2d8fa7d50d8645bbae8b8685c3f9ecb7f5f4d", [:mix], [{:erlex, ">= 0.2.8", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm", "b34527202e6eb8cee198efec110996c25c5898f43a4094df157f8d28f27d9efe"}, "dotenvy": {:hex, :dotenvy, "1.1.1", "00e318f3c51de9fafc4b48598447e386f19204dc18ca69886905bb8f8b08b667", [:mix], [], "hexpm", "c8269471b5701e9e56dc86509c1199ded2b33dce088c3471afcfef7839766d8e"}, "earmark_parser": {:hex, :earmark_parser, "1.4.44", "f20830dd6b5c77afe2b063777ddbbff09f9759396500cdbe7523efd58d7a339c", [:mix], [], "hexpm", "4778ac752b4701a5599215f7030989c989ffdc4f6df457c5f36938cc2d2a2750"}, @@ -58,6 +60,7 @@ "retry": {:hex, :retry, "0.19.0", "aeb326d87f62295d950f41e1255fe6f43280a1b390d36e280b7c9b00601ccbc2", [:mix], [], "hexpm", "85ef376aa60007e7bff565c366310966ec1bd38078765a0e7f20ec8a220d02ca"}, "sentry": {:hex, :sentry, "12.0.3", "0d5f681b4a7b57c4df16a37f4b90a554d10e577dad01d4457c844ffa24800861", [:mix], [{:finch, "~> 0.21", [hex: :finch, repo: "hexpm", optional: true]}, {:hackney, "~> 1.8", [hex: :hackney, repo: "hexpm", optional: true]}, {:igniter, "~> 0.5", [hex: :igniter, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: true]}, {:nimble_options, "~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_ownership, "~> 0.3.0 or ~> 1.0", [hex: :nimble_ownership, repo: "hexpm", optional: false]}, {:opentelemetry, ">= 0.0.0", [hex: :opentelemetry, repo: "hexpm", optional: true]}, {:opentelemetry_api, ">= 0.0.0", [hex: :opentelemetry_api, repo: "hexpm", optional: true]}, {:opentelemetry_exporter, ">= 0.0.0", [hex: :opentelemetry_exporter, repo: "hexpm", optional: true]}, {:opentelemetry_semantic_conventions, ">= 0.0.0", [hex: :opentelemetry_semantic_conventions, repo: "hexpm", optional: true]}, {:phoenix, "~> 1.6", [hex: :phoenix, repo: "hexpm", optional: true]}, {:phoenix_live_view, "~> 0.20 or ~> 1.0", [hex: :phoenix_live_view, repo: "hexpm", optional: true]}, {:plug, "~> 1.6", [hex: :plug, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: true]}], "hexpm", "3c2f571ea43e8d3c893c5ca88ab1d1b9501f02925d094fc23811ce5b7f228b65"}, "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"}, + "statistex": {:hex, :statistex, "1.1.0", "7fec1eb2f580a0d2c1a05ed27396a084ab064a40cfc84246dbfb0c72a5c761e5", [:mix], [], "hexpm", "f5950ea26ad43246ba2cce54324ac394a4e7408fdcf98b8e230f503a0cba9cf5"}, "stream_data": {:hex, :stream_data, "1.3.0", "bde37905530aff386dea1ddd86ecbf00e6642dc074ceffc10b7d4e41dfd6aac9", [:mix], [], "hexpm", "3cc552e286e817dca43c98044c706eec9318083a1480c52ae2688b08e2936e3c"}, "stream_split": {:hex, :stream_split, "0.1.7", "2d3fd1fd21697da7f91926768d65f79409086052c9ec7ae593987388f52425f8", [:mix], [], "hexpm", "1dc072ff507a64404a0ad7af90df97096183fee8eeac7b300320cea7c4679147"}, "telemetry": {:hex, :telemetry, "1.4.1", "ab6de178e2b29b58e8256b92b382ea3f590a47152ca3651ea857a6cae05ac423", [:rebar3], [], "hexpm", "2172e05a27531d3d31dd9782841065c50dd5c3c7699d95266b2edd54c2dafa1c"}, diff --git a/packages/sync-service/scripts/subquery_logical_time_memory.exs b/packages/sync-service/scripts/subquery_logical_time_memory.exs new file mode 100644 index 0000000000..2158c2ee96 --- /dev/null +++ b/packages/sync-service/scripts/subquery_logical_time_memory.exs @@ -0,0 +1,312 @@ +# SubqueryIndex memory snapshot. +# +# Run from packages/sync-service: +# +# mix run --no-start scripts/subquery_logical_time_memory.exs +# +# `--no-start` skips booting the sync-service application; this script only +# uses the SubqueryIndex / MultiTimeView modules and a small simulated +# consumer-side state, all in-process. +# +# Reproduces the "Logical" columns of the memory table in +# docs/rfcs/subquery-index.md (§Memory Savings Prototype). The corresponding +# "current model" numbers in that table were generated by a separate +# prototype that simulated the pre-RFC per-shape MapSet implementation; we +# don't reproduce them here because the old model code has been removed +# from this branch. +# +# Measured per scenario: +# * ETS: SubqueryIndex bag table + MultiTimeView set table. +# * Consumers: the compact `%{subquery_ref => %{subquery_id, time}}` map +# each shape holds, plus any in-flight ActiveMove structs. +# * Total: ETS + Consumers. + +alias Electric.Replication.Eval.Parser.{Func, Ref} +alias Electric.Shapes.Consumer.Subqueries.ActiveMove +alias Electric.Shapes.DnfPlan +alias Electric.Shapes.Filter +alias Electric.Shapes.Filter.Indexes.SubqueryIndex +alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView +alias Electric.Shapes.Filter.WhereCondition + +defmodule MemBench do + @field "par_id" + @subquery_ref ["$sublink", "0"] + @wordsize :erlang.system_info(:wordsize) + + def make_plan(polarity \\ :positive) do + testexpr = %Ref{path: [@field], type: :int8} + ref = %Ref{path: @subquery_ref, type: {:array, :int8}} + + ast = %Func{ + name: "sublink_membership_check", + args: [testexpr, ref], + type: :bool + } + + %DnfPlan{ + disjuncts: [], + disjuncts_positions: [], + position_count: 1, + positions: %{ + 0 => %{ + ast: ast, + sql: "fake", + is_subquery: true, + negated: polarity == :negated, + dependency_index: 0, + subquery_ref: @subquery_ref, + tag_columns: [@field] + } + }, + dependency_positions: %{0 => [0]}, + dependency_disjuncts: %{}, + dependency_polarities: %{0 => polarity} + } + end + + def opt do + %{ + operation: "subquery", + field: @field, + testexpr: %Ref{path: [@field], type: :int8}, + subquery_ref: @subquery_ref, + dep_index: 0, + polarity: :positive, + and_where: nil + } + end + + # Build a Filter with one subquery containing `value_count` base values, + # `shape_count` shapes attached, all at time 0. + def build_steady(shape_count, value_count) do + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + index = filter.subquery_index + dep = "dep_a" + + values = for v <- 1..value_count, do: v + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + + shape_ids = + for i <- 1..shape_count do + shape_id = "shape_#{i}" + SubqueryIndex.register_shape(index, shape_id, make_plan(), [dep]) + SubqueryIndex.add_shape(filter, condition_id, shape_id, opt(), []) + SubqueryIndex.mark_ready(index, shape_id) + shape_id + end + + consumer_states = + for _ <- 1..shape_count, do: %{@subquery_ref => %{subquery_id: dep, time: 0}} + + %{filter: filter, dep: dep, shape_ids: shape_ids, consumer_states: consumer_states} + end + + # Layer on `added_count` values across `time_steps` logical timesteps. + # Each added value enters the MTV via mark_in at a logical time spread + # uniformly across [1, time_steps]. Old times are retained because not + # every consumer has advanced past them. + def apply_advanced(state, added_count, time_steps) do + %{filter: filter, dep: dep, shape_ids: shape_ids, consumer_states: consumer_states} = + state + + index = filter.subquery_index + mtv = index.multi_time_view + + base_value = 1_000_000 + + for i <- 1..added_count do + value = base_value + i + # Spread the adds across the available time steps. Floor division so + # each step gets a roughly equal slice. + step = div((i - 1) * time_steps, added_count) + 1 + MultiTimeView.mark_in(mtv, dep, value, step) + SubqueryIndex.add_positive_route(index, dep, value) + end + + # Advance each consumer to a logical time spread across [0, time_steps]. + # This leaves a non-trivial retention window so histories accumulate. + consumer_states = + consumer_states + |> Enum.with_index(0) + |> Enum.map(fn {state, idx} -> + target = div(idx * time_steps, length(shape_ids)) + Map.update!(state, @subquery_ref, &%{&1 | time: target}) + end) + + %{state | consumer_states: consumer_states} + end + + # Add `move_count` in-flight ActiveMove structs. `total_moved_values` is + # split evenly across the moves. + def apply_active_moves(state, move_count, total_moved_values) do + dep = state.dep + values_per_move = max(1, div(total_moved_values, max(move_count, 1))) + + active_moves = + for i <- 1..move_count do + from_t = i + to_t = i + 1 + + move_in_values = + for j <- 1..values_per_move do + value = 2_000_000 + i * values_per_move + j + {value, "v#{value}"} + end + + %ActiveMove{ + subquery_id: dep, + dep_index: 0, + subquery_ref: @subquery_ref, + from_time: from_t, + to_time: to_t, + move_in_values: move_in_values, + move_out_values: [], + txids: [10_000 + i], + buffered_txn_count: 0, + buffered_txns: [] + } + end + + Map.put(state, :active_moves, active_moves) + end + + defp ets_info_bytes(table) do + case :ets.info(table, :memory) do + words when is_integer(words) -> words * @wordsize + _ -> 0 + end + end + + def ets_bytes(filter) do + ets_info_bytes(filter.subquery_index.table) + + ets_info_bytes(filter.subquery_index.counters) + end + + def mtv_bytes(filter) do + ets_info_bytes(filter.subquery_index.multi_time_view) + end + + def consumer_bytes(state) do + consumer_size = + state.consumer_states + |> Enum.map(&(:erts_debug.size(&1) * @wordsize)) + |> Enum.sum() + + move_size = + Map.get(state, :active_moves, []) + |> Enum.map(&(:erts_debug.size(&1) * @wordsize)) + |> Enum.sum() + + consumer_size + move_size + end + + def measure(label, state) do + ets = ets_bytes(state.filter) + mtv_bytes(state.filter) + consumers = consumer_bytes(state) + total = ets + consumers + + %{ + label: label, + ets: ets, + consumers: consumers, + total: total + } + end + + def human(bytes) when bytes < 1024, do: "#{bytes} B" + + def human(bytes) when bytes < 1024 * 1024, + do: :erlang.float_to_binary(bytes / 1024, decimals: 1) <> " KiB" + + def human(bytes), do: :erlang.float_to_binary(bytes / 1_048_576, decimals: 2) <> " MiB" + + def print_table(rows) do + IO.puts("") + + IO.puts( + "| Scenario | Logical total | Logical ETS | Logical consumers |" + ) + + IO.puts( + "|----------|---------------|-------------|-------------------|" + ) + + for r <- rows do + IO.puts( + "| #{r.label} | #{human(r.total)} | #{human(r.ets)} | #{human(r.consumers)} |" + ) + end + + IO.puts("") + end +end + +# ============================================================================ +# Scenarios +# ============================================================================ + +scenarios = [ + fn -> + MemBench.build_steady(1, 1_000) + |> then(&MemBench.measure("1 shape, 1k values, steady", &1)) + end, + fn -> + MemBench.build_steady(10, 1_000) + |> then(&MemBench.measure("10 shapes, 1k values, steady", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> then(&MemBench.measure("100 shapes, 1k values, steady", &1)) + end, + fn -> + MemBench.build_steady(100, 10_000) + |> then(&MemBench.measure("100 shapes, 10k values, steady", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(100, 10) + |> then(&MemBench.measure("100 shapes, 1k base, 100 added x 10 advanced", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(100, 99) + |> then(&MemBench.measure("100 shapes, 1k base, 100 added x 99 advanced", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(100, 10) + |> MemBench.apply_active_moves(10, 100) + |> then(&MemBench.measure("100 shapes, 1k base, 100 added x 10 active move", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(1_000, 99) + |> MemBench.apply_active_moves(99, 1_000) + |> then(&MemBench.measure("100 shapes, 1k base, 1k added x 99 active move", &1)) + end +] + +results = + for scenario <- scenarios do + # Each scenario builds its own private ETS; force a GC between to keep + # the numbers comparable. + :erlang.garbage_collect() + result = scenario.() + :erlang.garbage_collect() + result + end + +IO.puts("# SubqueryIndex memory snapshot") +IO.puts("") + +IO.puts( + "OTP: #{:erlang.system_info(:otp_release)} Wordsize: #{:erlang.system_info(:wordsize)} bytes" +) + +IO.puts("Elixir: #{System.version()}") + +MemBench.print_table(results) diff --git a/packages/sync-service/subquery-index-prompt.md b/packages/sync-service/subquery-index-prompt.md new file mode 100644 index 0000000000..a75af9a287 --- /dev/null +++ b/packages/sync-service/subquery-index-prompt.md @@ -0,0 +1,203 @@ +# Shared Subquery Views with Logical-Time Reads + +## Summary + +Electric v1.6 introduced per-shape subquery indexing so consumers can keep +boolean subquery shapes live while dependency rows move across `WHERE` +boundaries. That solved correctness, but it made memory scale with the number of +shape consumers. Each consumer can keep its own materialized dependency view in +the `SubqueryIndex`, and while move-ins are buffered it can also hold both a +before and after view. + +This RFC proposes replacing per-consumer materialized subquery views with one +shared, versioned view per subquery. Consumers do not copy the view. +Instead, they read the shared view at a logical time. + + +## Background + +Related implementation work: + +- Commit: https://github.com/electric-sql/electric/commit/a04b25962cdb7ca86c4434585b6f74c758e1a31b +- PR: https://github.com/electric-sql/electric/pull/4051 +- issue: https://github.com/electric-sql/electric/issues/4279 +- Current index: `packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex` + +The v1.6 work lets shapes with boolean combinations around subqueries remain +live when dependency rows move. The key correctness problem is that consumers can +temporarily disagree about a subquery's membership while one consumer has +processed a move and another has not. + +The current implementation handles that by letting each shape consumer seed and +update exact per-shape membership rows. That keeps each consumer correct, but it +duplicates the same view across many shapes. During move-in buffering, the +consumer also carries before and after views so it can convert buffered +transactions and build the move-in query. + +## Problem + +The memory problem is broader than value-keyed routing rows in `SubqueryIndex`. +There are at least two duplicated memory pools: + +1. `SubqueryIndex` membership and routing rows, currently keyed by shape. +2. Consumer/materializer views, including before and after views during active + move-in buffering. + +Adding a reverse index such as `shape_handle -> all values` would make removal +faster, but it would increase memory. + +## Definitions + +### Subquery + +Each subquery gets it's own shape. If the select statement differs at all we count it as a different subquery, even if the difference is just in a constant. So: +- SELECT id FROM users WHERE company_id=7 +- SELECT id FROM users WHERE company_id=8 +are two different subqueries and each get their own subquery_id (the handle for the subquery shape) + +### Subquery Group + +A subquery group is a set of subqueries that have the same field and polarity at a particular node in the filter tree. + +So for example the two subqueries in the two shapes below are differnt subqueries (because they differ by the company_id constant) but they are in the same subquery group because they have the same field (user_id) and polarity (:positive) at the same node in the filter tree: +WHERE user_id IN (SELECT id FROM users WHERE company_id=7) +WHERE user_id IN (SELECT id FROM users WHERE company_id=8) + +A subquery_id may appear in multiple subquery groups if it appears at multiple nodes in the filter tree. For the subquery is the same (has the same subquery_id) in the two shapes below but falls into different subquery groups because it appears at a differnt node in the filter tree: +WHERE user_id IN (SELECT id FROM users WHERE company_id=7) +WHERE project_id=4 AND user_id IN (SELECT id FROM users WHERE company_id=7) + +## Goals + +- Reduce memory footprint of subqueries significantly while remaining consitant and performant +- have near O(1) performance for: + - subquery addition and removal, including subquery group addition and removal where needed + - row processing by the where clause filter (so for afffected_shapes in the SubqueryIndex) even when there are thousands of subqueries in a subquery group +- Store one shared materialized view per subquery. +- Support exact membership reads at separate logical times. +- Preserve positive, negated, AND, OR, and NOT subquery correctness from v1.6. + +## Non-Goals + +- Do not change the client wire protocol. + +## Proposal + +### Components + +#### SubqueryIndex.MultiTimeView + +The MultiTimeView is an Materialized view of a subquery, queryable at multiple points in time. + +It's implimented as an ETS table (one ETS table per stack_id) + +subquery_id, value -> list(times) + +the meaning of the result: + doesn't exist - the value is not a member of the subquery for all logical times + [] - the value is a member of the subquery for all logical times + [:out, 9] - the value was out of the set before 9 and in the set from time 9 and above + [:out, 9, 11] - the value was out of the set before 9 and in the set from 9 to 10 and out the set again from time 11 and above + [:in, 9] - the value was in of the set before 9 and out the set from time 9 and above + [:in, 9, 11] - the value was in of the set before 9 and out the set from 9 to 10 and in the set again from time 11 and above + +note: the list(times) structure above has been chosen for memory efficientcy, but if you can think of a smaller structure let me know. for example if `[]` takes up more space than `true` then we should use `true` since this will be the most common case and we want to be memory efficient. + +so for subquery_id, value - [:in, 9, 11] + +member?(subquery_id, value, time: 8) = true +member?(subquery_id, value, time: 9) = false +member?(subquery_id, value, time: 10) = false +member?(subquery_id, value, time: 11) = true + +rather than specifying a time you can also ask for membership across all times: + +member_at_some_time?(subquery_id, value) = true +member_at_all_times?(subquery_id, value) = false + +These are useful for the where clause filter which needs to keep the filter broad enough so that all consumers get all the changes they need while they may be at any of the logical times. + +For each subquery there will be a minimum logical time needed (the minimum in-flight logical time for the subquery) which the SubqueryProgressMonitor will set on the MultiTimeView. This allows the MultiTimeViewETS table to be compacted for memory and performace efficientcy. For any given list(times) it can be compacted by removing times from before the minimum in-flight logical time, making sure to update the :in/:out marker at the beginning of the list appropriately or removing it if there are no times left. + +Compacting should happen: +- when the list is read (e.g. when member? for the value is called) +- when the list is written to (e.g. when a value is moved in or out) +- when an async compaction routine is run (the design of this will need to be discussed) + +Removing a subquery should not involve a full ETS table scan as this will be too slow with lots of subqueries. If the ETS table is orderd we should be able to find the first item for the subqery, delete that, then find the next, and continue until the whole subquery is gone. That means it scales with the number of values (which is acceptable) rather than the number of subqueries. + + +#### SubqueryIndex + +This is a complete re-write of the existing SubqueryIndex that delegates some of it's resposibility to the the MultiTimeView. + +Since there may be many subqueries in a subquery group, the SubqueryIndex should keep: + +subquery_group_id, value -> list(child_node_id) + +where: +subquery_group_id is a number (whatever is smallest in memory) and represents {node_id, field, polarity} but to save memory (as it's going to be repeated lots in the ETS table) we keep it small and also store: +subquery_group_id -> {node_id, field, polarity} and +{node_id, field, polarity} -> subquery_group_id + +and there's one child_node_id per subquery_id for the group. child_node_id is smaller in memory so we keep that in places where it's going to be repeated lots in the ETS table (e.g. in `subquery_group_id, value -> list(child_node_id)`) + + +So for `afffected_shapes` for a particular value, we'd look up the list of child_node_ids from the subquery_group_id, value pair then lookup the subquery_ids from the child_node_ids then for each subquery_id: + +if MultiTimeView.member_at_some_time?(subquery_id, value) do + WhereCondition.affected_shapes(child_node_id) +else + MapSet.new() +end + +For a given {subquery_id, :negative} pair the affected shaped will be: + +if MultiTimeView.member_at_all_times?(subquery_id, value) do + MapSet.new() +else + WhereCondition.affected_shapes(child_node_id) +end + +This will ensure that the rows are included for all available times. + +If the MultiTimeView has not been marked ready by the Materializer yet, the SubqueryIndex should return WhereCondition.affected_shapes(child_node_id) + +Removal of a subquery must not scale with the total number of shapes or the number of subqueries in the group, but can scale with the number of values for the subquery. This can be achived by getting the getting the values for the subquery from the MultiTimeView (as discussed above in the MultiTimeView section when talking about subquery removal) - whilst iterating though those values we can also delete those values in the SubqueryIndex for all the groups that it's in. + +#### Materializer + +This is the existing Materializer. It will just need to be updated to: +- populate the SubqueryIndex when the Materializer has initialised (it has a full materialized view). This should be at logical time 0. +- increment logical time for each `{:materializer_changes` message it sends to outer consumers, and include the new logical time in that message +- before the `{:materializer_changes` message is sent, the SubqueryIndex should be updated with the changes giving the new logical time as the time of the change + +#### Logical Time + +Logical Time is monotonically incrementing counter per subquery. + +This needs to be a memory efficient data staructure that can be incremented indefinately. If it needs to wrap we need to make sure we use appropriate conparison functions when comparing times. Wrapping is an acceptable solution since there will only ever be so many moves in flight for any given subquery and memory would explode due to that before wrapping would cause comparison failures. + +#### SubqueryProgressMonitor + +This can be a separate process that the outer consumer calls to acknoledge that it's finished with a logical time for a subquery. The SubqueryProgressMonitor can then keep track of the minimum in-flight logical time for each subquery and set that on the MultiTimeView so that the MultiTimeView can compact it's ETS table for memory and performance efficientcy. + +The SubqueryProgressMonitor can be implimented as an ETS table ordered by subquery_id then logical time with an index to where an outer shape_id entry is so that when an outer consumer acks a logical time for a subquery, the outer shape can be found in the the ordered list and removed and replaced with the acked time. The minimum of theses times is the minimum in-flight logical time for the subquery. This should mean that updating a outer shape's logical time is O(1) and reading the minimum in-flight logical time is O(1). The SubqueryProgressMonitor should notify the MultiTimeView when the minimum in-flight logical time for a subquery changes so that the MultiTimeView can compact it's ETS table. + +The SubqueryProgressMonitor must know about all shapes for a subquery (so for example if it's not seen an ack from one of them it needs to know the minimum time is still 0) or a subquery and have those shapes removed + +#### Consumer EventProcessors + +These should be updated so that rather than holding views of the subquery, they just hold the logical time. so the before and after views should instead just be the before and after logical times. +- `convert_change` should have a function passed to it that access MultiTimeView.member? at the specified time +- the move-in query needs entire views at specific times and so should call MultiTimeView.get(time) and care should be made to not keep this in memory for too long, perhaps we should GC the consumer process afterwards, or perhaps the task process that runs the query should call MultiTimeView.get(time) so that the memory is freed when the process ends + +### Concurrency model + +Reads and writes to the MultiTimeView and SubqueryIndex ETS tables will mostly not be concurrent: +- add_shape and remove_shape will happen on the ShapeLogCollector process +- add_value and remove_value will happen while the ShapeLogCollector process is blocked so acts as if it were on the ShapeLogCollector process (ShapeLogCollector calls the Consumer which calls the Materializer which calls the SubqueryIndex to add/remove values, all synchronously) +- a Materializer seeding a subquery will happen when the Materializer is ready (so asyncronously to the ShapeLogCollector process) but will then call mark_ready on the SubqueryIndex which is an atomic process +- read of MultiTimeView may happen async by a consumer, but will be a read at a specific logical time so concurrentcy should not be an issue +- the mimimum in-flight logical time for a subquery will be updated by the SubqueryProgressMonitor async, but this will just update a single number, so concurrentcy should not be an issue + diff --git a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs b/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs deleted file mode 100644 index 0978b3ce3e..0000000000 --- a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs +++ /dev/null @@ -1,957 +0,0 @@ -defmodule Electric.Shapes.Consumer.EventHandler.SubqueriesTest do - use ExUnit.Case, async: true - - alias Electric.Postgres.Lsn - alias Electric.Replication.Changes - alias Electric.Replication.Changes.Transaction - alias Electric.Shapes.Consumer.Effects - alias Electric.Shapes.Consumer.EventHandler - alias Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering - alias Electric.Shapes.Consumer.EventHandler.Subqueries.Steady - alias Electric.Shapes.Consumer.Subqueries.ActiveMove - alias Electric.Shapes.Consumer.Subqueries.RefResolver - alias Electric.Shapes.Consumer.Subqueries.ShapeInfo - alias Electric.Shapes.DnfPlan - alias Electric.Shapes.Shape - - @inspector Support.StubInspector.new( - tables: ["parent", "child"], - columns: [ - %{name: "id", type: "int8", pk_position: 0, type_id: {20, 1}}, - %{name: "value", type: "text", pk_position: nil, type_id: {28, 1}}, - %{name: "parent_id", type: "int8", pk_position: nil, type_id: {20, 1}}, - %{name: "name", type: "text", pk_position: nil, type_id: {28, 1}} - ] - ) - - describe "Subquery handler" do - test "converts transactions against the current subquery view" do - handler = new_handler(subquery_view: MapSet.new([1])) - - assert {:ok, %Steady{}, plan} = - EventHandler.handle_event( - handler, - txn(50, [child_insert("1", "1"), child_insert("2", "2")]) - ) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "1"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _} - ] = plan - end - - test "still converts root transactions when dependency moves are configured to invalidate" do - handler = - new_handler( - subquery_view: MapSet.new([1]), - dependency_move_policy: :invalidate_on_dependency_move - ) - - assert {:ok, %Steady{}, plan} = - EventHandler.handle_event( - handler, - txn(50, [child_insert("1", "1"), child_insert("2", "2")]) - ) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "1"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _} - ] = plan - end - - test "returns unsupported_subquery when dependency moves are configured to invalidate" do - handler = new_handler(dependency_move_policy: :invalidate_on_dependency_move) - dep_handle = dep_handle(handler) - - assert {:error, :unsupported_subquery} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - end - - test "negated subquery turns dependency move-in into an outer move-out" do - handler = new_handler(shape: negated_shape()) - dep_handle = dep_handle(handler) - - assert {:ok, %Steady{views: %{["$sublink", "0"] => view}} = _handler, plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert view == MapSet.new([1]) - - # Case D: negated move-in completes immediately — effects_for_complete - # adds the value to the index (deferred to completion for NOT IN broadening) - assert [ - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: [%{pos: 0}]}} - }, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{1, "1"}]} - ] = plan - end - - test "negated subquery turns dependency move-out into a buffered outer move-in" do - handler = new_handler(shape: negated_shape(), subquery_view: MapSet.new([1])) - dep_handle = dep_handle(handler) - - # Case B: negated move-out → remove the value when buffering starts so the - # negated index reflects the post-move exclusion set while buffering. - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{1, "1"}]}} - ) - - assert [ - %Effects.SubscribeGlobalLsn{}, - %Effects.RemoveFromSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.StartMoveInQuery{} - ] = plan - - assert %Buffering{ - active_move: %ActiveMove{ - views_before_move: %{["$sublink", "0"] => before_view}, - views_after_move: %{["$sublink", "0"] => after_view} - } - } = handler - - assert before_view == MapSet.new([1]) - assert after_view == MapSet.new() - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - # Case B: negated move-out → no further index effect at complete because - # the buffering-start removal already matches the post-splice dependency view. - assert {:ok, %Steady{views: %{["$sublink", "0"] => view}}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view == MapSet.new() - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "splices buffered transactions around the snapshot visibility boundary" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, - [ - %Effects.SubscribeGlobalLsn{}, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.StartMoveInQuery{} - ]} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_insert("10", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(150, [child_insert("11", "1")])) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "11"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "splices move-in query rows between emitted pre and post boundary changes" do - handler = new_handler(subquery_view: MapSet.new([1])) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_insert("10", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(150, [child_insert("11", "2")])) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([1, 2]) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "10"}}] - }, - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "11"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "splices updates that become a delete before the boundary and an insert after it" do - handler = new_handler(subquery_view: MapSet.new([1])) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_update("10", "1", "2")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(150, [child_update("11", "3", "2")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([1, 2]) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.DeletedRecord{old_record: %{"id" => "10"}}] - }, - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "11"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "uses lsn updates to splice at the current buffer tail" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(120, [child_insert("10", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "waits for an lsn update even when the move-in query completes with an empty buffer" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "keeps an empty stored move-in snapshot as an effect so execution can clean it up" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20), row_count: 0, row_bytes: 0) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 0, - row_bytes: 0 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "uses an lsn update that arrived before the move-in query completed" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "keeps the newest seen lsn when an older update arrives later" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "defers queued move outs until after splice and starts the next move in" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert %Buffering{ - active_move: %ActiveMove{ - values: [{2, "2"}], - views_before_move: views_before, - views_after_move: views_after - } - } = handler - - assert view_for(views_before) == MapSet.new() - assert view_for(views_after) == MapSet.new([2]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: [%{pos: 0}]}} - }, - %Effects.RemoveFromSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{2, "2"}]}, - %Effects.StartMoveInQuery{} - ] = plan - end - - test "queued second move-in emits buffering effects only after it is dequeued" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, - [ - %Effects.SubscribeGlobalLsn{}, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.StartMoveInQuery{} - ]} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}, queue: queue} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {[{2, "2"}], _} = Map.fetch!(queue.move_in, 0) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{values: [{2, "2"}]}} = _handler, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{2, "2"}]}, - %Effects.StartMoveInQuery{} - ] = plan - end - - test "chained move-in resolves without needing a new lsn broadcast" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - # First splice completes, second move-in starts - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - # Second move-in resolves with no further lsn broadcasts - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {200, 300, []}}) - - assert {:ok, %Steady{views: views}, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([2]) - end - - test "applies a queued move out for the active move-in value after splice" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view_for(views) == MapSet.new() - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: [%{pos: 0}]}} - }, - %Effects.RemoveFromSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "batches consecutive move ins into a single active move in" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, - %{move_in: [{1, "1"}, {2, "2"}], move_out: []}} - ) - - assert %Buffering{ - active_move: %ActiveMove{ - values: [{1, "1"}, {2, "2"}], - views_before_move: views_before, - views_after_move: views_after - } - } = handler - - assert view_for(views_before) == MapSet.new() - assert view_for(views_after) == MapSet.new([1, 2]) - end - - test "cancels pending inverse ops while buffering" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{2, "2"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "merges queued move outs into a single control message after splice" do - handler = new_handler(subquery_view: MapSet.new([2])) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{2, "2"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view_for(views) == MapSet.new() - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: patterns}} - }, - %Effects.RemoveFromSubqueryIndex{values: values}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - - assert length(patterns) == 2 - assert length(values) == 2 - end - - test "returns {:error, :buffer_overflow} when buffered transactions exceed the limit" do - handler = new_handler(buffer_max_transactions: 3) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_insert("1", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(51, [child_insert("2", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(52, [child_insert("3", "1")])) - - assert {:error, :buffer_overflow} = - EventHandler.handle_event(handler, txn(53, [child_insert("4", "1")])) - end - - test "returns truncate error on TruncatedRelation while steady" do - handler = new_handler(subquery_view: MapSet.new([1])) - - assert {:error, {:truncate, 1}} = - EventHandler.handle_event(handler, txn(1, [child_truncate()])) - end - - test "returns truncate error while buffering once splice completes" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_truncate()])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:error, {:truncate, 50}} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - end - - test "raises on dependency handle mismatch" do - assert_raise ArgumentError, ~r/unexpected dependency handle/, fn -> - new_handler() - |> EventHandler.handle_event( - {:materializer_changes, "wrong", %{move_in: [], move_out: []}} - ) - end - end - - test "raises on query callbacks while steady" do - handler = new_handler() - - assert_raise ArgumentError, ~r/no move-in is buffering/, fn -> - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - end - - assert_raise ArgumentError, ~r/no move-in is buffering/, fn -> - EventHandler.handle_event(handler, move_in_complete(lsn(1), row_count: 0, row_bytes: 0)) - end - end - end - - # -- Helpers -- - - defp new_handler(opts \\ []) do - shape = Keyword.get(opts, :shape, shape()) - {:ok, dnf_plan} = DnfPlan.compile(shape) - dep_handle = hd(shape.shape_dependencies_handles) - - %Steady{ - shape_info: %ShapeInfo{ - shape: shape, - stack_id: "stack-id", - shape_handle: "shape-handle", - dnf_plan: dnf_plan, - ref_resolver: - RefResolver.new(%{dep_handle => {0, ["$sublink", "0"]}}, %{0 => ["$sublink", "0"]}), - buffer_max_transactions: Keyword.get(opts, :buffer_max_transactions, 1000), - dependency_move_policy: - Keyword.get(opts, :dependency_move_policy, :stream_dependency_moves) - }, - views: %{["$sublink", "0"] => Keyword.get(opts, :subquery_view, MapSet.new())} - } - end - - defp dep_handle(handler) do - handler.shape_info.ref_resolver.handle_to_ref |> Map.keys() |> hd() - end - - defp view_for(views, ref \\ ["$sublink", "0"]) when is_map(views) do - views[ref] - end - - defp shape do - Shape.new!("child", - where: "parent_id IN (SELECT id FROM public.parent WHERE value = 'keep')", - inspector: @inspector, - feature_flags: ["allow_subqueries"] - ) - |> fill_handles() - end - - defp negated_shape do - Shape.new!("child", - where: "parent_id NOT IN (SELECT id FROM public.parent WHERE value = 'keep')", - inspector: @inspector, - feature_flags: ["allow_subqueries"] - ) - |> fill_handles() - end - - defp fill_handles(shape) do - filled_deps = Enum.map(shape.shape_dependencies, &fill_handles/1) - handles = Enum.map(filled_deps, &Shape.generate_id/1) - %{shape | shape_dependencies: filled_deps, shape_dependencies_handles: handles} - end - - defp txn(xid, changes) do - %Transaction{ - xid: xid, - changes: changes, - num_changes: length(changes), - lsn: lsn(xid), - last_log_offset: Electric.Replication.LogOffset.new(lsn(xid), max(length(changes) - 1, 0)) - } - end - - defp lsn(value), do: Lsn.from_integer(value) - defp global_last_seen_lsn(value), do: {:global_last_seen_lsn, value} - - defp move_in_complete(lsn, opts \\ []) do - {:query_move_in_complete, Keyword.get(opts, :snapshot_name, "move-in-snapshot"), - Keyword.get(opts, :row_count, 1), Keyword.get(opts, :row_bytes, 100), lsn} - end - - defp child_insert(id, parent_id) do - %Changes.NewRecord{ - relation: {"public", "child"}, - record: %{"id" => id, "parent_id" => parent_id, "name" => "child-#{id}"} - } - |> Changes.fill_key(["id"]) - end - - defp child_truncate do - %Changes.TruncatedRelation{relation: {"public", "child"}} - end - - defp child_update(id, old_parent_id, new_parent_id) do - Changes.UpdatedRecord.new( - relation: {"public", "child"}, - old_record: %{"id" => id, "parent_id" => old_parent_id, "name" => "child-#{id}-old"}, - record: %{"id" => id, "parent_id" => new_parent_id, "name" => "child-#{id}-new"} - ) - |> Changes.fill_key(["id"]) - end -end diff --git a/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs b/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs index 06f9cbbd48..e786877a9c 100644 --- a/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs @@ -11,6 +11,13 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do alias Electric.Shapes.ConsumerRegistry alias Electric.Replication.LogOffset alias Electric.Shapes.Consumer.Materializer + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + + defp link_values(%{stack_id: stack_id, shape_handle: shape_handle}) do + mtv = MultiTimeView.for_stack(stack_id) + time = MultiTimeView.current_time(mtv, shape_handle) || 0 + mtv |> MultiTimeView.values(shape_handle, time) |> MapSet.new() + end @moduletag :tmp_dir @@ -98,7 +105,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "3"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1, 2, 3]) + assert link_values(ctx) == MapSet.new([1, 2, 3]) end describe "materializing non-pk selected columns" do @@ -109,7 +116,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "1", record: %{"value" => "1"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1]) + assert link_values(ctx) == MapSet.new([1]) assert_receive {:materializer_changes, _, %{move_in: [{1, "1"}]}} end @@ -151,7 +158,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "on-load insert of a new value is seen & does not cause a move-in", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -171,7 +178,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([11]) + assert link_values(ctx) == MapSet.new([11]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{11, "11"}]}} end @@ -185,7 +192,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ] test "on-load update of a value is seen & does not cause events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([11]) + assert link_values(ctx) == MapSet.new([11]) refute_received {:materializer_changes, _, _} end @@ -198,7 +205,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.DeletedRecord{old_record: %{"id" => "1", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -210,7 +217,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "on-load delete of a value is seen & does not cause events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) refute_received {:materializer_changes, _, _} end @@ -224,7 +231,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.NewRecord{record: %{"id" => "2", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -236,7 +243,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update of a value to a present value causes just a move-out", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) Materializer.new_changes( ctx, @@ -249,7 +256,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_received {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -261,7 +268,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update of a value to a non-present value causes a move-in", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) Materializer.new_changes( ctx, @@ -274,7 +281,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) assert_received {:materializer_changes, _, %{move_in: [{20, "20"}]}} end @@ -287,7 +294,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update between otherwise present values causes no events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) Materializer.new_changes( ctx, @@ -300,7 +307,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) refute_received {:materializer_changes, _, _} end @@ -317,7 +324,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.DeletedRecord{old_record: %{"id" => "1", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -334,7 +341,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.NewRecord{record: %{"id" => "3", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -345,7 +352,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "insert of a PK we've already seen raises", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) pid = GenServer.whereis(Materializer.name(ctx)) Process.unlink(pid) @@ -364,7 +371,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "delete of a PK we've not seen throws an error", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) pid = GenServer.whereis(Materializer.name(ctx)) Process.unlink(pid) @@ -390,7 +397,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes the primary key is handled correctly", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where the PK changes from "1" to "2" Materializer.new_changes( @@ -404,7 +411,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{20, "20"}]}} end @@ -416,7 +423,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes the primary key but keeps the same value", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where the PK changes but tracked value stays the same Materializer.new_changes( @@ -431,7 +438,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ) # Value should still be present - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # No events since the tracked value didn't change refute_received {:materializer_changes, _, _} @@ -449,7 +456,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes PK and tag updates tag indices correctly", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where PK changes and tag changes Materializer.new_changes( @@ -465,7 +472,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{20, "20"}]}} # move_out for old tag should find nothing (old_key fully removed) @@ -473,7 +480,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag_a"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) refute_received {:materializer_changes, _, _} # move_out for new tag should remove the row using the new key @@ -481,7 +488,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag_b"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) assert_receive {:materializer_changes, _, %{move_out: [{20, "20"}], move_in: []}} end @@ -497,7 +504,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes PK but keeps same tag cleans up stale tag entry", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where PK changes but tag stays the same Materializer.new_changes( @@ -513,7 +520,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{20, "20"}]}} # move_out for tag_a should remove the row using the new key, not crash @@ -522,7 +529,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag_a"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) assert_receive {:materializer_changes, _, %{move_out: [{20, "20"}], move_in: []}} end @@ -559,7 +566,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "1"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1, 2]) + assert link_values(ctx) == MapSet.new([1, 2]) assert_receive {:materializer_changes, _, %{move_in: move_in}} assert Enum.sort(move_in) == [{1, "1"}, {2, "2"}] @@ -574,7 +581,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.UpdatedRecord{key: "1", record: %{"other" => "1"}, old_record: %{"other" => "0"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1, 3]) + assert link_values(ctx) == MapSet.new([1, 3]) assert_receive {:materializer_changes, _, %{move_out: [{2, "2"}], move_in: [{3, "3"}]}} end @@ -684,7 +691,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update with tag change but unchanged value updates tags without events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where tags change but the tracked value stays the same Materializer.new_changes( @@ -701,7 +708,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ) # Value should still be present - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # No move events should be emitted since the value didn't change refute_received {:materializer_changes, _, _} @@ -714,7 +721,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "tag is updated so subsequent move_out for old tag finds nothing", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update that changes the tag from old_tag to new_tag but keeps value the same Materializer.new_changes( @@ -739,7 +746,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value should still be present (row wasn't removed) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # No move events since the row was already moved to new_tag refute_received {:materializer_changes, _, _} @@ -752,7 +759,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "move_out for new tag after tag update removes the row", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update that changes the tag from old_tag to new_tag Materializer.new_changes( @@ -776,7 +783,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value should be gone - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) # Should emit move_out event assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} @@ -792,7 +799,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "multiple rows with same tag, one updates tag, move_out only affects remaining", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) # Row 1 moves from tag_a to tag_b, row 2 stays in tag_a Materializer.new_changes( @@ -816,7 +823,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Only value 10 should remain (row 1 is now under tag_b) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Should emit move_out only for row 2's value assert_receive {:materializer_changes, _, %{move_out: [{20, "20"}]}} @@ -834,7 +841,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "30"}, move_tags: ["tag1"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20, 30]) + assert link_values(ctx) == MapSet.new([10, 20, 30]) assert_receive {:materializer_changes, _, %{move_in: _}} # Send move_out event to remove rows with tag1 @@ -842,7 +849,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag1"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: move_out}} assert Enum.sort(move_out) == [{10, "10"}, {30, "30"}] end @@ -856,7 +863,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "30"}, move_tags: ["tag3"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20, 30]) + assert link_values(ctx) == MapSet.new([10, 20, 30]) assert_receive {:materializer_changes, _, %{move_in: _}} # Remove rows with tag1 or tag3 @@ -869,7 +876,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: move_out}} assert Enum.sort(move_out) == [{10, "10"}, {30, "30"}] end @@ -881,7 +888,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "1", record: %{"value" => "10"}, move_tags: ["tag1"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Try to remove rows with non-existent tag @@ -889,7 +896,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "non_existent"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -902,7 +909,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "2", record: %{"value" => "10"}, move_tags: ["tag2"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Remove only tag1 row @@ -911,7 +918,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value 10 should still be present because key "2" still has it - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -926,7 +933,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Both values should be present after on-load - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) # Now send move_out event to remove rows with tag1 Materializer.new_changes(ctx, [ @@ -934,7 +941,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Only value 20 should remain after move_out - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -949,7 +956,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Value 10 should be present (from both rows) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Remove rows with tag1 Materializer.new_changes(ctx, [ @@ -957,7 +964,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value 10 should still be present because key "2" still has it - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -972,14 +979,14 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "on-load tags with multiple rows sharing same tag can all be removed", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20, 30]) + assert link_values(ctx) == MapSet.new([10, 20, 30]) # Remove all rows with tag1 Materializer.new_changes(ctx, [ %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag1"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([30]) + assert link_values(ctx) == MapSet.new([30]) assert_receive {:materializer_changes, _, %{move_out: move_out}} assert Enum.sort(move_out) == [{10, "10"}, {20, "20"}] end @@ -993,7 +1000,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Only value 20 should remain after on-load processing of move_out - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) refute_received {:materializer_changes, _, _} end @@ -1006,7 +1013,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Value 10 should still be present because key "2" still has it - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -1037,7 +1044,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do Materializer.new_changes(ctx, range) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) {range, _writer} = Storage.append_control_message!( @@ -1047,7 +1054,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do Materializer.new_changes(ctx, range) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() end end @@ -1067,7 +1074,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Row is not included because no disjunct has all positions active - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} end @@ -1084,7 +1091,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # First disjunct "hash_a/" has position 0 active → included - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} end @@ -1101,7 +1108,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} # Move-in at position 0 with value "hash_a" @@ -1110,7 +1117,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Now position 0 is true, first disjunct "hash_a/" is satisfied - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} end @@ -1127,7 +1134,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Move-out at position 0 - but position 1 still holds via second disjunct @@ -1136,7 +1143,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Row should still be included because disjunct "/hash_b" at position 1 is still true - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -1153,7 +1160,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Move-out at position 1 - now no disjunct holds @@ -1161,7 +1168,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 1, value: "hash_b"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -1178,7 +1185,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Move-in at position 1 - row was already included via position 0 @@ -1187,7 +1194,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # No value count change - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -1205,7 +1212,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Position 1 is false, so the disjunct is not satisfied - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} end @@ -1222,7 +1229,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} # Move-in at position 0 makes both positions active @@ -1230,7 +1237,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-in", patterns: [%{pos: 0, value: "hash_a"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} end @@ -1253,7 +1260,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) assert_receive {:materializer_changes, _, %{move_in: _}} # Move-out only for hash_x at position 0 @@ -1261,7 +1268,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "hash_x"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end end @@ -1372,7 +1379,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(mat_ctx) == MapSet.new([10]) + assert link_values(mat_ctx) == MapSet.new([10]) end end diff --git a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs index 150088c1d1..a822f40671 100644 --- a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs @@ -5,125 +5,115 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do @dep 0 + defp view(values), do: fn value -> value in values end + test "drops redundant move outs for values absent from the base view" do - queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_out: [{1, "1"}]}, MapSet.new()) + queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_out: [{1, "1"}]}, view([])) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "drops redundant move ins for values already present in the base view" do - queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_in: [{1, "1"}]}, MapSet.new([1])) + queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_in: [{1, "1"}]}, view([1])) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "cancels a pending move in with a later move out for the same value" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, MapSet.new()) - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new()) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, view([])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, view([])) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "cancels a pending move out with a later move in for the same value" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, view([1])) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "merges repeated move ins and keeps the terminal tuple" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "01"}]}, MapSet.new()) - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], move_out: []}, MapSet.new()) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "01"}]}, view([])) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], move_out: []}, view([])) - assert %MoveQueue{move_in: %{0 => {[{1, "1"}], _}}, move_out: empty_out} = queue - assert empty_out == %{} + assert {%{move_in_values: [{1, "1"}], move_out_values: []}, _queue} = MoveQueue.pop_next(queue) end test "merges repeated move outs and keeps the terminal tuple" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "01"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}], move_in: []}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "01"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}], move_in: []}, view([1])) - assert %MoveQueue{move_out: %{0 => {[{1, "1"}], _}}, move_in: empty_in} = queue - assert empty_in == %{} + assert {%{move_in_values: [], move_out_values: [{1, "1"}]}, _queue} = MoveQueue.pop_next(queue) end - test "orders surviving move outs before move ins" do + test "pop_next returns one combined batch per dep carrying both kinds" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, view([1])) + + assert { + %{ + dep_index: 0, + move_in_values: [{2, "2"}, {3, "3"}], + move_out_values: [{1, "1"}] + }, + queue + } = MoveQueue.pop_next(queue) - assert %MoveQueue{ - move_out: %{0 => {[{1, "1"}], _}}, - move_in: %{0 => {[{2, "2"}], _}} - } = queue + assert nil == MoveQueue.pop_next(queue) end - test "uses the provided base view when reducing buffering follow-up moves" do + test "carries the first from_time and the max to_time per dep" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_out: [{2, "2"}]}, MapSet.new([1])) + |> MoveQueue.enqueue( + @dep, + %{move_in: [{1, "1"}], from_time: 5, to_time: 6}, + view([]) + ) + |> MoveQueue.enqueue( + @dep, + %{move_in: [{2, "2"}], from_time: 6, to_time: 9}, + view([]) + ) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert {%{from_time: 5, to_time: 9}, _queue} = MoveQueue.pop_next(queue) end - test "pop_next returns the whole move out batch before the move in batch" do + test "accumulates txids from successive enqueues per dependency" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, MapSet.new([1])) - - assert {{:move_out, 0, [{1, "1"}], []}, queue} = MoveQueue.pop_next(queue) - assert queue.move_out == %{} - assert {[{2, "2"}, {3, "3"}], _} = Map.fetch!(queue.move_in, 0) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], txids: [10]}, view([])) + |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], txids: [20]}, view([])) - assert {{:move_in, 0, [{2, "2"}, {3, "3"}], []}, queue} = MoveQueue.pop_next(queue) - assert queue.move_out == %{} - assert queue.move_in == %{} - assert nil == MoveQueue.pop_next(queue) + assert {%{move_in_values: _, txids: [10, 20]}, _queue} = MoveQueue.pop_next(queue) end - test "accumulates txids from successive enqueues per dependency" do + test "pops the lowest-indexed dep first across deps" do queue = MoveQueue.new() - |> MoveQueue.enqueue( - @dep, - %{move_in: [{1, "1"}], txids: [10]}, - MapSet.new() - ) - |> MoveQueue.enqueue( - @dep, - %{move_in: [{2, "2"}], txids: [20]}, - MapSet.new() - ) + |> MoveQueue.enqueue(1, %{move_in: [{2, "2"}]}, view([])) + |> MoveQueue.enqueue(0, %{move_in: [{1, "1"}]}, view([])) - assert {{:move_in, 0, _, [10, 20]}, _queue} = MoveQueue.pop_next(queue) + assert {%{dep_index: 0}, queue} = MoveQueue.pop_next(queue) + assert {%{dep_index: 1}, _queue} = MoveQueue.pop_next(queue) end test "length counts queued values across both batches" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, view([1])) assert 3 == MoveQueue.length(queue) end diff --git a/packages/sync-service/test/electric/shapes/consumer_test.exs b/packages/sync-service/test/electric/shapes/consumer_test.exs index 40b27576c5..e53b9c2627 100644 --- a/packages/sync-service/test/electric/shapes/consumer_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer_test.exs @@ -2263,39 +2263,36 @@ defmodule Electric.Shapes.ConsumerTest do ] = get_log_items_from_storage(LogOffset.last_before_real_offsets(), shape_storage) end - test "consumer startup seeds the stack-scoped subquery index", ctx do + test "consumer startup registers with the stack-scoped subquery index", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView {shape_handle, _} = ShapeCache.get_or_create_shape_handle(@shape_with_subquery, ctx.stack_id) :started = ShapeCache.await_snapshot_start(shape_handle, ctx.stack_id) - # The consumer should have seeded the SubqueryIndex during initialization index = SubqueryIndex.for_stack(ctx.stack_id) assert index != nil - # The shape should be registered with positions (by Filter.add_shape) + # Filter.add_shape registers the outer shape with the index. After + # the consumer's initial materialisation lands, the shape is no + # longer in fallback, the dep view is ready at logical time 0, and + # the dep handle has been recorded for the shape. assert SubqueryIndex.has_positions?(index, shape_handle) + refute SubqueryIndex.fallback?(index, shape_handle) - # The shape should be marked ready (no longer in fallback) once - # the consumer has seeded the index. After await_snapshot_start returns - # the consumer has completed initialization including subquery seeding. - {:ok, _shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) - - # The consumer seeds the index via SubqueryIndex.for_stack, but the - # index is also modified by the Filter (which runs in the - # ShapeLogCollector process). Check that the shape has positions - # and that membership entries are correct (empty views for a fresh shape). - positions = SubqueryIndex.positions_for_shape(index, shape_handle) - assert length(positions) > 0 + {:ok, shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) + [dep_handle] = shape.shape_dependencies_handles - # Verify the index is accessible and has retained node registrations. - assert positions == SubqueryIndex.positions_for_shape(index, shape_handle) + assert MultiTimeView.ready?(index.multi_time_view, dep_handle) + assert MultiTimeView.current_time(index.multi_time_view, dep_handle) == 0 end - test "consumer steady dependency move_in adds value to the subquery index", ctx do + test "consumer steady dependency move_in advances the shape's subquery index time", + ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView parent = self() @@ -2317,10 +2314,12 @@ defmodule Electric.Shapes.ConsumerTest do :started = ShapeCache.await_snapshot_start(shape_handle, ctx.stack_id) index = SubqueryIndex.for_stack(ctx.stack_id) - {:ok, _shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) + mtv = index.multi_time_view + {:ok, shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) + [dep_handle] = shape.shape_dependencies_handles - # Before any dependency changes, the index has empty membership - refute SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) + # Before any dep moves: dep view is empty at logical time 0. + refute MultiTimeView.member?(mtv, dep_handle, 1, 0) # Send a new record for the dependency table to trigger a move_in ShapeLogCollector.handle_event( @@ -2334,14 +2333,12 @@ defmodule Electric.Shapes.ConsumerTest do ctx.stack_id ) - # Wait for the consumer to process the event and request a move_in query assert_receive {:query_requested, consumer_pid} - # During buffering, the value should have been added to the index - # (union for positive dependency: before ∪ after) - assert SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) + # While the move-in query is buffering, the dep view at time 0 is + # unchanged — value 1 is still not a member there. + refute MultiTimeView.member?(mtv, dep_handle, 1, 0) - # Complete the move_in query to transition back to steady state send(consumer_pid, {:pg_snapshot_known, {100, 300, []}}) shape_storage = Storage.for_shape(shape_handle, ctx.storage) @@ -2366,13 +2363,15 @@ defmodule Electric.Shapes.ConsumerTest do Lsn.from_integer(100) ) - # Allow the consumer to process the completion assert :ok = LsnTracker.broadcast_last_seen_lsn(ctx.stack_id, 100) ref = Shapes.Consumer.register_for_changes(ctx.stack_id, shape_handle) assert_receive {^ref, :new_changes, _offset}, @receive_timeout - # After move_in completes, value should still be in the index (now steady state) - assert SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) + # After splice the materializer's logical time has advanced past + # the move-in, and value 1 is now visible at the new current time. + current = MultiTimeView.current_time(mtv, dep_handle) + assert current > 0 + assert MultiTimeView.member?(mtv, dep_handle, 1, current) end test "consumer cleanup removes shape rows from the subquery index", ctx do diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/compactor_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/compactor_test.exs new file mode 100644 index 0000000000..04cd05ea8b --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/compactor_test.exs @@ -0,0 +1,61 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.CompactorTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.Compactor + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor + + setup do + stack_id = "compactor-test-#{System.unique_integer([:positive])}" + Electric.ProcessRegistry.start_link(stack_id: stack_id) + {:ok, _} = ProgressMonitor.start_link(stack_id: stack_id) + {:ok, compactor} = Compactor.start_link(stack_id: stack_id, interval_ms: 3_600_000) + + mtv = MultiTimeView.new(stack_id: stack_id) + %{stack_id: stack_id, mtv: mtv, compactor: compactor} + end + + test "advances min_required_time and drops empty histories from MTV", %{ + stack_id: stack_id, + mtv: mtv + } do + MultiTimeView.init_subquery(mtv, :sq, [1, 2]) + MultiTimeView.mark_ready(mtv, :sq) + MultiTimeView.mark_out(mtv, :sq, 1, 5) + MultiTimeView.mark_in(mtv, :sq, 3, 7) + + :ok = ProgressMonitor.register_consumer(stack_id, :sq, "shape-a", self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 6, :sq, "shape-a") + + :ok = Compactor.compact_now(stack_id) + + # Value 1 was out for the entire retained window (>= 7) so its row is gone. + assert MultiTimeView.values(mtv, :sq) |> Enum.sort() == [2, 3] + end + + test "GCs positive-routing rows for values whose history compacts away", %{ + stack_id: stack_id, + mtv: _mtv + } do + # Build a real SubqueryIndex (with its own MTV) so add_positive_route / + # remove_positive_route can be exercised end-to-end. The compactor finds + # the index via SubqueryIndex.for_stack/1. + _index = SubqueryIndex.new(stack_id: stack_id) + index = SubqueryIndex.for_stack(stack_id) + mtv = index.multi_time_view + + MultiTimeView.init_subquery(mtv, :sq2, [10]) + MultiTimeView.mark_ready(mtv, :sq2) + MultiTimeView.mark_out(mtv, :sq2, 10, 4) + + SubqueryIndex.add_positive_route(index, :sq2, 10) + + :ok = ProgressMonitor.register_consumer(stack_id, :sq2, "shape-b", self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 5, :sq2, "shape-b") + + :ok = Compactor.compact_now(stack_id) + + refute :ets.member(index.table, {:positive, :sq2, 10}) + end +end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/history_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/history_test.exs new file mode 100644 index 0000000000..5d69a1de24 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/history_test.exs @@ -0,0 +1,177 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.HistoryTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.History + + describe "new/0" do + test "is the empty history — member at every retained time" do + assert History.new() == [] + end + end + + describe "member?/2" do + test "[] is in at every time" do + assert History.member?([], 0) + assert History.member?([], 1_000) + end + + test "nil is out at every time" do + refute History.member?(nil, 0) + refute History.member?(nil, 1_000) + end + + test "[:out, t] is out before t and in from t onwards" do + h = [:out, 9] + + refute History.member?(h, 0) + refute History.member?(h, 8) + assert History.member?(h, 9) + assert History.member?(h, 100) + end + + test "[:in, t] is in before t and out from t onwards" do + h = [:in, 9] + + assert History.member?(h, 0) + assert History.member?(h, 8) + refute History.member?(h, 9) + refute History.member?(h, 100) + end + + test "[:out, a, b] toggles in at a then out at b" do + h = [:out, 9, 11] + + refute History.member?(h, 8) + assert History.member?(h, 9) + assert History.member?(h, 10) + refute History.member?(h, 11) + refute History.member?(h, 100) + end + + test "[:in, a, b] toggles out at a then in at b" do + h = [:in, 9, 11] + + assert History.member?(h, 8) + refute History.member?(h, 9) + refute History.member?(h, 10) + assert History.member?(h, 11) + assert History.member?(h, 100) + end + end + + describe "member_at_some_time?/1" do + test "true for any non-nil history" do + assert History.member_at_some_time?([]) + assert History.member_at_some_time?([:out, 9]) + assert History.member_at_some_time?([:in, 9]) + assert History.member_at_some_time?([:out, 9, 11]) + assert History.member_at_some_time?([:in, 9, 11]) + end + + test "false for nil" do + refute History.member_at_some_time?(nil) + end + end + + describe "member_at_all_times?/1" do + test "true only for the empty history" do + assert History.member_at_all_times?([]) + end + + test "false for any history with toggles, and for nil" do + refute History.member_at_all_times?(nil) + refute History.member_at_all_times?([:out, 9]) + refute History.member_at_all_times?([:in, 9]) + refute History.member_at_all_times?([:out, 9, 11]) + refute History.member_at_all_times?([:in, 9, 11]) + end + end + + describe "mark_in/2" do + test "promotes nil to [:out, t] — a value seen for the first time" do + assert History.mark_in(nil, 5) == [:out, 5] + end + + test "leaves [] alone — already in" do + assert History.mark_in([], 5) == [] + end + + test "appends a toggle when current state is :out" do + assert History.mark_in([:in, 5], 10) == [:in, 5, 10] + assert History.mark_in([:out, 5, 8], 10) == [:out, 5, 8, 10] + end + + test "is a no-op when current state is already :in" do + assert History.mark_in([:out, 5], 10) == [:out, 5] + assert History.mark_in([:in, 5, 8], 10) == [:in, 5, 8] + end + end + + describe "mark_out/2" do + test "is a no-op on nil — there is nothing to remove" do + assert History.mark_out(nil, 5) == nil + end + + test "transitions [] to [:in, t] — out of the always-in baseline" do + assert History.mark_out([], 5) == [:in, 5] + end + + test "appends a toggle when current state is :in" do + assert History.mark_out([:out, 5], 10) == [:out, 5, 10] + assert History.mark_out([:in, 5, 8], 10) == [:in, 5, 8, 10] + end + + test "is a no-op when current state is already :out" do + assert History.mark_out([:in, 5], 10) == [:in, 5] + assert History.mark_out([:out, 5, 8], 10) == [:out, 5, 8] + end + end + + describe "compact/2" do + test "[] always stays []" do + assert History.compact([], 0) == [] + assert History.compact([], 1_000_000) == [] + end + + test "nil always stays nil" do + assert History.compact(nil, 0) == nil + assert History.compact(nil, 1_000_000) == nil + end + + test "keeps everything when min_required_time precedes the first toggle" do + assert History.compact([:out, 9, 11], 8) == [:out, 9, 11] + assert History.compact([:in, 9], 0) == [:in, 9] + end + + test "folds a toggle at min_required_time into the initial state" do + # [:out, 9]: out before 9, in from 9 onwards. + # retain from 9 onwards -> always in. + assert History.compact([:out, 9], 9) == [] + + # [:in, 11]: in before 11, out from 11 onwards. + # retain from 11 onwards -> always out -> row can be deleted. + assert History.compact([:in, 11], 11) == nil + end + + test "preserves membership across folded toggles" do + # [:out, 9, 11]: out before 9, in from 9..10, out from 11. + # retain from 10 onwards: at time 10 value is in, flips out at 11. + assert History.compact([:out, 9, 11], 10) == [:in, 11] + + # [:in, 9, 11]: in before 9, out from 9..10, in from 11. + # retain from 10 onwards: at time 10 value is out, flips in at 11. + assert History.compact([:in, 9, 11], 10) == [:out, 11] + end + + test "returns nil when retained window is entirely out" do + assert History.compact([:out, 9, 11], 12) == nil + assert History.compact([:in, 5], 10) == nil + end + + test "returns [] when retained window is entirely in" do + # [:out, 9, 11, 20]: out, in at 9, out at 11, in at 20. + # retain from 20 onwards -> always in. + assert History.compact([:out, 9, 11, 20], 20) == [] + end + end +end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/multi_time_view_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/multi_time_view_test.exs new file mode 100644 index 0000000000..1bc016d010 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/multi_time_view_test.exs @@ -0,0 +1,289 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeViewTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + + setup do + %{view: MultiTimeView.new()} + end + + describe "init_subquery/3" do + test "starts the subquery at logical time 0", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + assert MultiTimeView.current_time(view, :s7) == 0 + end + + test "makes the initial values members at time 0", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + + assert MultiTimeView.member?(view, :s7, 10, 0) + assert MultiTimeView.member?(view, :s7, 20, 0) + end + + test "values outside the initial set are not members", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + refute MultiTimeView.member?(view, :s7, 30, 0) + end + + test "does not mark the subquery as ready", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + refute MultiTimeView.ready?(view, :s7) + end + + test "keeps subqueries isolated from each other", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.init_subquery(view, :s8, [20]) + + assert MultiTimeView.member?(view, :s7, 10, 0) + refute MultiTimeView.member?(view, :s7, 20, 0) + assert MultiTimeView.member?(view, :s8, 20, 0) + refute MultiTimeView.member?(view, :s8, 10, 0) + end + end + + describe "mark_ready/2 and ready?/2" do + test "an unknown subquery is not ready", %{view: view} do + refute MultiTimeView.ready?(view, :s7) + end + + test "a subquery is not ready immediately after init", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.ready?(view, :s7) + end + + test "becomes ready after mark_ready", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_ready(view, :s7) + assert MultiTimeView.ready?(view, :s7) + end + end + + describe "mark_in/4" do + test "adds a value as a member from the transition time onwards", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 30, 1) + + refute MultiTimeView.member?(view, :s7, 30, 0) + assert MultiTimeView.member?(view, :s7, 30, 1) + assert MultiTimeView.member?(view, :s7, 30, 5) + end + + test "advances the current logical time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 30, 3) + + assert MultiTimeView.current_time(view, :s7) == 3 + end + + test "is a no-op when the value is already a member", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 10, 1) + + assert MultiTimeView.member_at_all_times?(view, :s7, 10) + end + end + + describe "mark_out/4" do + test "removes a value from the transition time onwards", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 2) + + assert MultiTimeView.member?(view, :s7, 20, 0) + assert MultiTimeView.member?(view, :s7, 20, 1) + refute MultiTimeView.member?(view, :s7, 20, 2) + refute MultiTimeView.member?(view, :s7, 20, 5) + end + + test "advances the current logical time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 2) + + assert MultiTimeView.current_time(view, :s7) == 2 + end + + test "is a no-op when the value was never a member", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_out(view, :s7, 99, 1) + + refute MultiTimeView.member_at_some_time?(view, :s7, 99) + end + end + + describe "member?/4" do + test "is false for values never seen", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.member?(view, :s7, 99, 0) + end + + test "tracks add-then-remove transitions correctly", %{view: view} do + MultiTimeView.init_subquery(view, :s7, []) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 3) + + refute MultiTimeView.member?(view, :s7, 30, 0) + assert MultiTimeView.member?(view, :s7, 30, 1) + assert MultiTimeView.member?(view, :s7, 30, 2) + refute MultiTimeView.member?(view, :s7, 30, 3) + end + end + + describe "member_at_some_time?/3" do + test "true for values currently present", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + assert MultiTimeView.member_at_some_time?(view, :s7, 10) + end + + test "true for values that were members at any retained time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, []) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 2) + + # 30 is no longer present at the current time, but is still retained. + assert MultiTimeView.member_at_some_time?(view, :s7, 30) + end + + test "false for values never seen", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.member_at_some_time?(view, :s7, 99) + end + end + + describe "member_at_all_times?/3" do + test "true when the value has no toggles in the retained window", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + assert MultiTimeView.member_at_all_times?(view, :s7, 10) + end + + test "false once a transition has been recorded", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_out(view, :s7, 10, 1) + + refute MultiTimeView.member_at_all_times?(view, :s7, 10) + end + + test "false for values never seen", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.member_at_all_times?(view, :s7, 99) + end + end + + describe "values/2" do + test "returns every value with retained membership", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_in(view, :s7, 30, 1) + + assert MultiTimeView.values(view, :s7) |> Enum.sort() == [10, 20, 30] + end + + test "includes values that have been removed but are still retained", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 1) + + assert MultiTimeView.values(view, :s7) |> Enum.sort() == [10, 20] + end + + test "returns an empty list for an unknown subquery", %{view: view} do + assert MultiTimeView.values(view, :unknown) == [] + end + end + + describe "values/3" do + test "returns only members at the given logical time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 20, 2) + + assert MultiTimeView.values(view, :s7, 0) |> Enum.sort() == [10, 20] + assert MultiTimeView.values(view, :s7, 1) |> Enum.sort() == [10, 20, 30] + assert MultiTimeView.values(view, :s7, 2) |> Enum.sort() == [10, 30] + end + end + + describe "current_time/2" do + test "is nil for an unknown subquery", %{view: view} do + assert MultiTimeView.current_time(view, :unknown) == nil + end + + test "is the highest time written so far", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 4) + + assert MultiTimeView.current_time(view, :s7) == 4 + end + end + + describe "set_min_required_time/3" do + test "folds toggles at or before the new min into the initial state", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_out(view, :s7, 10, 1) + MultiTimeView.mark_in(view, :s7, 10, 3) + + MultiTimeView.set_min_required_time(view, :s7, 3) + + # Value 10 is in for the entire retained window after compaction. + assert MultiTimeView.member_at_all_times?(view, :s7, 10) + end + + test "drops rows for values that are out for the whole retained window", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 1) + + MultiTimeView.set_min_required_time(view, :s7, 2) + + refute MultiTimeView.member_at_some_time?(view, :s7, 20) + assert MultiTimeView.values(view, :s7) == [10] + end + + test "preserves membership for retained times after compaction", %{view: view} do + MultiTimeView.init_subquery(view, :s7, []) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 5) + + MultiTimeView.set_min_required_time(view, :s7, 3) + + # 30 was in from time 1, including at time 3 (the new min) and 4. + assert MultiTimeView.member?(view, :s7, 30, 3) + assert MultiTimeView.member?(view, :s7, 30, 4) + refute MultiTimeView.member?(view, :s7, 30, 5) + end + end + + describe "remove_subquery/2" do + test "deletes all rows for the subquery", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_ready(view, :s7) + MultiTimeView.mark_in(view, :s7, 30, 1) + + MultiTimeView.remove_subquery(view, :s7) + + refute MultiTimeView.ready?(view, :s7) + assert MultiTimeView.values(view, :s7) == [] + assert MultiTimeView.current_time(view, :s7) == nil + end + + test "leaves other subqueries untouched", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.init_subquery(view, :s8, [20]) + MultiTimeView.mark_ready(view, :s8) + + MultiTimeView.remove_subquery(view, :s7) + + assert MultiTimeView.values(view, :s8) == [20] + assert MultiTimeView.ready?(view, :s8) + end + end + + describe "for_stack/1" do + test "returns the table when one was created for the stack" do + stack_id = "stack-#{System.unique_integer([:positive])}" + _view = MultiTimeView.new(stack_id: stack_id) + + assert MultiTimeView.for_stack(stack_id) != nil + end + + test "returns nil when no table exists for the stack" do + assert MultiTimeView.for_stack("nope-#{System.unique_integer([:positive])}") == nil + end + end +end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/progress_monitor_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/progress_monitor_test.exs new file mode 100644 index 0000000000..7ed4f75e75 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/progress_monitor_test.exs @@ -0,0 +1,199 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitorTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor + + setup do + stack_id = "stack-#{System.unique_integer([:positive])}" + start_supervised!({ProgressMonitor, stack_id: stack_id}) + %{stack_id: stack_id} + end + + describe "register_consumer/5" do + test "sets the min required time to the consumer's initial time", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 3) + assert ProgressMonitor.min_required_time(stack_id, :s7) == 3 + end + + test "a second, earlier consumer lowers the min", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 5) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 2) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 2 + end + + test "isolates min by subquery", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 1) + :ok = ProgressMonitor.register_consumer(stack_id, :s8, :shape_a, self(), 4) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 1 + assert ProgressMonitor.min_required_time(stack_id, :s8) == 4 + end + + test "re-registering replaces the previous entry", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 1) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 4) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 4 + end + + test "marks the consumer as registered", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + assert ProgressMonitor.registered?(stack_id, :s7, :shape_a) + refute ProgressMonitor.registered?(stack_id, :s7, :shape_b) + end + end + + describe "notify_processed_up_to/4" do + test "advances the min when the limiting consumer moves on", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 0, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 1 + end + + test "does not advance the min when a slower consumer pins an older time", %{ + stack_id: stack_id + } do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 0) + + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 0, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 0 + end + + test "is monotonic — an earlier time does not regress the required time", %{ + stack_id: stack_id + } do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 5, :s7, :shape_a) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 2, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 6 + end + + test "is a no-op for an unknown consumer (race-safe)", %{stack_id: stack_id} do + assert :ok = ProgressMonitor.notify_processed_up_to(stack_id, 0, :s7, :shape_a) + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + end + + describe "unregister_consumer/3" do + test "releases the pinned time", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 5) + + :ok = ProgressMonitor.unregister_consumer(stack_id, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 5 + refute ProgressMonitor.registered?(stack_id, :s7, :shape_a) + end + + test "is idempotent", %{stack_id: stack_id} do + :ok = ProgressMonitor.unregister_consumer(stack_id, :s7, :shape_a) + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + + test "clears the min when the last consumer unregisters", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.unregister_consumer(stack_id, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + end + + describe "consumer process death" do + test "automatically releases the pinned time when the consumer pid dies", %{ + stack_id: stack_id + } do + {pid, ref} = spawn_consumer() + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, pid, 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 5) + + stop_consumer(pid, ref) + + assert eventually(fn -> + ProgressMonitor.min_required_time(stack_id, :s7) == 5 + end) + + refute ProgressMonitor.registered?(stack_id, :s7, :shape_a) + end + + test "releases every registration for that pid", %{stack_id: stack_id} do + {pid, ref} = spawn_consumer() + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, pid, 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s8, :shape_a, pid, 0) + + stop_consumer(pid, ref) + + assert eventually(fn -> + not ProgressMonitor.registered?(stack_id, :s7, :shape_a) and + not ProgressMonitor.registered?(stack_id, :s8, :shape_a) + end) + end + end + + describe "for_stack/1" do + test "returns the ETS table name for a started stack", %{stack_id: stack_id} do + assert ProgressMonitor.for_stack(stack_id) != nil + end + + test "returns nil when no monitor exists for the stack" do + assert ProgressMonitor.for_stack("nope-#{System.unique_integer([:positive])}") == nil + end + end + + describe "min_required_time/2" do + test "returns nil when no consumers are registered", %{stack_id: stack_id} do + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + + test "can be read by table name", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 2) + + table = ProgressMonitor.for_stack(stack_id) + assert ProgressMonitor.min_required_time(table, :s7) == 2 + end + end + + defp spawn_consumer do + parent = self() + + pid = + spawn(fn -> + ref = make_ref() + send(parent, {:consumer_ready, ref, self()}) + + receive do + {:stop, ^ref} -> :ok + end + end) + + receive do + {:consumer_ready, ref, ^pid} -> {pid, ref} + end + end + + defp stop_consumer(pid, ref) do + mon = Process.monitor(pid) + send(pid, {:stop, ref}) + + receive do + {:DOWN, ^mon, :process, ^pid, _} -> :ok + end + end + + defp eventually(fun, attempts \\ 50, sleep_ms \\ 5) do + if fun.() do + true + else + if attempts <= 0 do + false + else + Process.sleep(sleep_ms) + eventually(fun, attempts - 1, sleep_ms) + end + end + end +end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs new file mode 100644 index 0000000000..4317ddc518 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs @@ -0,0 +1,581 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do + use ExUnit.Case, async: true + + alias Electric.Replication.Eval.Parser + alias Electric.Replication.Eval.Parser.{Func, Ref} + alias Electric.Shapes.DnfPlan + alias Electric.Shapes.Filter + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.WhereCondition + + @subquery_ref ["$sublink", "0"] + @other_subquery_ref ["$sublink", "1"] + @field "par_id" + @other_field "id" + @dep_handle_a "dep_a" + @dep_handle_b "dep_b" + + setup do + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + index = filter.subquery_index + + %{ + filter: filter, + index: index, + table: index.table, + mtv: index.multi_time_view, + condition_id: condition_id + } + end + + describe "register_shape/4" do + test "marks the shape as fallback for both polarities", %{index: index} do + SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) + + assert SubqueryIndex.fallback?(index, "s1") + + SubqueryIndex.register_shape( + index, + "s2", + make_plan(polarity: :negated), + [@dep_handle_a] + ) + + assert SubqueryIndex.fallback?(index, "s2") + end + end + + describe "unregister_shape/2" do + test "drops dep_handle and fallback rows", %{index: index} do + SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) + SubqueryIndex.unregister_shape(index, "s1") + + refute SubqueryIndex.fallback?(index, "s1") + end + end + + describe "add_shape/5 (positive)" do + test "creates a single child for the first shape in a group + subquery", %{ + filter: filter, + index: index, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + + assert SubqueryIndex.has_positions?(index, "s1") + end + + test "two shapes sharing the same group + subquery share a single child", %{ + filter: filter, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + register_node_shape(filter, condition_id, "s2") + + children = + :ets.match(table, {{:shape_child, "s1"}, {:"$1", :_}}) ++ + :ets.match(table, {{:shape_child, "s2"}, {:"$1", :_}}) + + assert children |> Enum.uniq() |> length() == 1 + end + + test "shapes with the same group but different subqueries land on different children", %{ + filter: filter, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1", dep_handles: [@dep_handle_a]) + + register_node_shape(filter, condition_id, "s2", dep_handles: [@dep_handle_b]) + + [[c1]] = :ets.match(table, {{:shape_child, "s1"}, {:"$1", :_}}) + [[c2]] = :ets.match(table, {{:shape_child, "s2"}, {:"$1", :_}}) + + assert c1 != c2 + end + + test "first-child creation seeds positive routing from MultiTimeView", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10, 20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + + [[_, group_id]] = + :ets.match(table, {{:group, condition_id, @field, :"$1"}, :"$2"}) + + assert :ets.match(table, {{:positive, group_id, :"$1"}, :_}) |> Enum.sort() == + [[10], [20]] |> Enum.sort() + end + + test "adding a second shape to an existing child does not duplicate positive routes", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10, 20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + + before_routes = :ets.match(table, {{:positive, :_, :_}, :_}) + register_node_shape(filter, condition_id, "s2") + after_routes = :ets.match(table, {{:positive, :_, :_}, :_}) + + assert Enum.sort(before_routes) == Enum.sort(after_routes) + end + end + + describe "add_shape/5 (negated)" do + test "stores one group-keyed routing row regardless of subquery value count", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [1, 2, 3, 4, 5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "n1", polarity: :negated) + + assert :ets.match(table, {{:negated, :_}, :_}) |> length() == 1 + assert :ets.match(table, {{:positive, :_, :_}, :_}) == [] + end + end + + describe "affected_shapes/4 (positive routing)" do + test "routes shapes by value-keyed positive routing", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + SubqueryIndex.mark_ready(index, "s1") + + assert MapSet.new(["s1"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5"} + ) + end + + test "diverging consumer times: filter over-routes both shapes", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do + # The filter is time-unaware by design: it routes on the retained + # window, so a value that is a member at *some* time fans out to every + # attached shape. Per-consumer correctness is left to + # `Shape.convert_change`, which evaluates membership at the consumer's + # own logical time(s) — including both `from_time` and `to_time` + # during a buffered move. + MultiTimeView.init_subquery(mtv, @dep_handle_a, []) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s_old") + register_node_shape(filter, condition_id, "s_new") + + MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) + SubqueryIndex.add_positive_route(index, @dep_handle_a, 30) + + SubqueryIndex.mark_ready(index, "s_old") + SubqueryIndex.mark_ready(index, "s_new") + + assert MapSet.new(["s_old", "s_new"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "30"} + ) + end + + test "returns only shapes registered under the requested field key", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "local") + + register_node_shape(filter, condition_id, "other_field", + field: @other_field, + subquery_ref: @other_subquery_ref, + dep_handles: [@dep_handle_a] + ) + + for shape <- ~w(local other_field) do + SubqueryIndex.mark_ready(index, shape) + end + + assert MapSet.new(["local"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5", "id" => "5"} + ) + end + + test "delegates an and_where tail to the child WhereCondition", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "tail", + and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) + ) + + SubqueryIndex.mark_ready(index, "tail") + + assert MapSet.new(["tail"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5", "name" => "keep_me"} + ) + + assert MapSet.new() == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5", "name" => "discard"} + ) + end + + test "routes unseeded shapes via the per-node fallback rows", %{ + filter: filter, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "unseeded") + + assert MapSet.new(["unseeded"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "999"} + ) + end + end + + describe "affected_shapes/4 (negated routing)" do + test "prunes the child when the value is a member at every retained time", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "n1", polarity: :negated) + SubqueryIndex.mark_ready(index, "n1") + + assert MapSet.new() == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5"} + ) + + assert MapSet.new(["n1"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "99"} + ) + end + + test "keeps the child when the value is only a member at some retained time", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, []) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "n1", polarity: :negated) + MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) + + SubqueryIndex.mark_ready(index, "n1") + + assert MapSet.new(["n1"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "30"} + ) + end + end + + describe "all_shape_ids/3" do + test "returns shapes for the requested field key only", %{ + filter: filter, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + + register_node_shape(filter, condition_id, "s2", + and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) + ) + + register_node_shape(filter, condition_id, "other", + field: @other_field, + subquery_ref: @other_subquery_ref + ) + + assert MapSet.new(["s1", "s2"]) == + SubqueryIndex.all_shape_ids(filter, condition_id, @field) + end + end + + describe "add_positive_route/3 and remove_positive_route/3" do + test "mutate routing without touching per-shape rows", %{ + filter: filter, + index: index, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, []) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + + shape_rows_before = :ets.lookup(table, {:shape_child, "s1"}) + + SubqueryIndex.add_positive_route(index, @dep_handle_a, 42) + + [[group_id]] = :ets.match(table, {{:group, condition_id, @field, :positive}, :"$1"}) + assert :ets.lookup(table, {:positive, group_id, 42}) |> length() == 1 + + SubqueryIndex.remove_positive_route(index, @dep_handle_a, 42) + assert :ets.lookup(table, {:positive, group_id, 42}) == [] + + assert :ets.lookup(table, {:shape_child, "s1"}) == shape_rows_before + end + end + + describe "remove_shape/5" do + test "leaves shared child intact when other shapes remain", %{ + filter: filter, + index: index, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + register_node_shape(filter, condition_id, "s2") + + assert :ok = + SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) + + assert MapSet.new(["s2"]) == SubqueryIndex.all_shape_ids(filter, condition_id, @field) + refute SubqueryIndex.has_positions?(index, "s1") + end + + test "cleans the child and positive routes when the last shape leaves", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10, 20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + register_node_shape(filter, condition_id, "s2") + + assert :ok = + SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) + + assert :deleted = + SubqueryIndex.remove_shape(filter, condition_id, "s2", subquery_optimisation(), []) + + assert :ets.match(table, {{:positive, :_, :_}, :_}) == [] + assert :ets.match(table, {{:child_meta, :_}, :_}) == [] + end + + test "tracks emptiness per field key", %{ + filter: filter, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + + register_node_shape(filter, condition_id, "s2", + field: @other_field, + subquery_ref: @other_subquery_ref + ) + + assert :deleted = + SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) + + assert MapSet.new(["s2"]) == + SubqueryIndex.all_shape_ids(filter, condition_id, @other_field) + + assert :deleted = + SubqueryIndex.remove_shape( + filter, + condition_id, + "s2", + subquery_optimisation(field: @other_field, subquery_ref: @other_subquery_ref), + [] + ) + end + end + + describe "remove_subquery/3" do + test "cascades to every child and participant for that subquery only", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10]) + MultiTimeView.init_subquery(mtv, @dep_handle_b, [20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + MultiTimeView.mark_ready(mtv, @dep_handle_b) + + register_node_shape(filter, condition_id, "s_a", dep_handles: [@dep_handle_a]) + + register_node_shape(filter, condition_id, "s_b", + field: @other_field, + subquery_ref: @other_subquery_ref, + dep_handles: [@dep_handle_b] + ) + + SubqueryIndex.remove_subquery(index, @dep_handle_a) + + refute SubqueryIndex.has_positions?(index, "s_a") + assert SubqueryIndex.has_positions?(index, "s_b") + refute MultiTimeView.member_at_some_time?(mtv, @dep_handle_a, 10) + assert MultiTimeView.member_at_some_time?(mtv, @dep_handle_b, 20) + end + end + + describe "mark_ready/2 and fallback?/2" do + test "mark_ready clears fallback and per-node fallback rows", %{ + filter: filter, + index: index, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + assert SubqueryIndex.fallback?(index, "s1") + assert :ets.match(table, {{:node_fallback, :_, :_}, {:_, "s1"}}) != [] + + SubqueryIndex.mark_ready(index, "s1") + + refute SubqueryIndex.fallback?(index, "s1") + assert :ets.match(table, {{:node_fallback, :_, :_}, {:_, "s1"}}) == [] + end + end + + describe "for_stack/1" do + test "returns the index when one was created for the stack" do + stack_id = "test-stack-#{System.unique_integer([:positive])}" + _index = SubqueryIndex.new(stack_id: stack_id) + assert %SubqueryIndex{} = SubqueryIndex.for_stack(stack_id) + end + + test "returns nil for unknown stack" do + assert SubqueryIndex.for_stack("nonexistent-stack-#{System.unique_integer([:positive])}") == + nil + end + end + + defp register_node_shape(filter, condition_id, shape_id, opts \\ []) do + dep_handles = Keyword.get(opts, :dep_handles, [@dep_handle_a]) + SubqueryIndex.register_shape(filter.subquery_index, shape_id, make_plan(opts), dep_handles) + + :ok = + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + subquery_optimisation(opts), + [] + ) + end + + defp subquery_optimisation(opts \\ []) do + field = Keyword.get(opts, :field, @field) + + %{ + operation: "subquery", + field: field, + testexpr: %Ref{path: [field], type: :int8}, + subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), + dep_index: Keyword.get(opts, :dep_index, 0), + polarity: Keyword.get(opts, :polarity, :positive), + and_where: Keyword.get(opts, :and_where) + } + end + + defp make_plan(opts \\ []) do + polarity = Keyword.get(opts, :polarity, :positive) + dep_index = Keyword.get(opts, :dep_index, 0) + subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) + field = Keyword.get(opts, :field, @field) + + testexpr = %Ref{path: [field], type: :int8} + ref = %Ref{path: subquery_ref, type: {:array, :int8}} + + ast = %Func{ + name: "sublink_membership_check", + args: [testexpr, ref], + type: :bool + } + + %DnfPlan{ + disjuncts: [], + disjuncts_positions: [], + position_count: 1, + positions: %{ + 0 => %{ + ast: ast, + sql: "fake", + is_subquery: true, + negated: polarity == :negated, + dependency_index: dep_index, + subquery_ref: subquery_ref, + tag_columns: [field] + } + }, + dependency_positions: %{dep_index => [0]}, + dependency_disjuncts: %{}, + dependency_polarities: %{dep_index => polarity} + } + end + + defp where(query, refs), do: Parser.parse_and_validate_expression!(query, refs: refs) +end diff --git a/packages/sync-service/test/electric/shapes/filter/subquery_index_test.exs b/packages/sync-service/test/electric/shapes/filter/subquery_index_test.exs deleted file mode 100644 index bc5987ee59..0000000000 --- a/packages/sync-service/test/electric/shapes/filter/subquery_index_test.exs +++ /dev/null @@ -1,228 +0,0 @@ -defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do - use ExUnit.Case - - alias Electric.Replication.Eval.Parser.{Func, Ref} - alias Electric.Shapes.DnfPlan - alias Electric.Shapes.Filter - alias Electric.Shapes.Filter.Indexes.SubqueryIndex - alias Electric.Shapes.Filter.WhereCondition - - @subquery_ref ["$sublink", "0"] - @field "par_id" - - setup do - filter = Filter.new() - condition_id = make_ref() - WhereCondition.init(filter, condition_id) - - %{ - filter: filter, - table: Filter.subquery_index(filter), - condition_id: condition_id - } - end - - describe "shape-level metadata" do - test "register_shape stores polarity and fallback used by exact evaluation", %{table: table} do - SubqueryIndex.register_shape(table, "s1", make_plan()) - - assert SubqueryIndex.fallback?(table, "s1") - assert SubqueryIndex.membership_or_fallback?(table, "s1", @subquery_ref, 99) - - SubqueryIndex.register_shape(table, "s2", make_plan(polarity: :negated)) - - refute SubqueryIndex.membership_or_fallback?(table, "s2", @subquery_ref, 99) - end - - test "unregister_shape removes exact membership metadata", %{table: table} do - SubqueryIndex.register_shape(table, "s1", make_plan()) - SubqueryIndex.add_value(table, "s1", @subquery_ref, 0, 5) - - assert SubqueryIndex.member?(table, "s1", @subquery_ref, 5) - - SubqueryIndex.unregister_shape(table, "s1") - - refute SubqueryIndex.member?(table, "s1", @subquery_ref, 5) - refute SubqueryIndex.fallback?(table, "s1") - end - end - - describe "node registration and updates" do - test "add_shape registers node mappings for a dependency", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - - assert SubqueryIndex.has_positions?(table, "s1") - assert [{^condition_id, @field}] = SubqueryIndex.positions_for_shape(table, "s1") - end - - test "multiple shapes on the same node infer emptiness from node registrations", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - register_node_shape(filter, table, condition_id, "s2") - register_node_shape(filter, table, condition_id, "s3") - - assert [ - {{:node_shape, {^condition_id, @field}}, {"s1", 0, :positive, _, []}}, - {{:node_shape, {^condition_id, @field}}, {"s2", 0, :positive, _, []}}, - {{:node_shape, {^condition_id, @field}}, {"s3", 0, :positive, _, []}} - ] = Enum.sort(:ets.lookup(table, {:node_shape, {condition_id, @field}})) - - assert :ok = - SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) - - assert MapSet.new(["s2", "s3"]) == SubqueryIndex.all_shape_ids(filter, condition_id, @field) - - assert :ok = - SubqueryIndex.remove_shape(filter, condition_id, "s2", subquery_optimisation(), []) - - assert :deleted = - SubqueryIndex.remove_shape(filter, condition_id, "s3", subquery_optimisation(), []) - - assert [] == :ets.lookup(table, {:node_shape, {condition_id, @field}}) - assert [] == :ets.lookup(table, {:node_meta, {condition_id, @field}}) - end - - test "seed_membership updates node-local routing and exact membership", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - - SubqueryIndex.seed_membership(table, "s1", @subquery_ref, 0, MapSet.new([5])) - SubqueryIndex.mark_ready(table, "s1") - - assert SubqueryIndex.member?(table, "s1", @subquery_ref, 5) - - assert MapSet.new(["s1"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5"} - ) - end - - test "negated nodes use local complement semantics", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1", polarity: :negated) - - SubqueryIndex.seed_membership(table, "s1", @subquery_ref, 0, MapSet.new([5])) - SubqueryIndex.mark_ready(table, "s1") - - refute MapSet.member?( - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5"} - ), - "s1" - ) - - assert MapSet.member?( - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "99"} - ), - "s1" - ) - end - - test "remove_shape clears node registrations", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - SubqueryIndex.add_value(table, "s1", @subquery_ref, 0, 5) - - assert :deleted = - SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) - - refute SubqueryIndex.has_positions?(table, "s1") - - SubqueryIndex.unregister_shape(table, "s1") - - refute SubqueryIndex.fallback?(table, "s1") - end - end - - describe "stack lookup" do - test "stores and retrieves table ref by stack_id" do - table = SubqueryIndex.new(stack_id: "test-stack-123") - assert SubqueryIndex.for_stack("test-stack-123") == table - end - - test "returns nil for unknown stack" do - assert SubqueryIndex.for_stack("nonexistent-stack") == nil - end - end - - defp register_node_shape(filter, table, condition_id, shape_id, opts \\ []) do - SubqueryIndex.register_shape(table, shape_id, make_plan(opts)) - :ok = SubqueryIndex.add_shape(filter, condition_id, shape_id, subquery_optimisation(opts), []) - end - - defp subquery_optimisation(opts \\ []) do - field = Keyword.get(opts, :field, @field) - - %{ - operation: "subquery", - field: field, - testexpr: %Ref{path: [field], type: :int8}, - subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), - dep_index: Keyword.get(opts, :dep_index, 0), - polarity: Keyword.get(opts, :polarity, :positive), - and_where: Keyword.get(opts, :and_where) - } - end - - defp make_plan(opts \\ []) do - polarity = Keyword.get(opts, :polarity, :positive) - dep_index = Keyword.get(opts, :dep_index, 0) - subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) - field = Keyword.get(opts, :field, @field) - - testexpr = %Ref{path: [field], type: :int8} - ref = %Ref{path: subquery_ref, type: {:array, :int8}} - - ast = %Func{ - name: "sublink_membership_check", - args: [testexpr, ref], - type: :bool - } - - %DnfPlan{ - disjuncts: [], - disjuncts_positions: [], - position_count: 1, - positions: %{ - 0 => %{ - ast: ast, - sql: "fake", - is_subquery: true, - negated: polarity == :negated, - dependency_index: dep_index, - subquery_ref: subquery_ref, - tag_columns: [field] - } - }, - dependency_positions: %{dep_index => [0]}, - dependency_disjuncts: %{}, - dependency_polarities: %{dep_index => polarity} - } - end -end diff --git a/packages/sync-service/test/electric/shapes/filter/subquery_node_test.exs b/packages/sync-service/test/electric/shapes/filter/subquery_node_test.exs deleted file mode 100644 index 580c3394de..0000000000 --- a/packages/sync-service/test/electric/shapes/filter/subquery_node_test.exs +++ /dev/null @@ -1,235 +0,0 @@ -defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexNodeTest do - use ExUnit.Case - - alias Electric.Replication.Eval.Parser - alias Electric.Replication.Eval.Parser.{Func, Ref} - alias Electric.Shapes.DnfPlan - alias Electric.Shapes.Filter - alias Electric.Shapes.Filter.Indexes.SubqueryIndex - alias Electric.Shapes.Filter.WhereCondition - - @subquery_ref ["$sublink", "0"] - @field "par_id" - @other_field "id" - - setup do - filter = Filter.new() - condition_id = make_ref() - WhereCondition.init(filter, condition_id) - - %{ - filter: filter, - condition_id: condition_id, - reverse_index: Filter.subquery_index(filter) - } - end - - describe "affected_shapes/4" do - test "returns only shapes registered under the current field key", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "local_shape") - - register_node_shape(filter, reverse_index, condition_id, "other_field_shape", - field: @other_field - ) - - seed_shape(reverse_index, "local_shape", [5]) - seed_shape(reverse_index, "other_field_shape", [5]) - - assert MapSet.new(["local_shape"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5", "id" => "5"} - ) - end - - test "delegates matching candidates to the child where condition", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape( - filter, - reverse_index, - condition_id, - "shape_with_exact_tail", - and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) - ) - - seed_shape(reverse_index, "shape_with_exact_tail", [5]) - - assert MapSet.new(["shape_with_exact_tail"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5", "name" => "keep_me"} - ) - - assert MapSet.new() == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5", "name" => "discard"} - ) - end - - test "routes unseeded shapes once traversal reaches the node", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "unseeded_shape") - - assert MapSet.new(["unseeded_shape"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "999"} - ) - end - end - - describe "all_shape_ids/3" do - test "returns only the shape ids for the requested field key", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "shape1") - - register_node_shape( - filter, - reverse_index, - condition_id, - "shape2", - and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) - ) - - register_node_shape(filter, reverse_index, condition_id, "other_field_shape", - field: @other_field - ) - - assert MapSet.new(["shape1", "shape2"]) == - SubqueryIndex.all_shape_ids(filter, condition_id, @field) - end - end - - describe "remove_shape/4" do - test "tracks emptiness per field key", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "shape1") - register_node_shape(filter, reverse_index, condition_id, "shape2", field: @other_field) - - assert :deleted = - SubqueryIndex.remove_shape( - filter, - condition_id, - "shape1", - subquery_optimisation(), - [] - ) - - refute SubqueryIndex.has_positions?(reverse_index, "shape1") - - assert MapSet.new(["shape2"]) == - SubqueryIndex.all_shape_ids(filter, condition_id, @other_field) - - assert :deleted = - SubqueryIndex.remove_shape( - filter, - condition_id, - "shape2", - subquery_optimisation(field: @other_field), - [] - ) - end - end - - defp register_node_shape(filter, reverse_index, condition_id, shape_id, opts \\ []) do - SubqueryIndex.register_shape(reverse_index, shape_id, make_plan(opts)) - - :ok = - SubqueryIndex.add_shape( - filter, - condition_id, - shape_id, - subquery_optimisation(opts), - [] - ) - end - - defp seed_shape(reverse_index, shape_id, values) do - SubqueryIndex.seed_membership( - reverse_index, - shape_id, - @subquery_ref, - 0, - MapSet.new(values) - ) - - SubqueryIndex.mark_ready(reverse_index, shape_id) - end - - defp subquery_optimisation(opts \\ []) do - %{ - operation: "subquery", - field: Keyword.get(opts, :field, @field), - testexpr: %Ref{path: [Keyword.get(opts, :field, @field)], type: :int8}, - subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), - dep_index: Keyword.get(opts, :dep_index, 0), - polarity: Keyword.get(opts, :polarity, :positive), - and_where: Keyword.get(opts, :and_where) - } - end - - defp where(query, refs) do - Parser.parse_and_validate_expression!(query, refs: refs) - end - - defp make_plan(opts) do - polarity = Keyword.get(opts, :polarity, :positive) - dep_index = Keyword.get(opts, :dep_index, 0) - subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) - field = Keyword.get(opts, :field, @field) - - testexpr = %Ref{path: [field], type: :int8} - ref = %Ref{path: subquery_ref, type: {:array, :int8}} - - ast = %Func{ - name: "sublink_membership_check", - args: [testexpr, ref], - type: :bool - } - - %DnfPlan{ - disjuncts: [], - disjuncts_positions: [], - position_count: 1, - positions: %{ - 0 => %{ - ast: ast, - sql: "fake", - is_subquery: true, - negated: polarity == :negated, - dependency_index: dep_index, - subquery_ref: subquery_ref, - tag_columns: [field] - } - }, - dependency_positions: %{dep_index => [0]}, - dependency_disjuncts: %{}, - dependency_polarities: %{dep_index => polarity} - } - end -end diff --git a/packages/sync-service/test/electric/shapes/filter_test.exs b/packages/sync-service/test/electric/shapes/filter_test.exs index ca768df1ae..81af5ad0d1 100644 --- a/packages/sync-service/test/electric/shapes/filter_test.exs +++ b/packages/sync-service/test/electric/shapes/filter_test.exs @@ -630,33 +630,6 @@ defmodule Electric.Shapes.FilterTest do end) end - test "Filter.remove_shape/2 removes seeded subquery index state" do - filter = Filter.new() - state_before = snapshot_filter_ets(filter) - shape_id = "seeded-shape" - - shape = - Shape.new!("table", - where: "id IN (SELECT id FROM another_table)", - inspector: @inspector, - feature_flags: ["allow_subqueries"] - ) - - Filter.add_shape(filter, shape_id, shape) - - index = Filter.subquery_index(filter) - subquery_ref = ["$sublink", "0"] - - SubqueryIndex.seed_membership(index, shape_id, subquery_ref, 0, MapSet.new([5])) - SubqueryIndex.mark_ready(index, shape_id) - - assert snapshot_filter_ets(filter) != state_before - - Filter.remove_shape(filter, shape_id) - - assert snapshot_filter_ets(filter) == state_before - end - # Captures the full state of all ETS tables in a filter for comparison defp snapshot_filter_ets(filter) do %{ @@ -665,7 +638,8 @@ defmodule Electric.Shapes.FilterTest do where_cond: :ets.tab2list(filter.where_cond_table) |> Enum.sort(), eq_index: :ets.tab2list(filter.eq_index_table) |> Enum.sort(), incl_index: :ets.tab2list(filter.incl_index_table) |> Enum.sort(), - subquery_index: :ets.tab2list(filter.subquery_index) |> Enum.sort() + subquery_index: :ets.tab2list(filter.subquery_index.table) |> Enum.sort(), + multi_time_view: :ets.tab2list(filter.subquery_index.multi_time_view) |> Enum.sort() } end @@ -931,6 +905,8 @@ defmodule Electric.Shapes.FilterTest do end describe "subquery shapes routing in filter" do + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + import Support.DbSetup import Support.DbStructureSetup import Support.ComponentSetup @@ -943,6 +919,34 @@ defmodule Electric.Shapes.FilterTest do :with_sql_execute ] + # Test helper: emulate the pre-RFC `SubqueryIndex.seed_membership/5` + # against the new shared-view API. Looks up the subquery_id that + # `Filter.add_shape` stored for the shape's dependency (handles either + # a real dep_handle or the `{shape_handle, dep_index}` fallback that + # `register_shape` falls back to when `shape_dependencies_handles` + # isn't populated), seeds `MultiTimeView` with the values, and wires + # positive routing rows so `affected_shapes/2` can find each value. + defp seed_membership(filter, shape_id, _shape, subquery_ref, values) do + index = Filter.subquery_index(filter) + mtv = index.multi_time_view + dep_index = subquery_ref |> List.last() |> String.to_integer() + {subquery_id, _polarity} = SubqueryIndex.lookup_dep!(index, shape_id, dep_index) + + MultiTimeView.init_subquery(mtv, subquery_id, values) + MultiTimeView.mark_ready(mtv, subquery_id) + + for v <- values do + :ok = SubqueryIndex.add_positive_route(index, subquery_id, v) + end + + :ok + end + + # Test helper: emulate the pre-RFC per-value `SubqueryIndex.add_value/5`. + defp add_value(filter, shape_id, shape, subquery_ref, value) do + seed_membership(filter, shape_id, shape, subquery_ref, [value]) + end + @tag with_sql: [ "CREATE TABLE IF NOT EXISTS parent (id INT PRIMARY KEY)", "CREATE TABLE IF NOT EXISTS child (id INT PRIMARY KEY, par_id INT REFERENCES parent(id))" @@ -1127,7 +1131,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] for value <- [1, 2, 3] do - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, value) + add_value(filter, "shape1", shape, subquery_ref, value) end SubqueryIndex.mark_ready(index, "shape1") @@ -1170,9 +1174,17 @@ defmodule Electric.Shapes.FilterTest do "CREATE TABLE IF NOT EXISTS or_parent (id INT PRIMARY KEY)", "CREATE TABLE IF NOT EXISTS or_child (id INT PRIMARY KEY, par_id INT REFERENCES or_parent(id), value TEXT NOT NULL)" ] - test "mixed OR+subquery shape falls back to other_shapes verification", %{ - inspector: inspector - } do + test "mixed OR+subquery shape filters through other_shapes via the MTV conservative callback", + %{ + inspector: inspector + } do + # A mixed OR shape lands in the catch-all `other_shapes` path because + # neither side of the OR is individually optimisable. The filter + # evaluates the residual using + # `WhereClause.subquery_member_conservative_from_index/2`, which + # consults MTV with `member_at_some_time?` for the positive sublink + # — so values provably outside the retained window are pruned here + # instead of over-routing. {:ok, shape} = Shape.new("or_child", inspector: inspector, @@ -1186,7 +1198,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] for value <- [1, 2, 3] do - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, value) + add_value(filter, "shape1", shape, subquery_ref, value) end SubqueryIndex.mark_ready(index, "shape1") @@ -1210,6 +1222,8 @@ defmodule Electric.Shapes.FilterTest do record: %{"id" => "50", "par_id" => "99", "value" => "other"} } + # Conservatively pruned: par_id=99 is absent from MTV at every + # retained time and the LIKE branch doesn't match. assert Filter.affected_shapes(filter, insert_no_match) == MapSet.new([]) end @@ -1264,7 +1278,7 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, 1) + add_value(filter, "shape1", shape, subquery_ref, 1) SubqueryIndex.mark_ready(index, "shape1") wrong_subquery_value = %NewRecord{ @@ -1309,8 +1323,8 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, 1) - SubqueryIndex.add_value(index, "shape2", subquery_ref, 0, 1) + add_value(filter, "shape1", shape1, subquery_ref, 1) + add_value(filter, "shape2", shape2, subquery_ref, 1) SubqueryIndex.mark_ready(index, "shape1") SubqueryIndex.mark_ready(index, "shape2") @@ -1340,14 +1354,21 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, 1) + add_value(filter, "shape1", shape, subquery_ref, 1) SubqueryIndex.mark_ready(index, "shape1") - assert :ets.tab2list(index) != [] + # Before remove, the shape routes records whose value is in the + # subquery view. + hit = %NewRecord{relation: {"public", "child"}, record: %{"id" => "1", "par_id" => "9"}} + assert Filter.affected_shapes(filter, hit) == MapSet.new(["shape1"]) + assert SubqueryIndex.has_positions?(index, "shape1") Filter.remove_shape(filter, "shape1") - assert :ets.tab2list(index) == [] + # After remove, the shape no longer routes and its index entries + # are gone. + assert Filter.affected_shapes(filter, hit) == MapSet.new([]) + refute SubqueryIndex.has_positions?(index, "shape1") end @tag with_sql: [ @@ -1369,13 +1390,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] # Seed membership with value 1 (parent id 1 matches the subquery "WHERE value = 'keep'") - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([1]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1])) SubqueryIndex.mark_ready(index, "shape1") @@ -1426,13 +1441,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] # Seed the membership view with values {1, 2} - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([1, 2]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1, 2])) SubqueryIndex.mark_ready(index, "shape1") @@ -1480,13 +1489,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] # Seed membership with a tuple value {10, 20} - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([{10, 20}]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([{10, 20}])) SubqueryIndex.mark_ready(index, "shape1") @@ -1529,13 +1532,7 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([1, 2]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1, 2])) SubqueryIndex.mark_ready(index, "shape1") @@ -1596,13 +1593,7 @@ defmodule Electric.Shapes.FilterTest do # seeded and marked ready. subquery_ref = ["$sublink", "0"] - SubqueryIndex.seed_membership( - index, - "indexed_s", - subquery_ref, - 0, - MapSet.new([1]) - ) + seed_membership(filter, "indexed_s", indexed_shape, subquery_ref, MapSet.new([1])) SubqueryIndex.mark_ready(index, "indexed_s") @@ -1634,5 +1625,97 @@ defmodule Electric.Shapes.FilterTest do assert Filter.affected_shapes(filter, insert_other) == MapSet.new([]) end + + @tag with_sql: [ + "CREATE TABLE IF NOT EXISTS neg_res_parent (id INT PRIMARY KEY)", + "CREATE TABLE IF NOT EXISTS neg_res_child (id INT PRIMARY KEY, name TEXT NOT NULL, par_id INT REFERENCES neg_res_parent(id))" + ] + test "residual NOT IN sublink prunes via member_at_all_times? when value is always a member", + %{inspector: inspector} do + # `LIKE … OR NOT IN (…)` is unoptimisable (LIKE isn't indexable), so + # the shape lands in the `other_shapes` path with a negated sublink in + # the residual. The conservative callback uses + # `MultiTimeView.member_at_all_times?/3` so values provably present + # at every retained time get excluded here (NOT IN is false at every + # consumer time) instead of over-routing. + {:ok, shape} = + Shape.new("neg_res_child", + inspector: inspector, + where: "name LIKE 'never%' OR par_id NOT IN (SELECT id FROM neg_res_parent)" + ) + + filter = Filter.new() |> Filter.add_shape("shape1", shape) + index = Filter.subquery_index(filter) + subquery_ref = ["$sublink", "0"] + + # Seed MTV with values that are always-members (empty history). + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1, 2])) + SubqueryIndex.mark_ready(index, "shape1") + + # par_id=1 is in MTV at every retained time → NOT IN is false → exclude. + insert_always_member = %NewRecord{ + relation: {"public", "neg_res_child"}, + record: %{"id" => "10", "par_id" => "1", "name" => "x"} + } + + assert Filter.affected_shapes(filter, insert_always_member) == MapSet.new([]) + + # par_id=99 is unknown to MTV → conservatively-not-a-member → NOT IN is + # true → include. + insert_unknown = %NewRecord{ + relation: {"public", "neg_res_child"}, + record: %{"id" => "11", "par_id" => "99", "name" => "x"} + } + + assert Filter.affected_shapes(filter, insert_unknown) == MapSet.new(["shape1"]) + end + + @tag with_sql: [ + "CREATE TABLE IF NOT EXISTS amb_parent (id INT PRIMARY KEY)", + "CREATE TABLE IF NOT EXISTS amb_child (id INT PRIMARY KEY, name TEXT NOT NULL, par_id INT REFERENCES amb_parent(id))" + ] + test "ambiguous value (in at some retained times, out at others) routes conservatively in both polarities", + %{inspector: inspector} do + # A value with a non-empty MTV history sits between always-in and + # always-out — `member_at_some_time?` is true *and* + # `member_at_all_times?` is false. Both polarities of conservative + # callback should err on the side of include. + {:ok, pos_shape} = + Shape.new("amb_child", + inspector: inspector, + where: "name LIKE 'never%' OR par_id IN (SELECT id FROM amb_parent)" + ) + + {:ok, neg_shape} = + Shape.new("amb_child", + inspector: inspector, + where: "name LIKE 'never%' OR par_id NOT IN (SELECT id FROM amb_parent)" + ) + + filter = + Filter.new() + |> Filter.add_shape("pos", pos_shape) + |> Filter.add_shape("neg", neg_shape) + + index = Filter.subquery_index(filter) + mtv = index.multi_time_view + + # Build an ambiguous history for value 1: in at time 0, out from time 1. + {dep_handle, :positive} = SubqueryIndex.lookup_dep!(index, "pos", 0) + MultiTimeView.init_subquery(mtv, dep_handle, [1]) + MultiTimeView.mark_out(mtv, dep_handle, 1, 1) + MultiTimeView.mark_ready(mtv, dep_handle) + + SubqueryIndex.mark_ready(index, "pos") + SubqueryIndex.mark_ready(index, "neg") + + # par_id=1 ambiguous → conservative include for both polarities. + insert_ambiguous = %NewRecord{ + relation: {"public", "amb_child"}, + record: %{"id" => "10", "par_id" => "1", "name" => "x"} + } + + assert Filter.affected_shapes(filter, insert_ambiguous) == MapSet.new(["pos", "neg"]) + end end end diff --git a/packages/sync-service/test/electric/shapes/querying_test.exs b/packages/sync-service/test/electric/shapes/querying_test.exs index 4e3660ad83..4cd38e5c3a 100644 --- a/packages/sync-service/test/electric/shapes/querying_test.exs +++ b/packages/sync-service/test/electric/shapes/querying_test.exs @@ -21,6 +21,15 @@ defmodule Electric.Shapes.QueryingTest do @stack_id "test_stack" @shape_handle "test_shape" + # Build a `values_for/2` resolver from two view maps, matching the pre-RFC + # `move_in_where_clause/5` call shape used throughout these tests. + defp values_for_from_views(views_before, views_after) do + fn + ref, :before -> Map.get(views_before, ref, []) + ref, :after -> Map.get(views_after, ref, []) + end + end + describe "stream_initial_data/4" do test "should give information about the table and the result stream", %{db_conn: conn} do Postgrex.query!( @@ -629,8 +638,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -674,8 +682,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -727,8 +734,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -788,8 +794,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -834,8 +839,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -880,8 +884,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -929,8 +932,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), where.used_refs ) @@ -960,8 +962,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 1, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), where.used_refs ) @@ -1002,8 +1003,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), where.used_refs ) @@ -1027,8 +1027,10 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - %{["$sublink", "0"] => MapSet.new([1, 2, 3])}, - %{["$sublink", "0"] => MapSet.new([3])}, + values_for_from_views( + %{["$sublink", "0"] => MapSet.new([1, 2, 3])}, + %{["$sublink", "0"] => MapSet.new([3])} + ), where.used_refs ) @@ -1048,8 +1050,10 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - %{["$sublink", "0"] => MapSet.new([5, 6])}, - %{["$sublink", "0"] => MapSet.new([6])}, + values_for_from_views( + %{["$sublink", "0"] => MapSet.new([5, 6])}, + %{["$sublink", "0"] => MapSet.new([6])} + ), where.used_refs )