[Bug]: Switch operator lacks support of virtualized changeset

[dmitmaloukas]

Describe the bug 🐞

There is no Switch operator version for virtualized changesets IChangeSet<TObject, TKey, VirtualContext<TObject>>.

In this case default reactive Switch operator is used, so the test fails.

 [Fact]
 public void SwitchSet_Switch_Virtualized()
 {
     BehaviorSubject<bool> enableSubj = new(true);
     SourceCache<int, int> cache = new(x => x);
     cache.Edit(updater => updater.AddOrUpdate(Enumerable.Range(0, 100)));
     SortExpressionComparer<int> comparer = SortExpressionComparer<int>.Ascending(x => x);

     int count = 10;

     IObservable<VirtualRequest> virtualReqStream
         = Observable.Return(new VirtualRequest(0, count)).Concat(Observable.Never<VirtualRequest>());

     IObservable<IChangeSet<int, int, VirtualContext<int>>> virtualizedChangeSet = cache.Connect()
         .SortAndVirtualize(comparer, virtualReqStream);

     IObservable<IChangeSet<int, int, VirtualContext<int>>> testStream = enableSubj
         .Select(enable =>
         {
             if (!enable)
                 return Observable.Empty<IChangeSet<int, int, VirtualContext<int>>>();

             return virtualizedChangeSet;
         })
         .Switch();

     TestScheduler scheduler = new();

     ITestableObserver<IChangeSet<int, int, VirtualContext<int>>> observer
         = scheduler.CreateObserver<IChangeSet<int, int, VirtualContext<int>>>();

     testStream.Subscribe(observer);

     enableSubj.OnNext(false);

     Assert.Collection(
         observer.Messages,
         msg =>
         {
             Assert.Equal(NotificationKind.OnNext, msg.Value.Kind);
             Assert.Equal(count, msg.Value.Value.Adds);
             Assert.Equal(0, msg.Value.Value.Removes);
         },
         msg =>
         {
             Assert.Equal(NotificationKind.OnNext, msg.Value.Kind);
             Assert.Equal(0, msg.Value.Value.Adds);
             Assert.Equal(count, msg.Value.Value.Removes);
         }
     );
 }

DynamicData Version

9.4.33

[JakenVeina]

There actually is a .Switch() operator for virtualized changesets: it's the same one as for regular changesets.

public static IObservable<IChangeSet<TObject, TKey>> Switch<TObject, TKey>(
        this IObservable<IObservableCache<TObject, TKey>> sources)
    where TObject : notnull
    where TKey : notnull;

IChangeSet<TObject, TKey, VirtualContext<TObject>> inherits from IChangeSet<TObject, TKey>, so the same operator can serve both. The tricky part is that .NET won't resolve to this operator for IChangeSet<TObject, TKey, VirtualContext<TObject>>, since the native RX operator is an equally-valid candidate, from an outside perspective. The even trickier part is that we've never noticed this because from WITHIN the DynamicData namespaces, where we do all our work, the DynamicData operator resolves as a better candidate.

You can work around this easily enough by just adding an alias extension somewhere.

public static IObservable<IChangeSet<TObject, TKey>> Switch<TObject, TKey>(
            this IObservable<IChangeSet<TObject, TKey, VirtualContext<TObject>>> source)
        where TObject : notnull
        where TKey : notnull
    => ObservableCacheEx.Switch(source);

We can add this internally as well, for future releases.

Alternatively, just directly cast your query, or move your .SortAndVirtualize() down the query, below the .Switch().

[dmitmaloukas]

@JakenVeina
Doesn't the suggested approach break resolution of Bind operator, which internally uses BindVirtualized?

    public static IObservable<IChangeSet<TObject, TKey>> Bind<[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.All)] TObject, TKey>(
        this IObservable<IChangeSet<TObject, TKey, VirtualContext<TObject>>> source,
        IList<TObject> targetList)
        where TObject : notnull
        where TKey : notnull =>
        new BindVirtualized<TObject, TKey>(source, targetList, null).Run();

[JakenVeina]

I wouldn't say it "breaks" anything. It means that inserting a .Switch() between a .Virtualize() and a .Bind() would cause the .Bind() to longer receive the VirtualContext<> data, which in turn means .Bind() won't be able to sort the items going into the target list.... which makes no sense, given that they should either already be sorted, or that you should be using a .SortAndBind(), if sorting matters to you. Hell, if virtualization matters to you, you should be using .BindAndVirtualize().

I probably need to take a closer look at what .Virtualize() is actually intended to be doing here. I don't see any reason why .Bind() should need any context about virtualization, as virtualization is fundamentally what all DD streams already are.

You're the one encountering this issue, what behavior are YOU interested in here?

[dmitmaloukas]

Sorry for the late reply.

Let me explain what I'm trying to achieve.

I faced a performance issue, so I'm trying to optimize my data flow (where order of the items matters).
The core of the problem is the large number of items I have to handle and the number of asynchronous transformations I have to perform with each item.
I cannot get rid of transformations, so I use SortAndVirtualize operator as early as possible to reduce the number of items to be processed.
The order of items may be changed dynamically in my case, so I was expecting it to be preserved by Bind operator.

you should be using a .SortAndBind(), if sorting matters to you.

I understand your point, but it still looks a bit clunky to use SortAndVirtualize and then SortAndBind and to pass them the same comparer observable, while the changeset already contains everything required for proper binding.

I probably need to take a closer look at what .Virtualize() is actually intended to be doing here. I don't see any reason why .Bind() should need any context about virtualization, as virtualization is fundamentally what all DD streams already are.

I assume the goal of having VirtualContext embedded in the ChangeSet is to make comparer change atomic for both virtualization and binding. It avoids processing of intermediate state when the changeset is virtualized with a new comparer but SortAndBind operator is still using the old comparer or vice versa.