Frame ghosting

[AIisCool]

As I'm sure we've all seen, despite AMAZING results, SEEDVR2 does have an issue; whether it's hardcoded in the model or something to correct via code: when there is medium to fast motion between frames, you can see ghosting/afterimage of the previous frame which leads to distortions for really fast motion. I can also see a similar ghosting/distortion issue with Topaz's Starlight model.

So the point of this discussion is: What is causing it and how do we correct it if such a task is possible?

[AIisCool]

@naxci1 Tried those exact setting and the issue is still there for me.

Input:

Output:

You can see his left arm is distorting with the plant behind him. If you wish to try with that video, I uploaded it here.

[AIisCool]

Not sure why the link did not work but I tried again.

No corruption in the input video, it only becomes distorted after processing.

[AIisCool]

No, the FPS is 23.976. As for deinterlacing it's from a Blu-ray download that I resized the resolution down by 50%. That said I have seen the same distortion issue with pure progressive video so interlacing isn't the cause.

Not sure why mega isn't working but I tried proton this time.

[AIisCool]

I don't think that's the case.

Yes there is motion blur, but in the SEEDVR2 video there is green distortions that wasn't in the original:

I tried another video. You can see the pattern on her skirt is all distorted after SEEDVR processing. Yes there is motion and it's blurry as she moves, but in the original the pattern isn't distorted.

Input:

SEEDVR2:

[mindplay-dk]

I don't think that's the case.

Yes there is motion blur, but in the SEEDVR2 video there is green distortions that wasn't in the original:

Huh, now in this example I definitely see what you're talking about - it's as though it's painted the leaves from the plant onto his arm.

If I had to guess, I'd say the input is probably too blurry for the model to make sense of, and so it makes a "wrong best guess".

On the skirt in the other example, I don't see a problem - just looks like motion blur in the input and the output, right?

[AIisCool]

Yes the leaves into his arm is the issue I've been trying to convey. As for her skirt, while yes it's in motion, the pattern looses it's definition that while in motion in the original, doesn't become blurred/distorted.

[AIisCool]

Here is the clip I used with the skirt: https://drive.proton.me/urls/9KR6XDHWK8#IGaFvCez57Lg

Have to go for a while, be back later.

[AIisCool]

@adrientoupet curious to know your thoughts on this. Am I misinterpreting motion blur or are distortions happening?

[adrientoupet]

Apologies @AIisCool I'm not sure - and I won't look into this right now. I'm currently focused on finishing the release I'm working on, to clear a lot of of other tickets that have been opened for a while. Once that's done, I'll go back to that thread/your question and will help troubleshoot, if you all don't find a solution before me. Thanks for your patience.

[AIisCool]

No worries! I am always impressed by how much you've improved SeedVR2 in such a short time.

[adrientoupet]

It would be even more impressive if it had been officially released and I had time to make an updated video about it to document everything :) but thank you, appreciate the support... hopefully we'll get there soon!

[AIisCool]

You're welcome, I wish I was far more knowledgeable and able to assist.

[AIisCool]

Hi @adrientoupet I had a thought about the distortions. When I tried upscaling a video using UltimateSDUpscale and tied in the WAN 2.2 model and it's VAE, it didn't have any distortions. I wonder if this may help, if it's possible to somehow integrate WAN's temporal consistency. This is of course assuming the distortions can't be fixed via coding and isn't just part of the SEEDVR model.

[adrientoupet]

Thank you @AIisCool - I suspect the distortions should be fixable in the code, but I'll report back once I had a chance to dive deeper into it.
Out of curiosity, how does ultimateSDUpscale compare to SeedVR2 ?

[AIisCool]

It was decent, but took too much to try and keep it from changing the input. That said the one thing I liked was that it wasn't causing color banding, which I see happening on the output of SEEDVR2 at times even though I have it set to save as ProRes 4444XQ

Input:

SEEDVR2:

I even tried using DVO Clarity to reduce the banding in the input clip, but it still comes out with more pronounced banding from SEEDVR2.

I think it's due to SEEDVR2 not being trained on degraded input, but perhaps there's a way to reduce the banding?

[AIisCool]

@adrientoupet I found another good example of this while testing SEEDVR2 on some traditional 2D animation.

Everything is fine in this frame:

But then the next fame has the "ghost" of the outlines of the previous around the character:

Not present on the original input:

I've uploaded the original clip here should you wish to test with it.

[mindplay-dk]

I wonder if this is an edge case for animation, which doesn't always contain "motion" in the traditional sense.

I wonder if it's even a good idea to have the same model trained on both film and animation? Wildly different types of motion... like, I wonder what the model learns from seeing duplicated frames, which are often present in (esp. traditional) animated content? 🤔

[mindplay-dk]

Oh, interesting, this particular clip in fact does not appear to have duplicate frames - it was created at the native frame rate of the original film. Hmm, I wonder if the artifacting you're seeing here is something it learned from animated content with duplicate frames? It's not unlikely that (the wrong kind of) compression could create artifacts like that while attempting to compensate for choppy motion?

[adrientoupet]

Hi @AIisCool - thank you for sharing the clip, I'll give it a go and report back.

[adrientoupet]

Tried it, I'm able to reproduce the issue. Unfortunately I don't have a fix for it just yet.
Need to look into temporal overlap/prepend as discussed here #217 (comment) to see if that could help, or do further investigations. That will be for next time as I don't have bandwidth to look into it for the coming release. Thank you

[AIisCool]

Tried it, I'm able to reproduce the issue. Unfortunately I don't have a fix for it just yet. Need to look into temporal overlap/prepend as discussed here #217 (comment) to see if that could help, or do further investigations. That will be for next time as I don't have bandwidth to look into it for the coming release. Thank you

I'm glad it's reproducible, and not just a "me" issue.
No, thank you for continuing to put endless hours into improving this! Please no rush. I hope that whether it comes in November, December, or January, it is fixable.

[AIisCool]

@adrientoupet I had a thought, though it's of course completely uneducated on the matter. I notice that SEEDVR2 makes immobile objects very clear, along with the backgrounds that are stationary become very clean/clear. Perhaps it's overcompensating and that is the cause of the "ghosting" when something/someone moves and is then for super-fast movement it cannot deal with the temporal shift?

[adrientoupet]

I'm curious to see if you're getting more luck with the last version ? Using batch size long enough to be temporally consistent but not too long that they create issues, enable uniform batch size, add temporal overlap, prepend... Let me know when you have a chance, thank you.

[R3G4L101]

On 2.5 I am getting the after image ghost trails with movement. Makes the quality look distorted. Previous nightly build doesn't have this effect on the same videos.

[adrientoupet]

Can you share workflow ,& video please so I can troubleshoot?

[R3G4L101]

Here you go, workflow attached at bottom. I took another look and it is in both, but the trailing is more pronounced in 2.5. I never noticed from before because the trailing is more subtle in motion. Used the same seed for both.


SeedVR2.5 Video TEST.json

[adrientoupet]

Thank you very much @R3G4L101 - Any chance you can upload your input footage somewhere so I can troubleshoot on the same thing? Cheers

[R3G4L101]

SeedVR2.Comparison.mp4

Comparison vid

[adamfluty]

Hi, I am also having this issue. I have both detelecined (30 -> 24 fps) and deinterlaced the raw DVD rip. It is clean to start with as seen in the first image. Batching at all, whether 9 or 121, it creates major "leak" between frames. 90% of the time its previous frames being retained and I have even seen future frames leak in as well.
Ready-to-upscale

Batch 9

Here you can see where the brim of his hat and fingers will be in the NEXT frame

With batch 121, the issue is more localized to the moving character, not the space around them

Batch 0 makes each frame flawless but with bad shimmering in the video, as you would expect.
I am not tiling, and I have played with the temporal overlap as well with no improvement.

[thehhmdb]

I've noticed this issue as well.

[thehhmdb]

Did anyone find any fixes or workarounds for the frame ghosting?

[cd1188]

I have the same problem when people are talking. There are slightly blurry and darker frames in the face.
It's definitely not motionblur! Even with constantly moving objects, there is a flicker.
I set the batch size to 51 and I feel like every fourth frame is sharp and then the following frames become blurry...
Seems to be a pattern of 4 or 5 frames?
As a test I have also set the temporal-overlap to 0 and its the same bad flicker.
I noticed that when I remove a frame at the beginning of the video, the sharper images also shift by one frame.
I had already considered rendering the video four times and then extracting the sharp images, but I don't know if that will work.
I think that would break the motion consistency if I combine different videos.

Edit: extracting every 4th frame is difficult, because the pattern is only consisten within each batch.
The problem is that each batch has a odd frame count and after each batch, you have to shift the extraction pattern for one frame.
-> temporal overlap of 1 helps here to be in the pattern of 4 frames for the next batch

Edit2:
The Idee from extracting each 4th frame from 4 different videos with shifted start-frame is also not good.. you see now shimmering on each frame.

Edit3:
My bad workaround for the moment is to convert a 15fps clip to 60fps with 4 identical frames in a row and render with temporal overlap of 1 frame and batch size of 45. That way each batch starts within a new 4frame-window which is important!... after that I remove 3/4 of the frames by convert it back to 15fps. it's a waste of time to render that way and you loose a bit of motion consistency, but the ghosting is gone and some fast frames are now way sharper with more detail. Hope SeedVR3 does fix this problem.

Here is an example from my tests If I convert a 160p-15fps clip to 60fps without interplation and upscale to 960p with a batch-size of 45 with temporal overlap 1.
Would be good to have an option to deactivate this 4-frame-analyse-system in SeedVR as a workaround.