Skip to content

VLC Video sync and cleanup - #1732

Closed
taynaron wants to merge 311 commits into
YARC-Official:masterfrom
taynaron:background-video-sync
Closed

taynaron wants to merge 311 commits into
YARC-Official:masterfrom
taynaron:background-video-sync

Conversation

@taynaron

@taynaron taynaron commented Oct 6, 2026

Copy link
Copy Markdown

Follow-up to #1679. Review after that's merged for a much smaller PR.

Unity and VLC videos now have:

  • all songs respect the video_start_time tag
  • correct video aspect ratios

VLC videos now support these features:

  • song speeds are respected
  • pause/resume works pretty smoothly - rewinds to rewind point immediately upon pause, as matching rewind behavior exactly led to video lag.
  • looping in practice mode retains video sync
  • speed changes in practice retains video sync
  • slight fade in/out for videos that start after/end before the song
  • support for looping videos (some videos are made to play continuously)

poliwhirlge and others added 30 commits July 18, 2026 20:25
…to a game mode where the currently selected engine doesn't exist (YARC-Official#1554)
* Add ability to view the score screen for a replay

* Don't play vox samples if viewing a replay

* Add high score detection to view replay score

* Check replay consistency when viewing score screen

* Don't show completed/cleared on replays

* Move inconsistent replay warning to pill
- Also fix some replay loading stuff
- Also some localization, needs more
…icial#1558)

* Improve audio synchronization

* Refine audio synchronization

* Fix transient sync delta after resume

* Improve audio synchronization on resume and fix teardown hangs

* Fix audio teardown and latency compensation

* More improvements

* Remove resume audio debug logs

* Document sync correction behavior

* Keep sync correction state aligned with playback

* Reduce playback sync polling overhead

* Use ring buffer for sync correction history

* Improve audio sync and calibration timing

* Unify audio playback preparation

* Improve audio sync correction

* More cleanup

* Handle playback latency for metronome sample channel

* Improved scheduling of metronome hits

* More oneshot refactoring

* Refactor position handling to simplify song runner

* Simplify handling playback/seek latency

* Fix calibration input boxes for negatives

* Fix calibration issue

* First implementation of DSP backed metronome

* Improved oneshot design

* Update YARG.Core playback sync API

* Remove test cowbell asset

* More cleanup

* Much simplified song runner

* More cleanup

* More cleanup

* More cleanup

* docs: clarify song sync latency compensation and desync handling

* docs: update deadband filter explanation in song_sync.md

* docs: clarify transition handling vs feedback adjustments in song_sync.md

* docs: remove fluctuations detail from transition latency note

* Fix metronome issues on linux

* Improve Freeverb and Quickplay mic startup

* Refactor Quickplay microphone lifecycle

* Updated song sync.md

* Add audio delay verification utility

* Better fix for mic latency

* Add BASS position clock stability test

* Fix microphone restarts and metronome playback

* Restart profile microphones when loading screen finishes

* Better song runner

* Measure BASS position startup with plain PCM stream

* Measure BASS position startup with plain PCM stream

* More testing

* Simpler handling of play -> getPosition delay

* Metronome tests

* Fix extra loud volume when star power enabled

* Route tempo stream through a mixer

* Route metronome through new playback mixer

* Improved metronome scheduling and fix another whammy desync

* Route crowd claps through playback mixer

* Restore DSP version of oneshot channel

* Latency correction for macos

* Song sync update

* speed changes in practice mode no longer flush

* Remove audio delay verification utility

* Fix crowd chatter playing when it shouldn't

* Fix calibration

* Improve BASS startup latency measurement

* Restore dev YARG.Core pointer

* Update song synchronization documentation

* Replace LaTeX equations in sync docs

* Remove audio delay verification utility

* Remove microphone diagnostic logging

---------

Co-authored-by: Phil Olson <polson@github.com>
The default behavior of automatic interface selection remains, but now
you can override it if it didn't choose the correct network.
* Reapply active filters when opening music library

* Don't reapply filters a redundant time when coming from Difficulty Select
* Remove dead playlist hashes from disk when songs are removed from disk and library is rescanned

(cherry picked from commit 3185230)

* Only remove dead playlist hashes from disk when the playlist is manually edited
* Add keyboard shortcut for search

* Add explicit menu binds for search

* Highlight search field when focused

* Add border to search bar

* Clean up prefab spam

* Toggle with the keybinding

* Remove duplicate Search key in en-US.json

* Reset YARG.Core submodule pointer to match dev
…rts (YARC-Official#1515)

A player who launched a song as HARM3 and later played as VOCALS could
open a two-part song, switch the instrument to HARMONY, and confirm as
HARM3 — a part that doesn't exist, producing a song with no notes.

The out-of-range check in ChangePlayer compared profile.HarmonyIndex,
whose getter returns 0 whenever CurrentInstrument != Harmony (that guard
is load-bearing: VocalsPlayer indexes Parts[HarmonyIndex] for VOCALS
players too). With the player on VOCALS the check could never fire, so
the stale stored index survived until the instrument switch exposed it.

Use YargProfile.ResolveHarmonyIndex (added in the corresponding
YARG.Core change), which validates the stored value regardless of the
current instrument and recomputes the effective index per song from the
player's last explicit selection: clamped to the highest available part
on songs with fewer parts, restored in full on songs with enough — the
same retention DifficultyFallback gives Expert+. Also resolved in
CreateInstrumentMenu right after an instrument switch, covering the
path from the report.

Requires a YARG.Core version containing ResolveHarmonyIndex.
…ARC-Official#1565)

* Refresh star sort grouping after instrument or difficulty changes

* Add current difficulty high score options and sync star sorting with display
* ci: add workflow to auto-label PRs needing rebase

* fix(ci): update label-conflicts action to v3.1.0 commit SHA
* refactor(ci): replace 3rd party action with inline gh CLI script

* feat(ci): add explicit logging for open PR count and conflict labeling

* feat(ci): add retry loop for UNKNOWN PR mergeability status

* fix(ci): report conflict label failures
wyrdough and others added 29 commits October 5, 2026 22:34
LibVLCSharp's Constants.cs resolves UnityPlugin to the bare name
"VLCUnityPlugin" on every platform but iOS. On macOS the real file is
libVLCUnityPlugin.dylib, so that DllImport fails to dlopen and VLC init
dies with a bare DllNotFoundException. Our own OnLoad.cs has a separate
UnityPlugin constant that does branch correctly; Constants.cs does not,
and it is compiled into the prebuilt LibVLCSharp.dll, so it cannot be
fixed from this repo.

This is a universal (x86_64 + arm64) stub that re-exports the real dylib
under the name Constants.cs asks for, built with:

  ld -arch <arch> -dylib -o VLCUnityPlugin_<arch>.dylib \
    -reexport_library libVLCUnityPlugin.dylib \
    -install_name @rpath/VLCUnityPlugin.dylib \
    -platform_version macos <min> <sdk> \
    -syslibroot $(xcrun --show-sdk-path) -lSystem
  install_name_tool -change @rpath/libVLCUnityPlugin.dylib \
    @loader_path/libVLCUnityPlugin.dylib VLCUnityPlugin_<arch>.dylib
  lipo -create VLCUnityPlugin_x86_64.dylib VLCUnityPlugin_arm64.dylib \
    -output VLCUnityPlugin.dylib

@loader_path rather than @rpath so the shim finds its dependency without
the host process having to set up an rpath. Rebuild it if
libVLCUnityPlugin.dylib is ever updated.

The importer settings are scoped to macOS only, so no other platform
picks this up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Update path setting message for clarity
Independent correctness fixes in the two wrapper classes. No policy
change in BackgroundManager; each of these is wrong on its own terms.

VLCMediaPlayer.Pause() called MediaPlayer.Pause(), which is
libvlc_media_player_pause -- a TOGGLE, not a pause. On a player that has
not reached Playing yet (a Pause issued in the same frame as a Play, say)
it starts playback instead of stopping it. Use SetPause(true), which is
unconditional. The sync layer later in this branch depends on this:
ApplyVideoPaused issues a Pause whenever the game wants the video paused,
without first checking what the player is doing.

VLCMediaPlayer.SetTime discarded SeekTo's return value, so a seek issued
with no active input to seek within -- after Stop(), for instance --
failed silently and the caller went on believing the player had moved.
Log a warning when it reports failure. Note that a seek issued while
merely paused still returns true and is still dropped, so the warning's
absence does not prove a seek landed.

The same call now passes fast: false explicitly. It is already SeekTo's
default, but landing accuracy is the entire purpose of this method and a
keyframe-snapped seek can be seconds out on a long-GOP video, so the
intent should not rest on a default.

YargVideoPlayer's targetTexture setter passed the RenderTexture straight
to YargLogger.LogFormatDebug, which has no formatter for that type and
throws. Log value.name instead.

YargVideoPlayer.OnVLCTextureResized forwarded every OnTextureResized to
prepareCompleted. That event is raised on each decoded-size change, not
just the first, so a mid-song resolution change re-entered the caller's
prepare handler and re-ran the whole initial setup underneath a playing
player. Latch it to one fire per Prepare(), matching the Unity path,
whose OnUnityVideoPrepared already self-unregisters after the first fire.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SongTime carries AudioCalibration, which absorbs the device's output
latency when AccountForHardwareLatency is on. Driving the video off it
offset the video from the highway and vocals lane by
(AudioCalibration - VideoCalibration) * speed, and left VideoCalibration
with no effect on the video at all.

VisualTime is the clock everything else on screen uses. Adds
GameManager.GetVideoPlaybackTime and uses it at all three places the
video reads the song clock: BackgroundManager.Update, SetSongTime, and
song load. The file-position conversion is the same SongOffset shift
either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libavcodec's frame threading delays decoder output by thread_count - 1
frames by design, and hardware decoders buffer more on top. At 24-30fps
that delay is the whole of the offset the video sits at, and no seek or
resume timing moves it.

Each song's media is opened with :avcodec-threads=1 and :low-delay, plus
:no-audio as insurance against a badly muxed file reaching the speakers.
Both decoder options are serialized on BackgroundManager and on by
default, so single-threaded decode can be turned off on a machine it
starves. YargVideoPlayer.Prepare now takes per-media options; the Unity
path ignores them.

Measured settled lag, four videos at 100%: 0.085-0.102s before,
0.005-0.024s after. A video whose start sits mid-GOP keeps paying for the
decode from its keyframe: 0.166s -> 0.087s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YargVideoPlayer raised seekCompleted straight from the time setter, on
the premise that libVLC's SetTime is synchronous and has no completion
callback. Neither holds: the seek is asynchronous, and libVLC 4 reports
completion through libvlc_media_player_watch_time's on_seek. Every seek
was announced before it had happened, releasing WaitForSongVideo's hold
on the song early.

A seek now completes when on_seek reports it finished and a picture has
been presented after that, bounded by a 0.35s deadline since a caller may
be holding the song on it. There is deliberately no proximity test: a
seek lands on the nearest decodable frame, so a mid-GOP target settles
short and a distance test never passes. framesDelivered counts libVLC's
presented pictures, which is also what the clock re-anchors on.

The watch is registered by YargVideoPlayer on VLCMediaPlayer's public
MediaPlayer, so no vendored code changes. libVLC asserts at player
deletion if a watcher is still attached, which aborts the Editor. Scene
unload destroys GameObjects one at a time, so YargVideoPlayer's own
OnDisable cannot guarantee it runs first; VlcPlayerTeardownHook, added at
runtime to the VLC player's GameObject, removes the watch in OnDisable,
which Unity runs before any OnDestroy on that same object.

Also reads MediaPlayer.Time in microseconds rather than VLCMediaPlayer's
millisecond-truncated Time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YargVideoPlayer stopped the player after prepare, and OnVideoPrepared's
start-position seek came after that. A seek on a stopped libVLC player is
silently dropped -- Stop() releases the input entirely -- so
video_start_time was ignored.

The player is now left playing through prepare. OnVideoPrepared seeks to
the start position while it still is, then pauses, parking the video on
its first frame. A negative start time is clamped to 0: it delays when
the video starts rather than naming a position before the file begins.

Update()'s start block plays, then seeks to the song's position
unconditionally. Resuming a cold pause leaves a large stream spinning up
at partial rate for seconds; the seek flushes and refills the decoder.
It must follow the Play, since a seek issued while paused is dropped.

The venue stays behind its curtain until the decoder has presented
REVEAL_MIN_FRAMES pictures, so a slow spin-up is hidden rather than
shown. All play/pause now goes through SetVideoPlaying: Play and Pause
apply asynchronously in issue order, so one choke point keeps the order
reasoned about in one place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Background videos were stretched to fill whatever rect the consumer
mapped them onto. Every consumer of targetTexture -- the venue RawImage,
the yarground screen material -- maps the whole texture onto a fixed
rect, and VLC decodes at the video's native resolution, so a 4:3 video on
a 16:9 screen came out horizontally stretched.

YargVideoPlayer now owns the externally-supplied RenderTexture and both
backends fit themselves into it:

  - The built-in player does it at decode time via
    VideoAspectRatio.FitInside, on both the Prepare() path and the VLC
    fallback path.
  - The VLC path blits OutputTexture into the target, centered at the
    source's own aspect ratio, clearing the remainder to black.

The blit runs in LateUpdate. VLCMediaPlayer fetches libVLC's newest
picture in its own Update and the two have no defined order; blitting in
Update ran before the fetch on every measured frame, so the venue always
showed the previous frame's picture, about 17ms late at 60fps.

targetTexture's setter used to drop the supplied texture under VLC; it
now keeps it, so consumers see one stable texture across both backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ine lag

A VLC picture reaches the screen a fixed interval after libVLC presents it,
so a video aimed exactly at the song shows late. Aim every video target
0.08s ahead on macOS and 0.04s elsewhere (about two and one 24fps frames),
scaled by song speed since the lag is wall-clock. Unity's player has no such
lag and gets none.

A constant rather than frame-rate based: libVLC reports 0 fps for
fragmented MP4s, whose headers carry no sample count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@taynaron taynaron closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.