Recover the total length for formats that omit contentLength

[lpaiu-cs]

Fixes #234.

Cause

extractFormat lets itag 18 through without a contentLength while skipping every other format missing one, and itag 18 is the only format left when a client's response is SABR-gated — so on those videos it is the whole of playback.

PersistentHttpStream normally recovers a missing length by adopting Content-Length from the first response. But YoutubePersistentHttpStream sets useHeadersForRange() to false and carries the range in the query instead, so the response is a plain 200 whose Content-Length describes the chunk, not the resource. The stream ends up believing it is one buffer long and stops at the first range boundary:

Client [TVHTML5] failed: Something went wrong when decoding the track.
        at ...container.mpeg.MpegAudioTrack.process(MpegAudioTrack.java:52)
Caused by:
        at org.apache.commons.io.IOUtils.readFully(IOUtils.java:2120)
        at ...container.mpeg.MpegAacTrackConsumer.consume(MpegAacTrackConsumer.java:68)

The loss lines up with BUFFER_SIZE (11862014): 8CFh_-qtzeg is 12164812 bytes, so one buffer is 97.5% of it, against 291s of 298s (97.7%) actually played.

video	duration	played	lost
8CFh_-qtzeg	298s	291s (×2)	~7s
i2eeqThRBlQ	289s	277s (×4)	~12s

Change

Ask for the total with a zero-length ranged request, which does come back as a 206 carrying Content-Range. Only for formats that arrive without a length, and it falls back to the previous behaviour whenever the probe does not yield a usable value.

Two smaller variants I tried first, and why they do not work

You may reach for these, so recording the results:

1. Read Content-Range off the existing response (override createContentInputStream). Does not work — the range goes in the query, so YouTube answers 200 with no Content-Range at all. Deployed and measured: still 292s of 298s.
2. Also send the range as a header so the response becomes a 206 (override getConnectRequest). Does not compile — getConnectRequest() is private in the released lavaplayer, though it is protected on lavaplayer main. Would become viable on a future bump.

If you would rather have the probe live in YoutubePersistentHttpStream than in YoutubeAudioTrack, that reads better to me too and I am happy to move it — I left it here only because that is the form I have measured.

Verification

Measured in production: the two videos above played 299s/298s and 289s/289s with no EOF in the container reader, against 291s and 277s on the same build without the change, and 25 distinct tracks logged a recovered length. That was with a slightly more defensive version of this same probe; the form in this PR is deployed now and I will confirm it here once it has played a full track.

[lpaiu-cs]

Verified in production now.

DEBUG YoutubeAudioTrack : Recovered content length 12164812 for
  https://...googlevideo.com/videoplayback?...&itag=18&...&dur=297.424&...&c=TVHTML5

8CFh_-qtzeg (298s track) played 07:18:48 → 07:23:47 = 299s, against 291s on the same build without this change. No MpegAacTrackConsumer EOF during playback. The only exception in the window is the unrelated metadata-stage login wall two seconds before the track started, which does not involve TVHTML5.

[lpaiu-cs]

Confirmed in production with the exact code in this PR.

8CFh_-qtzeg (298s) played 298s, against 291s on the same build without the change. No MpegAacTrackConsumer EOF in the window.

The range sequence walks to the true end of the file rather than stopping a buffer in:

range=0-11862014
range=142493-12004507
range=12026824-12164812      <- 12164812 is the real total, from Content-Range

For contrast, variant 1 from the description (reading Content-Range off the existing query-ranged response) was deployed the same way and gave 292s of 298s, since that response is a 200 without the header.

[devoxin]

This doesn't sound right, though. Lavaplayer should have provisions to keep going even when the content length is signaled to be unknown. Your fix isn't a hack/workaround but, this would signal an issue on Lavaplayer's side that would need fixing as there is a specific value to denote unknown clen

[lpaiu-cs]

That is a fair reading, and I think you are right that something on the Lavaplayer side is off — but the problem is not that Lavaplayer fails to cope with an unknown length. It is that the length never stays unknown.

PersistentHttpStream.attemptConnect:

if (contentLength == Units.CONTENT_LENGTH_UNKNOWN) {
    Header header = currentResponse.getFirstHeader("Content-Length");

    if (header != null) {
        contentLength = Long.parseLong(header.getValue());
    }
}

That is reasonable when the request was for the whole resource. But YoutubePersistentHttpStream sets useHeadersForRange() to false and puts the range in the query instead, so the first connect is already a partial request that YouTube answers with a plain 200 — no Content-Range, nothing marking it as partial. Lavaplayer cannot tell, adopts the chunk length as the resource length, and the sentinel is gone after the first connect. From then on getNextRangeUrl()'s clamp and internalRead's rangeEnd == contentLength check are all working off one buffer's worth.

That is why the loss tracks BUFFER_SIZE exactly rather than looking like a random truncation.

I measured this rather than inferring it: the earlier variant of this PR read Content-Range off that same response and changed nothing (292s of 298s), which is what you would expect if the response really is an unmarked 200.

So the Lavaplayer-side fix would be to not adopt Content-Length as the resource length when the subclass has made a partial request — the subclass knows, the base class does not. Something like consulting useHeadersForRange()/getConnectUrl(), or having the subclass opt out. I have not opened anything on Lavaplayer for this; happy to if you would like, or to leave it with you.

For this repo there is also a smaller thing in the same area: getNextRangeUrl() does if (rangeEnd > contentLength) rangeEnd = contentLength, which with the sentinel sets rangeEnd to -1 and emits range=0--1 on the first request. Harmless once a length is known, but it is another spot where CONTENT_LENGTH_UNKNOWN is treated as a number.

Happy to hold this PR while the Lavaplayer side is decided — I am carrying it locally either way and will report if a soak turns up anything. The measurement stands as posted: 298s of 298s with the code in this PR, against 291s without it.

[devoxin]

I think that's a fair assessment. I'm happy to merge this PR. I'm not too keen on making another request, specifically one to probe the content length, as it gives YouTube another angle to block us or break youtube-source but the alternative is having a track that doesn't play all the way through; neither is an ideal situation, but one has the benefit of fixing the other, and the risks are probably not greater than they are now.

[lpaiu-cs]

Following up on this — found the root cause and it's actually a lavaplayer issue, fixed there in lavaplayer#198. Measured end-to-end and the track plays in full now. #237 reverts this once that's merged.

Thanks for the pointer!