Do not take a partial response's Content-Length as the resource length

[lpaiu-cs]

Fixes #197.

useHeadersForRange() exists so a subclass can ask for part of a resource by rewriting the URL instead of sending a Range header. The server answers such a request with an ordinary 200 — nothing marks it partial — and its Content-Length describes only the part that came back.

attemptConnect fills in an unknown length from that header, so the unknown marker is replaced by the size of one window on the first connect, and the stream stops there believing it has reached the end.

This skips the header when the request was narrowed, leaving the length unknown as it was. Requests that were not narrowed are unaffected.

Measured

On a host where this reproduces, reading a real URL to EOF and counting bytes. youtube-source is the caller that hits this: its YoutubePersistentHttpStream sets useHeadersForRange() to false, and its itag 18 fallback arrives with no contentLength of its own. True size 752848816:

lavaplayer	youtube-source	bytes read
stock	stock	11862016
stock	+ range continuation	11862016
this patch	stock	752848816
this patch	+ range continuation	752848816

Rows two and four are there because I had assumed the subclass would also need work to keep going without a length. It does not — with the length left unknown, the existing rangeEnd == contentLength check in YoutubePersistentHttpStream.internalRead already falls through to reconnecting for the next window. CONTENT_LENGTH_UNKNOWN is Long.MAX_VALUE, so the clamp in getNextRangeUrl() does not fire either. Only this change is needed.

Effect downstream

youtube-source currently carries a workaround for this (lavalink-devs/youtube-source#235): it issues a Range: bytes=0-0 before streaming purely to learn the total. Its maintainer merged that with the reservation that a request no real client makes gives YouTube one more thing to fingerprint. With this, that request can be dropped.

The stock-lavaplayer rows above are the current behaviour, i.e. a track whose last ~7 to ~12 seconds are silently lost while the player reports it as finished.

[lpaiu-cs]

End-to-end now, on a live Lavalink rather than a harness.

Built Lavalink 4.2.2 from source against a 2.2.6 checkout carrying this change, and ran youtube-source main with the Range: bytes=0-0 probe from lavalink-devs/youtube-source#235 taken back out — so the total is never learned anywhere, and the only thing keeping playback going is this patch.

A 298s track played 297s and ended finished. Same track was losing 7s before. Over the run:

decode exceptions ("Something went wrong when decoding the track")   0
"Recovered content length" (the probe, now removed)                  0

The range sequence for that track is the part I would point at:

range=0-11862014
range=142493-12004507
range=12026824-23888838      <- asks past the end of the file

That last window is unclamped because the length was still unknown when it was built, and the server simply returns the remainder. With the probe in place the same track ended range=12026824-12164812, clamped to a known total. So the stream really does run the whole way without ever learning the length, which is what the sentinel was for.

The only error in the window is an unrelated metadata-stage login wall two seconds before playback started, which does not involve the streaming client.

For the record on the earlier numbers in the description: those came from a standalone harness driving YoutubePersistentHttpStream directly and counting bytes. This is the same change measured through actual playback instead.

[devoxin]

Something I must ask -- is range clamped exactly to the content's length when it is known?

YouTube deployed a change years ago that meant invalid, or too large ranges (including those with a range end larger than the content length itself) would automatically throttle the stream down to about 80KB/s. Setting the correct range would unthrottle it (alongside the other numerous challenges and parameters.)

Does this behaviour remain unaffected or are we going to run into issues with throttling?

[lpaiu-cs]

@devoxin
Good point.

Clamping when the length is known is untouched — getNextRangeUrl()
still clamps to contentLength. This only changes the path where a length never
arrives at all, which in practice is just the itag 18 fallback.

On throttling: I ran this on the host where the truncation shows up. Same URL,
three runs each — a window sitting entirely inside the file, one clamped exactly
to the total, and one a full buffer past the end. All of them land in the same
4-5.5 MB/s band, and the overshooting window wasn't slower than the clamped one.
Sustained reads didn't degrade partway through either. Only the final window ever
overshoots; every earlier one is exactly BUFFER_SIZE and well inside the file.

The 80KB/s behaviour seems to have turned into a hard rejection somewhere along
the way — corrupting n or dropping it gives a straight 403 on these URLs rather
than a slow stream. One limit on that though: the URLs we get always carry
ratebypass=yes, and it's part of sparams, so I can't strip it to see what
happens without it.

One other thing while I was in there — of 107 itag 18 streams I looked at, 16%
carried gir, and those bring a clen parameter holding the full length. Where
it shows up it's exact, but it isn't on every video, so I didn't build anything
on it.