Repository navigation
stereoscopic video support #249
Description
Activity
Some of us will be at TPAC so we can join you if you'd like to discuss this topic.
Yes, happy to add this to the Media WG agenda for Friday at TPAC.
Reacted by Rik CabanierIt would be helpful to describe how stereo information is coded into video container formats or codecs, with source links to associated standards (MPEG, RFC codec strings, etc.), so we can start to evaluate with implementers whether there is a clear path to interop for this sort of query.
Stereo information can be encoded into the stream in a few different ways:
- Google Spherical Video: https://github.com/google/spatial-media/blob/master/docs/spherical-video-rfc.md#StereoMode
- Google Spherical Video V2 (st3b box): https://github.com/google/spatial-media/blob/master/docs/spherical-video-v2-rfc.md
- WebM stereo element: https://www.webmproject.org/docs/container/
- Apple VEXU box for MV-HEVC: https://developer.apple.com/av-foundation/Stereo-Video-ISOBMFF-Extensions.pdf
- ... and possibly others which I am not aware of
Having thought about this some more, I think the stereoMode may need a fourth value for use with MV-HEVC. We propose that this value would be "multiview". This value could be used with any codec that supports multiple views, not just MV-HEVC.
This is especially important since MV-HEVC is backwards compatible, and most browsers will just decode MV-HEVC as HEVC, and only show one of the views. Using this "multiview" value and receiving a "supported" reply means that the browser is able to decode at least two views and send them to the left and right eye as indicated by the stream.
I am not very familiar with this domain, but I did look through some of the specifications you linked above.
- Google Spherical Video V2 has 5 possible enum values for stereo layout.
- webm has four.
- Apple VEXU has a different model with 4 boolean flags.
This suggests that stereoscopic support is specific to the container type, and may be better served by constructing a query with a mime type + codec string.
Isn't the mime type already part of the query, but in a different field? Why add it here?
Or, did you mean that the stereoscopic property should be a part of the string of numbers in the codec string?In its simplest form, a stereoscopic video is just a double-wide video where the left-eye data is in the left half of the frame and the right-eye data in the right eye of the frame. This can be done in any video format regardless of codec and container.
How you tell the decoder that that the video actually is stereoscopic does vary from format to format though. We think that keeping this separate from the mime is a worthwhile simplification.
left-right and top-bottom are by far the most common formats, and mv-hevc is new and quickly growing, which is I suggest we start with those formats. It is my assumption that we can easily add more values later if required.
Very brief summary of the TPAC conversation (please correct if I'm missing something):
Basically there were two concerns:
- MediaCapabilities should not worry about rendering, answers should be based on what the browser understands, and not what it can render.
- Are the values suggested here ( mono/left-right/top-bottom/multiview ) the right ones. Other standards have different ones, or maybe we can just reduce it to a bool?
After some consideration, I do agree that MediaCapabilities needs to be able to provide answers independently of the output device. This would be more like how HDR is handled, and less like spatial audio is handled. However, I wonder if we can make a compromise here: Maybe we can document the expectation that IF MediaCapabilities says "supported" for a particular stereoscopic type AND the corresponding media query (TBD) says the display can do it, then we expect the browser to show the stereoscopic video stereoscopically. The additional clarity would help both web developers and browser developers I think.
As for which values to use. Anecdotally, these four values are the most common that I see. Also don't think that we can get away with using a bool, because it turns out that there are a lot of ways to put a two frames into one (webm defines 11, but only 4 are supported), and people have been doing this for a long time, but most of those formats aren't really in use anymore. If we reduce the value to a bool, then we would need to specify what that bool means, and what do we do if the browser only supports left-right, but the video is top-bottom.
The next most common value is right-left. This encoding used to be more common because there is an attachment you can buy for some lenses which uses mirrors to record a stereoscopic image on the film/sensor. However, these mirrors usually cause the left image to be recorded on the right side and vice versa. AFAIK, nobody makes such devices anymore.
I believe the concern was that rendering capabilities should be surfaced through CSS Media Queries like it was done for HDR. Similarly, if media has stereo capabilities, they could be surfaced like for HDR.
I think an explainer with the problem statement will likely guide the media and css groups on what spec additions are needed.
I agree, writing an explainer sounds like a good next step.
We (Meta) would like to be able to ask the browser if it can play stereoscopic content.
Stereoscopic content comes in many forms, but the most common ones are "left-right" (meaning that he left half of the picture should be shown to the left eye, and the right half of the picture should be shown to the right eye) and "top-bottom". (top half goes to the left eye, bottom half goes to the right eye.)
I'd like to add a field to the video description called "stereoMode", which can would be optional. If present it should have one of the values "mono", "left-right" or "top-bottom".
Most regular browsers should return supported: false when the "left-right" or "top-bottom" values are used. Only browsers that can show different images to different eyes, such browsers in VR, or on 3d-TVs should return supported for such queries.
Since most browsers will not support the "stereoMode" field, any website that really wants to know if the browsers supports stereoScopic videos will need to make a query where they set stereoMode to something nonsensical. If that doesn't fail, then the browser doesn't support the stereoScopic field, and thus does not support stereoscopic videos either.