Playback model for Media Overlays #2943
Description
Section
https://w3c.github.io/epub-specs/epub34/rs/#sec-behaviors-loading
Describe the problem
We're currently in the process of implementing support for Media Overlays in the Readium Swift toolkit, which means that we're carefully reading the spec but also creating examples for edge cases.
The current spec is rather light for the playback model, in EPUB Reading Systems 3.4 it says:
Playback MUST start at the media overlay element which corresponds to the desired EPUB content document starting point. Note that the start of an EPUB content document could correspond to an element at the start or in the middle of a media overlay.
When the media overlay document finishes playing, the reading system SHOULD load the next EPUB content document (as specified in the package document spine) and also load its corresponding media overlay document, provided that one is given.
On the authoring side, the spec says:
If an EPUB content document is wholly or partially referenced by a media overlay document, then its manifest item element MUST specify a media-overlay attribute. The attribute MUST reference the ID [xml] of the manifest item for the corresponding media overlay document.
The media-overlay attribute MUST only be specified on manifest item elements for EPUB content documents.
Manifest items for media overlay documents MUST have the media type application/smil+xml.
In the spec, nothing forbids a single SMIL document to cover multiple items from the <spine> and it's even hinted as a possibility:
Note that the start of an EPUB content document could correspond to an element at the start or in the middle of a media overlay.
Let's imagine that I have 3 HTML documents with a single SMIL:
- first document references the SMIL
- second document doesn't references the SMIL
- third document references the SMIL
Let's say that I start playback from the first document. Reading between the lines:
- "When the media overlay document finishes playing" means that I go through the entire SMIL, which will skip the second document and go through every element related to the third document
- "the reading system SHOULD load the next EPUB content document (as specified in the package document spine)" feels much more ambiguous and I'm not sure how to understand this one
What's the next EPUB content document in this context?
- Am I at the end of the publication since the SMIL brought me to the third and last document?
- Or should I consider that I was previously in chapter 1 since this is where I started the playback?
With the second option, I would have to:
- play the SMIL entirely
- then use TTS for the second document
- then play the SMIL partially for all elements related to the third document
That's super weird.
With the first option, my experience would vary greatly based on where I start the playback:
- on the first document it would go through the entire SMIL and that's all
- on the second document it would go through this document with TTS then partially play the SMIL
- and on the third document it would partially play the SMIL
There's a bit of an inconsistency, but it doesn't feel nearly as bad as the first one.
Just to make things a little more fun, we also need to consider the case where the SMIL goes back and forth (#2940).
Let's take the same example that I've used in the other issue:
<smil xmlns="http://www.w3.org/ns/SMIL" xmlns:epub="http://www.idpf.org/2007/ops" version="3.0">
<body>
<par epub:type="panel">
<text src="page1.xhtml#panel1"/>
<audio src="audio/page1-panel1-description.mp3"/>
</par>
<par epub:type="panel">
<text src="page2.xhtml#panel2"/>
<audio src="audio/page2-panel1-description.mp3"/>
</par>
<par epub:type="panel">
<text src="page1.xhtml#panel3"/>
<audio src="audio/page1-panel2-description.mp3"/>
</par>
</body>
</smil>In this example, we go back and forth:
- page 1
- then page 2
- then back to page 1
What's the "next EPUB content document" in this case?
Is it page 2 because when the Media Overlay finishes playing I'm back on page 1?
If that's the case, the playback model would mean:
- playing the SMIL entirely (both page 1 and page 2)
- then playing the SMIL again partially (starting at panel 2)
- but I would be back again to page 1 and stuck in a loop
Just to be crystal clear: this use case is valid for some publications. In highly illustrated publications, the reading order can finish on the same page of the spread where you started (I could probably find an example in Promethea from Alan Moore).
Describe the fix or new feature you propose
We need to clarify a few things in our playback model:
- confirm that once we start the playback of a Media Overlay, we go entirely through it
- define what the "next EPUB content document" means in this context
- and add a rule where we skip adjacent content documents if they reference the same SMIL