Spreads and Media Overlays

[HadrienGardeur]

Section

https://w3c.github.io/epub-specs/epub34/authoring/#sec-media-overlays

Describe the problem

With pre-paginated EPUB files, a number of publications can either have:

• content that requires reading back and forth across the spread
• and/or content that overlaps across the spread (partially present in both pages)

This is true for comics, but it's also something that you can find in travel guides, textbooks, cookbooks and art books just to name a few.

In the current version of the specification, we don't provide any hint whatsoever on how this should be handled by authors or RS.

For content that goes back and forth (left, right then left again for example), it's technically possible to create a single SMIL that goes back and forth between two resources. I've created such an example and while it didn't raise any warning or error in EPUBCheck, I could find a RS supporting this correctly:

• Apple Books didn't play any audio (not sure why) and didn't went back to the left page when playing
• Thorium did play the audio properly but it got stuck in a loop (most likely because the same SMIL was referenced on both pages)

For elements that are visually present on both pages of the spread, that's a different story altogether.

Unlike region-based navigation, we don't have a concept for "synthetic regions" in SMIL.

At an abstract-level we need to convey the following information:

• the spread is needed to synchronize with an <audio> element
• and that a specific <audio> element is tied to two <text> fragments instead of a single one

This goes against our content model and it's unclear how this could be addressed.

Describe the fix or new feature you propose

We need a dedicated section for spreads and clear guidelines for authors and reading systems.

For content where we go back and forth, I think that having the SMIL associated to both pages of the spread is fine but this requires guidelines for RS. For example:

• how to avoid loops (which could be tricky if the RS follows the <spine> and then fetches the associated SMIL)
• where the playback should start in such cases for the second page in the spread (first <par> referencing the resource in the SMIL?)

For content that exists across the spread, we could either tell authors to use a single centered page instead (but that's very impactful on smaller devices) or we could explore how Media Overlays could evolve to handle this use case.

[HadrienGardeur]

I've written this issue with the current content model in mind but this is also an issue for <img> in SMIL (#2919 and #2883):

• with XHTML-based content, we need to reference two fragments using <text> for a single <audio> element
• with images, we need to reference two fragments using <img> for a single <audio> and/or <text> element

[sueneu]

Content that overlaps a spread is common in children's picture books. This can lead to an unsatisfactory reading experience when spreads are not rendered on screen even without overlays.

Most devices can be re-oriented to horizontal (landscape) by the user. Does that make it less troublesome to suggest single centered pages?

Two situations come to mind that might require different solutions.

1. Dialog across a spread.
   For instance, Charactor1 says something on the left hand page, Character2 replies on the right hand page, then Charactor1 responds on the left, etc.
   Could the rendering bounce between the pages as the audio plays? Like for frame-by-frame comics?

2. Single illustration, sometimes with text, that crosses the spread.
   In this case, using a single centered page would make the media overlay easier to implement and understand, but could lead to a jarring change visually in the scale of rendering the book. I'd love to see an animation moving across the spread as the media overlay plays.

[HadrienGardeur]

I've created a test case for content that goes back and forth in the spread: https://github.com/HadrienGardeur/accessible-epub-comics/tree/main/spreads

[HadrienGardeur]

Here's a video for Thorium Desktop where we can notice three issues:

• it loops back
• the third panel is never read properly
• there's a positioning issue (this could be completely unrelated)

Back.and.forth.in.spreads.with.Thorium.Desktop.mov

I think that this issue can be easily explained:

• RS expect the <spine> to be the primary place that defines progression in a publication
• when using Media Overlays, the SMIL for the current ressource is used

In this specific case, this is probably what happens:

• Thorium Desktop sees that there's a SMIL for page 1 and starts the audio playback from the first element referencing page 1
• at some point in the playback it sees that the SMIL now references page 2 and navigates to page 2 (not needed in this case because of spreads)
• the SMIL for page 2 is fetched from the first element referencing page 2
• but when it sees a reference back to page 1 it goes back to step 1

To support this use case, you need to treat SMIL very differently and in a way it needs to become what drives the progression throughout the publication.

[HadrienGardeur]

@sueneu your two use cases are exactly consistent with what I described at the top of this issue

Dialog across a spread.
For instance, Charactor1 says something on the left hand page, Character2 replies on the right hand page, then Charactor1 responds on the left, etc.
Could the rendering bounce between the pages as the audio plays? Like for frame-by-frame comics?

I'll need to find a better example to illustrate it, but here's a test EPUB that I built on top of my example of an accessible comic: https://github.com/HadrienGardeur/accessible-epub-comics/tree/main/spreads

The SMIL looks like this:

<smil xmlns="http://www.w3.org/ns/SMIL" xmlns:epub="http://www.idpf.org/2007/ops" version="3.0">
  <body>
    <par epub:type="panel">
      <text src="page1.xhtml#panel1"/>
      <audio src="audio/page1-panel1-description.mp3"/>
    </par>
    <par epub:type="panel">
      <text src="page2.xhtml#panel2"/>
      <audio src="audio/page2-panel1-description.mp3"/>
    </par>
    <par epub:type="panel">
      <text src="page1.xhtml#panel3"/>
      <audio src="audio/page1-panel2-description.mp3"/>
    </par>
  </body>
</smil>

As you can see I go back and forth between pages:

• panel in the left page
• then panel in the right page
• then back to a panel in the left page

Single illustration, sometimes with text, that crosses the spread.
In this case, using a single centered page would make the media overlay easier to implement and understand, but could lead to a jarring change visually in the scale of rendering the book. I'd love to see an animation moving across the spread as the media overlay plays.

There's a good illustration for that one in Region-based navigation.

[HadrienGardeur]

Cross posting this on both issues, but here's a rather extreme example of going back and forth in the spread:

[smoores-dev]

I'm pretty sure that Storyteller handles this "correctly" (but I think it does so by not properly abiding by the spec). When you're playing audio, even on the read view, Storyteller takes the audio progression as the source of truth for progression (under the assumption that it would be more jarring to jump around in the audio than in the text). This means that Storyteller would navigate "back and forth", and then continue forth onto the next panel.

[dalerrogers]

@HadrienGardeur: I work in multimedia. In a video file, the main timeline is king. I can pull in resources from wherever I want. In EPUB, my understanding is that the only file that bridges all of the content is the opf. The XHTML files are discrete units of content. However, the opf is not a file to be executed. If is more of a reference file for the reading system. My understanding is that SMIL documents are tied to a content document (XHTML). Since I'm not an expert with SMIL, can a SMIL document stand on its own? Can it be called from the Spine section in the opf and can the SMIL document have its own "timeline" that uses XHTML resources as it sees fit?

[HadrienGardeur]

Quick update on this one:

• For the case where we go back and forth between pages in a spread, I think that I've mostly addressed this use case with Playback model for Media Overlays #2943.
  To summarize it: "we discover SMIL through the <spine> but once it takes over, SMIL MUST decide how we navigate through the publication".
• For the second use-case (visual elements that are partially drawn on both pages of the spread), I really don't see how this could be handled with our spec currently.

The only place where we currently work with a region drawn across the <spine> is in Region-based navigation with synthetic regions: https://idpf.org/epub/renditions/region-nav/epub-region-nav-20150826.html#sec-3.5.2

This is what it looks like:

<li>
   <span>
      <a href="page22.xhtml#xywh=percent:75,0,25,25"></a>   // left region
      <a href="page23.xhtml#xywh=percent:0,0,25,25"></a>    // right region
   </span>
</li>

An equivalent would require:

• a new container element
• which allows two <text> or two <img> elements

I'll need to explore SMIL to see if anything could be a good fit, but this radically changes the current content model that we have in our spec and opens up all sorts of new problems (how do we deal with synthetic regions that are not adjacent?).