Access modes for synchronized text and audio

[jonaslil]

Section

https://w3c.github.io/publ-a11y/package-metadata-authoring-guide/#accessMode-sync

Describe the problem

In the section about auditory content, accessMode auditory is defined as follows:

When at least some of the information necessary to read a publication is encoded in auditory form, an auditory access mode is declared ...

In the sections about primary access modes, "information necessary to read the publication" refers to the main, authored content of the book, excluding for example publisher information, decorative images, and mood music or background audio.

By contrast, prerecorded narration in synchronized text-and-audio books typically conveys the (full) authored text. So based on the definition above, such publications would appear to meet the criteria for accessMode auditory alongside accessMode textual.

The subsection on Synchronized text and audio, however, describes audio playback as an "extra feature" and indicates that such books should not be assigned an auditory access mode, while still requiring an auditory sufficient access mode because the user can listen to the complete publication.

This seems difficult to reconcile with the earlier criteria for assigning access modes. For the other modalities, the presence of authored information in a modality is sufficient to declare an access mode, regardless of whether the same information is available through another modality.

Using the same logic, we could argue that a media overlays publication should not have accessMode textual because the content is also available in auditory form.

The argument relies on treating audio as an “optional” or “extra” feature. It is not entirely clear how this should be evaluated - for some users, prerecorded audio may be essential - or why it would affect the assignment of access modes.

Describe the fix or new feature you propose

Clarify whether media overlays publications should declare an auditory access mode, or why not under the current definition.

[mattgarrish]

For the other modalities, the presence of authored information in a modality is sufficient to declare an access mode, regardless of whether the same information is available through another modality.

Primary access modes mean you have to be able to process information in the listed form otherwise you won't be able to read the work - there's no "or" condition that says one or the other of what the author lists. Media overlays in the general case of full text and full audio synchronization are an enhancement - they are an affordance that lets you read the content in another form, and access modes don't take into account affordances.

Or put another way, media overlays are not embedded media in the work that the author requires the reader to access - and you can't even access the synchronized audio except through this extra affordance and a device that can play it back. Not every reader will have a device that is capable of giving them access to the audio version, and it certainly isn't critical that they be able to access the audio.

That's why we say not to list an auditory access mode. The unique case for epub is that it can almost mimic an audiobook. When the epub is audio with bare text synchronization, then there's no way to read the work except auditorily. In that case, if you don't have a reading system capable of rendering the audio you cannot read the publication, so that's when it becomes the primary access mode.

[GeorgeKerscher]

This sounds right to me. We say in DAIOSY that audio is king and what we mean is the main mode of presentation and control is through the audio. In EPUB, text is king, because it is the main mode of presentation and control.

[mattgarrish]

Right, I'm doing some reworking of the access mode sections after the discussion yesterday. I think it might be best to consistently call what we name access modes right now as the primary access modes. That should make it easier to differentiate how to express the default nature of the book from the sufficient access modes, which should focus on how the content can be read.

I'm going to merge the two sections under a single "Access modes" heading, too, and add an explainer so that we can fully detail their purposes in one spot.

All that to say that the next PR will hopefully add some additional clarity on this problem and why we have to pick one primary access mode for these kinds of two-in-one books that epub allows.

[HadrienGardeur]

The main issue in EPUB is figuring out whether an EPUB Media Overlay has partial audio or full audio. If your file has full audio, I think it's fair to say that neither audio or text is king, both are valid options.

For example in our upcoming Thorium Mobile app, we provide two main options to open an EPUB:

• read
• or listen

We remember this option per book, so if you've read book A but listened to book B, we'll remember that.

When you "read", it displays text by default and you can turn read aloud on whenever you want.
When you "listen", it opens audiobook-like controls.
In both cases, we use Media Overlays by default (when present) but fallback on TTS. Users are also free to use TTS with their preferred voice over Media Overlays as well.

[mattgarrish]

both are valid options

Yes, but this is what sufficient access modes capture. That's why regardless of how the primary access mode(s) are specified, there's always a sufficient access mode of auditory if you have the full audio available via a media overlay.

We don't live in an oral tradition where books are narrated and then a text equivalent is created after for people who can't hear the content. That's the only realistic scenario where you would list an auditory primary access mode when both presentations are available, but any content created this way would be exceedingly small in comparison to the other way around. That's why when it comes to defining the primary access mode(s) when two presentations are provided it's not so much that text is king as the production for visual readers is king because that's what the author wrote.

And keep in mind that we're downplaying primary access modes at this point because they only give a picture of the content as it was originally authored, minus any accessibility affordances, not all the ways it can be read. The metadata has been dropped from required to recommended in the 1.2 standard. The sufficient access modes are more important because they paint the complete picture, which in the case of a book with synchronized audio is to state that there is both the default reading mode (textual/visual/textual+visual/etc.) and another that is auditory (plus potentially others).

[HadrienGardeur]

Yes, but this is what sufficient access modes capture. That's why regardless of how the primary access mode(s) are specified, there's always a sufficient access mode of auditory if you have the full audio available via a media overlay.

Agree that accessModeSufficient is ideally the metadata to convey this information. In practice though, it's not always documented.

That's why we're considering looking at additional ways to detect full vs partial audio in Readium toolkits. For example if every spine item has a Media Overlay, that's usually a fairly good sign for coverage.

[samimaattaCelia]

Isn't this document referencing 7.2.1 auditory (latest editor's draft of Schema.org Accessibility Properties for Discoverability Vocabulary), which states

7.2.1 auditory

Indicates that the resource contains information encoded in auditory form.

It would seem weird to me that a publication that has text and audio content wouldn't have the "auditory" metadata assigned to it.

Also:

Primary access modes

There isn't any mention of "primary" in the Schema.org's accessibility metadata and introducing such concepts here seems to complicate it further.

AccessMode only tells you that there is some text or some audio (that have information). This metadata doesn't tell you if they are sufficient to understand the whole publication. If anything, AccessModeSufficient is closest to the meaning of "primary access mode" - and there are multiple primary access modes if the whole publication can be understood in textual or auditory form (as it is in the document right now). There is no hierarchy in the metadata, so I don't think it is fruitful to think about which encoding is whose "king" in this regard.

I think this dicussion makes the point of #786 much clearer, which is good!

[mattgarrish]

AccessMode only tells you that there is some text or some audio (that have information).

That's what it tells you about the authored content, not including affordances.

A media overlay for a full text work is a separate representation provided to make it accessible to those who cannot see the content or need help following the text. It duplicates the information, in other words; it does not provide new information that must be accessed in order to read it.

Access modes indicate that you must be able to process content in the listed representations otherwise the work will be inaccessible to you. It's not a selective list of senses that could be useful.

The only times that a media overlay has essential audio is when it isn't synchronized to full text or the very rare case of text being synchronized to an oral work (in which case the text becomes the affordance).

[samimaattaCelia]

Could you prodive some citations to the metadata where these conclusions could be derived from? To me the accessibility metadata make no difference between authored content and affordances.

Just trying to wrap my head around this: If there was an ebook with only a partial media overlay, for example the index or the appendices are not read aloud, what would the metadata be for this title? This ebook wouldn't have AccessModeSufficient values auditory or textual, auditory and by this issue's section it also wouldn't have AccessMode auditory. In this case the user wouldn't know that the ebook contains audio at all.

If there was a children's picture book with only images as pages, but the visual text and images would be read aloud and described in audio form, what would this title's accessibility metadata (AccessMode and AccessModeSufficient) be? I would say the AccessModes are visual and auditory. The AccessModeSufficient would be visual and auditory. Based on this issue's section, I would leave one of AccessModes out, but which one?

[jonaslil]

Good questions by @samimaattaCelia.

Even if image descriptions don't count for declaring a textual access mode (which may be debatable), complete audio representations of a work are a different matter. They exist as separate publications, seen as manifestations of the same work as the printed or ebook edition (if such editions exist). This is fundamentally different from image descriptions.

For libraries serving print-disabled persons, audiobooks have traditionally been the default format. At Celia, textual ebooks are used for specific purposes, but they're not what most of our patrons are used to. If they receive a book that combines text and audio, they may see the text as an extra enhancement rather than the primary mode. This is reflected in the term MTM uses for synchronized text and audio books in their service: "talbok med text" ("audiobook with text"). Our colleagues in Denmark and Norway use equivalent terms.

We use a Finnish term that translates roughly as "combination book": the book combines text and audio, without deciding which mode is primary. As Hadrien's example shows, such books can be used purely as audiobooks - nothing forces you to engage with the text. And vice versa. Some devices and apps used in the special library sphere do not even provide access to the text or visual content.

Audio is also increasingly central in mainstream publishing. In the Nordic countries, audiobooks have become the default digital format for trade publishing. Digital books are mainly consumed in subscription services known as audio book services, although ebooks are also available there. When one of these services recently introduced synchronized audio and text, it was presented as a way to "erase the boundaries between formats" and "make audiobooks even more engaging" - not as an accessibility enhancement to the text.

My point is that when an audiobook (access mode auditory) and a textual book (access mode textual) are combined into one publication, it should declare both access modes. Declaring only textual would be misleading to both our library patrons and mainstream users.

We also need to consider publishers who create this metadata. If auditory simply means "contains pre-recorded audio", this is not hard to explain and apply. If we complicate it much beyond that, we risk losing clarity and reliable data.

A balanced view may be that the the affordance lies not in the audio alone or the text alone, but in their combination with synchronization. That is why we declare synchronizedAudioText as an accessibility feature.

Regardless of how the work originated, the end result from a user perspective is the same: a book with full synchronized text and audio. The metadata should reflect that. Accessibility metadata describes how users can access the EPUB publication - not how the work was created.

[mattgarrish]

A good starting place would be to read the metadata techniques for epub: https://www.w3.org/TR/epub-a11y-tech-11/#sec-discovery

But that document is one of the sources for this one and it's the guidance that has been in place for 10 years now. How to handle media overlays isn't new or something we've made up just now.

The metadata also goes back further than schema.org as it was submitted to them by Benetech/IMS global, a process that many of us in this group were part of. If anyone knows how the metadata is expected to be applied, it's this group. We're not trying to come up with novel interpretations of other people's metadata. This stuff was created exactly so that ebooks could be described more accessibly.

But if you want to go back further, access mode was defined in the Access for All metadata standard as (emphasis mine):

An access mode through which the intellectual content of a described resource or adaptation is communicated; if adaptations for the resource are known, the access modes of those adaptations are not included.

https://www.imsglobal.org/accessibility/afav3p0pd//AfA3p0_DRDinfoModel_v1p0pd.html#toc-7

The sole purpose of access modes is to describe how you would have to read the content if no adaptations are available. It's not giving all the possible ways to read the work. That's what sufficient access modes do and why they are the more important to list.

So for the example you cited of a visual work, the metadata in the epub would be:

<meta property="schema:accessMode">visual</meta>
<meta property="schema:accessModeSufficient">visual</meta>
<meta property="schema:accessModeSufficient">auditory</meta>
<meta property="schema:accessibilityFeature">synchronizedAudioText</meta>

The access modes tells you this is a visual work. The sufficient access mode tells you you can read it visually or auditorily, but knowing it is a visual work you can also expect that you're not going to get exactly the same information from it unless you can also see the content. You might only get the words in the images read out. Or you might get a brief description of the images. (Edit just to say that whether text alternatives are provided is also likely to be in the list of features but I'm not going to try and explain every possibility for this case.)

[samimaattaCelia]

Thanks for providing the reading! We share the goal of providing understandable metadata to the readers of ebooks. It seems confusing that the notion of "adaptation" has been dropped in Schema.org when it comes to the AccessMode metadata, when it was relevant in 2012. Why was this done?

Is the aim of using the AccessMode metadata that there would always only be one entry per publication? It seems to boil down to it.

On the children's book example

I'm now confused about the AccessModeSufficient. In the EPUB Accessibility Techniques 1.1, section on sufficient access modes it states that

-- When adding a schema:accessModeSufficient property with only a single value, all information in the publication must be available in that mode.

All of the information should be in audio form for <meta property="schema:accessModeSufficient">auditory</meta>. I wouldn't include this metadata if only the visual text is narrated, but the images are not described in audio as well.

I also wouldn't include <meta property="schema:accessModeSufficient">auditory</meta> for a publication with only a partial media overlay that is fully textual, but otherwise narrated, not counting indeces or the appendix. Based on the logic for the children's book example, it should be included?

[mattgarrish]

It seems confusing that the notion of "adaptation" has been dropped in Schema.org

Yeah, I don't really like that change, either. It's going back a lot of years now, but I seem to recall there was disagreement about whether the definition should focus on what is excluded from its calculation. Making it an implementation detail for epub does make it generally ambiguous if you read only the schema.org definition. @madeleinerothberg or @clapierre, do you recall if it was to make a certain vocal member of the group happy or some other reason that we did this?

Even the examples referenced from the schema.org page for accessMode still had the restriction present: https://github.com/daisy/epub-revision-a11y/wiki/ePub-3.1-Accessibility--Proposal-To-Schema.org

Is the aim of using the AccessMode metadata that there would always only be one entry per publication?

No, there can be multiple depending on what kind of content is in the book. If it's text and images, you would list a textual and a visual access mode, for example.

But access mode isn't as helpful as knowing the sufficient access modes and there we do want people to focus on setting which single sets are viable. It's more useful to know the entire work can be read by text or auditorily than to get the mix of all the access modes. You can always infer that one sufficient mode is the sum of all the access modes.

Media overlays just upsets the balance because it introduces audio and there's no way to separate the two presentations. So if you set auditory for a media overlay, then you can't be sure if auditory is being set because the only audio comes from the overlay or whether there is additional embedded audio that you'd have to be able to hear to read the work. That's why it's a unique case where if it duplicates the intellectual content in another form then it has to be considered an affordance.

I wouldn't include this metadata if only the visual text is narrated, but the images are not described in audio as well.

I also wouldn't include <meta property="schema:accessModeSufficient">auditory</meta> for a publication with only a partial media overlay that is fully textual

Yes, those are both correct. The overlay has to provide an equivalent auditory experience before you would list is as sufficient. If the audio limits you to only being able to read part of the work, then it's not sufficient. I was just trying to keep things simple for this discussion.

[martinpub]

Thanks for raising this issue @jonaslil, and for the background and explanation provided by @mattgarrish and others in this thread.

At MTM we have also struggled a little bit with following the logic of accessMode for full text full audio books, like MTM's "Talking books with text", for example. It seems like the division between "authored content" and "accessibility affordances" is important to this understanding, but a clear distinction between these two is not apparent from a universal design perspective.