Monday, January 20, 2014

Personalized web for the assistive technology

Sometimes the web authors provide a different content for screen readers than they do that for sighted users. That could be an additional content like "skip to content" navigational links or set of landmarks to create a more reliable document structure. In some cases the web author might want to remove a content, for instance, duping links, or make extra tricks to keep the content accessible if, for example, the author gets out of the standard HTML. In ideal case of course the content is supposed to be quite the same but because of design issues and HTML imperfection it doesn't really happen. The web repletes with examples of special content for the assistive technology.

The need of alternative content

Usually authors don't need to put significant changes into a web page to make it accessible. Keeping in mind usability aspects and following best practices is often enough for good results. In other words this is the kind of changes (except some ARIA) that is supposed to be useful for everybody. Sure it doesn't count the web pages having large pieces of ARIA but that's rather an area of large web apps and custom widgets.

That's how it is but nowadays tendency is getting changed and certain web apps want to provide whole portions of alternative content for the assistive technology.

A few examples are good for demo proposes.

Couple examples

Shared video example. If blind and sighted users want to share the device to watch the movie then it might be good idea to have audio descriptions shown (announced) for the blind user only.

Camera apps. It may be another use case of separate content for blind and sighted users. A camera app shows what's on the camera and may have graphic-only interface like green rectangle showing the thing that is currently in focus. A screen reader user may benefit if the face detection software beeped when face gets in focus since it gives a good chance of nice picture.

Another example can be QR code reader software which helps to find QR code label on the product. In general all camera apps may benefit from giving special instructions for assistive technology users.

Integration vs personalization

So alternative content for assistive technology can be a part of the web app design. A next question is how the alternative content can be added into the web page. Would it be one integral app explicitly and implicitly containing special content or will it be personalized version of the app designed for the assistive technology.

Both approaches have own benefits and disadvantages. So that personalization approach wins in performance since potentially the app doesn't need to combine two different versions into one (and actually it is a big concern of web apps vendors from what I hear). Benefit of integration is people get all-in-one solution what mean users share the app and usually have same experience.

Privacy concerns

A big issue of personalization the people talks about is privacy. If you want to have a personalized version of the web site then you have to tell the web site you use the assistive technology.

The idea of sharing personal information is not comfortable in general for many people and it's quite understandable. But you need to keep in mind that those who wants to know whether the user uses the assistive technology quite likely have a way to detect this. These are solutions like screen reader sniffing flash plugin or JS script based on the difference in content navigation. For example, it didn't take much time for me to write a simple script (can be found attached to Mozilla bug) to detect the NVDA screen reader. These solutions are not perfect and may give false positives but I'm pretty sure they can be improved if somebody wanted.

On the another hand if the user says he wants a specialized version for assistive technology then it doesn't necessary mean the user has the assistive technology running and of course it doesn't necessary mean the user have to share what kind of assistive technology he uses.

So of course there's a privacy concern but it's not bigger than, say, privacy concern of Geolocation API. The difference in sharing and not sharing is rather seeming. After all the user decides.

Tech party

Not sure which approach will get dominant, perhaps that will be some reasonable mix of them. Anyway it would be a good idea to provide web authors the different techniques to implement either approach.



Just a method to detect the presence of assistive technology such as a screen reader allow the web app to load specialized scripts and build personalized version of the app. The idea in implementation can be quite similar to Geolocation API. If the web app wants to know whether an assistive technology is running then the user gets asked if he's ok to share this info. If the user agrees then personalized app is loaded.


It can be used to hide certain parts of the web page from the assistive technology and (quite a recent change) it can be used to create web page portions for assistive technology only (not visible/operable/etc for sighted users).

Actually at this point ARIA spec doesn't allow aria-hidden to create web content for AT. However w3c pushed this option into the law by resolving HTML5 bug. I admit that next ARIA spec should get in sync with the change. It's worth to notice however that WHATWG spec hasn't been changed yet and probably it won't be.

Also ARIA recommends very limited usage of aria-hidden:

Authors MAY, with caution, use aria-hidden to hide visibly rendered content from assistive technologies only if the act of hiding this content is intended to improve the experience for users of assistive technologies by removing redundant or extraneous content. Authors using aria-hidden to hide visible content from screen readers MUST ensure that identical or equivalent meaning and functionality is exposed to assistive technologies.
I suppose it may be considered as temporary advice and will be neutralized as soon as the web gets more apps having special versions for the assistive technology.

So here's a demo of the approach

  <div aria-hidden="true">An ordinal version</div>
  <div hidden aria-hidden="false">An assistive technology version</div>

The minus of the aria-hidden="true" is the user navigable content is hidden from the assistive technology. You may want to read the post why I didn't fell in love with aria-hidden="true" as it's stated. My opinion haven't really changed over years.

The minus of aria-hidden="false" is AT visible only content is not keyboard navigable until AT has special support of it, it's not focusable and has neither dimensions nor position what is a certain restriction for AT. Also it's good to note, screen readers like Orca would ignore aria-hidden="false" content in general because it requires a virtual buffer feature for content navigation but Orca doesn't have it in question.

So aria-hidden has bunch of disadvantages but I admit it has one good benefit which is its simplicity for the web author.

CSS media


CSS media features are designed to provide styling depending on features the device has. So if screen reader is detected then the web app can show or hide portions of the page when it makes sense for screen reader users, for example,
@media (screenreader) {
  .sr {
    display: none;
<div class="sr">hidden from AT</div>

As you see it's very easy to change the web page and personalize it for the assistive technology user. This technique doesn't have disadvantages of aria-hidden because all content is rendered on the screen. So that the content has position and size, it's focusable and navigable by standard ways.

Nevertheless CSS media is also disputable approach (check out Mozilla bug for details). Note, afaik it's not supported by web browsers yet.

Two screens option

Same screen approach looks quite appealing but it doesn't answer to all designing needs the web authors want. For example, in the case of shared screen to watch the movie it might be a nice option if audio descriptions were shown only for those who needs them.

This idea leads to the option to render CSS media styles virtually. Technically speaking that means the presence of two screens, i.e. one version is rendered on the display (that's a movie), the second version is rendered in memory, for assistive technology only (subtitles). Nowadays a similar thing is implemented for HTML 5 canvas shadow content: it's not visible on the display but the assistive technology literally sees it, i.e. it have an access to layout, position and dimensions, and it can navigate it.

It's not clear however how to share input devices like mouse and keyboard, but probably it could be nicely resolved this or that way. This approach is quite close to aria-hidden technique since it allows same design techniques but it doesn't have disadvantages of it because the content is still rendered.

The web is going to change the way it handles the assistive technology. Quite soon I think.

Monday, January 13, 2014

Accessible Mozilla: tech overview of Firefox 26 and Firefox 27

Here's a list of accessibility improvements for assistive technology in Firefox 26 (release) and Firefox 27 (beta).


HTML label gets a label accessible regardless used CSS styles (see bug), for example,

<label style="display: block; overflow: hidden;">Label</label>

LABEL_FOR relation is exposed on the label accessible implicitly associated with the control (see bug):



In Firefox 27 we started to create a generic accessible tree for MathML elements, you can read here for details.

Text interface

We've got a bunch of text interface fixes including regression fixes introduced in Firefox 25 timeframe.

Arabic and Herbew character bounds are exposed correctly now (see bug).

XUL link accessible (used by Firefox UI) implements text interface (see bug).

Text selection

We reworked the TEXT_SELECTION_CHANGE eventing. In particular that allowed to fix events in case of selection spanned through several text objects like HTML paragraphs (see bug) and one pretty old Thunderbird bug where caret move events were missed (see bug).


Firefox 27 has got IAccessible2::relationTargetsOfType (see bug) method from latest IAccessible2 release as well as new CONTAINING_DOCUMENT/TAB_PANE/APPLICATION relations (see bug). Check out this post for summary of IA2 1.3 release changes.


ISimpleDOMDocument interface is implemented as tear off in Firefox 27. The change is similar to the one we did a year ago so it shouldn't hurt anybody. Just in case.

Tuesday, November 5, 2013

IAccessible2 relation API

Before IAccessible2 Mozilla practiced all manner of extensions of MSAA. I don't mean I exalt the forethought of MSAA API, I'd rather say this is an example of necessity is the mother of invention. But fortunately MSAA was extensible enough to workaround some lacks of it. Firefox ignored MSAA spec occasionally, implemented tricks and - what a surprise - some of them are in use by modern screen readers nowadays. 

All aforesaid is applicable to accessible relations. MSAA has accNavigate method to navigate the accessibles in tree hierarchy and layout order. Firefox introduced a bunch of extra constants for this method to expose accessible relations. That was a neat approach so that after IAccessible2 invention and implementation the assistive technology wasn't in hurry to switch to the right API. accNavigate was light and simple and had only one disadvantage: it didn't allow you to get multiple targets. IA2 relations in turn didn't have this limitation but API appeared heavy and Firefox implementation was quite slow those days. Firefox 4 implemented a fast version but it didn't change things. That made me think Firefox wasn't a stumbling block here. So

IAccessible2 1.3 refreshed relations API and we've got relationTargetsByType method which allows to get all targets of the given relation type. In other words this is a multiple target version of MSAA accNavigate: plain and simple. The method was implemented in Firefox 27.

I'm curious how fast the assistive technology will pick it up. Otherwise do we need those multiple targets at all?

Tuesday, October 29, 2013

MathML accessibility APIs

Browsers traditionally overlook MathML accessibility besides MathML implementation itself by some of them. Because of that there is a bunch of 3d parties software to display MathML and expose math content to assistive technology. They do a good job but what relates to screen readers support it's far from being perfect.

So MathPlayer works with IE only. MathJax, a cross browser math renderer, claims they do accessibility by means of MathPlayer. Last time I checked MathPlayer did a trick by adding accessible properties like accessible name to MathML nodes in accessible tree. There's a bunch of problems with this approach like the user can't control the speech output. However I must admit it makes screen readers to read math.

Main problem here there's no right API and they have to use existing APIs which is Procrustean bed for math. The browser and assistive technology have to invent something that describes math well. Until that there's no right tool to succeed. I should notice that above-mentioned primarily applies to Windows and Linux worlds. Much to my surprise WebKit has got a math accessibility API on OS X last year. I didn't hear VoiceOver picked it up yet but as soon as it does the MathML content should become accessible on Mac.

In Firefox we have a meta bug to track MathML accessibility work. As the first step on this way Firefox 27 started to create generic accessible objects for MathML elements. It's not so valuable by itself because a generic accessible don't expose any math semantic but it allows the assistive technology to navigate the math and watch for tree mutations. Next we could follow WebKit effort by extending ATK and IAccessible2 APIs to make math accessible on Windows and Linux but I have been told libraries that screen readers rely on to process MathML speak also MathML language. It might be unwise to split MathML into atoms of high level API to make the assistive technology to reconstruct MathML on its side.

Gecko exposes ISimpleDOMNode interface providing a direct access to DOM for assistive technology. I never was a devotee of ISimpleDOM because I believe that high-level APIs like IAccessible2 are more efficient. But in case of MathML it's apparently not true. Having said that I think it'd be good to have something more sophisticated than plain DOM to implement, for example, an extended math navigation. Otherwise I think AT have to learn some MathML.

So we stopped at this point for now. Assistive technology can navigate the MathML tree, get MathML markup and feed it to utility libraries processing the math. It looks good for a start, at least after years of keeping silence. Ostap Bender would say the ice has broken, ladies and gentlemen of the jury!

If you have ideas, thoughts to share you're welcome to comment our meta bug.

Thursday, September 5, 2013

Accessible Mozilla: Tech overview of Firefox 25


* We don't map aria-atomic:false to accessibility API anymore (see bug). UAIG is flexible on this and allows a fork in implementation: either ignore it or map it. Ignoring seems more reasonable so we did it this way. And here's why. Assistive technology relies on container-atomic object attribute exposed on every child of atomic region. In other words if there's no container-atomic object attribute then the accessible is not contained by atomic region. It's worth to notice it works pretty well if the users deals with nested aria-atomic regions, i.e. when atomic regions contain not atomic areas.

* We've got smarter about accessible tree creation, we started to ignore whitespace accessibles between ARIA grid cells (see bug). These whitespaces can be unintentionally introduced by the author when he creates the web page. They serve the pure layout job like they add some visual space between cells. What's worse any whitespaces are not expected under the grid control hierarchy and thus can confuse the assistive technology.

* ARIA column and row headers are not selectable by default anymore. Technically speaking if you do not specify aria-selected then SELECTABLE state is not exposed (see bug). The idea behind that is ARIA column/row headers are either not interactive at all or they are used to select the whole column/row at once.

* ARIA textbox exposes accessible value constructed from the content beneath the element (see bug).

* ARIA tablist is no longer a live region (see bug). The bug popped up from the wild: the users run into the problem. So we've got the bug fixed and asked for an update to UAIG spec.

* ARIA listbox exposes FOCUSABLE state regardless how it manages its children. That means if the listbox is not DOM focusable and its items are focusable instead (tabindex on items approach) then we enforce FOCUSABLE state on the listbox accessible. The trick is primarily introduced for MSAA environment where screen readers relies on FOCUSABLE state to differentiate HTML:ul from HTML:select since they share same MSAA role (see bug).

* ARIA treegrid like ARIA grid are editable by default now (see bug).

* In some cases we didn't treat 'undefined' value as absent value, for example, ARIA checkbox with aria-checked="undefined" was marked as not checkable. We fixed it.


longdesc attribute is exposed in Firefox context menu (see bug) as "View description" menu item. Right click on the image having a longdesc allows you to see it in new tab.

Text interface

A major change in this release is ATK and IAccessible text traversal by lines got a new heart. We still observe some problems (and even some serious ones) but definitely it must be a good update.


* We started to fire SELECTION state_change evens on items of selectable widgets like combobox or listbox (see bug). On MSAA level these events are partially duped by selection event but having only it AT had no way to detect which items was deselected.

* We fixed a nasty focus bug observed at start page: we missed the focus event when search textbox is moved through the page (see bug).

* In case of HTML checkbox input we fire state_change event whenever checkbox value is changed (previously we were restricted to user interactions), see bug.

Anything else

* We fixed hit testing for Firefox UI menus and popup (see bug) on Windows. Now each menu and popup hosted in own window returns HWND of that window and when the window receives WM_GET_OBJECT message then it answers it by returning the menu/popup accessible. If you take a hit test on the returned accessible then you get kids of menus/popups.

* We've got rid of XPCOM constant BOUNDARY_ATTRIBUTE_RANGE. It doesn't seem useful and practical.

Wednesday, July 10, 2013

Accessible Mozilla: Tech overview of Firefox 24

Here's a list of Firefox 24 (currently Aurora) improvements for assistive technology.


We supported one more way to create hierarchy for ARIA lists and trees (see bug). Now you can do

  <ul role="list">
    <li role="listitem">Item 1
      <ul role="group">
        <li role="listitem">Item 1A</li>
        <li role="listitem">Item 1B</li>
    <li role="listitem">Item 2
      <ul role="group">
        <li role="listitem">Item 2A</li>
        <li role="listitem">Item 2B</li>          

So following this pattern allows you to get for free the group position for items and relations between them.

It's worth to say that Firefox supports another kind of hierarchy for ARIA trees for a long time:

  <div role="tree">
    <div role="treeitem">Item 1</div>
    <div role="group">
      <div role="treeitem">Item 1A</div>
      <div role="treeitem">Item 1B</div>
    <div role="treeitem">Item 2</div>
    <div role="group">
      <div role="treeitem">Item 2A</div>
      <div role="treeitem">Item 2B</div>          

Here ARIA group element defines a sub group of items for the item preceding to it.

You can use either hierarchy that suites better for you. Note, second approach doesn't work in case of ARIA lists.


datalist attribute on the widget makes the accessible object to expose HASPOPUP state in addition to SUPPORTS_AUTOCOMPLETION state we used to have (see bug).

Action interface works on HTML textarea element now.

HTML td elements may be pointed by headers attribute, i.e. HTML td can serve as row/column header. In other words the following pattern is supported:

    <tr><td id="juice">Juice</td></tr>
    <tr><td headers="juice">Orange</td></tr>
    <tr><td headers="juice">Apple</td></tr>

Moving at the "orange" or "apple" cell by screen reader you will hear that this is orange or apple juice.

Text work

getTextAfterOffset and getTextBeforeOffset methods for word boundaries were improved a bit (see bug and bug). There are still problems left.


We did first steps on the way to implement new IAccessible2 version released couple months ago. We've got IA2_RELATION_NODE_PARENT_OF relation exposed on ARIA and XUL trees (thanks to Zach who volunteered for this work).

Note, alternatively to IAccessible2 relation interface you can obtain this relation by MSAA IAccessible::accNavigation method using 0x1010 constant. Refer to our source code for all possible values (no docs sorry).

Another bit of IA2 support was IA2_STATE_CHECKABLE (thanks to Marcos A. Di Pietro for the fix). You can see this state on all checkbox-like widgets. Prior to this state the checkable:true object attribute was used for that. Now we expose state and object attribute both.

User interface

Menu items of Firefox UI have correct state now, i.e. no OFFSCREEN state for visible menu items anymore (see bug).

Pinned tabs in Firefox UI expose IA2_STATE_PINNED now.

Thursday, June 13, 2013

Accessible Mozilla: Tech overview of Firefox 23

Here's a list of core accessibility improvements in oncoming Firefox 23 (will be available as beta 24 June).


A non standard extension of ARIA was introduced. Now Firefox recognize a "key" ARIA role which is mapped to push button role on IAccessible2/ATK layer. It is supposed to be used for on-screen keyboards and dialpads. AT can use  xml-roles object attribute to pick up that role.


* A major change of this release is we don't longer create a flat hierarchy for HTML selects (see bug). Say you have a HTML select having optgroup elements like

    <optgroup label="group">

then we used to have an accessible hierarchy like:

  combobox - HTML select
   option - HTML option (item)
   heading - HTML optgroup
   option- HTML option (subitem)

Firefox 23 exposes the following hierarchy:

  combobox - HTML select
   option - HTML option (item)
   grouping - HTML optgroup
     option - HTML option (subitem)

* Another change is HTML5 input@type=range is now accessible. It's exposed as a slider role and implements the value interface, i.e. you can obtain min, max and current values and step. Also it fires value change events.

Nasty bugs

A bug where JAWS lost the virtual buffer was fixed. The patch was back ported to Firefox 22 and Firefox 21.

Another bug was that we lost inactive document occasionally. So if the user didn't interact with the document and the document wasn't self updated via JavaScript then it could be not accessible for screen readers (see bug).

ISimpleDOM support

Yet another ISimpleDOM interface (ISimpleDOMText) was tear offed as continuation of this work.

Other bug fixes

* A selection bug where we wrongly claimed that unfocused text area has a selection was fixed.

* We don't prune children anymore for number of roles like text entry, menu items and etc if they have complex trees. In other words, if for example a text entry has anything else than a plain text then you will see that in hierarchy (see bug). So if you do

  <div role="textbox"><span role="button"></span></div>

then you will get a button inside the text entry.