IAccessible2 relation API

Before IAccessible2 Mozilla practiced all manner of extensions of MSAA. I don't mean I exalt the forethought of MSAA API, I'd rather say this is an example of necessity is the mother of invention. But fortunately MSAA was extensible enough to workaround some lacks of it. Firefox ignored MSAA spec occasionally, implemented tricks and - what a surprise - some of them are in use by modern screen readers nowadays. 

All aforesaid is applicable to accessible relations. MSAA has accNavigate method to navigate the accessibles in tree hierarchy and layout order. Firefox introduced a bunch of extra constants for this method to expose accessible relations. That was a neat approach so that after IAccessible2 invention and implementation the assistive technology wasn't in hurry to switch to the right API. accNavigate was light and simple and had only one disadvantage: it didn't allow you to get multiple targets. IA2 relations in turn didn't have this limitation but API appeared heavy and Firefox implementation was quite slow those days. Firefox 4 implemented a fast version but it didn't change things. That made me think Firefox wasn't a stumbling block here. So

IAccessible2 1.3 refreshed relations API and we've got relationTargetsByType method which allows to get all targets of the given relation type. In other words this is a multiple target version of MSAA accNavigate: plain and simple. The method was implemented in Firefox 27.

I'm curious how fast the assistive technology will pick it up. Otherwise do we need those multiple targets at all?

MathML accessibility APIs

Browsers traditionally overlook MathML accessibility besides MathML implementation itself by some of them. Because of that there is a bunch of 3d parties software to display MathML and expose math content to assistive technology. They do a good job but what relates to screen readers support it's far from being perfect.

So MathPlayer works with IE only. MathJax, a cross browser math renderer, claims they do accessibility by means of MathPlayer. Last time I checked MathPlayer did a trick by adding accessible properties like accessible name to MathML nodes in accessible tree. There's a bunch of problems with this approach like the user can't control the speech output. However I must admit it makes screen readers to read math.

Main problem here there's no right API and they have to use existing APIs which is Procrustean bed for math. The browser and assistive technology have to invent something that describes math well. Until that there's no right tool to succeed. I should notice that above-mentioned primarily applies to Windows and Linux worlds. Much to my surprise WebKit has got a math accessibility API on OS X last year. I didn't hear VoiceOver picked it up yet but as soon as it does the MathML content should become accessible on Mac.

In Firefox we have a meta bug to track MathML accessibility work. As the first step on this way Firefox 27 started to create generic accessible objects for MathML elements. It's not so valuable by itself because a generic accessible don't expose any math semantic but it allows the assistive technology to navigate the math and watch for tree mutations. Next we could follow WebKit effort by extending ATK and IAccessible2 APIs to make math accessible on Windows and Linux but I have been told libraries that screen readers rely on to process MathML speak also MathML language. It might be unwise to split MathML into atoms of high level API to make the assistive technology to reconstruct MathML on its side.

Gecko exposes ISimpleDOMNode interface providing a direct access to DOM for assistive technology. I never was a devotee of ISimpleDOM because I believe that high-level APIs like IAccessible2 are more efficient. But in case of MathML it's apparently not true. Having said that I think it'd be good to have something more sophisticated than plain DOM to implement, for example, an extended math navigation. Otherwise I think AT have to learn some MathML.

So we stopped at this point for now. Assistive technology can navigate the MathML tree, get MathML markup and feed it to utility libraries processing the math. It looks good for a start, at least after years of keeping silence. Ostap Bender would say the ice has broken, ladies and gentlemen of the jury!

If you have ideas, thoughts to share you're welcome to comment our meta bug.

Accessible Mozilla: Tech overview of Firefox 25

ARIA

* We don't map aria-atomic:false to accessibility API anymore (see bug). UAIG is flexible on this and allows a fork in implementation: either ignore it or map it. Ignoring seems more reasonable so we did it this way. And here's why. Assistive technology relies on container-atomic object attribute exposed on every child of atomic region. In other words if there's no container-atomic object attribute then the accessible is not contained by atomic region. It's worth to notice it works pretty well if the users deals with nested aria-atomic regions, i.e. when atomic regions contain not atomic areas.

* We've got smarter about accessible tree creation, we started to ignore whitespace accessibles between ARIA grid cells (see bug). These whitespaces can be unintentionally introduced by the author when he creates the web page. They serve the pure layout job like they add some visual space between cells. What's worse any whitespaces are not expected under the grid control hierarchy and thus can confuse the assistive technology.

* ARIA column and row headers are not selectable by default anymore. Technically speaking if you do not specify aria-selected then SELECTABLE state is not exposed (see bug). The idea behind that is ARIA column/row headers are either not interactive at all or they are used to select the whole column/row at once.

* ARIA textbox exposes accessible value constructed from the content beneath the element (see bug).

* ARIA tablist is no longer a live region (see bug). The bug popped up from the wild: the users run into the problem. So we've got the bug fixed and asked for an update to UAIG spec.

* ARIA listbox exposes FOCUSABLE state regardless how it manages its children. That means if the listbox is not DOM focusable and its items are focusable instead (tabindex on items approach) then we enforce FOCUSABLE state on the listbox accessible. The trick is primarily introduced for MSAA environment where screen readers relies on FOCUSABLE state to differentiate HTML:ul from HTML:select since they share same MSAA role (see bug).

* ARIA treegrid like ARIA grid are editable by default now (see bug).

* In some cases we didn't treat 'undefined' value as absent value, for example, ARIA checkbox with aria-checked="undefined" was marked as not checkable. We fixed it.

HTML

longdesc attribute is exposed in Firefox context menu (see bug) as "View description" menu item. Right click on the image having a longdesc allows you to see it in new tab.

Text interface

A major change in this release is ATK and IAccessible text traversal by lines got a new heart. We still observe some problems (and even some serious ones) but definitely it must be a good update.

Events

* We started to fire SELECTION state_change evens on items of selectable widgets like combobox or listbox (see bug). On MSAA level these events are partially duped by selection event but having only it AT had no way to detect which items was deselected.

* We fixed a nasty focus bug observed at google.com start page: we missed the focus event when search textbox is moved through the page (see bug).

* In case of HTML checkbox input we fire state_change event whenever checkbox value is changed (previously we were restricted to user interactions), see bug.

Anything else

* We fixed hit testing for Firefox UI menus and popup (see bug) on Windows. Now each menu and popup hosted in own window returns HWND of that window and when the window receives WM_GET_OBJECT message then it answers it by returning the menu/popup accessible. If you take a hit test on the returned accessible then you get kids of menus/popups.

* We've got rid of XPCOM constant BOUNDARY_ATTRIBUTE_RANGE. It doesn't seem useful and practical.

Accessible Mozilla: Tech overview of Firefox 24

Here's a list of Firefox 24 (currently Aurora) improvements for assistive technology.

ARIA

We supported one more way to create hierarchy for ARIA lists and trees (see bug). Now you can do

  <ul role="list">
    <li role="listitem">Item 1
      <ul role="group">
        <li role="listitem">Item 1A</li>
        <li role="listitem">Item 1B</li>
      </ul>
    </li>
    <li role="listitem">Item 2
      <ul role="group">
        <li role="listitem">Item 2A</li>
        <li role="listitem">Item 2B</li>          
      </ul>
    </li>
  </li> 

So following this pattern allows you to get for free the group position for items and relations between them.

It's worth to say that Firefox supports another kind of hierarchy for ARIA trees for a long time:

  <div role="tree">
    <div role="treeitem">Item 1</div>
    <div role="group">
      <div role="treeitem">Item 1A</div>
      <div role="treeitem">Item 1B</div>
    </div>
    <div role="treeitem">Item 2</div>
    <div role="group">
      <div role="treeitem">Item 2A</div>
      <div role="treeitem">Item 2B</div>          
    </div>
  </div> 

Here ARIA group element defines a sub group of items for the item preceding to it.

You can use either hierarchy that suites better for you. Note, second approach doesn't work in case of ARIA lists.

HTML

datalist attribute on the widget makes the accessible object to expose HASPOPUP state in addition to SUPPORTS_AUTOCOMPLETION state we used to have (see bug).

Action interface works on HTML textarea element now.

HTML td elements may be pointed by headers attribute, i.e. HTML td can serve as row/column header. In other words the following pattern is supported:

  <table>
    <tr><td id="juice">Juice</td></tr>
    <tr><td headers="juice">Orange</td></tr>
    <tr><td headers="juice">Apple</td></tr>
  </table>

Moving at the "orange" or "apple" cell by screen reader you will hear that this is orange or apple juice.

Text work

getTextAfterOffset and getTextBeforeOffset methods for word boundaries were improved a bit (see bug and bug). There are still problems left.

IAccessible2

We did first steps on the way to implement new IAccessible2 version released couple months ago. We've got IA2_RELATION_NODE_PARENT_OF relation exposed on ARIA and XUL trees (thanks to Zach who volunteered for this work).

Note, alternatively to IAccessible2 relation interface you can obtain this relation by MSAA IAccessible::accNavigation method using 0x1010 constant. Refer to our source code for all possible values (no docs sorry).

Another bit of IA2 support was IA2_STATE_CHECKABLE (thanks to Marcos A. Di Pietro for the fix). You can see this state on all checkbox-like widgets. Prior to this state the checkable:true object attribute was used for that. Now we expose state and object attribute both.

User interface

Menu items of Firefox UI have correct state now, i.e. no OFFSCREEN state for visible menu items anymore (see bug).

Pinned tabs in Firefox UI expose IA2_STATE_PINNED now.

Accessible Mozilla: Tech overview of Firefox 23

Here's a list of core accessibility improvements in oncoming Firefox 23 (will be available as beta 24 June).

ARIA

A non standard extension of ARIA was introduced. Now Firefox recognize a "key" ARIA role which is mapped to push button role on IAccessible2/ATK layer. It is supposed to be used for on-screen keyboards and dialpads. AT can use  xml-roles object attribute to pick up that role.

HTML

* A major change of this release is we don't longer create a flat hierarchy for HTML selects (see bug). Say you have a HTML select having optgroup elements like

  <select>
    <option>item
    <optgroup label="group">
      <option>subitem
    </optgroup>
  </select>

then we used to have an accessible hierarchy like:

  combobox - HTML select
   option - HTML option (item)
   heading - HTML optgroup
   option- HTML option (subitem)

Firefox 23 exposes the following hierarchy:

  combobox - HTML select
   option - HTML option (item)
   grouping - HTML optgroup
     option - HTML option (subitem)


* Another change is HTML5 input@type=range is now accessible. It's exposed as a slider role and implements the value interface, i.e. you can obtain min, max and current values and step. Also it fires value change events.

Nasty bugs

A bug where JAWS lost the virtual buffer was fixed. The patch was back ported to Firefox 22 and Firefox 21.

Another bug was that we lost inactive document occasionally. So if the user didn't interact with the document and the document wasn't self updated via JavaScript then it could be not accessible for screen readers (see bug).

ISimpleDOM support

Yet another ISimpleDOM interface (ISimpleDOMText) was tear offed as continuation of this work.

Other bug fixes

* A selection bug where we wrongly claimed that unfocused text area has a selection was fixed.

* We don't prune children anymore for number of roles like text entry, menu items and etc if they have complex trees. In other words, if for example a text entry has anything else than a plain text then you will see that in hierarchy (see bug). So if you do

  <div role="textbox"><span role="button"></span></div>

then you will get a button inside the text entry.

IAccessible2. New features and performance.

After months of long discussions the IAccessible2 group released 1.3 version. It was probably most disputable release after the initial shipping. A number of assistive technology vendors requested a number of changes for better integration with the browser. So it took a time to find a common multiplier for all ideas.

So updated IAccessible2 API require some engineering love and thus it will be implemented in Firefox by parts. It'd be helpful to know which features are most crucial and needed for you as AT developer so we can give them a priority. To let us know please leave a comment in the bug or just ping me saying that. Alternatively you can vote for or comment right into the interesting bug (see sections below).

Roles

IA2_ROLE_COMPLEMENTARY_CONTENT is supposed for HTML5 aside element and WAI ARIA complementray role (see Mozilla bug).

States

A new IA2_STATE_PINNED state will be exposed on pinned tabs in Firefox UI (see Mozilla bug).

Technically speaking IA2_STATE_CHECKABLE was introduced in place of hacks used to workaround a lack of this state. For example Firefox exposes checkable object attribute (IAccessible2 hack) and STATE_SYSTEM_MARQUEED (pure MSAA approach). See Mozilla bug.

Relations

There is a bunch of new relations introduced to simplify a navigation through the hierarchy. Now you can quickly move to

• nearest containing document, instance of IAccessibleDocument (IA2_RELATION_CONTAINING_DOCUMENT)
• a containing tab pane, instance of IAccessible2_2 (IA2_RELATION_CONTAINING_TAB_PANE)
• a containing window document, instance of IAccessibleDocument (IA2_RELATION_CONTAINING_WINDOW)
• an application accessible, instance of IAccessibleApplication (IA2_RELATION_CONTAINING_APPLICATION)

Note, IServiceProvider is an alternative to described relations (strictly speaking a subsest of described relations). See Mozilla bug.

Also now you can get a next and previous elements in tab order (IA2_RELATION_NEXT_TABBABLE and IA2_RELATION_PREVIOUS_TABBABLE). It's useful for AT that implement own in-page navigation (see Mozilla bug).

New IA2_RELATION_NODE_PARENT_OF is a dual relation to IA2_RELATION_NODE_CHILD_OF. It's useful to get all logical children of the accessible (as you know a logical parent-child relation may be different from parent-child relation defined by accessible tree). Example is aria-owns (like aria-owns tree) and flat structured hierarchies like XUL tree (used for message list in Thunderbird) or ARIA tree where the hierarchy is supplied by aria-level like:

  <ul role="tree">
    <li role="treeitem" aria-level="1">Item 1</li>
    <li role="treeitem" aria-level="2">Item 1.1</li>
  </ul>

See Mozilla bug.

 

IAccessibleDocument

IAccessibleDocument is a new interface that allows you to get an anchor target, i.e. an element that the page was scrolled to after the page was loaded. Also an object of this interface is returned by relations like IA2_RELATION_CONTAINING_DOCUMENT and IA2_RELATION_CONTAINING_WINDOW. See Mozilla bug.

 

Manage the media content

The idea was to manage the media content by accessible actions (IA2 group got parsimonious for a new interface so the existing mechanism was reused).

If a screen reader starts, say, a subtitled video then it invokes IA2_ACTION_OPEN action. The browsers plays a portion of subtitled video and stops it until a screen reader invokes IA2_ACTION_COMPLETE action. This action is used to notify the browser that screen reader consumed the portion of played video subtitles, for example, it finished reading of them. If screen reader is no longer interesting to manage the media playback then it invokes IA2_ACTION_CLOSE action.

If more than one AT tries to manage the same media (i.e. AT that invoked 'open' action) then the browser holds on the video playback until all AT invoke 'complete' action. See Mozilla bug.

Performance

After IAccessible2 implementation by Firefox and AT it got clear that IAccessible2 API can be improved in certain ways to make the intercommunication between the browser and AT faster. So number of changes was introduced for performance matter.

Object attribute by name

Now you can obtain object attributes by name rather than get them all together and then parse the string (see Mozilla bug).

Relation by type

Now you have a method to get relation targets by type. You don't need to get all relations when you need a single one (see Mozilla bug).

Accessible containing the caret

You have a fast way to get a text accessible containing a caret in the subtree and the caret offset in  the text accessible. Previously the algorithm to find an accessible containing the caret failed on some cases. The idea of that approach was that a screen reader should crawl down the hierarchy asking for caret position until it reaches a deepest text accessible having the caret. It failed if you meet non text accessible on the way. As a workaround the screen readers navigated children. While the workaround worked well but it was really slow (see Mozilla bug).

Text attributes

The change is similar to object attributes. A light alternative to the method used to get all text attributes was added.  Now you can specify a string filter and get attributes you need (see Mozilla bug).

Get all hyperlinks

Usually AT needs all hyperlinks at once. Getting them one by one is unperformant (especially for out of process AT) and requires a cache implementation on the browser side. So a new method to get all hyperlinks has been introduced (see Mozilla bug).

Accessible Mozilla: Tech overview of Firefox 22

Firefox 22 reached beta status (it will be released June 24). It's time to list accessibility improvements we made for this version.

ARIA

ARIA role="note" doesn't allow name from subtree (bug) anymore. The bug caused JAWS, for example, to announce role="note" content twice.

HTML

* HTML radio group position doesn't count hidden radio elements (bug). So if the page contains hidden input@type="radio" then a screen reader doesn't take them into account announcing the number of radios.

* HTML input@type="file" changed its hierachy. Now it contains a push button and a label. Be careful if you have dependences on this hierarchy (see bug).

* HTML5 header and footer has changed their mapping according to HTML spec:

footer element that is not a descendant of an article or section element. contentinfo role;

header element that is not a descendant of an article or section element.  banner role.

XUL

XUL label@value element now implements text interface (partially). You can obtain a text between offsets but you can't get it by words for example (refer to bug). XUL label is used wide in Firefox user interface (for example, in Options dialog). Let us know if you have problems with new implementation.

ATK

RELATION_NODE_PARENT_OF has been implemented. It's exposed for aria-owns markup and XUL trees (used in Thunderbird and Firefox bookmarks).

Text interface

As I wrote before we started text interface reimplementation. Firefox 22 got improved getTextAt and getTextBefore offsets at word boundaries. Note, in case of getTextAt we had to mimic WebKit behavior rather than follow the ATK spec to keep Orca working.

Attention. It might be important

* Document load event may be fired a bit later than we used to do that, it will be fired right after all events contained in the queue at the time when document got loaded (see bug).

* IServiceProvider interface is implemented as a tear off (bug).

Firefox preferences for screen readers

I realized that nobody yet shared that new user preferences in Firefox were introduced for screen readers so I do it here.

There are couple preferences to make screen readers support smoother. And all of them - you might be not surprised - are Adobe Flash related. Flash is headache for both assistive technology vendors and for Firefox developers because flash is tightly related with hangs, crashes and security problems. Unfortunately we need to balance between usability and stability when dealing with Flash so we provide a bunch of preferences to let the user make a choice.

No plugin sandboxing for JAWS

You probably heard or even encountered yourself that JAWS started to hang occasionally in Firefox 14. It was a regression from bug where OOPP (out of process plugins) option was forced on by default on Windows Vista and later. Since it was done on security proposes then we had to choose between making Firefox unstable for JAWS users and keeping JAWS working. So we let users to decide and introduced a preference

  dom.ipc.plugins.enabled.a11y

The preference is not public and you should add it manually via about:config. For that you need to type about:config into address bar and then choose 'New' -> 'Boolean' items from the context menu. As far as I know you don't need any actions on Flash side because Adobe turns sandboxing off as long as Firefox's one is off.

AFAIK JAWS fixed the issue on their side: they started to deal with flash content as separate document. It is definitely different user experience but it must be better than security hazard. So we have an open bug to remove this pref. If you have a reason why we shouldn't do this then please comment into the bug.

Note, the preference was introduced in Firefox 18 and it has an affect only if JAWS is running.

Delayed plugin creation

We had another report about hanging screen readers (Windows Eyes and JAWS) on web pages with Flash. To workaround the problem we introduced a new boolean preference:

  accessibility.delay_plugins

This pref makes the browser to delay a plugin accessible creation. As a rule it saves from hangs. There's accompanying numeric preference used to specify a value of the delay in seconds:

  accessibility.delay_plugin_time

By default the value is 10 seconds. This means you can't interact with Flash content during 10 secs after the plugin was loaded on the web page. It's definitely bad user experience but it's still an option to keep screen readers working.

You should be aware that user entered value of accessibility.delay_plugins preference is ignored as long as JAWS or Windows Eyes screen reader is detected. In this case a plugin accessible creation is delayed automatically.

This feature was introduced in Firefox 22.

UIA text vs ATK / IAccessible2

After IAccessible2 comparison to ATK text it'd be good to get a quick look at ATK / IAccessible2 APIs and UIA text. Firefox doesn't have UIA implementation yet and there's no nearest plans about it but it's worth to compare these API because one day UIA text might become a good alternative to IAccessible2 I think.

UIA text is closer to user actions since UIA has concept of range which can be moved like a cursor through the web page. You can move it (and extend it) by characters, words, lines the same way as the user would do. And then you can get a text the range is spanned to. This means you won't ever run into restrictions of accessible tree and embedded characters as you probably do in case of IAccesisble or ATK APIs. A couple examples might be good to explain what I mean.

Example #1

If accessible tree is DOM based i.e. it's close to DOM hierarchy then ATK / IA2 text interface implementation might be tricky. For example, the following HTML paragraph

<p>hello
  <aside style="position:absolute;top:0px;left:0px;">meine freund</aside>
my friend</p>

can have the accessible tree

paragraph (html:p)
  text leaf ('hello' text)
  section (html:aside)
    text leaf ('meine freund' text)
  text leaf ('my friend' text)

If the browser is not smart enough then it doesn't remove embedded character designating html:aside element from the text of parent html:p element. In other words if the paragraph text is "hello*my friend" (where * is an embedded character for html:aside) then a screen reader have to to deal with it and it should ignore out of flow content somehow. If the screen reader is not smart enough to ignore that then it will move the user through "meine freund" text when the user moves through the paragraph text.

Example #2

In case of Firefox implementation which tends to use embedded characters for everything you can observe another kind of weird behavior. Screen reader must be smart to move by words, etc because embedded characters are used for inline objects like anchors. For instance if you have

<p>hel<a>lo</a></p>

then screen reader must juggle offsets to detect that this paragraph technically consists of one word. You can get into similar troubles when an anchor is spanned through multiple lines. So if you move by lines then the end offset pointing after embedded character never says to you whether a line end is in the middle or after the embedded object. You need to look into embedded object to detect that. It makes a screen reader logic not performant and not trivial.

A summary 

In short UIA lets you to move through the page in correspondence to web page layout while ATK and IAccessible2 allow you to move in correspondence to the accessible tree. Sometimes it makes a difference. 

So accessible tree dependent approach makes the text implementation not trivial on certain platforms (granted, on Gecko). I'm sure that everything (or mostly everything) can be implemented right on the browser side or can be workarounded by screen readers but implementation in either case must be not seamless. Note, somebody told me that Webkit has a nice ATK text implementation (or nicer than Gecko's one? I don't recall). So it's an attestation it's doable, just can be not easy.

I should notice we didn't prototyped anything yet. Of course before making any judgements (did I?) we need to implement it and screen readers should adopt it. And only after that I have a right to say whether it was so good as it looks. On the other hand ATK appeared years ago, IAccessible2 just adopted and simplified ATK ideas so that Microsoft had enough time after MSAA to invent something nice. So I'm ready to believe they did it.

ATK text pitfalls

As soon as I ensured myself I've got a good understanding of ATK text they put me back into reality. One more time I must admit myself that ATK text is unknowable like the universe. Seriously, shortly after I started the work on fixing ATK text bugs in Firefox then Orca, a Linux screen reader, suddenly felt bad. I've been suggested to compare Firefox and GEdit to see if there's a difference in implementations. So did I and then I realized that results depend on whence you start the ATK spec reading (btw, GEdit implementation doesn't always follow the spec). If you read the spec from beginning (a first sentence) then you get one result. If you read it from the end (a second sentence) then you might conclude that a different result is expected. I filed a bug against ATK. But let's read it again together, I might be missing something.

Let's consider an example: "a funny word".

* atk_text_get_text_at_offset for BOUNDARY_WORD_END

The returned string is from the word end before the offset to the word end at or after the offset.

I think you will agree that there's no word end *before* 0 offset so it can be treated as an author error. ATK doesn't say how error values should be handled so I guess any reasonable return value is allowed. Firefox returned a ('', 0, 0) triplet and that confused Orca.

Read the spec next:

The returned string will contain the word at the offset if the offset is inside a word.

This means we should return a ('a', 0, 1) triple because 0 offset is inside 'a' word (btw, that's what GEdit did).


* atk_text_get_text_at_offset for BOUNDARY_WORD_START

It is a dual problem to the issue above for the offset equal to a text length. Spec says:

The returned string is from the word start at or before the offset to the word start after the offset.

and

The returned string will contain the word at the offset if the offset is inside a word.

There's no word start after the offset but the same time the offset is inside 'word' word. Reading next.


* atk_text_get_text_after_offset for BOUNDARY_WORD_END

The returned string is from the word end at or after the offset to the next work end.

It might be not evident but 0 offset is a word end offset. A proof by contradiction. If 0 offset is not the end offset then

get_text_at_offset(0, BOUNDARY_WORD_END)

in case of single word (like 'word') should return an empty text. But this contradicts to get_text_at_offset method name semantic and Orca expectations (see the case above). Therefore 0 offset is a word end.

Then it means that the method at 0 offset should return the first word ('a' in our example). But the second sentence says that it must be a second word ('funny' in our case).

The returned string will contain the word after the offset if the offset is inside a word.

* atk_text_get_text_before_offset for BOUNDARY_WORD_START.

This is a dual problem to get_text_after_offset (word end boundary) case. Let's take an offset equal to text length.

The returned string is from the word start before the word start before or at the offset to the word start before or at the offset.

Text length offset is a word start offset. A proof is by analogy (see above). That means that 3d word is expected ('text' in our case). However the second sentence says that it should be a 2nd word ('funny' in our case):

The returned string will contain the word before the offset if the offset is inside a word.

IAccessible2 text vs ATK text

We started our accessible text rework in Firefox. It's time to revisit our IAccessible2 text implementation and compare it to ATK text.

Both IAccessible2 and ATK allows you to navigate the text by characters, words and lines. To do so they both provide these three methods:

• get text *at* offset
• get text *after* offset
• get text *before* offset

It makes clear that these methods are used to get the text at, after or before offsets. It's true in IAccessible2 world but it's no so univocal in case of ATK. ATK provides bunch of tricky boundary types that may change things. Below I will example the difference.

A difference #1. IAccessible2 getTextAtOffset may return nothing when you asked for a word. This happens when the given offset is between words (see the spec).

If the index is valid, but no suitable word (or other text type) is found, a NULL pointer is returned.

ATK always returns a word, no matter where you are because it requires to return the text between two word start or word end offsets (check out the spec).

A difference #2. IAccessible2 getTextAfter/BeforeOffset always return the requested lexem after/before the given offset.

Returns the substring of the specified text type that is located after/before the given character and does not include it. The result of this method should be same as a result for IAccessibleText::textAtOffset with a suitably increased/decreased index value.

ATK get_text_after/before_offset methods may return a lexem at the offset under certain circumstances. For example here's the case of word end boundary in get_text_after_offset method:

If the boundary_type is ATK_TEXT_BOUNDARY_WORD_END the returned string is from the word end at or after the offset to the next work end.

In short ATK and IAccessible2 supply two resembling but different concepts of text traversal. ATK allows you to move through the whole text by any method. For example, text_get_text_at_offset works nice if you pass the end offset you obtained at previous call. In IAccessible2 word you need to use a couple consisting of getTextAtOffset and getTextBefore/AfterOffset to move backward/forward. I'd say IAccessible2 text is simplified (a human friendly) version of ATK text. I won't object if somebody says that ATK looks more powerful. But the same time I'm not sure screen readers need this power. 

Anyway IAccessible2 text  looked so close to ATK and thus originally we mapped IAccessible2 boundary constants into ATK constants and we ended up with shared logic between these APIs. It was done several years ago and we never revisited this code.

Now we are going to change. IAccessible2 consumers please keep the track of our progress to catch regressions early.

Accessible text support in Firefox

So getting back shortly after ATK text insight to share our plans about text support in Firefox. In fact we started the work already and couple patches were landed into Firefox 22 (currently in Aurora stage). The primary work is targeted to Firefox 23 and Firefox 24.

We need to know if the changes we introduce deliver any problems for screen readers. Recently we figured out that we broke Orca. We cherished our bugs through years and probably screen readers learned how to live with them. Putting a screen reader into fresh water might make it feel bad. I guess that's what happens with Orca.

So it's worth to give the assistive technology and the users a notice and ask them to keep eyes open. And of course be ready to report bugs against us.

 

An overview

The text problem is wide and consists of several parts (or has several layers like ogres or onions).

Caret navigation

Caret navigation in Firefox is not good in general. The behavior is inconsistent and

accidental. From time to time I observe weirdness myself writing a next blog post. The caret may move somewhere I didn't want to or it may hang somewhere in the text and I need to use mouse to move it. I don't exclude a possibility that Google Blogger product "helps" the Firefox but I don't recall any web app for rich text editing running really smooth in Firefox. If you deal with lists, tables or control elements then as a rule you should be ready to surprises.

We can't bypass a caret navigation problems, for example, recently I fixed one bug but it's likely we won't pay much attention until UIA text implementation.

ATK and IAccessible2 text

These interfaces are very similar. IA2 interface is simplified (I'd say a human friendly) version of ATK. Actually this is a reason that these interfaces have shared implementation in Firefox. We have a *lot* of bugs related to this implementation.

UIA text

UIA has a different concept of accessible text support and likely requires a separate implementation. We don't have immediate plans on it.

 

Technical part

In short our plans is ATK/IAccessible2 text bugs. On technical layer the work is scoped by following items:

* Fix ATK/IAccessible2 getTextAt/Before/After method logic. Check out our understanding of ATK text to make sure our understanding is the same with yours one.
* Fix soft line breaks problem. Currently start and end of lines produced by soft line break share the same offset. So we never know which line we should return when that offset is given.
* Do not use embedded characters for inline elements.

Please keep your eyes open and let us know if you observe a problem when reading the text in your favorite screen reader.

An easy way to understanding the ATK text

As far as I remember myself I've always been in touch with ATK text interface implementation in Mozilla. I started from writing and reviewing some patches in far 2006 year. But I didn't really understand that piece of code so I wasn't sure that a change here don't break things there. At some point we decided that we should get an automated test coverage for the text interface before we do any serious work in this area. At least that allowed us to be sure to a certain extent we don't regress badly from a single bug fix. And then I helped my colleagues in test suite creation. As part of this work we caught a bug in ATK spec (thanks to Evan Yan, a Mozilla community member and those times Sun engineer). So that wasn't easy. It wouldn't be a lie if I said that I've never seen a more complicated API for the things it's designed for.

I should notice that roughly speaking IAccessible2 text interface implementation in Firefox is done via ATK text interface. So having a bad implementation in ATK we deliver all bugs right to IAccessible2 screen readers. It's a hot problem in other words. Recently I've felt myself brave enough (again) to say: we should stop this shame. And I started to look at the code and the spec trying to untwist things. And then I realized I still don't have a good perception of ATK text. First of all I thought it'd be good to add some drawings to stingy ATK spec to let me and everybody else check easily whether the expected results are correct actually.

Preliminaries

ATK provides bunch of methods to get a text:

• atk_text_get_text_before_offset
• atk_text_get_text_at_offset
• atk_text_get_text_after_offset

Of course ATK provides a bunch of other methods but they are trivial and it doesn't make sense to even mention them. Each of methods above take AtkTextBoundary as an argument and its values are:

• char (trivial)
• word start and word end
• line start and line end
• sentence start and sentence end (not implemented in Firefox)

So we have get_text_before/at/after methods and word/line start/end offsets. This is a subject of the talk.

About terms: a chapter for the advanced

If you didn't planned to read the ATK spec or dig into details then you can skip this chapter and move to the pictures part. Otherwise this chapter might be useful since it has some clarifications.

First of all, here are some terms which are used in the spec but aren't defined there:

• word start offset - an offset where the word starts, for example, "hello, all" has two word start offsets: 0 for "hello" and 7 for "all";
• word end offset - 5 for "hello", i.e. an offset after 'o' character, and 10 for "all";
• inside word offset - any offset between word start and end offset (including boundaries), in our case these are 0-5 and 7-9 offsets;
• outside word offset - everything that is not inside a word, in our case this is only 6 offset.

It's pretty much the same for line start and line end offsets.

Also we need to mention edge cases: imaginary offsets. Say we have a paragraph:

  <p>hello</p>

and then we do

  gint startOffset = 0, endOffset = 0;
  atk_text_get_text_at_offset(accessible, 1, 
                              ATK_TEXT_BOUNDARY_WORD_START,
                              &startOffset, &endOffset);
  atk_text_get_text_at_offset(accessible, 1, 
                              ATK_TEXT_BOUNDARY_WORD_END,
                              &startOffset, &endOffset);

In both cases we expect "hello" string with (0, 5) start and end offsets otherwise there's no way traverse this paragraph by words. But actually it goes with spec: "The returned string will contain the word at the offset if the offset is inside a word". But this means that 0 and 5 offsets are both start and end offset the same time because "the returned string is from the word start at or before the offset to the word start after the offset" and "the returned string is from the word end before the offset to the word end at or after the offset".

Summarizing it all a zero offset (0) and a last offset (character count) are special offsets and can be treated as word start, word end, line start and line end offsets.

A Quick-n-Easy Guide

Update. The proposed algorithm must be corrected to handle edge offsets properly, see ATK text pitfalls. I won't spend time to update it since Joanie proposed ATK text simplification and hopefully it will be accepted in foreseeable future.

So we are ready to put the spec verbosity into nice pictures.

atk_text_get_text_at_offset

WORD_START and LINE_START boundaries are illustrated this way (X symbol designates the initial offset):

  or    

Move forward to the boundary and then (if was successful) move backward. The start offset is at or before the initial offset, the end offset is after the initial offset.

WORD_END and LINE_END boundaries:

or

Move backward to the boundary and then (if was successful) move forward. The start offset is before the initial offset, the end offset is at or after the initial offset.

atk_text_get_text_before_offset

WORD_START and LINE_START boundaries:

  or  

If the initial offset is the boundary then move backward to find the start offset. Otherwise move backward twice to pick up the start and end offsets.

WORD_END and LINE_END boundaries:

Move backward twice for start and end offsets.

atk_text_get_after_offset

WORD_START and LINE_START boundaries:

Move forward twice for start and end offsets.

WORD_END and LINE_END boundaries:

 or      

If the initial offset is the boundary then move forward to find the end offset. Otherwise move forward twice to pick up the start and end offsets.

That's all. Hallelujah to Love!

P.S. Well if I'm wrong in sayings above then you'd better say it otherwise this will be implemented in Firefox soon ;)

Accessible Mozilla: Tech overview of Firefox 21

Next week Firefox 21 reaches a beta status so it's time to list accessibility related changes we introduced for assistive technology. This release we were focused on under-the-hood improvements and ARIA bugs. Let's start.

ARIA

* We pick up accessible value of ARIA combobox in name computation. An example:

  <label for="test">Flash the screen 
    <div role="combobox">
      <div role="textbox"></div>
      <ul role="listbox" style="list-style-type:none;">
        <li role="option">1</li>
        <li role="option" aria-selected="true">2</li>
        <li role="option">3</li>
      </ul>
    </div>
    times.

  </label> 

In this case accessible name of the label is "Flash the screen 2 times" (refer to bug).

* ARIA grid has editable state by default. ARIA grid editable or readonly states (a case of aria-readonly attribute usage) are inherited by grid cells until aria-readonly attribute on a gridcell overrides it (see bug).

* We no longer expose hidden:false object attribute for aria-hidden="false" because ARIA group members concluded that aria-hidden mirrors CSS display:none (check out the bug for details). In particular this means

  <div aria-hidden="true">
    <input aria-hidden="false">
  </div>

HTML input element is still ARIA hidden.

Sure the author doesn't have any single reason to use aria-hidden="false" in his web app but you know people do crazy things every time. If we exposed it then it could be confusing for screen readers (if they wouldn't know that "false" value is an author error of course). However you may ask why ARIA spec doesn't like to treat "false" value as "true" value like it does for any other error value. It's less work for browsers, no burden for AT. You see that wouldn't bad. I do not know. Go ask your dad.

* We supported ARIA based text attributes. Now you can use aria-invalid="grammar" to mark that your text has a grammatical error. From AT perspective it means that we expose invalid:grammar text attribute.

Unfortunately ARIA spec doesn't look perfect about defining the aria-invalid. For example, it doesn't allow a list of values

  <p aria-invalid="spelling,grammar">my wrong text</p>

and it says nothing about aria-invalid inheritance which would allow you to do a trick

  <p aria-invalid="spelling">
    <span aria-invalid="grammar">my wrong text</span>
  </p>

to say that your text is misspelled and has grammatical error. Also it doesn't say how the browser should resolve collisions between aria-invalid and native spelling support which is also mapped into invalid text attribute.

HTML

* HTML5 main element was implemented. It's exposed as xml-roles:main object attribute on accessibility layer.

* We sorted out name computation for HTML input buttons:

• HTML input@type="button", @type="submit", type="reset" gets name from:
• @value attribute
• @title attribute 
• HTML input@type="image" gets name from:
• @alt attribute
• @value attribute
• @title attribute
• HTML input@type="image" having no valid @src attribute gets name from:
• @alt attribute
• @value attribute
• "Submit Query" - a visible label of the button

Everything else

* We did one more fix in our name computation algorithm and now we don't jam together a plain text and name coming from a control element. For example:

  <label>foo<input type="text" value="bar">baz</label>

The accessible name of the label is "foo bar baz". Visually "foo" and "baz" are separated from HTML input, thus we wrap the control's name by spaces when we compute the label name.

* We learned how to coalesce state change events so you should get only one event when the object changes its state fast enough, for example, it may happen during document loading when state busy of a document accessible is switched quickly.

* You can use magic offsets now to get text attributes.