How we leveled up accessibility testing

Early in our accessibility journey, we generally grouped accessibility testing into two categories: automated and manual. The former to quickly catch algorithmically-detectable low hanging fruit and the latter for everything else that required more time and attention to find. Automated fit well into rapid processes like continuous integration (CI), while manual required careful coordination with skilled testers to cover everything automated tools couldn’t.

Along the way we discovered there's actually a sweet spot between these two approaches. In partnership with Assistiv Labs, we combined better processes with end-to-end service testing towards a testing workflow that redefines automated and manual. It’s something in between, a hybrid approach that dramatically increases automated coverage and reduces manual coordination overhead.

Now we catch clear, easy-to-fix issues in hours instead of weeks. Our new processes automatically figure out what's urgent and send issues to the right teams, who typically resolve them within days.

The results have been both technical and cultural. We give accessibility feedback way faster, teams resolve bugs quicker, and engineers across Asana have started thinking about accessibility issues differently.

Building processes to prioritize accessibility bugs

At Asana’s scale, a strong prioritization framework supported by processes and governance principles is essential. Teams need to know what needs to be scheduled into their backlogs for when, and what needs attention right now.

But it’s easy to underestimate how hard it is to prioritize accessibility bugs. Priority is based on a dizzying matrix of unique factors including assistive technologies, browsers, impacted user groups, user feedback, product area, fix difficulty, workarounds, WCAG conformance, and historical context. Some bugs may have existed for a long time, but were only recently detected. Others appear with new features. And some are regressions—functionality that used to be accessible but now isn’t.

To create a framework, we formally defined different types of accessibility bugs so we could design streamlined workflows around each. An essential step was defining what constituted a regression vs. what didn’t.

regression (n)

A documented (is there any record of this working, like previous bugs, comment threads, screenshots, or screen recordings?), reproducible (is a colleague able to reproduce using the steps as outlined?) issue that was previously working and now no longer is.

Regressions can naturally receive high priority—the last thing anyone wants is to have accessibility backslide. This can greatly simplify prioritization in many cases.

However, there’s a catch. Regressions are defined by before and after snapshots. For example, a before snapshot could be a video of the functionality working and an after snapshot could be a video of a new problematic result. Bugs naturally have after snapshots. But finding the before snapshot usually requires deep digging.

Without a reliable source of before snapshots, it often wasn’t worth the time to investigate whether a bug could be prioritized as a regression. This is where a new kind of automation proved immensely helpful.

End-to-end accessibility automation

Fully automated tests are not new to accessibility testing or to Asana. Historically, axe DevTools and jsx-a11y for React provided us with broad, horizontal coverage. But they’re shallow. We felt the token 30% estimate for automated tests across the industry was fairly close to the WCAG criteria we were achieving. Limited coverage meant manual testing was still finding bugs that automated tools had missed.

We needed tools that could go deeper. Tools more closely aligned with our user research and governance principles. Which is what we found with Assistiv. Tests for Assistiv’s end-to-end service are written from the ground up by Assistiv engineers based on user flows and test parameters we provide. The suites incorporate keyboard shortcuts, real screen readers, browsers, and machine vision powered by the Assistiv Labs cloud. The result is more than simulation, with events transmitted to a machine the same way a human user would, maximizing coverage of WCAG and broader accessibility concerns.

End-to-end accessibility automation is radically different from traditional automation, interacting with Asana to accomplish tasks in the same way a manual tester would. It’s possible to cover up to 75% of WCAG criteria, depending on the test scenario. And there are still live, critical thinking experts watching over all of it. People from both Asana and Assistiv are involved in designing representative user actions and reviewing the outcomes, drastically raising the automated testing floor in terms of scope, frequency, and accuracy.

Combining forces

With a strong prioritization framework in place and new automation that switched from finding a minority of bugs to the majority, we implemented a powerful prioritization workflow.

First, automated tests are sync’d with Asana’s existing engineering pipeline. New issues are detected in near-real time and correlated with the code changes that likely caused them.

Next, an Assistiv engineer reviews to filter out false positives and writes up an issue in Asana’s backlog with contextualized user impact and remediation guidance. Because automated tests are running continuously, a before snapshot is readily available and regressions can be easily classified. We maintain automated Asana workflows that route the issue to the correct team.

In practice, this means that regressions are usually flagged within 24 hours of a deployment and documented in a way that is easy for engineers without a background in accessibility to understand. That allows our accessibility team to set an SLA for addressing regressions and leave product teams to it. No one has to make the case for which regression comes first or second or last. They’re just bugs that need attention.

This decentralization directly translates into a more sustainable program and more inclusive end user experiences. We have a lot more agency and impact, a lot of space for creativity because we're not constantly putting out fires. And that in turn means that we’re happier and more effective.

The ROI of faster feedback loops

A bug that goes undetected for weeks or months is expensive to fix. Someone has to triage it to the appropriate team and ensure it’s prioritized. The engineer it’s assigned to likely isn’t the same engineer who caused it. Even when they are, it’s difficult to dig up the forgotten context, shift gears, or coordinate with other teams to untangle technical debt that has sprung up in the intervening weeks and months.

Putting numbers to the problem, a highly-cited IBM study found that it costs 30 times more to discover bugs in production than those found during the design phase. That rings true.

When engineers deploy an update in the morning, any evidence of regressions will normally start popping up before the end of the work day. The feedback loop is so short that our end-to-end tests have been the first to alert us to generic UI bugs, not restricted to screen readers or keyboard navigation, on multiple occasions.

As engineers started receiving faster feedback from end-to-end testing, we noticed a reduction in the cognitive overhead of prioritization. The ambiguity went away. Engineers would think “I’m accountable for this because I shipped it this morning. It worked yesterday and it’s broken today, I need to fix it now.” Fixing accessibility bugs becomes like muscle memory.

At Asana accessibility bugs are just bugs. And they get fixed.

A better way to automate accessibility

Before adopting end-to-end accessibility testing, our accessibility team made significant headway. On the Operations side of things, design and engineering teams had documented review processes, regression definitions, and SLAs in place. When it came to testing, there were automated solutions for swift but shallow reports and manual QA for thorough but time-consuming audits.

With end-to-end, we have a technical solution that complements and sits atop existing efforts. Regressions are being documented sooner, in more detail, with more evidence. Virtually no one is wasting time on bug reports that aren’t actionable or reproducible.

Now we have a clear picture of which bugs are new and which are old, which user flows they affect and who is responsible for remediation. We can get ahead of the curve and focus on our vision for Asana’s accessibility in the future.

About the authors:  Cameron Cundiff (Technical Lead) and Jiaxin Zheng (Technical Program Manager) are both members of the Development Lifecycle group, dedicated to providing tooling that enables developers to bring ideas to production quickly, robustly, and delightfully.