Screen Reader Testing: A Protocol That Produces Comparable Results Release to Release
The WebAIM Million 2026 report scanned the top one million home pages in February 2026 and found that 95.9% had detectable WCAG 2 failures - up from 94.8% the year before, reversing six years of slow improvement. WebAIM states plainly that because only automatically detectable failures were counted, the rate of full WCAG 2 A/AA conformance was certainly lower than 4.1%. A page with zero automated errors is not an accessible page. It is a page where the scanner found nothing - and those two things are not the same.
That gap is where screen reader testing lives. Automated tools tell you whether certain markup is absent. They cannot tell you whether a dialog traps focus correctly, whether a live region announces at the right moment, or whether a custom combobox behaves the way a JAWS user expects. This post delivers a protocol for answering those questions: which assistive technology and browser pairs to test, in what priority, and a fixed script that produces comparable results release to release.
Why Automation Is Not Enough
The WebAIM Million 2026 found an average of 56.1 errors per page - a 10.1% increase in a single year, with page complexity rising 22.5% to an average of 1,437 elements. Users hit a detectable error on roughly one in every 26 elements. Those are the failures a scanner can see.
The failures it cannot see include: a skip link that is present in the DOM but whose target does not receive focus; a modal that announces its role but does not trap keyboard navigation; a form that labels its fields correctly but fails to associate error messages with the right input when validation fires. All of these pass automated checks. None of them are usable.
The ARIA data makes this concrete. Pages with ARIA present averaged 59.1 errors versus 42 on pages without ARIA (WebAIM Million 2026). WebAIM attributes this to page complexity - ARIA-heavy pages tend to be more complex - so treat it as correlation, not causation. But the pattern is consistent: more ARIA, more opportunity for the kind of interaction failure that only a screen reader exposes.
41.8% of pages had skipped heading levels and 7.5% had no headings at all (WebAIM Million 2026). 71.6% of screen reader users navigate long pages by headings first (WebAIM Screen Reader Survey #10, collected December 2023-January 2024, 1,539 responses). Those two facts together describe a large proportion of the web that is structurally unnavigable for its primary users - and no automated tool will tell you whether the headings that do exist make sense in context.
17.1% of pages had a skip link, and one in ten of those skip links was broken (WebAIM Million 2026). 5.7% of pages used role="menu" and 22% of those introduced accessibility barriers due to missing keyboard interactions. Both are manual-only catches.
Absence of complaints is not evidence of accessibility. The WebAIM Screen Reader Survey #10 found that 67% of screen reader users never or rarely contact a site owner when they encounter a barrier. They leave. Your support queue is not a signal.
The Test Matrix
Screen reader testing is not about one product. 71.6% of screen reader users use more than one screen reader, and 43% use three or more (WebAIM Survey #10). You are testing behaviours across a population, not certifying compatibility with a single tool.
The matrix below is tiered by coverage-to-cost ratio. Start with Tier 1. Add tiers as your risk profile, sector, and audience require.
| Tier | AT + Browser + OS | Rationale | Primary WCAG criteria |
|---|---|---|---|
| 1 — Minimum | NVDA + Chrome / Windows | NVDA is commonly used by 65.6% of respondents; Chrome is the dominant browser pair. Free, so no licence barrier for QA teams. | 1.3.1, 2.1.1, 4.1.2, 4.1.3 |
| 1 — Minimum | VoiceOver + Safari / iOS | 91.3% of screen reader users use mobile. iPhone/iPad accounts for 70.6% of mobile usage. VoiceOver + Safari is the dominant mobile pair at 58.2% primary mobile browser. | 1.3.1, 2.1.1, 4.1.2, 4.1.3 |
| 2 — Regulated / Procurement | JAWS + Chrome / Windows | The single most common desktop pair at 24.7%. Mandatory for public sector, financial services, and any product subject to procurement accessibility requirements. | 1.3.1, 2.1.1, 2.4.1, 2.4.3, 4.1.2, 4.1.3 |
| 2 — Regulated / Procurement | JAWS + Edge / Windows | Second JAWS pair at 11.4%. Relevant for enterprise and government contexts where Edge is the managed browser. | 1.3.1, 2.1.1, 4.1.2 |
| 3 — Extended | TalkBack + Chrome / Android | Android accounts for 27.6% of mobile screen reader usage. Add when Android share of your audience justifies the cost. | 1.3.1, 2.1.1, 4.1.2 |
| 3 — Extended | NVDA + Firefox / Windows | Fourth most common pair at 10.0%. Useful for catching Firefox-specific rendering differences. | 1.3.1, 4.1.2 |
| Audience-specific | Narrator / Windows | Primary for only 0.7% but commonly used by 37.3%. Test only if your audience is specifically Windows-first and you have evidence of Narrator use. | 4.1.2 |
| Audience-specific | Orca + Firefox / Linux | 2.4% primary usage. Test for Linux-specific deployments or developer tooling. | 4.1.2 |
EU market note. In Europe, NVDA (37.2%) outranks JAWS (29.7%) as the primary screen reader - the reverse of North America, where JAWS leads at 55.5% versus NVDA at 24.0% (WebAIM Survey #10). If your product is sold into EU markets, treating JAWS as the only priority misrepresents your actual user population. Start with NVDA.
The trade-off. Tier 1 covers the widest population at the lowest cost. Tier 2 adds the pair most likely to be cited in a formal audit or procurement evaluation. Tier 3 adds coverage at real cost in time and licences. Name that cost explicitly when scoping a sprint.
Keyboard-Only Pass First
Run a keyboard-only pass before you open a screen reader. If you skip this step, keyboard bugs get misdiagnosed as screen reader bugs, and you waste time chasing the wrong fix.
What to verify:
- Tab order follows a logical reading sequence. No element is skipped; no element appears out of document order.
- Visible focus indicator is present on every interactive element at all times (WCAG 2.4.7). Do not accept the browser default if it has been suppressed by
outline: none. - No keyboard traps. You can enter and exit every component using only the keyboard.
- Every control is reachable and actuable. Buttons fire on Enter and Space. Links fire on Enter. Custom widgets follow the ARIA Authoring Practices Guide keyboard patterns.
Only when the keyboard pass is clean should you add a screen reader to the session. A broken tab order will produce confusing announcements that look like AT bugs.
The Testing Protocol
This is a fixed script. Run it in the same order every time so results are comparable across releases. Assign pass/fail and record evidence (screenshot, video clip, or verbatim announcement) for each step.
Load the page. Confirm the screen reader announces a meaningful page title (WCAG 2.4.2). Confirm no unexpected content is announced on load — no focus-stealing dialogs, no auto-playing audio. Check that the document language is set correctly (lang attribute) so the AT uses the right voice and pronunciation rules.
Open the headings list (NVDA: Insert+F7 → Headings tab; JAWS: Insert+F6; VoiceOver: VO+U → Headings rotor). Verify: one <h1> that describes the page; no skipped levels (e.g. <h2> jumping to <h4>); every heading makes sense out of context. 88.8% of screen reader users find heading levels useful (WebAIM Survey #10). A heading list that reads like a table of contents is passing. One that reads like random labels is not.
Tab once from the top of the page. Confirm the skip link is visible or becomes visible on focus. Activate it. Confirm focus lands inside the <main> content area — not on the <main> element itself, but on the first focusable element or the container if it has tabindex="-1". This is the check that catches the one-in-ten broken skip links the WebAIM Million 2026 found.
Open the elements list and filter to links (NVDA: Insert+F7; JAWS: Insert+F7; VoiceOver: VO+U → Links rotor). Read every link in isolation. Each must make sense without surrounding context (WCAG 2.4.6). 'Read more', 'click here', and 'learn more' are failures. Then filter to buttons and repeat. Check that buttons are announced as buttons, not as links or unlabelled elements.
Tab through every form field. For each field, the screen reader must announce: the field's accessible name, its role (edit field, checkbox, combobox, etc.), its current state (checked, expanded, required), and whether it is required. Submit the form with at least one field empty or invalid. Confirm that error messages are announced — either via a live region or by focus moving to the error — and that each error is associated with the correct field via aria-describedby or equivalent. Map to WCAG 4.1.3 (Status Messages) and 1.3.1 (Info and Relationships).
Operate each widget using keyboard only. Confirm: the correct ARIA role is announced on focus; state changes (expanded/collapsed, selected, pressed) are announced immediately; arrow key navigation works as the APG pattern specifies; modal dialogs trap focus inside and return focus to the trigger element on close; comboboxes announce the number of options and the currently selected option. A role="menu" that does not support arrow-key navigation is a failure — this is the pattern behind the 22% barrier rate in the WebAIM Million 2026 data.
Trigger async updates: search results loading, form validation firing, toast notifications appearing, cart counts updating. Confirm that aria-live regions announce the change without stealing focus. Confirm that status messages (WCAG 4.1.3) are announced without requiring the user to navigate to them. Confirm that page sections that update do not cause the reading cursor to jump unexpectedly.
Navigate to each image. Informative images must have a meaningful text alternative announced. Decorative images must be silent (empty alt or aria-hidden="true"). Complex images (charts, diagrams) must have a longer description accessible from the page. For data tables: confirm column and row headers are announced when navigating cells. For media: confirm controls are keyboard-operable and labelled. (Alt text and naming are covered in depth in the sibling posts on ARIA labels and text alternatives publishing this week.)
Run the primary revenue or service journey from start to finish using only the screen reader and keyboard. For an e-commerce product: browse to a product, add to cart, proceed to checkout, complete payment. For a SaaS product: log in, complete the core task, log out. Record every point where you had to guess, backtrack, or use a workaround. These are the failures that matter most to regulators and users.
Operating Each Tool
Switching on a screen reader without learning its navigation model produces false positives. Here is the minimum you need before running the protocol.
NVDA (Windows) NVDA has two modes. Browse mode (the default) intercepts arrow keys for reading. Focus mode activates automatically when you enter a form field or widget, passing keystrokes to the application. The most common tester mistake is staying in browse mode inside a widget and concluding it is broken. Use Insert+F7 to open the Elements List (links, headings, form fields, buttons). Use Insert+Space to toggle modes manually when needed.
JAWS (Windows) JAWS uses a virtual cursor for reading and forms mode for interactive elements. Insert+F6 opens the Headings List. Insert+F7 opens the Links List. JAWS has more configuration options than NVDA; test with default verbosity settings unless you are specifically investigating verbosity behaviour. Note that JAWS+Chrome and JAWS+Edge can behave differently for the same page - test both if you are in Tier 2.
VoiceOver on macOS Enable Full Keyboard Access (System Settings -> Keyboard -> Keyboard Shortcuts -> Use keyboard navigation to move focus between controls). The VO modifier is Caps Lock or Control+Option. VO+U opens the Web Rotor - navigate between Headings, Links, Form Controls, and Landmarks using the arrow keys. VO+Shift+Down Arrow enters a web area. Test with Safari; VoiceOver+Chrome on macOS has known inconsistencies.
VoiceOver on iOS Enable VoiceOver via Settings -> Accessibility -> VoiceOver, or triple-click the side button if the shortcut is configured. Navigate by swiping right (next element) and left (previous element). Double-tap to activate. Open the Rotor by rotating two fingers on the screen - set it to Headings, Links, and Form Controls for the protocol. Test with Safari; it is the primary mobile browser at 58.2% of mobile screen reader users.
TalkBack (Android) Enable via Settings -> Accessibility -> TalkBack. Navigate by swiping right/left. Use the Reading Controls (swipe up then right, or down then right) to switch between navigation modes (headings, links, controls). Explore by touch to identify elements. Test with Chrome.
38% of screen reader users use braille output alongside speech (WebAIM Survey #10). Verbosity has a real cost. If your ARIA labels are long, redundant, or include implementation details ('button, clickable, press Enter to activate'), they are a problem for braille users reading at a fixed cell width. Keep labels short and meaningful.
Running It as a Process
A one-off test is not a process. Here is how to embed this into a development workflow.
Who does what:
- Developers run the keyboard-only pass on every pull request that touches interactive components.
- QA runs the full protocol on changed components per pull request, and the full end-to-end journey per release.
- Involve actual AT users. Expert testers are not a substitute for lived experience. Disabled users find issues that neither automated tools nor non-disabled testers identify. Document this testing separately - it is what regulators and courts increasingly expect to see.
What to record: For every test session, record: screen reader name and version, browser name and version, OS and version, the journey or component tested, pass/fail for each protocol step, and evidence (video or screenshot). Record the date. Screen readers update frequently and behaviour changes between versions - re-baseline when a major version ships.
What to do with the results: Feed the matrix into your accessibility statement or conformance claim. Under EN 301 549, demonstrating AT compatibility is part of demonstrating conformance - not an optional extra. A formal WCAG-EM audit (covered in our website accessibility audit guide) requires a representative sample of pages; your protocol results provide the AT-specific evidence that audit methodology expects.
Common False Positives
Browse vs. focus mode confusion. If NVDA or JAWS is in browse mode inside a widget, arrow keys will move the reading cursor rather than operating the widget. This looks like a broken widget. Switch to focus mode (Insert+Space in NVDA) and re-test before filing a bug.
Verbosity and punctuation settings. NVDA's verbosity settings control whether it announces punctuation, capital letters, and attribute values. A tester with maximum punctuation verbosity will hear things a user with default settings will not. Test with default settings.
Wrong browser pairing. NVDA+Safari on Windows is not a supported combination. VoiceOver+Chrome on macOS has known gaps. Using an unsupported pair produces results that reflect the pairing, not the product. Stick to the matrix.
Screen reader bug vs. application bug. Before filing a bug, reproduce it across at least two AT/browser pairs. If only one combination fails, the issue may be in the AT or browser, not your code. Document which pairs reproduce the issue and which do not.
Standards Grounding
Under the European Accessibility Act, EN 301 549 v3.2.1 - incorporating WCAG 2.1 AA - is the current harmonised standard. A draft of v4.1.0 was published in November 2025, and v4.1.1, which aligns with WCAG 2.2, is expected to be cited in the Official Journal later in 2026 - treat that as expected, not settled.
EN 301 549 goes beyond WCAG in requiring assistive-technology compatibility and closed-functionality provisions. That is why AT testing is part of demonstrating conformance, not a supplement to it. The WCAG criteria most directly exercised by this protocol are: 1.3.1 (Info and Relationships), 2.1.1 (Keyboard), 2.4.1 (Bypass Blocks), 2.4.3 (Focus Order), 2.4.7 (Focus Visible), 4.1.2 (Name, Role, Value), and 4.1.3 (Status Messages). The W3C WAI evaluation resources and the ARIA Authoring Practices Guide are the authoritative references for what correct behaviour looks like.
Closing: What the Protocol Is Actually For
Screen reader testing is not about generating a compliance artefact. It is about answering a specific question: can a person using assistive technology complete the task your product exists to support?
An authority or complainant under the EAA will ask exactly that question. They will not ask whether your automated scan was clean. They will ask whether the task can be completed. The 67% of screen reader users who never contact a site owner when they hit a barrier will not ask it at all - they will simply leave.
The protocol in this post is designed to give you a repeatable, evidence-based answer to that question, release after release. Run it. Record it. Fix what it finds. That is what screen reader testing is for.
Related reading
Alt Text Best Practices: A Decision Tree for Every Image Type
More than one in four images on popular home pages has missing, questionable, or repetitive alt text. The fix isn't "describe the image" - it's asking what job the image is doing. A decision tree for every image type.
ARIA Labels Done Right: The First Rule, Accessible Names, and the Mistakes Behind 2026's Regression
ARIA code is up 27% in a year, yet WCAG failures just reversed six years of progress. Here's what front-end teams get wrong about ARIA labels, accessible names, and when not to reach for ARIA at all.
Accessibility Conformance Reports (ACR/VPAT): The EU Procurement Guide for Vendors and Buyers
What a VPAT is, how it becomes an ACR, which edition to use for EU procurement under EN 301 549, and how to write one that survives scrutiny - plus a buyer-side checklist.