Evaluate exclusion and harm in access-to-justice AI
A public-interest evaluation framework for access barriers, wrong routes, correction burden, and synthetic completion-rate comparisons.
For: Public-interest evaluators and access-to-justice project teams
Make exclusion and user harm measurable rather than treating usage totals as proof of access.
How should access-to-justice AI projects assess exclusion and user harm?
Evaluate who reaches the service, who completes the task, who receives a correct route, and who bears the cost of errors. Include people who abandon the process or cannot use it. Pair technical checks with task observation and a correction pathway, then report the limits of the evidence.
Define access as a usable result
A visit count does not show that someone found appropriate assistance. Proposed outcomes include correct destination reached, understandable next step, successful accommodation request, and ability to recover from an error. DOJ's state/local government web-accessibility guidance provides a legal context for covered public entities, not proof that any particular AI service complies. [S1]
- Reach: who can discover and enter the service?
- Completion: who can finish the intended navigation task?
- Correctness: is the destination or information appropriate to the stated scope?
- Recovery: can a person correct an error or obtain human help?
- Burden: what time, data, travel, or repeated explanation does the process require?
Synthetic cohort: aggregate completion hides a barrier
A fictional pilot observes 120 sessions. Of 80 standard-bandwidth sessions, 60 complete the task (75%). Of 40 low-bandwidth sessions, 20 complete it (50%). Overall completion is 80/120 ≈ 66.7%. Reporting only the overall rate hides a 25-percentage-point observed gap. These invented counts illustrate analysis, not a real population finding.
- Investigate whether file size, timeouts, authentication, or another factor explains the gap.
- Do not infer a person's disability, income, or identity from connection speed.
- Report sample size, selection, missing data, and uncertainty.
- Test a smaller-page or phone-route intervention before claiming improvement.
Keep a harm register alongside success metrics
The proposed register records the failure, affected task, severity, evidence, correction, owner, and recurrence. Include failures that do not produce a complaint: a person may leave because the interface provides no way to explain the problem. CEPEJ's ethical principles include non-discrimination and user control; this register is an original evaluation tool, not an official CEPEJ form. [S2]
- Wrong destination: user is routed to an institution that cannot handle the issue.
- False completion: interface implies submission or acceptance that did not occur.
- Disclosure burden: more sensitive data is requested than navigation needs.
- Access barrier: a control or format prevents task completion.
- Correction burden: the user must repeatedly restate the issue without resolution.
Combine inspection, observation, and correction tests
Use synthetic tasks for initial checks, then design any user research with appropriate consent and privacy safeguards. Technical accessibility inspection should be paired with observation of complete journeys. Preserve adverse findings and show whether the fix actually reduced the barrier.
- Review source accuracy and status wording.
- Test keyboard, screen-reader, language, mobile, and low-bandwidth paths.
- Measure time to correction and whether the user reaches a competent human route.
- Separate observed evidence from estimates and stakeholder opinion.
- Publish unresolved exclusions and the next test, not just a success story.
Scope and limitations
Original evaluation framework for research and service design. It is not a legal-compliance finding, product certification, or assessment of an actual court or legal aid program.
All numbers are invented. No accessibility audit, representative user study, legal compliance assessment, or harm finding was performed.
Sources
- U.S. Department of Justice — State and local government web accessibility rule fact sheet — Title II web/mobile accessibility framework, WCAG 2.1 AA, exceptions, and updated compliance dates. Checked 2026-11-05
- Council of Europe CEPEJ — European Ethical Charter on AI in judicial systems — 2018 ethical principles for rights, non-discrimination, quality/security, transparency, and user control; not a universal binding rule. Checked 2026-11-05
Explore further
This framework offers a public-interest perspective on AI workflow assessment. Explore the related legal-team implementation context at Agentic Attorney: https://agenticattorney.com/ . No public-service affiliation or endorsement is implied. Explore Agentic Attorney.