When 97% accurate AI is too risky to deploy: Understanding Shift-AI
Shared responsibility is no accountability– the line was offered by one of our audience members at our side-event, “How to shift AI use in #LMIC #healthcare?” at #UNGA81 in New York. While the healthcare sector is increasingly adopting AI and telemedicine in its existing infrastructure, the task-shifting and outcomes can vary widely.
The Problem
Two programs can both call themselves “AI decision support for community health workers” and share almost nothing.
-
One drafts text a nurse edits before sending.
-
Another makes a diagnostic call a specialist used to own.
Donors can’t compare them, regulators can’t certify them, and ministries can’t budget for them.
As Neha Verma said in her opening, an AI tool can drift after deployment, behave differently across languages and populations, and fail with full confidence and no visible warning.
The Solution
SHIFT-AI asks five questions of any deployment to improve how we deploy AI in LMIC healthcare.
- What task is shifting (Scope)?
- From whom to whom (Human cadre)?
- At what autonomy level, L0 to L5, with what oversight loop (Intelligence)?
- What happens when it fails (Fallback)?
- What evidence is required before launch (Trust)?
Five Insights from the case studies shared at the SHIFT-AI event
1. The hard part is the cost of a rare error.
Soumyadipta Acharya walked through VectorCam, which turns a low-cost smartphone into a tool village health teams use to identify mosquito species. In a Uganda cluster RCT, VHTs identified about 73,000 mosquitoes over 12 months with PCR-confirmed accuracy above 95%, and Uganda’s Ministry of Health is now scaling it nationally. The tension is Anopheles stephensi, an invasive vector that breeds in cities. At 97% accuracy, one false positive could trigger a national emergency response, and the mosquito is discarded once photographed. The team’s answer: one tool, two oversight regimes. Human on the loop for routine counts, human in the loop with molecular confirmation for rare alerts.
2. The safest model may be the simpler one.
Jordan Shuff described how Visilant shifts anterior eye screening to CHWs with a 12th-grade education and one day of training. In head-to-head tests, generative and classical models both reached 94 to 95% accuracy. The generative model’s failures were catastrophic: hallucinations, emergencies labeled normal. Visilant chose seven classical models that check each other, and set miss tolerance by severity. A 5% miss rate is acceptable for early cataract. Zero is acceptable for anything that could blind someone within two weeks. Lower intelligence tier, highest trust tier, backed by a 2,000-patient pragmatic trial across five phone models and multiple Indian states.
3. A fallback that assumes specialists who don’t exist protects no one.
Ekyaalo, still early, would let a district hospital nurse perform a fine needle aspiration and scan the slide through a smartphone-mounted microscope. This matters because median time to breast cancer diagnosis in Uganda is 11 months. With roughly 18 WHO Level 1 certified pathologists in the country, “a pathologist reviews every slide” sounds safe and will not happen. Designing for real autonomy demands harder evidence, and it is where the value sits.
4. Users will rewrite your design.
Subhashree Dutta shared what the^delta learned funding the Antara Foundation’s WhatsApp assistant for maternal and child health workers in Madhya Pradesh. Workers wanted help mid-conversation with a patient, not a curriculum. Village anganwadi workers, who run village nutrition and childcare centres, used it more than the trained health staff it was built for. The heaviest users were the cadre with the least clinical training and fewest other places to get an answer.
5. Evaluation never ends, and accountability is unresolved.
Temina Madon laid out the Agency Fund’s four-level evaluation playbook: model, product, user, impact. She cited a Reach Digital Health RCT in which AI onboarding collected less complete data from mothers than the rules-based chatbot it was tested against. Neha described a commercial translation model whose provider-side update made translations into Assamese, a language spoken by about 15 million people in northeast India, clinically wrong. Intelehealth caught it through daily random sampling and an in-app flag doctors can hit.
Who is liable? Shared liability is emerging as the answer, and an audience member offered the line of the day: shared responsibility is no accountability.
The vote that surprised us
Temina asked which feels safer: an AI copilot drafting replies a CHW sends to pregnant women, or AI ranking which households a CHW should visit each morning. About 60% chose prioritization. Supporters said the CHW keeps her judgment and her relationship with the family, while drafting patient replies felt closer to diagnosis. The sharpest dissent came from someone who knows Kenya’s community health systems well: household data in many countries isn’t yet good enough to rank risk. Both groups came to the same answer: The intelligence layer is only as reliable as the data feeding it.
What we still can’t answer
No one is benchmarking models for LMIC health contexts, so every implementer builds and evaluates alone. The evidence burden falls on innovators, while governments sit at the table but rarely design or run the studies. And there is a real risk that finance ministries cite AI cost-effectiveness as a reason to hold health budgets flat, in countries where CHW wages are already contested. Temina’s counterpoint: officials who write off the frontline workforce change their minds when an RCT shows that same workforce delivering results with the right tool.
How you can use SHIFT-AI today
At Intelehealth, SHIFT-AI is now our AI procurement policy. No AI tool enters clinical use until the canvas is filled out and our committee confirms that intelligence level, oversight loop, fallback and trust tier fit together. We revisit it quarterly. Temina urged teams to use it as a target product profile, defining where you intend to go so that target sets the evidence bar. Prototype cheaply first, even with a human relaying GPT answers over WhatsApp.
Our ask: fill out the canvas for one tool you fund, build or regulate, and reply with where it broke. Every case it can’t handle makes the framework stronger. A big thanks to our panelists and to Nikesh Shah for moderating.
