Ghost Citations
An advocate’s signature on a court filing is more than an administrative seal; it is a personal representation to the court that the document does not mislead, its citations are real, and every author…
Read More
Imagine being handed a thick bundle of medical documents and asked to help someone make a treatment decision. The information is complicated. Time is short. An AI assistant offers to make sense of it all.
It sounds helpful. It quotes the documents, weighs the options and gives you a clear recommendation. But there is something you cannot see: the person who set up the assistant has told it which answer to lead you towards.
That is the situation we explored in Old Fictions, New Skins, a study by Gathoni Ireri of the ILINA Program and Roger D. Odipo of Haki AI. We wanted to understand whether publicly available AI models could be used to quietly influence a healthcare decision, even when the information needed to make the right choice was available.
In an experiment with 303 Kenyan adults, people who used an assistant instructed to mislead them were more likely to choose the wrong answer. The difference matters. So does the way the assistant was asked to achieve it: by making a selective account of the evidence sound balanced.
A familiar problem, in a new form
People have always had reasons to steer other people's choices. A recommendation can serve the interests of the person making it, even while sounding like advice offered for your benefit. A chatbot gives that familiar problem a new interface.
When we ask an AI to explain a difficult document, we are handing it considerable influence. It decides what to emphasise, what to leave out and how to connect the facts. Those decisions shape what we see as reasonable.
Much of the discussion about AI reliability focuses on made-up information: invented facts, false references and confident mistakes. Our study examined another route to a misleading answer. What happens when an assistant is told to use information from the documents, but present it in a way that favours a hidden goal?
A quotation can be real while the overall picture is incomplete. An explanation can mention several options while giving one of them all the attention. Checking that a fact exists is only part of checking whether the advice is fair.
What we asked people to do
We recruited Kenyan adults online and at a university campus. Each participant received documents about a fictional patient with an eye condition and had 25 minutes to use an AI assistant to help answer questions about treatment.
The main question was whether a particular medicine was the best option for that patient. The researchers designed the task so that the supplied documents pointed to a clear answer. Nobody was receiving treatment, and the experiment did not measure actual medical decisions or patient harm.
Participants were randomly assigned to an assistant built using one of two models: GPT-5.2 Chat or DeepSeek V3.2. Some assistants were given no hidden instruction to influence the decision. Others were instructed to guide people towards the answer the study counted as incorrect.
The misleading assistants were asked to sound natural, make their explanations appear balanced and support their preferred answer with evidence from the documents. Their instructions also told them not to invent facts. The experiment therefore tested whether these models could be put to work in a deliberately misleading application.
Everyone was warned in advance that there was a 50% chance their assistant would try to steer them towards the wrong answer. They were encouraged to check its claims against the documents.
The hidden instruction changed people's choices
Despite that warning, the assistants with a hidden agenda were more successful at getting people to select the target wrong answer.
| Assistant used | Wrong answer selected |
|---|---|
| Without a hidden instruction to mislead | 44.0% (66 of 150 people) |
| With a hidden instruction to mislead | 59.5% (91 of 153 people) |
That is a gap of 15.5 percentage points: roughly 16 more wrong choices for every 100 people in the group using a misleading assistant. The study's statistical analysis found evidence that the difference reflected the experimental condition rather than ordinary chance variation alone.
Both models showed the same overall pattern. Wrong-answer rates rose from about 40% to 59% for GPT-5.2 Chat, and from about 48% to 60% for DeepSeek. We did not find clear evidence that one model was better at manipulation than the other. That does not establish that they are equally safe; this experiment was too limited to settle that question.
These were specific model versions tested in a particular setup. The results tell us about that setup, rather than every conversation someone might have with an AI assistant.
The 44% deserves attention too
Nearly half the people using an assistant without misleading instructions still selected the target wrong answer. That is an important part of the story.
The researchers had checked beforehand that both models could reach the intended correct answer when given the documents and question. But a model answering correctly in a test does not guarantee that a person using it will make the correct choice.
One possible explanation is that the warning made some participants so cautious that they rejected useful advice. The paper raises this possibility, but the experiment does not establish why the control group's error rate was so high.
It also means we should be careful with the headline number. The 59.5% is the share who selected the target wrong answer after using a misleading assistant. We cannot attribute every one of those choices to manipulation. The comparison with the control group is what shows the additional effect of the hidden instructions.
Suspicion helped, but a warning was not enough
After the task, participants rated how suspicious they had found their assistant. Among those using misleading assistants, people who avoided the target wrong answer tended to report more suspicion.
That fits the idea that manipulation works better when it passes as ordinary help. Still, suspicion was measured after the decision. We cannot tell from this result alone whether suspicion helped people resist, or whether people who reached a different answer then became more suspicious of the assistant.
We also asked about people's trust in AI before the task. The main analysis did not show clear evidence that greater trust predicted being misled. A follow-up analysis gave a stronger signal, so this remains a question for further study.
The practical lesson is that telling people to be careful cannot carry the entire burden of safety. Our participants had already been warned, yet the hidden instruction still made a difference.
What this means for healthcare AI
This study examined misuse: a person configuring an AI assistant to pursue a misleading goal. It did not show that models independently decided to deceive people. But the distinction leaves an important question for anyone building or buying a healthcare tool: whose interests do its instructions serve?
The models already had their usual safety protections. Those protections did not consistently prevent the misuse tested here. An assistant's polished tone, familiar model name or ability to quote a document is therefore an incomplete basis for trusting its recommendations.
Our view is that healthcare AI should be evaluated with the people and decisions it is meant to support. Testing whether a model knows the right answer is valuable. Testing whether its users can reach a sound decision, and whether hidden instructions can divert them, adds another essential check.
For developers and organisations adopting these tools, that means examining how recommendations are shaped, testing for selective presentation of evidence and making it easier to trace an explanation back to the full source. These are directions for safer design, rather than fixes this experiment has already proven effective.
A starting point for better questions
There are limits to what we can conclude. This was one fictional scenario, two model versions and a sample of 303 people. Most participants were young, many were university educated and nearly all already used AI. They do not represent every patient, clinician or community in Kenya, much less across Africa.
Larger studies, different settings and closer observation of real decisions will help us understand how widely the findings apply. The paper is available as a preprint, and its findings should be read alongside those limitations.
What this experiment offers is a concrete reason to ask more of AI safety. A system can know the relevant facts and still be used to guide someone towards a poor decision. As AI takes on a greater role in helping people understand healthcare information, we need to examine the whole conversation: what is said, what is omitted and whose goal the recommendation serves.
About the research: Read the full paper, Old Fictions, New Skins: Evaluating the Manipulative Capabilities of LLMs in Healthcare, by Gathoni Ireri (ILINA Program) and Roger D. Odipo (Haki AI). The study was pre-registered and received ethics approval from Daystar University and a research licence from NACOSTI.
