The use of AI in medical settings has raised serious concerns, particularly regarding its accuracy and potential for errors. A recent case involving Rebecca Green, a patient who was falsely accused of taking illegal drugs due to an AI scribe's error, highlights the risks associated with this technology. Green's experience underscores the importance of human oversight and the need for doctors to carefully review AI-generated content.
AI scribes have become prevalent in healthcare, with an estimated 40% of GPs using them regularly. While they can reduce administrative burdens and allow doctors to focus more on patients, there are significant risks. These risks include the potential for AI to "hallucinate" false symptoms or conditions, as evidenced by Green's case. Such errors can have serious consequences, impacting a patient's health, career, and insurance status.
The issue of consent is also a critical concern. The Digital Rights Watch (DRW) report found discrepancies in how consent is obtained, with some doctors only providing a sign in the waiting room, which does not meet privacy law requirements. This lack of informed consent can deter patients from disclosing sensitive information, such as family violence, when AI is involved.
Furthermore, the regulation of AI scribes is a complex issue. The Therapeutic Goods Administration (TGA) has acknowledged that some AI scribes may be in breach of the law, as they are not considered medical devices. The TGA is currently reviewing AI scribes and will release its findings in the coming weeks, with any breaches being addressed.
The DRW is calling for the establishment of an AI Safety Institute to test and approve AI scribes, similar to New Zealand's approach. This would ensure that AI tools are safe and effective for use in medical settings. Additionally, compulsory checks and balances built into the software or during practice accreditation could help enforce safe usage.
In conclusion, while AI has the potential to revolutionize healthcare, it is crucial to address the risks and concerns associated with its use. Human oversight, informed consent, and proper regulation are essential to ensure the accuracy and safety of AI in medical settings. Only through these measures can we fully realize the benefits of AI while mitigating its potential harms.
