Using AI in Automation of Evidence Synthesis: What Does Research Tell Us?

Methods in Evidence Synthesis Seminar (MESS), Bristol Medical School, UK; 18th Feb, 2025

0:00 Introducing the guest
0:57 Conflict of interest
01:52 Why this session
02:52 Typology of AI - A Classification
05:49 Research on Using LLMs in PICO Development for Systematic Reviews and Clinical Practice Guidelines
07:51 Research on Using LLMs in Developing Search Strategies
10:47 LLM and Systematic Search
14:07 Using AI for Screening (Title and Abstract Screening vs Full Text Screening)
16:32 LLM and Screening
18:35 Quality of Input (Garbage In, Garbage Out): Human Expertise + File + Prompt Engineering
22:21 ClinicalTrials.gov's Drama (RIS vs CSV Output/Input) and Incomplete or Non-Standard Records
23:12 Rayyan's Drama when there is no Abstract
24:05 PRISMA 2020 accepts using automation tools.
24:35 Machine Learning Options in Covidence: RCT Classifier
25:12 Result of Using RCT Classifier in Covidence
25:36 Priority Screening and Progress Graph using Machine Learning in ASReview
26:38 Relevance Ranking and Screening Relevant Ones First in Covidence
27:22 Research on Assessment of Risk of Bias Using LLMs
28:55 LLM and Risk of Bias Assessment
30:15 Risk of Bias Assessment Tools Using LLMs in PICO Portal - Attention Map and Possible Issues
31:45 Research on Using LLMs for Data Extraction
33:40 LLM and Data Extraction (Gartlehner et al. 2024)
35:24 Scoping Review of Research on using LLM in All Systematic Review Steps till February 2024 (Lieberum et al. 2024)
36:49 Ethical and Legal Issues of Using LLMs in Evidence Synthesis: Copyright, Using User Data, Law and Ethics Behind Science and Tech, Slow Research & Rapid Development, and Lack of Policy and Practice Guidelines
39:43 What to expect in the future: Evaluation, Embedded, Less Hallucination & Errors, OpenAlex, Searching=Screening, Usability, Prompts, RAG, and Temperature Settings
43:34 Recommended Reading: NICE, ISPOR, Cochrane, ICASR
44:08 How to stay up to date on the topic of using AI in evidence synthesis
45:16 Start of Questions and Answers
45:32 Question: How far away are we from review done predominantly by AI?
47:07 Question: We are using Laser AI, and others are using Nested Knowledge. What are your thoughts on those platforms?
47:54 Question: You mentioned identifying unpublished systematic reviews at the beginning of the talk. Is that a tool based on PROSPERO protocols?
49:06 Question: You mentioned the full-text screening can be automated without help from reviewers. What happens to the linked reports of studies?
50:19 Question: Sometimes, you find the relevant studies during the full-text screening via checking the references. How about that?
51:09 Question: What is the environmental impact of using AI and LLMs?
54:16 Question: AI complimented me on my writing. Should I believe it? Advice on Writing Tasks
57:24 Ending the webinar

‪@CochraneCollab‬ ‪@cochraneaustria8744‬ ‪@cochranesouthafrica8821‬ ‪@CochraneUKcentre‬ ‪@cochraneaustralia9872‬ ‪@cochranetraining685‬ ‪@collaborationtube‬ ‪@campbellukireland4997‬ ‪@CochraneHealthEquity‬ ‪@JBIEBHC‬ ‪@ISPORorg‬ ‪@niceorguk‬ ‪@Rayyanapp‬ ‪@rayyan5358‬ ‪@covidenceapp‬ ‪@EvidencePartners‬ ‪@EvidencePrime‬ ‪@collaborationforenvironmen1886‬ ‪@centerclinicaltrials‬ ‪@africaevidencenetwork8310‬ ‪@instituteforevidence-based5743‬ ‪@theevidence-informedchirop8847‬ ‪@gcespgi‬ ‪@nestedknowledge‬ ‪@ASReview‬