Reduced Time Spent Creating Clinical Study Reports By 90%

A nonprofit dedicated to public education around clinical trials, partnered with NineTwoThree to streamline the creation of Plain Language Summaries (PLS) from complex clinical study reports (CSRs) with a custom AI-powered workflow.

AI-Powered Document Generation

A leading non-profit organization was creating plain-language summaries of clinical trial results for the public when they uncovered an opportunity to leverage AI.

Each summary takes 40 hours to reach the first draft, and much of this was painstakingly manual. They knew AI could help their long process, but didn’t know where to start. They also didn’t have an engineering team to figure out how to accomplish this (achievable) goal.

NineTwoThree was tasked with building a tool to parse this highly medical information, and assist the nonprofit team members in creating a six-page summary document.

In just three months, NineTwoThree delivered an ROI-positive solution. The final result is a perfect case study on how to use AI tactically to save time and money.

NineTwoThree was tasked with building a tool to parse this highly medical information, and assist the nonprofit team members in creating a six-page summary document.

In just three months, NineTwoThree delivered an ROI-positive solution. The final result is a perfect case study on how to use AI tactically to save time and money.

CONCEPT

Clinical Trials Made Accessible

Imagine counting on the results of a clinical trial due to an illness affecting you or someone you love, but not being able to understand the science and conclusions. This non-profit organization summarizes the findings in plain language for medical professionals, their patients, and their families.

To do this, they read highly medical and technical source documents called Clinical Study Reports (CSRs). These reports can be up to 1,000 pages long, and they take ~40 hours to reach the first draft.

Avoiding AI Pitfalls with High-Stakes Data

NineTwoThree’s extensive experience with building intuitive customer-facing applications and experiences made this partnership a no-brainer.

CHALLENGE

Avoiding AI Pitfalls with High-Stakes Data

There were a few constraints that made this a complicated process.

Each document has a strict and specific template

There is a significant amount of conditional logic the nonprofit uses to generate these documents

Accuracy is paramount; many people depend on these studies to make informed decisions about their health

APPROACH

Empowering Experts Not Replacing Them

NineTwoThree’s innovative approach helped to create a solution: an AI-assisted web app that parses these documents, and guides the team members through the process of creating the report.

It helps generate the six-page summary Word document, ensures it’s written at a fifth-grade reading level, and incorporates a human-in-the-loop to ensure the extracted data is accurate before document creation.

A True Partnership

To build this, NineTwoThree assembled a nimble team:

One full-stack engineer
One ML engineer
One product manager

Programming expertise, data science knowledge, and a user advocate.
NineTwoThree also went above and beyond a simple contract. They identified an easy way to automate the entire workflow, not just the data extraction process the client initially envisioned.

We View
All Partnerships This Way;
We’re Invested
In Your Success.

THE AI

Purpose-Built To Achieve internal KPIs

In enterprise and customer-facing AI workflows, it’s important to consider which metrics actually matter.

In this case, the customer wasn’t interested in a flashy chatbot or image generator.

Their chief metrics were saving their team time, and accuracy. That meant we couldn’t hand the entire parsing and creation process off to an LLM, whose non-deterministic outputs might lead to inaccurate or unreliable end results.

Instead, we focused on relying on AI to automate specific, repetitive tasks. We also ensured a human was in the loop to check output before the final document was created.

SOLUTION

A Trustworthy AI Workflow

LLM For Data Extraction

Human-In-The-Loop Web App

Automated Document Generation

LLM For Data Extraction

We built an LLM layer to parse the CSR documents. Notably, this LLM layer is for extraction, not summarization.

It finds specific pieces of information, like treatment administration (“capsule” or “IV”) and uses that information to generate the draft summary.

To ensure high confidence, the system breaks the large document into smaller chunks, and analyzes each chunk.

Two notable strengths of LLMs are text parsing and discovering intent. This is how NineTwoThree leverages AI - playing on its strengths, without fully relying on LLMs.

Human-In-The-Loop Web App

An LLM-powered intent system is only marginally more helpful than the previous manual process.

Plus, let’s be honest; there’s no use building a system that no one will use. We wanted to make sure the new automations had an intuitive web app to wrap around the logic.

This web app allowed the team to serve as the primary human reviewer. Medical writers can review and double-check the accuracy of every single data point extracted by the LLM before we create the summary document.

The UI has 80+ fields the user can check, a huge improvement over the manual process.

Automated Document Generation

After the medical writer verifies the data, it’s one-click to generate the plain language summary.

This summary uses a combination of the writer’s thoughts and opinions with the logic-based structure we mentioned earlier.

Logic-based code comes in handy here. Again, we don’t want to hand this part off to an LLM. We want deterministic output and reliable performance.

We know that every time we’re getting the same result, even with the LLM components mentioned above.

Impact

Quantifiable Results from Day One

What started as a fantastic idea turned into a successful effort to make critical health information accessible to the public. This entire process took three months, and the efficiency gains were incredible.

90%

A 90% reduction in document creation; from ~40 hours per document down to just 3-4 hours.

ROI

We can easily estimate this as ROI-positive by taking an average Medical Writer’s salary ($85,000) and calculating their 40-hour work week as costing $1634.

$1500

A 90% reduction gives a savings of $1500 per paper. In under a year, this project will be ROI-positive. In fact, we calculated this project will save at least 1800 hours per year for our client.

This flexible web app and framework can also expand to other templates and use cases in the industry.
NineTwoThree identified the opportunity to go beyond just data review and proposed building the full software solution that also automates the creation of the final Word document.

How can I get results like this?

While this partnership was a success, it wasn’t abnormal.

NineTwoThree’s innovative, proven approach to building AI solutions applies to many industries.

Reach out today if you’re interested in seeing how it can make your organization AI-first.

Contact founders

How can I get results like this?

Casey Balich, Director at Edwards lifesciences

The fact that they want to understand the business first before jumping into the project is extremely helpful; the team has become knowledgeable about what we do and what our clinicians need, making the project move more quickly.