Bell Eapen MD, PhD.

Bringing Digital health & Gen AI research to life!

Locally hosted LLMs

TL; DR: From my personal experiments (on an 8-year-old, i5 laptop with 16 GB RAM), locally hosted LLMs are extremely useful for many tasks that do not require much model-captured knowledge. 

Ollama

Image credit: I, Luc Viatour, CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0, via Wikimedia Commons

The era of relying solely on large language models (LLMs) for all-encompassing knowledge is evolving. As technology advances, the focus shifts towards more specialized and integrated systems. These systems combine the strengths of LLMs with real-time data access, domain-specific expertise, and interactive capabilities. This evolution aims to provide more accurate, context-aware, and up-to-date information, saving us time and addressing the limitations of static model knowledge.

I have started to realize that LLMs are more useful as language assistants who can summarize documents, write discharge summaries, and find relevant information from a patient’s medical record. The last one still has several unsolved limitations, and reliable diagnostic (or other) decision-making is still in the (distant?) future. In short, LLMs are becoming increasingly useful in healthcare as time-saving tools, but they are unlikely to replace us doctors as decision-makers soon. That raises an important question; Do locally hosted LLMs (or even the smaller models) have a role to play? I believe they do! 

Locally hosted large language models (LLMs) offer several key benefits. First, they provide enhanced data privacy and security, as all data remains on your local infrastructure, reducing the risk of breaches and unauthorized access. Second, they allow for greater customization and control over the hardware, software, and data used, enabling more tailored solutions. Additionally, locally hosted LLMs can operate offline, making them valuable in areas with unreliable internet access. Finally, they can reduce latency and potentially lower costs if you already have the necessary hardware. These advantages make locally hosted LLMs an attractive option for many users.  

The accessibility and ease of use offered by modern, user-friendly platforms like OLLAMA are significantly lowering the barriers for individuals seeking technical expertise in self-hosting large language models (LLMs). The availability of a range of open-source models on Hugging Face lowers the barrier even further. 

I have been doing some personal experiments with Ollama (on docker), Microsoft’s phi3: mini (language model) and all-minilm (embedding model), and I must say I am pleasantly surprised by the results! I have been using an 8-year-old, i5 laptop with 16 GB RAM. I have been using it as part of a project for democratizing Gen AI in healthcare, especially for resource-deprived areas (more about it here), and it does a decent job of vectorizing health records and answering questions based on RAG. I also made a helpful personal writing assistant that is RAG-based. I am curious to know if anybody else in my network is doing similar experiments with locally hosted LLMs on personal hardware. 

Published by Bell Eapen on | Permalink

LLM-in-the Loop CQL execution 

TL;DR: I have added experimental support for using LLMs to interpret DocumentReference-based definitions in CQL of the type:

define HasVisualFootExamThisMonth: 
exists( 
[DocumentReference] D 
where D.status.value = ‘current’ 
) 

 (Read Part I here)

Clinical Quality Language (CQL)

Image credit: Grufo, CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0, via Wikimedia Commons

CQL is a domain-specific language that allows clinicians and researchers to express queries and retrieve data from electronic health records (EHRs) in a standardized and interoperable way. CQL can be used to define clinical quality measures, decision support rules, cohort identification criteria, and other clinical logic. Though CQL can be based on the HL7 FHIR standard, which defines a common data model and terminology for health information, the language itself is schema-independent. CQL aims to improve the quality and efficiency of healthcare by enabling the reuse and sharing of clinical knowledge across different systems and platforms. Here is an example.

CQL is a high-level language that needs to be translated into a lower-level language that can be executed. To facilitate this process, CQL supports a mechanism for transforming CQL expressions into an intermediate representation called Expression Logical Model (ELM). ELM is a platform-independent XML/JSON format that preserves the semantics and structure of CQL expressions but removes the syntactic variations and ambiguities of natural languages. ELM can then be converted into executable formats such as SQL or FHIRPath, depending on the target data source and system. This way, CQL to ELM translation enables the portability and interoperability of clinical queries across different platforms and environments. Here is an example.

CQL can leverage FHIRPath to express queries over FHIR resources in a consistent and interoperable way. However, FHIRPath alone is not sufficient to handle the semantic variations and complexity of clinical data. For example, different systems may use different codes or terminologies to represent the same concept, such as diabetes or foot exam. To address this issue, CQL supports the use of terminology services, which are external services that provide mappings and translations between different code systems and value sets. CQL can invoke terminology services to resolve the codes and values used in the queries and align them with the ones used in the data source. This way, CQL can execute queries over FHIR data using both syntactic and semantic interoperability. 

One limitation of FHIRPath-based CQL execution is that it cannot handle assertions in the FHIR DocumentReference resource. I have forked the nodejs CQL execution engine to add a hook that can call an LLM when it encounters a DocumentReference here:

https://github.com/dermatologist/cql-execution 

I will post a link to an end-to-end application that uses this hook. Also, I have added experimental support for other FHIR servers to the VSAC enabled code service engine. Do comment below, if you find this useful and use it for your project! 

Published by Bell Eapen on | Permalink

Kedro for multimodal machine learning in healthcare 

Healthcare data is heterogenous with several types of data like reports, tabular data, and images. Combining multiple modalities of data into a single model can be challenging due to several reasons. One challenge is that the diverse types of data may have different structures, formats, and scales which can make it difficult to integrate them into a single model. Additionally, some modalities of data may be missing or incomplete, which can make it difficult to train a model effectively. Another challenge is that different modalities of data may require different types of pre-processing and feature extraction techniques, which can further complicate the integration process. Furthermore, the lack of large-scale, annotated datasets that have multiple modalities of data can also be a challenge. Despite these challenges, advances in deep learning, multi-task learning and transfer learning are making it possible to develop models that can effectively combine multiple modalities of data and achieve reliable performance. 

Pipelines Kedro for multimodal machine learning

Kedro for multimodal machine learning

Kedro is an open-source Python framework that helps data scientists and engineers organize their code, increase productivity and collaboration, and make it easier to deploy their models to production. It is built on top of popular libraries such as Pandas, TensorFlow and PySpark, and follows best practices from software engineering, such as modularity and code reusability. Kedro supplies a standardized structure for organizing code, handling data and configuration, and running experiments. It also includes built-in support for version control, logging, and testing, making it easy to implement reproducible and maintainable pipelines. Additionally, Kedro allows to easily deploy the pipeline on cloud platforms like AWS, GCP or Azure. This makes it a powerful tool for creating robust and scalable data science and data engineering pipelines. 

I have built a few kedro packages that can make multi-modal machine learning easy in healthcare. The packages supply prebuilt pipelines for preprocessing images, tabular and text data and build fusion models that can be trained on multi-modal data for easy deployment. The text preprocessing package currently supports BERT and CNN-text models. There is also a template that you can copy to build your own pipelines making use of the preprocessing pipelines that I have built. Any number and combination of data types are supported. Additionally, like any other kedro pipeline, these can be deployed on kubeflow and VertexAI. Do comment below if you find these tools useful in your research. 

Template for multi-modal machine learning in healthcare using Kedro. Combine reports, tabular data and images using various fusion methods.
https://github.com/dermatologist/kedro-multimodal
2 forks.
16 stars.
0 open issues.

Recent commits:

Dark Mode

kedro-multimodal (this link opens in a new window) by dermatologist (this link opens in a new window)

Template for multi-modal machine learning in healthcare using Kedro. Combine reports, tabular data and image using various fusion methods.

1 Subscriber 3 Watchers 0 Forks Check out this repository on GitHub.com (this link opens in a new window)

Published by Bell Eapen on | Permalink

Six things data scientists in healthcare should know

Healthcare, like most other fields, is eager to get on the data science bandwagon. Data scientists can make a huge difference in the way big data is utilized for clinical decision-making. However, there are paradigmatic differences in the way data scientists from quantitative fields view the world, compared to their clinical counterparts. This is especially true in the emerging fields of machine learning and artificial intelligence. This may lead to considerable inefficiencies. As a person trained in both fields, here is my take on this.

Data scientists
Credit: Dasaptaerwin, CC0, via Wikimedia Commons

Data scientists should focus on the problem and not the solutions

Data scientists are excited about the latest GPT or BERT. Data scientists tend to refine the model a bit more using 10 more GPUs! In the process, they tend to solve problems that do not exist. From my experience practicing medicine in extremely resource-poor areas, simple solutions are valued more than BERT running on Kubernetes! This is true in the developed world as well, and many teams may have fundamental data needs that need to be tackled first.

Explanation comes before prediction

Emerging machine learning methods prioritize prediction accuracy compromising on explainability in the process. Clinicians, in most cases, cannot use nor trust a model that arrives at a conclusion without showing how it reached there. Hence, in the clinical domain, a simple logistic regression model may be more acceptable than a deep learning neural network. Parsimony is the key and a bit of feature selection to ensure parsimony will be appreciated always.

You need to know the clinical terminologies

A basic understanding of the clinical terminologies and terminology systems such as SNOMED and ICD is vital. It helps in understanding the clinical community better. Any healthcare analytics to consider variations in terminologies and adopt a standard system for consistency. Any tool that data scientists build for the clinical community should have support for terminology systems.

Biostatistics is more pervasive than you think

Most healthcare professionals are trained in biostatistics. Hence, the thinking leans towards population, sampling, randomization, blindings and showing a ‘statistically significant’ difference. Moving towards machine learning needs a paradigmatic shift. It may be useful to have a discussion on this at the outset.

Classes are of unequal importance

In healthcare, finding one class (e.g. cancer) is more important than the other class (e.g. no cancer). One class may need active intervention to save lives. Hence, sensitivity and specificity are of vital importance than accuracy!

Life is precious!

In healthcare, there is no room for error. Some decisions may have disastrous consequences while few others may save lives. As a data scientist in the healthcare domain, you should be cognizant of the fact that healthcare data is different from banking/airline data.

Published by Bell Eapen on | Permalink

Open-source for healthcare

This post is meant to be an instruction guide for healthcare professionals who would like to join my projects on GitHub.

eHealth Programmer Girl

What is a contribution?

Contribution is not always coding. You can clean up stuff, add documentation, instructions for others to follow etc. Issues and feature requests should be posted under the ‘issues’ tab and general discussions under the ‘Discussions’ tab if one is available.

How do I contribute.

How do I develop

  • The .devcontainer folder will have the configuration for the docker container for development.
  • Version bump action (if present) will automatically bump version based on the following terms in a commit message: major/minor/patch. Avoid these words in the commit message unless you want to trigger the action.
  • Most repositories have GH actions to generate and deploy documentation and changelog.

What do I do next

  • My repositories (so far) are small enough for beginners to get the big picture and make meaningful contributions.
  • Don’t be discouraged if you make mistakes. That is how we all learn.

There’s no better time than now to choose a repo to contribute!

Published by Bell Eapen on | Permalink

Interoperability for Doctors and Healthcare professionals Part I

It is important for health information systems to talk to each other. Unfortunately they speak different languages. This article is not for eHealth experts to understand the nuances of interoperability (HIE), but for health care professionals to have an idea about what is out there and what can be expected in the future.

When we consider HIE we have to think about what is being exchanged (package), how the information is organized (format) and how it is being transported (protocol). Though it is not essential to know, few terms that you might recognize are: HL7, XML for format and HTTP, TCP/IP for protocol. (Have you heard of MLLP? Google it!) The donor has the information in a certain format and protocol while the recipient expects it in a particular format and protocol.

At the core of all HIE platforms such as MirthConnect or OrionHealth’s Rhapsody, is an engine that does this conversion. Format and protocol of donor to format and protocol of recipient. Simple eh?

Is pragmatic interoperability the best solution?- M. Martineau @eHealthMusings explores a pragmatic Rhapsody approach http://t.co/XMkMXAEdt7

— Orion Health Canada (@OrionHealthCA) December 11, 2014

Now most of these platforms have a user interface or IDE for making this connection. You can also introduce certain filters at this stage. Enterprise systems like Rhapsody presents an attractive visual interface, whereas open source solutions may not be very user friendly.

What else can the engine do? It usually keeps a log of all package deliveries and whether the delivery was successful. If the delivery failed, it can attempt again and alert the maintenance team through a console. The console can even be mobile as in rhapsody. Though the engine can store the package itself for a limited time, storing the package is not really its job.

The donors could be:

  • A single department in a hospital sending lab reports.
  • All departments in a hospital sending all sorts of information.
  • Several hospitals in a region.

The recipient could be:

  • Another department in the same hospital expecting a lab test report.
  • A family physician who wants real time access to the lab reports for his patients admitted in the hospital.
  • A researcher who wants to know blood sugar status of all the diabetes patients. (population health)

We need a separate layer between the engine and the recipient to support all these use cases. Let us call this layer mediator.

The mediator can pull data in real time from donors or store it in a local database. The first one is the federated model while the other one is the centralized model. Federated is slow but up-to-date while centralized is fast but not concurrent. Mixed model has both and is preferred. The so called clinical viewers are federated mediators with a web interface.

Emerging paradigms like NoSQL and RDF may be ideal for data representation in mixed model. I have discussed RDF before. Will discuss NoSQL soon!

Published by Bell Eapen on | Permalink

Psoriasis support : eHealth gaming tools for patient engagement
Psoriasis manum
Psoriasis manum (Photo credit: Wikipedia)

Here is the IFPA  survey to compare 17 different strategies and activities that can be used to advance psoriasis education, advocacy and awareness. Preliminary results of the survey will be presented on World Psoriasis Day and the final results will be announced at the 4th World Psoriasis & Psoriatic Arthritis Conference in Stockholm July 8-11, 2015.

I have listed below some of my random ideas on eHealth tools for patient engagement in psoriasis:

An Agent based model (ABM) offers visual simulations of complex systems that can be displayed on a web browser. Psoriasis disease process can be modelled using psoriasis patients as ‘turtles’ with the known probabilities of auto-remission, exacerbation, response to conventional treatments and response to newer drugs added to the model. The patients and caregivers could interact with the model to understand how the treatment decisions affect the quality of life. ABM could be an innovative and useful web based patient education tool that portrays the reality of psoriasis without giving any false promises. Those in the  patient’s circle of care and the patient would understand the odds of improving quality of life.

Psoriasis: The naked truth
Psoriasis: The naked truth (Photo credit: SomosMedicina)

An android or iPhone app to calculate and log the PASI score of the patient would be a less obtrusive disease monitoring tool. The app may be designed to send the log to patient’s caregiver. I have not checked the apple app store or google play, probably such apps already exist.

A ‘push’ strategy such as email alerts is unlikely to work for psoriatics. An innovative strategy game where the body is modelled as a kingdom and the immunological perturbations as a t-cell mutiny could be a useful engagement tool. Vascular and systemic changes could also be part of the game. The game would be web based and would continue for a long time with the patient required to login periodically to make strategic alterations (treatment choices). Everytime the patient login to the game, medication reminders would be displayed. The game would mimic reality with changes reflecting new clinical studies. New clinical studies that has an impact on the ‘game plan’ would be available under the ‘game resources’ for everyone to read. Reading and understanding these resources would improve the performance in the game.

Published by Bell Eapen on | Permalink