When social media signals matter: Improving AI reliability in pharmacovigilance

Pharmacovigilance teams are responsible for monitoring potential adverse events across a growing range of data sources. Social media can provide early signals, but it also creates a challenge, even for AI: posts are informal, ambiguous, constantly changing, and often far removed from the structured language used in clinical or regulatory settings.

For one leading pharmaceutical company, an existing adverse event monitoring system was already being used to scan unstructured sources, including social media. However, the model’s performance was less reliable once deployed in production than it had been during training. As language, topics, and user behavior shifted over time, the system needed a more robust way to detect and manage changes in incoming data.

The client wanted to explore how machine learning could improve the reliability of adverse event detection while supporting the standards expected in a highly regulated patient safety environment.

The challenge: When social media language changes faster than the model

Adverse event monitoring on social media is not a simple text classification task. Posts can be short, informal, sarcastic, misspelled, or incomplete. Users may refer to medicines indirectly, describe symptoms casually, or use language that differs significantly from the examples available during model training.

This created a practical reliability challenge. A model that performed well on historical training data could become less accurate when exposed to live production data, where the distribution of language and topics was constantly changing.

For a pharmacovigilance use case, this mattered. Potential adverse event mentions needed to be identified consistently enough to support expert review, while maintaining transparency and control over how AI-assisted outputs were generated.

How we helped: Building a pipeline to monitor model reliability

We developed a machine learning pipeline designed to improve the reliability of adverse event detection from unstructured social media text. A key focus was monitoring and adjusting for data distribution shifts between training and production, one of the main reasons AI models can underperform after deployment.

The pipeline helped identify when incoming social media data began to differ from the data the model had been trained on. This gave the client a stronger foundation for understanding model behavior over time and maintaining performance in a changing real-world environment.

To support testing and validation, we also built a proof-of-concept application with a simple interface. Internal stakeholders could enter example messages and observe the model’s predictions in real time, making it easier to test edge cases, review outputs, and discuss model behavior with both technical and domain experts.

The production-oriented design was shaped around the needs of patient safety specialists. The goal was not to replace expert pharmacovigilance review, but to support safety teams with a more reliable and transparent AI-assisted monitoring workflow.

Validating a more reliable approach to AI-assisted adverse event detection

The project showed that model reliability in social media pharmacovigilance could be improved by actively monitoring changes between training and production data.

By making distribution shifts visible and easier to manage, the solution gave the client a stronger basis for maintaining adverse event detection performance over time. It also created a more accessible way for stakeholders to test model predictions, understand system behavior, and evaluate whether the approach could support patient safety workflows.

The engagement helped establish a foundation for more robust AI-assisted pharmacovigilance, with clearer validation, improved transparency, and better alignment between machine learning outputs and expert safety review.

Key outcomes included:

• Improved monitoring of model performance between training and production environments.
• Greater visibility into data distribution shifts affecting adverse event detection.
• A proof-of-concept interface that made model testing easier for internal stakeholders.
• Better alignment between AI-assisted detection and patient safety expert workflows.
• A stronger foundation for reliable, transparent AI use in pharmacovigilance.