Standardizing Clinical Performance Evaluation with Multimodal GenAI

Solution:

To meet these challenges, we designed and implemented a secure, cloud-native Multimodal Generative AI Evaluation System built on AWS. The system combined video, audio, and text analysis through advanced AI models to produce objective and rich performance evaluations, while incorporating Retrieval-Augmented Generation (RAG) to ensure continuous accuracy improvement over time. The entire application was deployed on Amazon EKS to support scalable, containerized workloads, and Amazon RDS served as the foundation for both the RAG knowledge store and the backend database.

The solution was implemented in three key phases:

Phase 1: Data Integration and Processing

• All video and audio feeds from simulation rooms were time-synced and securely ingested into the cloud.
• Speech recognition and sentiment analysis were applied to identify tone, clarity, and emotional state — capturing both student confidence and patient sentiment in real time.
• Video data was processed using Gemini Pro, a multimodal model that detected non-verbal behaviors such as body posture, hand gestures, eye contact, and adherence to procedural protocols.
• All extracted text, transcripts, and structured observations were stored in Amazon RDS, enabling the RAG workflow to retrieve historical context, institutional rubrics, and prior feedback patterns to enrich each evaluation.

Phase 2: Intelligent Evaluation and Scoring

• The outputs from audio, video, and sentiment models were synthesized by Claude 3 Sonnet, an advanced reasoning model that integrated all data points into a single, coherent analysis.
• The model applied the client’s detailed evaluation rubrics as its internal system prompts, ensuring full alignment with institutional grading standards.
• Through the RAG pipeline, Claude 3 Sonnet accessed the latest rubric updates, domain guidelines, and evaluator feedback stored in RDS — creating a continuous feedback loop that enhanced scoring accuracy over time.
• For each simulation, the system produced a structured evaluation report including:
  1. Numeric scores across rubric categories (e.g., Clinical Reasoning – 4/5; Empathy – 5/5).
  2. Narrative feedback highlighting specific moments and improvement areas (e.g., “The student showed strong diagnostic reasoning but missed summarizing patient concerns before moving to examination.”).

Phase 3: Adaptability Across Simulation Setups

• The system was built to handle all levels of complexity — from simple one-on-one encounters to multi-student, multi-patient, or distributed multi-room simulations.
• In group sessions, each student’s performance metrics were isolated and tracked individually, ensuring scoring fairness even in collaborative settings.
  In distributed environments, synchronized metadata correlated activities from multiple cameras and microphones into a unified timeline, allowing the AI to evaluate the entire session as a single, cohesive event.
• Deployed on Amazon EKS, the platform automatically scaled to support large simulation volumes, parallel evaluation workloads, and peak assessment periods.
  Using RDS-backed RAG, the system continuously learned from evaluator corrections and institutional updates, steadily improving rubric alignment and evaluation precision.