MedImageInsight Foundation-Model Evaluation

Foundation models promise strong performance for free. Before committing the platform, the organization needed measured answers on its own radiologist-labeled data.

A controlled three-way benchmark on one metric harness: zero-shot prompting, a scikit-learn MLP adapter and a Keras adapter trained on frozen embeddings. A deliberate ablation isolated image normalization. Embeddings were cached once so adapters trained in seconds on CPU.

• AUC 0.62 → 0.86 by adapting frozen embeddings instead of prompting — a reusable evaluate → embed → adapt pattern.
• Surfaced a silent failure: without min–max normalization zero-shot AUC collapsed to 0.45, worse than chance.
• The in-house model (AUC ≈ 0.868) beat the adapted foundation model (≈ 0.862), preventing a costly platform bet.

• Commissioned the study as a build-vs-adopt decision gate instead of committing on vendor claims.
• Kept the in-house models and banked the adapter pattern for future decisions.
• Required the normalization ablation and held-out-vs-training comparison before accepting any headline number.