Research · Final Year Project · B.Sc. Computer Engineering
MedVLM-Report
Final Year Project — IEEE Conference Paper
$ python evaluate.py --config medvlm_report.yaml[Stage-1] DenseNet ensemble (x5) — chest X-ray findings[Stage-1] ResNet-18 (weak-sup) — breast ultrasound findings[Backbone] Swin + BLIP-2 Q-Former -> LoRA-adapted BioGPT decoder[Loss] Faithfulness-weighted detect-then-describe conditioningHeld-out studies evaluated: 1,703Mean detector AUC: 0.704 | Clinical F1: 0.451Explainability: Grad-CAM, attention rollout, sentence-to-region ✓
A unified vision-language framework that generates radiology reports for both chest X-ray and breast ultrasound from one shared decoder. The central contribution is a detect-then-describe conditioning mechanism: a Stage-1 finding detector's output is injected into the language decoder's prompt and constrained with a faithfulness-weighted loss, rather than relying on implicit visual grounding. Every report ships with a four-part explainability package — attention rollout, Grad-CAM, sentence-to-region grounding, and token/sentence rationale — evaluated across a 14-version ablation on 1,703 held-out studies.
Stack
- Vision-Language Models
- PyTorch
- BioGPT
- LoRA
- Swin Transformer
- Q-Former
- Explainable AI
Next project
KCSC — Kallar Central Sports Club