解剖学报

解剖学报 ›› 2025, Vol. 56 ›› Issue (1): 22-29.doi: 10.16098/j.issn.0529-1356.2025.01.003

• 肿瘤学专栏 • 上一篇    下一篇

基于大规模视觉语言模型的黑色素瘤诊断方法

赵家悦1,2  李诗曼1,2  章琛曦1,2*   

  1. 1.复旦大学基础医学院数字医学研究中心，上海200032; 2.上海市医学图像处理与计算机辅助手术重点实验室，上海200032
  • 收稿日期:2024-06-13 修回日期:2024-10-01 出版日期:2025-02-06 发布日期:2025-02-06
  • 通讯作者: 章琛曦 E-mail:chenxizhang@fudan.edu.cn

A melanoma diagnosis method based on large-scale vision-language models

ZHAO Jia-yue1,2  LI Shi-man1,2  ZHANG Chen-xi1,2*   

  1. 1.Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; 
    2.Shanghai Key Laboratory of Medical Imaging Computing and Computer-Assisted Intervention, Shanghai 200032, China
  • Received:2024-06-13 Revised:2024-10-01 Online:2025-02-06 Published:2025-02-06
  • Contact: ZHANG Chen-xi E-mail:chenxizhang@fudan.edu.cn

PDF

摘要：

目的 开发一个基于大规模视觉语言模型的黑色素瘤诊断框架，并探讨该框架用于黑色素瘤诊断的可行性和准确性。方法采用公开数据集Derm7pt，其数据集划分为训练集 (346例)，验证集 (161例) 和测试集 (320例)。提出了一个基于大规模视觉语言模型的黑色素瘤诊断框架，该诊断框架包括两个文本分支和一个视觉分支。在文本分支中，一个分支处理固定的临床提示；另一个分支则处理可学习的提示。这种设计旨在通过固定的临床提示引导和优化可学习提示的效果。视觉分支处理皮肤镜图像，通过微调图像编码来增强对黑色素瘤特征的识别能力。结果在Derm7pt数据集上，我们的方法在性能上优于现有其他方法。其接收者操作特征曲线下面积（AUC），准确率和F1-分数分别为87.35%，84.17%和84.01%。结论通过适当的微调策略，基于大规模视觉语言预训练模型的方法能够有效地适应黑色素瘤的诊断任务。这种方法可以作为医生的有力辅助工具，帮助他们做出更加准确的诊断决策。


关键词: 黑色素瘤, 大规模视觉语言模型, 微调, 诊断, 深度学习, 人

Abstract:

Objective To develop a melanoma diagnosis framework based on large-scale vision-language models, and to explore the feasibility and accuracy of the framework for melanoma diagnosis.   Methods The publicly available Derm7pt dataset, which was divided into a training set (346 cases), a validation set (161 cases), and a test set (320 cases) was utilized. A melanoma diagnosis framework based on large-scale vision-language models was proposed, comprising two text branches and one visual branch. In the text branches, one branch processed fixed clinical prompts, while the other handled learnable prompts. This design aimed to optimize the effectiveness of learnable prompts through guidance from fixed clinical prompts. The visual branch processed dermoscopic images and enhanced melanoma feature recognition through fine-tuning the image encoder.   Results On the Derm7pt dataset, our method  outperformd other existing method. It achieved an area under the receiver operating characteristic curve (AUC) of 87.35%, an accuracy of 84.17%, and an F1-score of 84.01%.   Conclusion The study demonstrates that with appropriate fine-tuning strategies, methods based on large-scale vision-language pre-trained models can effectively adapt to melanoma diagnosis tasks. This approach can serve as a powerful auxiliary tool for doctors, helping them make more accurate diagnostic decisions.

Key words: Melanoma, Large-scale vision-language model, Fine-tuning, Diagnosis, Deep learning, Human

中图分类号: