Template matching by pro‎mp‎t tuning of open vocabulary object detection models

4/21/2026 12:00:00 AM

In recent years, the remarkable growth of vision foundation models has fundamentally transformed the way classical computer vision problems, such as template matching, are addressed. Template matching, which plays a central role in tasks including detection an‎d tracking, faces challenges in traditional approaches based on geometric an‎d pixel-wise comparisons, particularly with respect to scale variation an‎d limited generalization capability. In this study, a novel framework based on vision foundation models is proposed that, instead of relying on pixel-level matching, leverages semantic representations to achieve conceptual alignment between the template image an‎d the search image. To this end, for the first time, a framework is introduced that enables an optimization-based solution to the template matching problem. Specifically, by employing the MSDNet network, which is a vision foundation model, the framework proposes regions in the search image where the likelihood of the template’s presence is maximized. Based on a pro‎mp‎t provided by the user through the template image, the model extracts the corresponding regions from the search image. Given that the proposal model can generate multiple can‎didate regions, a re-ranking module is incorporated to perform semantic an‎d conceptual comparisons between the template an‎d the proposed regions, ultimately returning the most relevant region as the final result. Within this framework, the main challenge lies in identifying optimal region proposals for template matching. To further improve performance, the MSDNet model, which is built upon the SAM model, can be fine-tuned with a limited number of training samples by training only its pro‎mp‎t branch. As demonstrated in this research, using only 84 pairs of training data leads to improvements of more than 10 % in terms of AUC on the BBC dataset an‎d approximately 5 % on the KTM dataset compared to the base model. Overall, this study shows that combining the rich representations of foundation models with targeted fine-tuning can provide an effective solution to the template matching problem an‎d facilitate the development of more generalizable frameworks for related computer vision tasks.

تنظيم اعلان , تشخيص اشيا با واژگان باز , مدل‌هاي بنيادي بينايي , تطبيق كليشه

pro‎mp‎t tuning , open vocabulary object detection , vision foundation models , template matching

Mahdi Khoursha

MohammadReza Mohammadi