شماره ركورد
31650
پديد آورنده
محمد مصطفي رستم خاني
عنوان
تقويت استدلال مدلهاي چندوجهي با استفاده از مدلهاي متن-تصوير و زباني بزرگ
مقطع تحصيلي
كارشناسي ارشد
رشته تحصيلي
هوش مصنوعي و رباتيكز
سال تحصيل
1401
تاريخ دفاع
1403/08/05
استاد راهنما
سيد صالح اعتمادي
استاد مشاور
-
دانشكده
مهندسي كامپيوتر
چكيده
ﺩﺭ ﺳﺎﻝﻫﺎﯼ ﺍﺧﯿﺮ ﻭ ﺑﻪ ﺧﺼﻮﺹ ﺑﺎ ﺍﺭﺍﺋﻪ GPT-4 ﻣﺪﻝﻫﺎﯼ ﺯﺑﺎﻧ ⁃ﺑﯿﻨﺎﯾﯽ ﺯﯾﺎﺩﯼ ﺗﻮﺳﻌﻪ ﭘﯿﺪﺍ ﮐﺮﺩﻧﺪ ﻭ ﺍﯾﻦ ﻣﺪﻝﻫﺎ ﭘﯿﺸﺮﻓﺖﻫﺎﯼ ﺯﯾﺎﺩﯼ ﺩﺍﺷﺘەﺍﻧﺪ. ﺍﯾﻦ ﻣﺪﻝﻫﺎ ﻣﯿﺒﺎﯾﺴﺖ ﻫﻢ ﻣﺘﻦ ﻭ ﻫﻢ ﺗﺼﻮﯾﺮ ﺭﺍ ﻣﺘﻮﺟﻪ ﺷﻮﻧﺪ. ﺁﻧﭽﻪ ﺑﻪ ﻋﻨﻮﺍﻥ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﮔﺎﻥ ﺁﻣﻮﺯﺷ ﺩﺭ ﺍﯾﻦ ﻣﺪﻝﻫﺎ ﺍﺳﺘﻔﺎﺩﻩ ﻣ ﺷﻮﺩ، ﺑﻪ ﺩﺭﮎ ﺳﺎﺩﻩ ﺍﺯ ﺗﺼﻮﯾﺮ ﻣ ﭘﺮﺩﺍﺯﻧﺪ. ﺩﺭ ﺻﻮﺭﺗ ﮐﻪ ﺩﺭ ﺑﺮﺧ ﺗﺼﺎﻭﯾﺮ، ﺍﻧﺴﺎﻥﻫﺎ ﺑﻪ ﺩﺭﮐ ﻣﺘﻔﺎﻭﺕ ﺍﺯ ﺁﻥ ﺗﺼﺎﻭﯾﺮ ﻧﯿﺎﺯ ﺩﺍﺭﻧﺪ. ﺑﻌﻀ ﺍﺯ ﺗﺼﺎﻭﯾﺮ ﺩﺍﺭﺍﯼ ﭼﻨﺪ ﻣﻔﻬﻮﻡ
ﻣ ﺑﺎﺷﻨﺪ. ﺍﺯ ﺟﻤﻠﻪ ﺍﯾﻦ ﻣﻮﺍﺭﺩ، ﺗﺼﺎﻭﯾﺮﯼ ﻫﺴﺘﻨﺪ ﮐﻪ ﺑﺎﻋﺚ ﺍﯾﺠﺎﺩ ﺗﻮﻫﻤ ﺍﺯ ﯾ ﺗﺼﻮﯾﺮ ﺩﯾ ﺮ ﺩﺭ ﺗﺼﻮﯾﺮ ﻣ ﺷﻮﻧﺪ.
ﺩﺭ ﺍﯾﻨﮕﻮﻧﻪ ﺗﺼﺎﻭﯾﺮ، ﯾ ﺗﻮﻫﻢ ﺍﺯ ﯾ ﺗﺼﻮﯾﺮ ﺩﯾ ﺮ ﻭﺟﻮﺩ ﺩﺍﺭﺩ. ﺑﺎ ﻧﮕﺎﻩ ﺍﻭﻟﯿﻪ ﺑﻪ ﺗﺼﻮﯾﺮ، ﻣﺎ ﻣ ﺗﻮﺍﻧﯿﻢ ﺑﻪ ﺭﺍﺣﺘ
ﺁﻧﭽﻪ ﺭﺍ ﮐﻪ ﺗﺼﻮﯾﺮ ﺑﻪ ﺻﺮﺍﺣﺖ ﻧﺸﺎﻥ ﻣ ﺩﻫﺪ، ﺩﺭﮎ ﮐﻨﯿﻢ. ﺍﻣﺎ ﺑﺮﺍﯼ ﺩﺭﮎ ﺗﻮﻫﻢ ﻣﻮﺟﻮﺩ ﺩﺭ ﺁﻥ ﻧﯿﺎﺯ ﺍﺳﺖ ﺑﺎ ﻧﮕﺎﻫ ﺩﯾ ﺮ ﺑﻪ ﺗﺼﻮﯾﺮ ﺗﻮﺟﻪ ﮐﻨﯿﻢ. ﺑﺮﺍﯼ ﻣﺜﺎﻝ ﺗﺼﻮﯾﺮﯼ ﺭﺍ ﻓﺮﺽ ﮐﻨﯿﺪ ﮐﻪ ﺩﺭ ﻧﮕﺎﻩ ﺍﻭﻟﯿﻪ، ﺩﺭﺧﺘﺎﻧ ﺍﺯ ﯾ ﺟﻨﮕﻞ ﺭﺍ ﻧﺸﺎﻥ ﻣ ﺩﻫﺪ. ﺍﻣﺎ ﺩﺭ ﻫﻤﯿﻦ ﺗﺼﻮﯾﺮ، ﺗﻮﻫﻤ ﺍﺯ ﯾ ﺧﺮﮔﻮﺵ ﻧﯿﺰ ﻭﺟﻮﺩ ﺩﺍﺷﺘﻪ ﺑﺎﺷﺪ ﮐﻪ ﻧﺎﺷ ﺍﺯ ﮐﻨﺎﺭ ﻫﻢ ﻗﺮﺍﺭﮔﯿﺮﯼ
ﺍﺟﺰﺍ ﻭ ﻧﯿﺰ ﺭﻧﮓ ﺑﻨﺪﯼ ﺁﻧﻬﺎ ﺑﺎﺷﺪ. ﺩﺭﮎ ﺍﯾﻨﮕﻮﻧﻪ ﺗﺼﺎﻭﯾﺮ ﮐﻪ ﺩﺍﺭﺍﯼ ﯾ ﻣﻔﻬﻮﻡ ﺁﺷ ﺎﺭ ﻭ ﯾ ﻣﻔﻬﻮﻡ ﻧﻬﺎﻥ ﻭ ﺗﻮﻫﻤ ﻫﺴﺘﻨﺪ، ﺑﺮﺍﯼ ﺍﻧﺴﺎﻥ ﺭﺍﺣﺖ ﻣﺤﺴﻮﺏ ﻣ ﺷﻮﺩ. ﺗﻨﻬﺎ ﮐﺎﻓ ﺍﺳﺖ ﻣﻘﺪﺍﺭ ﭼﺸﻤﺎﻥ ﺧﻮﺩ ﺭﺍ ﺑﺎﺯ ﯾﺎ ﺑﺴﺘﻪ ﮐﻨﺪ ﺗﺎ ﺑﺘﻮﺍﻧﺪ ﺗﻮﻫﻢ ﻣﻮﺟﻮﺩ ﺭﺍ ﺩﺭﯾﺎﺑﺪ. ﺑﻪ ﺍﯾﻦ ﻣﻮﺿﻮﻉ ﺩﺭ ﻣﺪﻝﻫﺎﯼ ﺯﺑﺎﻧ ⁃ﺑﯿﻨﺎﯾﯽ ﺑﻪ ﺧﻮﺑﯽ ﭘﺮﺩﺍﺧﺘﻪ ﻧﺸﺪەﺍﺳﺖ. ﺩﺭ ﺍﯾﻦ ﻧﻮﻉ ﻣﺴﺌﻠﻪ، ﻭﺭﻭﺩﯼ ﻣﺪﻝ ﯾ ﻋﮑﺲ ﺑﻮﺩﻩ ﻭ ﺑﻪ ﻋﻨﻮﺍﻥ ﺧﺮﻭﺟ ﺍﻧﺘﻈﺎﺭ ﺩﺍﺭﯾﻢ ﮐﻪ ﻣﺪﻝ، ﺗﻮﻫﻢ ﻣﻮﺟﻮﺩ ﺩﺭ ﺗﺼﻮﯾﺮ ﺭﺍ ﺑﻘﻬﻤﺪ. ﻣﺪﻝﻫﺎﯼ ﻣﻮﺟﻮﺩ ﻟﺒﻪ ﺩﺍﻧﺶ ﮐﻨﻮﻧ ﺩﺭ ﻓﻬﻤﯿﺪﻥ ﺍﯾﻦ ﻧﻮﻉ ﺗﺼﺎﻭﯾﺮ ﺩﺍﺭﺍﯼ ﻣﺸ ﻞ ﻫﺴﺘﻨﺪ. ﻣﺎ ﺑﺮﺍﯼ ﺍﯾﻦ ﻣﺴﺌﻠﻪ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﮔﺎﻧ ﺭﺍ ﺟﻤﻊ ﺁﻭﺭﯼ ﮐﺮﺩﻩ ﻭ ﺑﺎ ﺍﺳﺘﻔﺎﺩﻩ ﺍﺯ ﺩﻭ ﺭﻭﺵ ﺑﻪ ﺑﻬﺒﻮﺩ ﻣﺪﻝ ﻫﺎ ﺩﺭ ﻓﻬﻤﯿﺪﻥ ﺍﯾﻦ ﻧﻮﻉ ﺗﺼﺎﻭﯾﺮ ﮐﻤ ﮐﺮﺩﯾﻢ. ﯾ ﺑﺎ ﺍﺳﺘﻔﺎﺩﻩ ﺍﺯ ﻫﻤﺴﺎﺯﯼ ﺩﻗﯿﻖ ﻣﺪﻝ ﻫﺎ ﺑﺮ ﺭﻭﯼ ﺍﯾﻨﮕﻮﻧﻪ ﺗﺼﺎﻭﯾﺮ ﻭ ﺩﯾ ﺮﯼ ﺑﺎ ﺍﺳﺘﻔﺎﺩﻩ ﺍﺯ ﯾ ﻓﯿﻠﺘﺮ ﺳﺎﺩﻩ ﺭﻭﯼ ﺗﺼﺎﻭﯾﺮ. ﭘﯿﺸﻨﻬﺎﺩ ﺍﯾﻦ ﻓﯿﻠﺘﺮ ﺑﺎ ﺍﻗﺘﺒﺎﺱ ﺍﺯ ﭘﺪﯾﺪەﺍﯼ ﮐﻪ ﺩﺭ ﭼﺸﻢ ﺍﻧﺴﺎﻥ ﺑﺎ ﻧﯿﻤﻪ ﺑﺴﺘﻪ ﮐﺮﺩﻥ ﺁﻥ ﺭﺥ ﻣ ﺩﻫﺪ ﺑﻮﺩﻩ ﺍﺳﺖ. ﻣﺎ ﺑﺮﺍﯼ ﻫﻤﺴﺎﺯﯼ ﺩﻗﯿﻖ ﻣﺪﻝ ﻫﺎ ﺑﺮ ﺭﻭﯼ ﺗﺼﺎﻭﯾﺮ، ﯾ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﮔﺎﻥ ﺷﺎﻣﻞ 4 ﺯﯾﺮ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﻩ ﺟﻤﻊ ﺁﻭﺭﯼ ﮐﺮﺩﯾﻢ ﮐﻪ ﻫﺮ ﯾ ﻣﺮﺑﻮﻁ ﺑﻪ ﯾ ﺯﻣﯿﻨﻪ ﻣﺨﺘﻠﻒ ﻫﺴﺘﻨﺪ. ﺍﺯ ﺍﯾﻦ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﮔﺎﻥ، IllusionMNIST ﻭ IllusionFashionMNIST ﻭ IllusionAnimals ﻣﺮﺑﻮﻁ ﺑﻪ ﻃﺒﻘەﺑﻨﺪﯼ ﻫﺴﺘﻨﺪ ﻭ ﻫﺮ ﯾ ﺷﺎﻣﻞ 11 ﮐﻼﺱ ﻫﺴﺘﻨﺪ ﻭ ﺑﺮﺍﯼ ﺑﺮﺭﺳ ﻣﺴﺌﻠﻪ ﻓﺮﺍﺗﺮ ﺍﺯ ﺣﺎﻟﺖ ﻃﺒﻘﻪ ﺑﻨﺪﯼ ﻧﯿﺰ ﺍﺯ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﻩ IllusionChar ﺍﺳﺘﻔﺎﺩﻩ ﮐﺮﺩﯾﻢ ﮐﻪ ﺩﺭ ﺣﻮﺯﻩ OCR ﺍﺳﺖ ﻭ ﺷﺎﻣﻞ 3 ﺗﺎ 5 ﮐﺎﺭﺍﮐﺘﺮ ﺍﺳﺖ ﮐﻪ ﺑﻪ ﺻﻮﺭﺕ ﺗﺼﺎﺩﻓ ﺩﺭ ﮐﻨﺎﺭ ﯾ ﺪﯾ ﺮ ﻗﺮﺍﺭ ﮔﺮﻓﺘەﺍﻧﺪ. ﺩﺭ ﺁﻣﺎﺭﮔﺎﻥ ﻣﺮﺑﻮﻁ ﺑﻪ ﺍﯾﻦ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﮔﺎﻥ، ﻫﺮ ﮐﺪﺍﻡ ﺍﺯ ﺁﻧﻬﺎ ﺩﺍﺭﺍﯼ ﺩﻭ ﻗﺴﻤﺖ ﺁﻣﻮﺯﺵ ﻭ ﺁﺯﻣﻮﻥ ﻫﺴﺘﻨﺪ ﮐﻪ ﺁﻣﺎﺭﮔﺎﻥ ﻫﺮ ﮐﺪﺍﻡ ﺑﻪ ﺷﺮﺡ ﺭﻭﺑﺮﻭ ﺍﺳﺖ: ﺩﺭ ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﮔﺎﻥ IllusionMNIST ، 0693 ﺩﺍﺩﻩ ﺁﻣﻮﺯﺷ ﻭ 9121 ﺩﺍﺩﻩ ﺁﺯﻣﻮﻥ ﻭ ﻣﺠﻤﻮﻋﻪ
ﺩﺍﺩﮔﺎﻥ IllusionFashionMNIST ﺷﺎﻣﻞ 0033 ﺩﺍﺩﻩ ﺁﻣﻮﺯﺷ ﻭ 7621 ﺩﺍﺩﻩ ﺁﺯﻣﻮﻥ ﻭ IllusionAnimals ﺷﺎﻣﻞ 0033 ﺩﺍﺩﻩ ﺁﻣﻮﺯﺷ ﻭ 0011 ﺩﺍﺩﻩ ﺁﺯﻣﻮﻥ ﻭ IllusionChar ﺷﺎﻣﻞ 0099 ﺩﺍﺩﻩ ﺁﻣﻮﺯﺷ ﻭ 0033 ﺩﺍﺩﻩ ﺁﺯﻣﻮﻥ
ﺍﺳﺖ. ﻣﻌﻤﺎﻫﺎﯼ ﺗﺼﻮﯾﺮﯼ ﻧﯿﺰ ﮔﻮﻧەﺍﯼ ﺩﯾ ﺮ ﺍﺯ ﺗﺼﺎﻭﯾﺮ ﻫﺴﺘﻨﺪ ﮐﻪ ﻣﯿﺒﺎﯾﺴﺖ ﺑﺎ ﻧﮕﺎﻫ ﻓﺮﺍﺗﺮ ﺍﺯ ﻣﻌﻤﻮﻝ ﺑﻪ ﺁﻧﻬﺎ ﺗﻮﺟﻪ ﻧﻤﻮﺩﻩ
ﺗﺎ ﺑﺘﻮﺍﻥ ﻣﻔﻬﻮﻡ ﺁﻧﻬﺎ ﺭﺍ ﻣﺘﻮﺟﻪ ﺷﺪ. ﺑﺮﺍﯼ ﻣﺜﺎﻝ ﺗﺼﻮﯾﺮﯼ ﺭﺍ ﺩﺭ ﻧﻈﺮ ﺑ ﯿﺮﯾﺪ ﮐﻪ ﺩﺭ ﺁﻥ ﯾ ﮐﻤﺎﻥ ﺭﻧﮕ ﻭﺟﻮﺩ ﺩﺍﺭﺩ.
ﺩﺭ ﻧﮕﺎﻩ ﺍﻭﻝ ﻣﻤ ﻦ ﺍﺳﺖ ﺑﻪ ﻧﻈﺮ ﺑﺮﺳﺪ ﮐﻪ ﺗﺼﻮﯾﺮ ﻫﻤﺎﻥ ﮐﻤﺎﻥ ﺭﻧﮕ ﺭﺍ ﻧﺸﺎﻥ ﻣ ﺩﻫﺪ. ﺍﻣﺎ ﺑﺎ ﺍﻧﺪﮐ ﺩﻗﺖ ﻭ ﺗﺎﻣﻞ، ﻣ ﺗﻮﺍﻥ ﺣﺪﺱ ﺯﺩ ﮐﻪ ﺷﺎﯾﺪ، ﻣﻘﺼﻮﺩ ﺍﺯ ﺍﯾﻦ ﺗﺼﻮﯾﺮ، ﮐﻠﻤﻪ ﺭﻧﮕﯿﻦ ﮐﻤﺎﻥ ﺑﺎﺷﺪ. ﺩﺭ ﺍﯾﻨﮕﻮﻧﻪ ﺗﺼﺎﻭﯾﺮ ﻧﯿﺰ ﻣﯿﺒﺎﯾﺴﺖ ﺑﺎ ﻧﮕﺎﻫ ﻓﺮﺍﺗﺮ ﺍﺯ ﻣﻌﻤﻮﻝ ﺑﻪ ﺗﺼﺎﻭﯾﺮ ﻧﮕﺮﯾﺴﺖ ﻭ ﺍﻧﺪﮐ ﺍﺳﺘﺪﻻﻝ ﺩﺭ ﻣﻮﺭﺩ ﺑﺨﺶ ﺯﺑﺎﻧ ﮐﺮﺩ. ﺩﺭ ﺍﯾﻦ ﭘﮋﻭﻫﺶ ﻫﺪﻑ ﺑﺮ ﺍﯾﻦ ﺑﻮﺩﻩ ﮐﻪ ﺑﺮﺍﯼ ﻫﺮﮐﺪﺍﻡ ﺍﺯ ﻣﻮﺍﺭﺩ ﺫﮐﺮ ﺷﺪﻩ ﺩﺭ ﺑﺎﻻ، ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩەﺍﯼ ﺭﺍ ﺳﺎﺧﺘﻪ ﯾﺎ ﺟﻤﻊﺁﻭﺭﯼ ﮐﺮﺩﻩ ﻭ ﺗﻌﺪﺍﺩﯼ ﺍﺯ ﻣﺪﻝﻫﺎﯼ ﻣﺮﺯ ﺩﺍﻧﺶ ﺭﺍ ﺑﺮ ﺭﻭﯼ ﺁﻧﻬﺎ ﺍﺭﺯﯾﺎﺑﯽ ﮐﻨﯿﻢ. ﺁﺯﻣﺎﯾﺶﻫﺎﯼ ﺍﻧﺠﺎﻡ ﺷﺪﻩ ﺩﺭ ﺍﯾﻦ ﭘﮋﻭﻫﺶ، ﺗﻔﺎﻭﺕ ﺯﯾﺎﺩ ﺑﯿﻦ ﻋﻤﻠ ﺮﺩ ﺍﻧﺴﺎﻥ ﻭ ﺍﯾﻦ ﻣﺪﻝﻫﺎ ﺭﺍ ﻧﺸﺎﻥ ﻣ ﺩﻫﺪ. ﺩﺭ ﻣﻮﺭﺩ ﺗﺼﺎﻭﯾﺮ ﺩﺍﺭﺍﯼ ﺗﻮﻫﻢ، ﺭﻭﺵ ﺳﺎﺩﻩ ﻭ ﺩﺭ ﻋﯿﻦ ﺣﺎﻝ ﮐﺎﺭﺁﻣﺪﯼ ﮐﻪ ﺷﺎﻣﻞ ﺑﻪ ﮐﺎﺭ ﮔﯿﺮﯼ ﯾ ﭘﺎﻻﯾﻪ ﭘﺎﯾﯿﻦ ﮔﺬﺭ ﺭﻭﯼ ﺗﺼﻮﯾﺮ ﺍﺳﺖ ﺭﺍ ﺍﺭﺍﺋﻪ ﺩﺍﺩﯾﻢ. ﺁﺯﻣﺎﯾﺶﻫﺎ ﻣﻮﺛﺮ ﺑﻮﺩﻥ ﺍﯾﻦ ﺭﻭﺵ ﺭﺍ ﺩﺭ ﺗﺸﺨﯿﺺ ﺗﻮﻫﻢ ﺑﻪ ﮐﺎﺭ ﺭﻓﺘﻪ ﺩﺭ ﺗﺼﻮﯾﺮ ﻧﺸﺎﻥ ﻣ ﺩﻫﻨﺪ. ﺷﺎﯾﺎﻥ ﺫﮐﺮ ﺍﺳﺖ ﮐﻪ ﺍﯾﻦ ﺭﻭﺵ، ﻫﺰﯾﻨﻪ ﻣﺤﺎﺳﺒﺎﺗ ﺯﯾﺎﺩ ﻧﺪﺍﺭﺩ ﻭ ﻧﯿﺎﺯﯼ ﺑﻪ ﺁﻣﻮﺯﺷ ﻣﺠﺪﺩ ﻣﺪﻝ ﺩﺭ ﺁﻥ ﻭﺟﻮﺩ ﻧﺪﺍﺭﺩ. ﺑﺎ ﺑﻪ ﮐﺎﺭ ﮔﯿﺮﯼ ﺍﯾﻦ ﺭﻭﺵ ﺩﺭ ﺑﻬﺘﺮﯾﻦ ﺣﺎﻟﺖ، ﻣﻮﻓﻖ ﺑﻪ ﮐﺴﺐ ﺍﻣﺘﯿﺎﺯ F1 ﺑﺮﺍﺑﺮ ﺑﺎ 32̸49 ﮐﻪ ﺣﺘ ﺑﺎﻻﺗﺮ ﺍﺯ ﺍﻧﺴﺎﻥ ﮐﻪ ﺑﺮﺍﺑﺮ ﺑﺎ 5̸19 ﺍﺳﺖ، ﺭﺳﯿﺪﯾﻢ. ﺑﻪ ﻋﻨﻮﺍﻥ ﺭﻭﺷ ﺩﯾ ﺮ ﺑﺮﺍﯼ ﻣﻮﺍﺭﺩ ﺩﺍﺭﺍﯼ ﺗﻮﻫﻢ، ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩﮔﺎﻧ ﺭﺍ ﺑﺮﺍﯼ ﻫﻢﺳﺎﺯﯼ ﺩﻗﻴﻖ ﻣﺪﻝﻫﺎ ﺗﻮﻟﯿﺪ ﮐﺮﺩﯾﻢ. ﺳﭙﺲ ﺗﻌﺪﺍﺩﯼ ﺍﺯ ﻣﺪﻝﻫﺎ ﺭﺍ ﺑﺮ ﺭﻭﯼ ﺁﻧﻬﺎ ﻫﻢﺳﺎﺯﯼ ﺩﻗﻴﻖ ﮐﺮﺩﻩ ﻭ ﺩﺭﻧﻬﺎﯾﺖ ﻋﻤﻠ ﺮﺩ ﺁﻧﻬﺎ ﺭﺍ ﺑﺮﺍﯼ ﺗﺸﺨﯿﺺ ﺗﻮﻫﻢ ﻣﻮﺟﻮﺩ ﺩﺭ ﺗﺼﺎﻭﯾﺮ ﺍﺭﺯﯾﺎﺑﯽ ﮐﺮﺩﯾﻢ. ﺩﺭ ﺍﯾﻦ ﺣﺎﻟﺖ ﻧﯿﺰ ﺁﺯﻣﺎﯾﺶﻫﺎ ﻧﺸﺎﻥ ﺍﺯ ﺍﻓﺰﺍﯾﺶ ﻋﻤﻠ ﺮﺩ ﻣ ﺩﻫﺪ. ﺩﺭ ﺍﯾﻦ ﺣﺎﻟﺖ ﻧﯿﺰ ﺩﺭ ﺑﻬﺘﺮﯾﻦ ﺣﺎﻟﺖ ﻣﻮﻓﻖ ﺑﻪ ﮐﺴﺐ ﺍﻣﺘﯿﺎﺯ F1 ﺑﺮﺍﺑﺮ ﺑﺎ 53̸49 ﺷﺪﯾﻢ ﮐﻪ ﺑﺎﺯ ﻫﻢ ﺑﺎﻻﺗﺮ ﺍﺯ ﺍﻧﺴﺎﻥ ﻗﺮﺍﺭ ﺩﺍﺭﺩ. ﺍﯾﻦ ﻣﻮﺍﺭﺩ ﺗﻔﺎﻭﺕ ﺑﯿﻦ ﺩﺭﮎ ﻣﺪﻝﻫﺎ ﺍﺯ ﺗﺼﻮﯾﺮ ﺑﺎ ﺩﺭﮎ ﺍﻧﺴﺎﻥ ﺍﺯ ﺁﻥ ﺭﺍ ﻧﺸﺎﻥ ﻣ ﺩﻫﺪ. ﺑﺮﺍﯼ ﺍﺭﺯﯾﺎﺑﯽ ﻣﺪﻝﻫﺎﯼ ﺯﺑﺎﻧ ⁃ﺑﯿﻨﺎﯾﯽ ﺑﺮ ﺭﻭﯼ ﻣﻌﻤﺎﻫﺎﯼ ﺗﺼﻮﯾﺮﯼ ﻧﯿﺰ، ﻣﺠﻤﻮﻋﻪ ﺩﺍﺩەﺍﯼ ﺷﺎﻣﻞ 352 ﻧﻤﻮﻧﻪ ﮐﻪ ﺑﻪ ﺻﻮﺭﺕ ﺩﺳﺘ ﺟﻤﻊ ﺁﻭﺭﯼ ﺷﺪﻩ ﻭ ﺣﺎﺷﯿﻪ ﻧﻮﯾﺴ ﺷﺪەﺍﻧﺪ ﺭﺍ ﺍﺭﺍﺋﻪ ﮐﺮﺩﻩ ﻭ ﻋﻤﻠ ﺮﺩ ﺗﻌﺪﺍﺩﯼ ﺍﺯ ﻣﺪﻝﻫﺎﯼ ﻣﺮﺯ ﺩﺍﻧﺶ ﺭﺍ ﺑﺮ ﺭﻭﯼ ﺁﻧﻬﺎ ﺍﺭﺯﯾﺎﺑﯽ ﮐﺮﺩﯾﻢ. ﺍﯾﻦ ﺁﺯﻣﺎﯾﺶ ﻫﺎ، ﺗﻔﺎﻭﺕ
ﺯﯾﺎﺩ ﻋﻤﻠ ﺮﺩﯼ ﺑﯿﻦ ﻣﺪﻝﻫﺎﯼ ﻣﺘﻦ ﺑﺎﺯ ﻭ ﻣﺘﻦ ﺑﺴﺘﻪ ﺭﺍ ﻧﺸﺎﻥ ﻣ ﺩﻫﻨﺪ. ﺩﺭ ﺍﯾﻦ ﺣﺎﻟﺖ، ﺑﻬﺘﺮﯾﻦ ﺍﻣﺘﯿﺎﺯ ﮐﺴﺐ ﺷﺪﻩ،
ﺗﻮﺳﻂ ﻣﺪﻝ GPT-4o ﺑﻮﺩﻩ ﮐﻪ ﺍﻣﺘﯿﺎﺯ F1 ﻣﻌﺎﺩﻝ ﺑﺎ 91̸48 ﺭﺍ ﮐﺴﺐ ﮐﺮﺩﻩ ﺍﺳﺖ. ﺍﻫﻤﯿﺖ ﻣﻮﺿﻮﻉ ﭘﺮﺩﺍﺧﺘﻦ ﺑﻪ ﻣﻮﺍﺭﺩ ﺫﮐﺮ ﺷﺪﻩ ﺩﺭ ﺩﻭ ﺟﻬﺖ ﻣﻬﻢ ﺍﺳﺖ. ﺍﻭﻝ ﺍﯾﻨﮑﻪ ﺑﺎﻋﺚ ﺍﻓﺰﺍﯾﺶ ﺩﺭﮎ ﻣﺪﻝﻫﺎ ﺍﺯ ﺗﺼﻮﯾﺮ ﻭ ﺍﻓﺰﺍﯾﺶ ﻗﺪﺭﺕ ﺍﺳﺘﺪﻻﻝ ﺁﻧﻬﺎ ﻣ ﺷﻮﺩ. ﺩﻭﻡ ﺍﯾﻨﮑﻪ ﺍﺯ ﻣﻮﺍﺭﺩ ﺩﺍﺭﺍﯼ ﺗﻮﻫﻢ، ﻣ ﺗﻮﺍﻥ ﺩﺭ ﻧﻬﺎﻥﻧﮕﺎﺭﯼ ﺍﺳﺘﻔﺎﺩﻩ
ﮐﺮﺩ.
تاريخ ورود اطلاعات
1403/09/11
عنوان به انگليسي
Enhancing reasoning of Multimodal models using text-image and large language models
تاريخ بهره برداري
1/1/1900 12:00:00 AM
دانشجوي وارد كننده اطلاعات
محمدمصطفي رستم خاني
چكيده به لاتين
In recent years, particularly with the introduction of GPT-4, numerous vision-language models have been developed, demonstrating significant progress. These models are expected to understand both text and images. However, the datasets used for training these models often focus on simple in- terpretations of images. Some images, however, require a different level of human interpretation as they may convey multiple meanings. One such category includes images that create illusions of other images within themselves. In these cases, an illusion of another image is present. With an initial glance, we can easily comprehend what the image explicitly shows, but to understand the illusion within, we need to view it differently. For example, consider an image that initially appears to show trees in a forest, but upon closer inspection, reveals the illusion of a rabbit formed by the arrangement and colors of the elements.
Human perception of images with both an explicit and a hidden, illusory meaning is relatively in- tuitive; one simply needs to adjust their focus to detect the illusion. However, this aspect is under- explored in vision-language models. In this problem, the model receives an image as input and is expected to identify the illusion within it. Current state-of-the-art models face challenges in under- standing such images. For this, we created a dataset and improved models’ understanding of these types of images using two methods: one by fine-tuning models on these images and the other by ap- plying a simple filter inspired by the effect observed in human vision when squinting. To fine-tune models on these images, we collected a dataset with four subsets, each related to a different domain. Among these datasets, IllusionMNIST, IllusionFashionMNIST, and IllusionAnimals are used for classification, each containing 11 classes. To address cases beyond classification, we also utilized the IllusionChar dataset, relevant to OCR, consisting of 3 to 5 characters randomly arranged.
The statistics of each subset in this dataset are as follows: IllusionMNIST contains 3960 training samples and 1219 test samples; IllusionFashionMNIST includes 3300 training samples and 1267 test samples; IllusionAnimals has 3300 training samples and 1100 test samples; and IllusionChar comprises 9900 training samples and 3300 test samples.
Visual puzzles are another category that requires a broader perspective to grasp their meaning. For instance, consider an image with an arc of colors. At first glance, it may appear to show only the arc, but with some thought, one might deduce that the intended concept could be ”rainbow.” These images also necessitate looking beyond the surface and reasoning about the linguistic aspect.
The goal of this research is to create or collect datasets for each of the cases mentioned above and evaluate several state-of-the-art models on them. Experiments in this study reveal a significant performance gap between human understanding and model performance. For illusion images, we
propose a simple yet effective method using a low-pass filter on the image. Experiments demon- strate the effectiveness of this approach in detecting illusions in images, achieving an F1 score of 94.23, surpassing human performance of 91.5. This method requires minimal computational cost and does not necessitate retraining the model. In the best case, we achieved an F1 score of 94.35, again surpassing human performance.
For evaluating vision-language models on visual puzzles, we present a dataset of 253 samples col- lected and annotated manually, and we evaluated the performance of various state-of-the-art models on it. These experiments indicate a large performance disparity between open-source and closed- source models, with the highest F1 score of 84.19 achieved by the GPT-4o model.
The importance of addressing these issues is twofold: firstly, it enhances the models’ understanding and reasoning capabilities, and secondly, illusions can be leveraged in steganography.
كليدواژه هاي فارسي
ﻣﺪﻝﻫﺎﯼ ﺯﺑﺎن⁃ﺑﯿﻨﺎﯾﯽ , ﺩﺭﮎ ﺗﺼﻮﯾﺮ , ﺗﺼﺎﻭﯾﺮ ﺩﺍﺭﺍﯼ ﺗﻮﻫﻢ , ﻣﻌﻤﺎﻫﺎﯼ ﺗﺼﻮﯾﺮﯼ , ﭘﺎﻻيهﻫﺎﯼ ﭘﺎﯾﯿﻦ ﮔﺬﺭ
كليدواژه هاي لاتين
vision-language models , image understanding , illusion-based images , visual puzzles , low-pass filter
Author
Mohammadmostafa Rostamkhani
SuperVisor
Sayyed Sauleh Eetemadi