Improving Evaluation Metrics for Vision-and-Language Models

Priberam Labs

Instituto Superior Técnico, Anfiteatro PA2

Abstract:Evaluating image captions is essential for ensuring both linguistic fluency and accurate semantic alignment with visual content. While reference-free metrics such as CLIPScore have advanced automated caption evaluation, most existing work on learned evaluation metrics remains limited to pointwise English-centric assessments, with significant gaps in terms of reliability, interpretability, and multilingual inclusivity of vision-and-language evaluation metrics. In this seminar session I will explore extensions of current English-centric benchmarks to a multilingual scenario promoting the development of more inclusive frameworks. Additionally I will present two extensions from CLIPScore metric aiming to improve its interpretability and reliability in real world applications. Leveraging a model-agnostic conformal risk control framework, I will explore the calibration of CLIPScore distributions values for task-specific control variables tackling both granular assessment for individual word errors within captions, and the calibration of these raw distribution scores producing a more reliable interval for captioning evaluation by improving the correlation between uncertainty estimations and prediction errors. Bio:Gonçalo Gomes received a MSc degree in Data Science and Engineering, from Instituto Superior Técnico, Universidade de Lisboa. He is currently a second-year PhD student at the same institution and a junior researcher at the Human Language Technologies Lab of INESC-ID and also at SARDINE Labs of Instituto de Telecomunicações (IT). His research interests focus on developing more informative and trustworthy evaluation frameworks for vision-and-language applications, particularly envisioning a more inclusive AI frameworks for non-English environments.<a href="https://www.priberam.com/" title="https://www.priberam.com/" target="_blank" data-msys-clicktrack="0" rel="nofollow noopener noreferrer">www.priberam.com</a>

Our {communityGuidelinesLink} describe the sort of content we prohibit on Eventbrite. If you suspect an event may be in violation, you can report it to us so we can investigate.

To report other categories of prohibited or illegal content, submit {link}

Improving Evaluation Metrics for Vision-and-Language Models

More events from Priberam Labs

Discover more events from Priberam Labs, from Science & Tech to other experiences you might love.

Still looking for the right event?

Explore all events in Lisboa and filter by date, category, and more to find the perfect fit.

Improving Evaluation Metrics for Vision-and-Language Models

More events from Priberam Labs

Discover more events from Priberam Labs, from Science & Tech to other experiences you might love.

More Science & Tech events

Browse more Science & Tech events with different dates, prices, and formats to find your next great experience.

Still looking for the right event?

Explore all events in Lisboa and filter by date, category, and more to find the perfect fit.