Transparent MT Evaluation through Fine-grained Error Detection

Priberam Labs

Instituto Superior Técnico

Abstract:The transition from COMET to XCOMET marks a significant evolution in machine translation evaluation methodologies. COMET, a neural framework renowned for its prowess in training multilingual machine translation evaluation models, has long been celebrated for its state-of-the-art correlation with human judgments. Leveraging breakthroughs in cross-lingual pretrained language modeling, COMET excels in generating adaptable evaluation models capable of accurately predicting translation quality by assimilating information from both source input and target-language references. However, the landscape of machine translation evaluation is rapidly evolving, propelled by the emergence of generative large language models (LLMs) and the demand for more granular error analysis. In response to these challenges, XCOMET emerges as a groundbreaking solution, aiming to bridge the gap between traditional learned metrics and the evolving demands of the field. Unlike its predecessor, XCOMET integrates both sentence-level evaluation and error span detection capabilities, offering deeper insights into translation errors and their severity. In this presentation, we delve into the conceptualization and implementation of XCOMET, highlighting its novel features and advancements over COMET. By combining state-of-the-art sentence-level evaluation with error span detection, XCOMET enriches the quality assessment process, providing valuable insights into the nature and severity of translation errors. Through rigorous analysis and stress tests, we showcase the robustness of XCOMET in identifying critical errors and hallucinations, solidifying its position as a pioneering solution in the realm of machine translation evaluation.Bio:Ricardo Rei is a senior research scientist at Unbabel, specializing in machine translation and natural language processing. He is set to complete his Ph.D. in April, which has been a collaborative effort between Unbabel, INESC-ID/Tecnico, and CMU University. His doctoral research has been centered on machine translation evaluation, and he is the main developer behind the COMET evaluation framework, which has become the industry standard metric for assessing machine translation quality. With a keen interest in advancing the capabilities of multilingual large language models (LLMs), he has been at the forefront of research and development in this domain. When not immersed in research, Ricardo enjoys maintaining an active lifestyle, often found at the gym or riding the waves while surfing—a passion he has pursued since the age of nine.<a href="https://www.priberam.com/" title="https://www.priberam.com/" target="_blank" data-msys-clicktrack="0" rel="nofollow noopener noreferrer">www.priberam.com</a>

Our {communityGuidelinesLink} describe the sort of content we prohibit on Eventbrite. If you suspect an event may be in violation, you can report it to us so we can investigate.

To report other categories of prohibited or illegal content, submit {link}

Transparent MT Evaluation through Fine-grained Error Detection

More events from Priberam Labs

Discover more events from Priberam Labs, from Science & Tech to other experiences you might love.

Still looking for the right event?

Explore all events in Lisboa and filter by date, category, and more to find the perfect fit.

Transparent MT Evaluation through Fine-grained Error Detection

More events from Priberam Labs

Discover more events from Priberam Labs, from Science & Tech to other experiences you might love.

More Science & Tech events

Browse more Science & Tech events with different dates, prices, and formats to find your next great experience.

Still looking for the right event?

Explore all events in Lisboa and filter by date, category, and more to find the perfect fit.