Evaluating Neural Methods for Approximate String Matching and Duplicate Detection

Priberam

Abstract:
Duplicate detection concerns with identifying pairs of attributes/records that refer to the same real-world object, thus corresponding to a fundamental process when ensuring data quality in databases.
Existing methods to detect duplicate attributes can leverage heuristic string similarity measures based on characters or small character sequences, phonetic encoding techniques that match strings based on the way they sound, or hybrid techniques that combine different approaches.
However, these methods rely on common sub-strings in order to establish similarity, and they often do not effectively capture the character replacements involved in duplicate attributes due to transliterations or the use of different languages and/or alphabets.
This work follows on recent work regarding string matching using deep neural networks, tackling the aforementioned challenges by leveraging recurrent neural units for modeling sequences of characters in order to build semantic representations for the input strings. We consider several alternative neural architectures, e.g. combining recurrent units with attention or pooling operations, or instead based on the Transformer model, and assessed the impact of training data size and/or domain, specifically considering datasets describing collections of person names, organizations, or geographical locations. The obtained results show that the neural models can achieve superior results on all datasets, when compared to standard string similarity measures and even with relatively small amounts of training data, without the need of major tuning on the network parameters. Models trained on a specific domain were nonetheless shown to have problems in generalizing to other domains (e.g., models trained on a dataset composed of person names perform worse when evaluated on pairs of organization names), although some level of knowledge transfer across domains was still observed (e.g., neural cross-domain models were still able to outperform standard string similarity metrics).
Short Bio:
Luís Borges is a CMU Portugal student in Language Technologies, having started his PhD in 2019. His Master’s degree was focused on the application of Natural Language Processing and Deep Learning techniques on the topic of fake news detection. For his PhD, he is working on Neural Information Retrieval, also leveraging NLP and Deep Learning tools.
More info: <A HREF="http://labs.priberam.com/Academia-Partnerships/Seminars.aspx" TARGET="_blank" DATA-MSYS-CLICKTRACK="0" REL="nofollow noopener noreferrer">http://labs.priberam.com/Academia-Partnerships/Seminars.aspx</A>

Zoom

Our {communityGuidelinesLink} describe the sort of content we prohibit on Eventbrite. If you suspect an event may be in violation, you can report it to us so we can investigate.

To report other categories of prohibited or illegal content, submit {link}

Evaluating Neural Methods for Approximate String Matching and Duplicate Detection

More events from Priberam

Discover more events from Priberam, from Science & Tech to other experiences you might love.

Evaluating Neural Methods for Approximate String Matching and Duplicate Detection

More events from Priberam

Discover more events from Priberam, from Science & Tech to other experiences you might love.

You might also like...

Browse more events with different dates, prices, and formats to find your next great experience.