Sparse and Structured Visual Attention

Priberam

Abstract:
Visual attention mechanisms are widely used in multimodal tasks, such as image captioning and visual question answering (VQA), being softmax attention mechanism the standard choice. One drawback of softmax-based attention mechanisms is that they assign probability mass to all image regions, regardless of their adjacency structure and of their relevance to the text. To better link the image structure with the text, we replace the traditional softmax attention mechanism with two alternative sparsity-promoting transformations: sparsemax, which is able to select the relevant regions only (assigning zero weight to the rest), and a newly proposed Total-Variation Sparse Attention (TVmax), which further encourages the joint selection of adjacent spatial locations. Experiments in image captioning and VQA, using both LSTM and Transformer architectures, show gains in terms of human-rated caption quality, attention relevance, and VQA accuracy, with improved interpretability.
Short Bio:
Pedro is currently a PhD student at IT, advised by André Martins and Zita Marinho. His research interests include machine learning and natural language processing. Before that, he worked as an IT consultant for six months. He completed his master degree in aerospace at IST in 2017.
More info: <A HREF="http://labs.priberam.com/Academia-Partnerships/Seminars.aspx" TARGET="_blank" DATA-MSYS-CLICKTRACK="0" REL="nofollow noopener noreferrer">http://labs.priberam.com/Academia-Partnerships/Seminars.aspx</A>

Instituto Superior Técnico

Our {communityGuidelinesLink} describe the sort of content we prohibit on Eventbrite. If you suspect an event may be in violation, you can report it to us so we can investigate.

To report other categories of prohibited or illegal content, submit {link}

Sparse and Structured Visual Attention

More events from Priberam

Discover more events from Priberam, from Science & Tech to other experiences you might love.

Sparse and Structured Visual Attention

More events from Priberam

Discover more events from Priberam, from Science & Tech to other experiences you might love.

You might also like...

Browse more events with different dates, prices, and formats to find your next great experience.