Sparse and Structured Visual Attention

Priberam

Abstract:
Visual attention mechanisms are widely used in multimodal tasks, such as image captioning and visual question answering (VQA), being softmax attention mechanism the standard choice. One drawback of softmax-based attention mechanisms is that they assign probability mass to all image regions, regardless of their adjacency structure and of their relevance to the text. To better link the image structure with the text, we replace the traditional softmax attention mechanism with two alternative sparsity-promoting transformations: sparsemax, which is able to select the relevant regions only (assigning zero weight to the rest), and a newly proposed Total-Variation Sparse Attention (TVmax), which further encourages the joint selection of adjacent spatial locations. Experiments in image captioning and VQA, using both LSTM and Transformer architectures, show gains in terms of human-rated caption quality, attention relevance, and VQA accuracy, with improved interpretability.
Short Bio:
Pedro is currently a PhD student at IT, advised by André Martins and Zita Marinho. His research interests include machine learning and natural language processing. Before that, he worked as an IT consultant for six months. He completed his master degree in aerospace at IST in 2017.
More info: <A HREF="http://labs.priberam.com/Academia-Partnerships/Seminars.aspx" TARGET="_blank" DATA-MSYS-CLICKTRACK="0" REL="nofollow noopener noreferrer">http://labs.priberam.com/Academia-Partnerships/Seminars.aspx</A>

Sparse and Structured Visual Attention

More events from Priberam

Discover more events from Priberam, from Science & Tech to other experiences you might love.

Still looking for the right event?

Explore all events in Lisboa and filter by date, category, and more to find the perfect fit.

Sparse and Structured Visual Attention

More events from Priberam

Discover more events from Priberam, from Science & Tech to other experiences you might love.

You might also like...

Browse more events with different dates, prices, and formats to find your next great experience.

Still looking for the right event?

Explore all events in Lisboa and filter by date, category, and more to find the perfect fit.