Skip to content
Thimothée Mickus: Linear structures in Transformer Embedding Spaces episode artwork

Oct 15, 2022 48 min

Thimothée Mickus: Linear structures in Transformer Embedding Spaces

from Finnish Center for Artificial Intelligence FCAI Hosted by DASER

Abstract: The Transformer architecture has taken the NLP community by storm. A large body of work has subsequently focused on understanding how they behave. In this talk, I will focus on the fact that a Transformer embedding can be expressed as a sum of vector factors, owing to the use of residual connections across all sublayers. This view sheds light on seemingly disconnected observations often made in the literature: why do Transformer embedding spaces exhibit anisotropy? how does BERT next sentence prediction objective shapes its vector space? why do lower layers tend to fare better on lexical semantic tasks? How different are Transformer embeddings from pure bag-of-word representations? Are multi-head attention modules the most important components in a Transformer? Speakers: Timothee Mickus is a post doc at University of Helsinki, working with Jörg Tiedemann on the ERC Fotran. Previously, his PhD research topic was on distributional semantics and dictionaries: do dictionary definitions depict meaning in the same way as neural networks-based word vectors? Can we come up with quantitative ways of measuring how similar these two theories are? Affiliation: University of Helsinki

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Thimothée Mickus: Linear structures in Transformer Embedding Spaces

0:00 48:11

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of Finnish Center for Artificial Intelligence FCAI?

This episode is 48 minutes long.

When was this Finnish Center for Artificial Intelligence FCAI episode published?

This episode was published on October 15, 2022.

Can I download this Finnish Center for Artificial Intelligence FCAI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.