“because they have a sort of global attention across the…” — Dwarkesh Podcast at 34:33
“because they have a sort of global attention across the whole thing, they can more easily draw these connections. But you do need more data there so that you can kind of learn through data the sort of invariant local features. I've tried very hard to make transformers work for this problem”
0:00
2:37:29