Transformers Without Normalization: Dynamic Tanh Achieves Strong Performance

from Build Wiz AI Show · host Build Wiz AI

This podcast episode delves into the "Transformers without Normalization" paper, which introduces Dynamic Tanh (DyT) as a potential replacement for normalization layers in Transformers. DyT, a simple operation defined as tanh(αx) with a learnable parameter, aims to replicate the effects of Layer Norm without calculating activation statistics. Could DyT offer similar or better performance and improved efficiency, challenging the necessity of normalization in modern neural networks?

Episode metadata supplied by the publisher feed · Published Mar 24, 2025

Embed this episode

Attribution link and audio player

Ready to play

Transformers Without Normalization: Dynamic Tanh Achieves Strong Performance

0:00 11:46

1×

No transcript for this episode yet

We transcribe on demand. Request one and we'll notify you when it's ready — usually under 10 minutes.

Share this episode

Similar Episodes

No similar episodes found.

Similar Podcasts

CISO Perspectives (public) N2K Networks This season on CISO Perspectives, host Kim Jones explores some of the challenges of leading through uncertainty. We explore the complexity of the changing nature of regulation and working with the federal government, the evolution of privacy and fraud, and how emerging technologies like AI and quantum computing are changing cyber. When you don’t know what questions to ask, you’re afraid to ask, or don’t know who to ask, CISO Perspectives provides the foundation for learning in this brave new world.

Frequently Asked Questions

How long is this episode of Build Wiz AI Show?

This episode is 11 minutes long.

When was this Build Wiz AI Show episode published?

This episode was published on March 24, 2025.

Can I download this Build Wiz AI Show episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.

URL copied to clipboard!