RetroFed Forums
Your home away from modern life.
Search Categories Forum list Register Log in
About RetroFed
RetroFed

Welcome to the nostalgia-filled piefed instance of yesteryear.

Open Registration with Application questionnaire.

If you have a question or comment you can post in !meta@retrofed.com or message @DosDude@retrofed.com or @Die4Ever@retrofed.com

In case of downtime and/or emergency, we will post updates on @DosDude@mastodon.social

Some rules we try to uphold:

  • Be civil. Bigotry is not welcome. This includes (but is not limited to): transphobia, queerphobia, racism, ageism and ableism. We do not promote hate.
  • Communities need to be Retro or Retro-adjacent.
  • Please don't upload pornographic content. (use an external image host for that)
72 registered users
Administrators: DosDude, Die4Ever
Visit retrofed.com
Activation function and GLU variants for Transformer models | Tarique Anwar | Apr 18, 2022
Post Reply RetroFed Forums Forum Index » Machine Learning
AuthorMessage
ericjmorey programming.dev ericjmorey@programming.dev
Joined: 2023-06-17
Quote

Link: https://medium.com/@tariqanwarph/activation-function-and-glu-variants-for-transformer-models-a4fcbe85323f

Apr 18, 2022 | Tarique Anwar Writes:

The main reason for ReLu being used is that it is simple, fast, and empirically it seems to work well.

But with the emergence of Transformer based models, different variants of activation functions and GLU have been experimented with and do seem to perform better. Some of them are:

  • GeLU²
  • Swish¹
  • GLU³
  • GEGLU⁴
  • SwiGLU⁴

We will go over some of these in detail but before that let’s see where exactly are these activations utilized in a Transformer architecture.

Read Activation function and GLU variants for Transformer models
1   Original
Post Reply
Back to top
Powered by RetroFed · PieBB frontend · All times are Europe/Amsterdam