Skip to content

Self attention with relative positional embedding #7356

Open

Description

@vgrau98

Is your feature request related to a problem? Please describe.
Enable relative positional embedding in attention block

Describe the solution you'd like
A constructor argument specifying which type (or None) of relative embedding to use

Describe alternatives you've considered
NA

Additional context
Should be useful for some network implementation like SAM (https://arxiv.org/abs/2304.02643, #6357 )
PR opened #7346 with suggestion for decomposed relative positional embedding (https://arxiv.org/abs/2112.01526)

Activity

  1. added a commit that references this issue on Jan 18, 2024
    41fb3ff
  2. added a commit that references this issue on Jan 30, 2024
    fdbc611
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions