Struct GatedMlpConfig 

pub struct GatedMlpConfig {
    pub d_model: usize,
    pub d_intermediate: usize,
    pub multiple_of: usize,
    pub has_proj_bias: bool,
}

Configuration to create a GatedMlp.

Fields

d_model: usize

Input and output width (the residual stream), d_model.

d_intermediate: usize

Inner width before rounding — d_intermediate in the reference configs. Rounded up to a multiple of Self::multiple_of by Self::hidden.

multiple_of: usize

Round Self::d_intermediate up to a multiple of this.

has_proj_bias: bool

Whether fc1/fc2 carry a bias term.

Implementations

impl GatedMlpConfig

pub fn new(d_model: usize, d_intermediate: usize) -> GatedMlpConfig

Create a new instance of the config.

Arguments

Required Arguments

d_model

Input and output width (the residual stream), d_model.

d_intermediate

Inner width before rounding — d_intermediate in the reference configs. Rounded up to a multiple of Self::multiple_of by Self::hidden.

Default Arguments

multiple_of

Round Self::d_intermediate up to a multiple of this.

• Defaults to 128

has_proj_bias

Whether fc1/fc2 carry a bias term.

• Defaults to false

impl GatedMlpConfig

pub fn with_multiple_of(self, multiple_of: usize) -> GatedMlpConfig

Sets the value for the field multiple_of.

Round Self::d_intermediate up to a multiple of this.

• Defaults to 128

pub fn with_has_proj_bias(self, has_proj_bias: bool) -> GatedMlpConfig

Sets the value for the field has_proj_bias.

Whether fc1/fc2 carry a bias term.

• Defaults to false

impl GatedMlpConfig

pub fn hidden(&self) -> usize

The realised inner width: Self::d_intermediate rounded up to a multiple of Self::multiple_of.

pub fn init(&self, device: &Device) -> GatedMlp

Initialize a new GatedMlp module.

Trait Implementations

impl Clone for GatedMlpConfig

fn clone(&self) -> GatedMlpConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Config for GatedMlpConfig

fn save<P>(&self, file: P) -> Result<(), Error>

where P: AsRef<Path>,

Saves the configuration to a file.

fn load<P>(file: P) -> Result<Self, ConfigError>

where P: AsRef<Path>,

Loads the configuration from a file.

fn load_binary(data: &[u8]) -> Result<Self, ConfigError>

Loads the configuration from a binary buffer.

impl Debug for GatedMlpConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for GatedMlpConfig

fn deserialize<D>( deserializer: D, ) -> Result<GatedMlpConfig, <D as Deserializer<'de>>::Error>

where D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for GatedMlpConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Serialize for GatedMlpConfig

fn serialize<S>( &self, serializer: S, ) -> Result<<S as Serializer>::Ok, <S as Serializer>::Error>

where S: Serializer,

Serialize this value into the given Serde serializer.