Would the multi-head attention (Wv) not be equivalent to the chemical gradient changes?
(there are multiple matrices in multi-head attention, one for each attention head and what I imagine would be the equivalent of different gradients
This allows each attention head to learn different representations and focus on different aspects of the input sequence.)
And then the output produced after applying the concatenated (W0 or output projection), be equivalent to the different electrical outputs such as the spikes and passed to the next neuron equivalent or attention head?
Would the multi-head attention (Wv) not be equivalent to the chemical gradient changes?
(there are multiple matrices in multi-head attention, one for each attention head and what I imagine would be the equivalent of different gradients
This allows each attention head to learn different representations and focus on different aspects of the input sequence.)
And then the output produced after applying the concatenated (W0 or output projection), be equivalent to the different electrical outputs such as the spikes and passed to the next neuron equivalent or attention head?