Research/Blog
“Attention” in AI
- December 3, 2019
- Posted by: CellStrat Editor
- Category: Artificial Intelligence Deep Learning Technology
No Comments
Attention is the simple idea of focussing on salient parts of input by taking a weighted average of them, has proven to be the key factor in a wide class of #neuralnet models. #MultiheadAttention in particular has proven to be reason for the success of state-of-art natural language processing (#NLP) models such as #BERT and #Transformer based machine translation models. Multihead attention — which is essentially “multiple attention heads attending to different sections of input in parallel”. It enables expression of sophisticated functions beyond the weighted average of just one attention head.