Research/Blog
CELLSTRAT AI LAB – MINUTES FROM 16TH. FEB 2019 BLR MEETUP
- February 18, 2019
- Posted by: CellStrat Editor
- Category: Uncategorized
The AI Lab group continues to do wonders and present highly knowledgeable sessions to all its members and growing strength of junior to very senior level Machine Learning professionals and datascientists. More and more cutting-edge areas of Artificial Intelligence and Machine Learning are being researched and worked up-on.
Discussion Points in AI Lab session:
- Language translation model based on encoder-decoder RNN/LSTM architecture.
- LSTM architecture with Attention based on the research paper “Neural Machine Translation” by Dzmitri Bahdanau and “Attention based Machine Translation” by Minh Thang Luong & others.
![](http://www.cellstrat.com/wp-content/uploads/2019/02/image-4.png)
Both approaches were demonstrated through implementation in Tensorflow-Keras.
Three papers were reviewed that discuss various ways we can use to optimize/ reduce the size of the trained model on disk. Novel ways like Student-Teacher networks, Vector quantization and Drop-Neuron algorithm were explained lucidly with implementation.
Training thin deep networks following the student-teacher learning paradigm has received intensive attention because of its excellent performance. The student can be significantly smaller than the Teacher network in terms of depth & number of parameters. The guidance is provided by the teacher network based on hints in some form or the other.
The Drop Neuron logic works with simplifying the structure of Deep Neural Networks. By pruning connections of the network by making the weight space, one is able to drop connections. However, the model occupies the same amount of size as before in memory while doing a forward pass. Through this approach, the size of the network on disk is reduced, and the resultant model as a whole occupies less space in the RAM during a forward pass. The regularization makes use of li_regularizer and lo_regularizer.
The weights of a neural network take up all the space in memory assigned to the model. These weights are all slightly different floating point numbers, hence simple compression formats like zip don’t compress them well. This algorithm uses uniform quantization, non-uniform quantization, and K-means clustering to quantize the weights, so the space taken is reduced significantly.
![](http://www.cellstrat.com/wp-content/uploads/2019/02/image-5.png)