|
ИСТИНА |
Войти в систему Регистрация |
ИПМех РАН |
||
Variational Dropout Sparsifies Deep Neural Networks Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrovстатья
- Авторы: Molchanov Dmitry, Ashukha Arsenii, Vetrov Dmitry
- Сборник: Proceedings of the 34th International Conference on Machine Learning, PMLR
- Том: 70
- Год издания: 2017
- Первая страница: 2498
- Последняя страница: 2507
- Аннотация: We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy.
- Добавил в систему: Ashukha Arsenii