Sign-Aware Periodicity Metrics of User Engagement for Online Search Quality Evaluation

Автор: Drutsa Alexey
Сборник: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval
Год издания: 2015
Место издания: ACM
Первая страница: 779
Последняя страница: 782
DOI: 10.1145/2766462.2767814
Аннотация: Modern Internet companies improve evaluation criteria of their data-driven decision-making that is based on online controlled experiments (also known as A/B tests). The amplitude metrics of user engagement are known to be well sensitive to service changes, but they could not be used to determine, whether the treatment effect is positive or negative. We propose to overcome this sign-agnostic issue by paying attention to the phase of the corresponding DFT sine wave. We refine the amplitude metrics of the first frequency by the phase ones and formalize our intuition in several novel overall evaluation criteria. These criteria are then verified over A/B experiments on real users of Yandex. We find that our approach holds the sensitivity level of the amplitudes and makes their changes sign-aware w.r.t. the treatment effect.
Добавил в систему: Друца Алексей Валерьевич

	ИСТИНА	Войти в систему Регистрация
	ИПМех РАН
	Главная Поиск Статистика О проекте Помощь

ИСТИНА