A Simple and Effective Usage of Word Clusters for CBOW Model

Yukun Feng1, Chenlong Hu1, Hidetaka Kamigaito1, Hiroya Takamura2, Manabu Okumura1
1Tokyo Institute of Technology, 2The National Institute of Advanced Industrial Science and Technology (AIST)


We propose a simple and effective method for incorporating word clusters into the Continuous Bag-of-Words (CBOW) model. Specifically, we propose to replace infrequent input and output words in CBOW model with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of frequent words and a small amount of cluster embeddings, which will be fine-tuned in downstream tasks. We empirically show our replacing method works well on several downstream tasks. Through our analysis, we show that our method might be also useful for other similar models which produce word embeddings.