Baike.dev
All toolsAI codingTrendingOpen sourceNewsSubmit
Log in
Back to tool/Back to issues
#210·GloVe

Poor Results on Large Corpus

Author: KarahanSCreated Mar 31, 2023Updated Apr 11, 2023

Greetings,

I'm trying to train my own GloVe word embeddings for Turkish language using a corpus of size ~10 GB. I have enough disk capacity on my computer and 16 GB memory. I created the vocab.txt successfully, I can confirm that there is no problem with it. Now, I believe I successfully generated a cooccurrence matrix as well which is of size ~35 GB but afterwards shuffling took too long and suddenly terminated. In contrast to the cooccurrence generation step, shuffling seems non-responsive, it's not really printing anything to the console. Then I decided to train my model on an unshuffled cooccurrence matrix directly. I trained it for 20 iterations. Cost for each iteration was something like this (numbers are not precise but my point is that the cost increased for first 3 iterations and then gradually decreased to ~0.11):

itr=1   cost = ~2.5
itr=2   cost = ~10.5
itr=3   cost = ~14.5
itr=4   cost = ~12.5
itr=5   cost = ~10.5
           ...
itr=19 cost = ~0.14
itr=20 cost  = ~0.11

Then I loaded the word vectors using load_word2vec_format function provided by gensim. Tested the vectors with several analogy tasks and unfortunately, the results are terrible. So, here is my questions:

  1. How vital is shuffling? Can such terrible results be explained by the fact that I skipped the shuffling part?
  2. Or, isn't ~0.11 cost enough to produce some reasonable results? Should I have iterated longer?
  3. When I run the shuffling operation, I get an output like this:
bash
Using random seed 1680251209
SHUFFLING COOCCURRENCES
array size: 1020054732
Shuffling by chunks: processed 0 lines.

I tried to print out some local variables and saw that they are increasing. So, the program is actually running but it feels like it will run to forever (if not terminates due to some error). Is it really supposed to take that long (even longer than cooccurrence matrix generation)? I'm suspicious that my memory is not enough. If it's the case, is there any solution rather than simply switching to another hardware/remote server etc.? (Also, it would be really weird that my memory is enough for matrix generation but not for shuffling o.O')

Note: I'm training on Windows using Ubuntu WSL. FYI

Source: stanfordnlp/GloVe

View original on GitHubView discussion on GitHub