arXMLiv 08.2019 - Word Embeddings; Token Model

Part of the arXMLiv project at the KWARC research group

Author

Deyan Ginev

Release

This page documents: 08.2019
Latest: 08.2019

Accessibility and License

The content of this Dataset is licensed to SIGMathLing members for research and tool development purposes.

Access is restricted to SIGMathLing members under the SIGMathLing Non-Disclosure-Agreement as for most arXiv articles, the right of distribution was only given (or assumed) to arXiv itself.

An 15.2 billion token model for the arXMLiv 08.2019 dataset, including subformula lexemes
- token_model.zip
300 dimensional GloVe word embeddings for the arXMLiv 08.2019 dataset
- glove.arxmliv.15B.300d.zip and vocab.arxmliv.zip
the main arXMLiv dataset is available separately here

Token Model Statistics

subset	documents	paragraphs
no_problem	150,701	6,071,920
warning_1	500,000	36,130,694
warning_2	328,127	24,285,351
error	395,711	31,155,136
complete	1,374,539	97,643,101

subset	words	formulas	inline cite
no_problem	619,051,536	25,210,637	4,248,840
warning_1	2,917,283,935	212,113,899	18,553,611
warning_2	1,937,516,458	140,094,708	12,590,335
error	2,307,007,544	163,290,748	14,200,445
complete	7,780,859,473	540,709,992	49,593,231

GloVe Model Statistics

subset	tokens	unique words	unique words (freq 5+ )
complete	15,192,564,807	2,782,667	989,136

Citing this Resource

Please cite the main dataset when using the word embeddings, as they are generated and distributed jointly. Instructions here

Download

Download links. This is a temporary solution as we are in the process of migrating the data.
SIGMathLing members only. Joining is free and mostly a legal checkmark on our end - all researchers welcome!

Generated via

Generation Parameters

token model distributed as 4 subsets - no_problem, warning_1, warning_2 and error. complete model is derived via:

   cat token_model_no_problem.txt \
       token_model_warning_1.txt token_model_warning_2.txt \
       token_model_error.txt > token_model_complete.txt

llamapun v0.3.4, corpus_token_model example used for token model extraction
- processed logical paragraphs, abstracts, captions and keywords, ignore all other content (e.g. tables, bibliography, others)
- excluded paragraphs containing latexml errors (marked via a ltx_ERROR HTML class); also excluded when words over 25 characters were encountered.
- used llamapun math-aware word tokenization, with sub-formula math lexemes (improved robustness since 2018)
- marked up inline citations replaced with citationelement token
- numeric literals replaced with NUM token (both in text and formulas)
- internal references replaced with ref token (e.g. Figure ref)
- textual punctuation is included as-is, while mathematical punctuation is annotated via the latexml-generated lexemes.
- words are downcased, while math content is kept cased, to mitigate lexical ambiguity.
- an update was provided on Sep 29, 2019, improving tokenization of apostrophe-including constructs such as 's.
GloVe repository at sha 07d59d
- build/vocab_count -min-count 5
- build/cooccur -memory 48.0 -window-size 15
- build/shuffle -memory 48.0
- build/glove -threads 30 -x-max 100 -iter 50 -vector-size 300 -binary 2

Examples and baselines

GloVe in-built evaluation (non-expert tasks e.g. language, relationships, geography)

NEW; 2019 model
- Total accuracy: 37.76% (7017/18322)
- Highest score: “gram3-comparative.txt”, 77.33% (1047/1332)
2018 GloVe embeddings
- Total accuracy: 35.48% (6298/17750)
- Highest score: “gram3-comparative.txt”, 76.65% (1021/1332)
demo baseline: text8 demo (first 100M characters of Wikipedia)
- Total accuracy: 23.62% (4211/17827)
- Highest score: “gram6-nationality-adjective.txt”, 58.65% (892/1521)

Measuring word analogy

In a cloned GloVe repository, start via:

python2 eval/python/word_analogy.py --vocab_file vocab.arxmliv.txt --vectors_file glove.arxmliv.15B.300d.txt

abelian is to group as disjoint is to ?
- Top hit: union, cosine distance 0.649029
convex is to concave as positive is to ?
- Top hit: negative, cosine distance 0.812031
finite is to infinte as abelian is to ?
- Top hit: nonabelian, cosine distance 0.689419
quantum is to classical as bottom is to ?
- Top hit: middle, cosine distance 0.770132
- Close second: top, cosine distance 0.758245
eq is to proves as figure is to ?
- Top hit: shows, cosine distance 0.675003
italic_x is to italic_y as italic_a is to ?
- Top hit: italic_b, cosine distance 0.912827

Nearest word vectors