JointStorage (deeplearning4j-nlp 0.4-rc3.8 API)

java.lang.Object
- org.deeplearning4j.models.embeddings.inmemory.JointStorage<T>

All Implemented Interfaces:

Serializable, WeightLookupTable<T>, VocabCache<T>
```
public class JointStorage<T extends SequenceElement>
extends Object
implements WeightLookupTable<T>, VocabCache<T>
```
This is going to be primitive implementation of joint WeightLookupTable, used for ParagraphVectors and Word2Vec joint training. Main idea of this implementation nr.1: in some cases you have to train corpus for 2 vocabs instead of 1. Or you need to extend vocab, and you can use few separate instead of rebuilding one big WeightLookupTable which can double used memory. WORK IS IN PROGRESS, PLEASE DO NOT USE

Author:

[email protected]

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class JointStorage.Builder<T extends SequenceElement>

Nested Classes
Modifier and Type	Class and Description
`static class`	`JointStorage.Builder<T extends SequenceElement>`

Field Summary

Fields
Modifier and Type Field and Description

protected Long tableId

Fields
Modifier and Type	Field and Description
`protected Long`	`tableId`

Constructor Summary

Constructors
Constructor and Description

JointStorage()

Constructors
Constructor and Description
`JointStorage()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addToken(SequenceElement word)` Adds a token to the cache
`void`	`addWordToIndex(int index, String word)`
`boolean`	`containsWord(String word)` Returns true if the cache contains the given word
`int`	`docAppearedIn(String word)` Count of documents a word appeared in
`T`	`elementAtIndex(int index)` Returns SequenceElement at the given index or null
`int`	`getVectorLength()` Returns outcome vector length
`org.nd4j.linalg.api.ndarray.INDArray`	`getWeights()`
`boolean`	`hasToken(String token)` Returns whether the cache contains this token or not
`void`	`importVocabulary(VocabCache<T> vocabCache)` imports vocabulary
`void`	`incrementDocCount(String word, int howMuch)` Increment the document count
`void`	`incrementTotalDocCount()` Increment the doc count
`void`	`incrementTotalDocCount(int by)` Increment the doc count
`void`	`incrementWordCount(String word)` Increment the count for the given word
`void`	`incrementWordCount(String word, int increment)` Increment the count for the given word by the amount increment
`int`	`indexOf(String word)` Returns the index of a given word
`void`	`iterate(T w1, T w2)` Iterate on the given 2 vocab words
`void`	`iterateSample(T w1, T w2, AtomicLong nextRandom, double alpha)` Iterate on the given 2 vocab words
`int`	`layerSize()` The layer size for the lookup table
`org.nd4j.linalg.api.ndarray.INDArray`	`loadCodes(int[] codes)` Loads the co-occurrences for the given codes
`void`	`loadVocab()` Load vocab
`int`	`numWords()` Returns number of words in all underlying vocabularies
`void`	`plotVocab()` Render the words via tsne
`void`	`plotVocab(Tsne tsne)` Render the words via TSNE
`void`	`putCode(int codeIndex, org.nd4j.linalg.api.ndarray.INDArray code)`
`void`	`putVector(String word, org.nd4j.linalg.api.ndarray.INDArray vector)` Inserts a word vector
`void`	`putVocabWord(String word)` Inserts the word as a vocab word (it gets the vocab word from the internal token store).
`void`	`removeElement(String label)` Removes element with specified label from vocabulary Please note: Huffman index should be updated after element removal
`void`	`removeElement(T element)` Removes specified element from vocabulary Please note: Huffman index should be updated after element removal
`void`	`resetWeights()` Reset the weights of the cache
`void`	`resetWeights(boolean reset)` Clear out all weights regardless
`void`	`saveVocab()` Saves the vocab: this allow for reuse of word frequencies
`void`	`setCountForDoc(String word, int count)` Set the count for the number of documents the word appears in
`void`	`setLearningRate(double lr)` Sets the learning rate
`T`	`tokenFor(String word)` Returns the token (again not necessarily in the vocab) for this word
`Collection<T>`	`tokens()` All of the tokens in the cache, (not necessarily apart of the vocab)
`int`	`totalNumberOfDocs()` Returns the total of number of documents encountered in the corpus
`long`	`totalWordOccurrences()` The total number of word occurrences
`void`	`updateWordsOccurencies()` Updates counters
`org.nd4j.linalg.api.ndarray.INDArray`	`vector(String word)`
`Iterator<org.nd4j.linalg.api.ndarray.INDArray>`	`vectors()` Iterates through all of the vectors in the cache
`boolean`	`vocabExists()` Vocab exists already
`Collection<T>`	`vocabWords()` Returns all SequenceElements in this JointStorage instance.
`String`	`wordAtIndex(int index)` Returns the word contained at the given index or null
`T`	`wordFor(String word)`
`int`	`wordFrequency(String word)` Returns the number of times the word has occurred
`Collection<String>`	`words()` Returns all of the words in the vocab

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.deeplearning4j.models.embeddings.WeightLookupTable
getTableId, setTableId

- Field Detail
  - tableId
```
protected Long tableId
```
- Constructor Detail
  - JointStorage
```
public JointStorage()
```
- Method Detail
  - loadVocab
```
public void loadVocab()
```
    Description copied from interface: VocabCache
    
    Load vocab
    
    Specified by:
    
    loadVocab in interface VocabCache<T extends SequenceElement>
  - vocabExists
```
public boolean vocabExists()
```
    Description copied from interface: VocabCache
    
    Vocab exists already
    
    Specified by:
    
    vocabExists in interface VocabCache<T extends SequenceElement>
    
    Returns:
  - saveVocab
```
public void saveVocab()
```
    Description copied from interface: VocabCache
    
    Saves the vocab: this allow for reuse of word frequencies
    
    Specified by:
    
    saveVocab in interface VocabCache<T extends SequenceElement>
  - words
```
public Collection<String> words()
```
    Description copied from interface: VocabCache
    
    Returns all of the words in the vocab
    
    Specified by:
    
    words in interface VocabCache<T extends SequenceElement>
  - incrementWordCount
```
public void incrementWordCount(String word)
```
    Description copied from interface: VocabCache
    
    Increment the count for the given word
    
    Specified by:
    
    incrementWordCount in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to increment the count for
  - incrementWordCount
```
public void incrementWordCount(String word,
                               int increment)
```
    Description copied from interface: VocabCache
    
    Increment the count for the given word by the amount increment
    
    Specified by:
    
    incrementWordCount in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to increment the count for
    
    increment - the amount to increment by
  - wordFrequency
```
public int wordFrequency(String word)
```
    Description copied from interface: VocabCache
    
    Returns the number of times the word has occurred
    
    Specified by:
    
    wordFrequency in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to retrieve the occurrence frequency for
    
    Returns:
    
    0 if hasn't occurred or the number of times the word occurs
  - containsWord
```
public boolean containsWord(String word)
```
    Description copied from interface: VocabCache
    
    Returns true if the cache contains the given word
    
    Specified by:
    
    containsWord in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to check for
    
    Returns:
  - wordAtIndex
```
public String wordAtIndex(int index)
```
    Description copied from interface: VocabCache
    
    Returns the word contained at the given index or null
    
    Specified by:
    
    wordAtIndex in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    index - the index of the word to get
    
    Returns:
    
    the word at the given index
  - elementAtIndex
```
public T elementAtIndex(int index)
```
    Description copied from interface: VocabCache
    
    Returns SequenceElement at the given index or null
    
    Specified by:
    
    elementAtIndex in interface VocabCache<T extends SequenceElement>
    
    Returns:
  - indexOf
```
public int indexOf(String word)
```
    Description copied from interface: VocabCache
    
    Returns the index of a given word
    
    Specified by:
    
    indexOf in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the index of a given word
    
    Returns:
    
    the index of a given word or -1 if not found
  - vocabWords
```
public Collection<T> vocabWords()
```
    Returns all SequenceElements in this JointStorage instance. Please note, if used in distributed environment this can cause OOM exceptions. Use with caution, or use iterator instead
    
    Specified by:
    
    vocabWords in interface VocabCache<T extends SequenceElement>
    
    Returns:
    
    Collection of all VocabWords in this JointStorage
  - totalWordOccurrences
```
public long totalWordOccurrences()
```
    Description copied from interface: VocabCache
    
    The total number of word occurrences
    
    Specified by:
    
    totalWordOccurrences in interface VocabCache<T extends SequenceElement>
    
    Returns:
    
    the total number of word occurrences
  - wordFor
```
public T wordFor(String word)
```
    Specified by:
    
    wordFor in interface VocabCache<T extends SequenceElement>
    
    Returns:
  - addWordToIndex
```
public void addWordToIndex(int index,
                           String word)
```
    Specified by:
    
    addWordToIndex in interface VocabCache<T extends SequenceElement>
  - putVocabWord
```
public void putVocabWord(String word)
```
    Description copied from interface: VocabCache
    
    Inserts the word as a vocab word (it gets the vocab word from the internal token store). Note that the index must be set on the token.
    
    Specified by:
    
    putVocabWord in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to add to the vocab
  - numWords
```
public int numWords()
```
    Returns number of words in all underlying vocabularies
    
    Specified by:
    
    numWords in interface VocabCache<T extends SequenceElement>
    
    Returns:
  - docAppearedIn
```
public int docAppearedIn(String word)
```
    Description copied from interface: VocabCache
    
    Count of documents a word appeared in
    
    Specified by:
    
    docAppearedIn in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the number of documents the word appeared in
    
    Returns:
  - incrementDocCount
```
public void incrementDocCount(String word,
                              int howMuch)
```
    Description copied from interface: VocabCache
    
    Increment the document count
    
    Specified by:
    
    incrementDocCount in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to increment by
  - setCountForDoc
```
public void setCountForDoc(String word,
                           int count)
```
    Description copied from interface: VocabCache
    
    Set the count for the number of documents the word appears in
    
    Specified by:
    
    setCountForDoc in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to set the count for
    
    count - the count of the word
  - totalNumberOfDocs
```
public int totalNumberOfDocs()
```
    Description copied from interface: VocabCache
    
    Returns the total of number of documents encountered in the corpus
    
    Specified by:
    
    totalNumberOfDocs in interface VocabCache<T extends SequenceElement>
    
    Returns:
    
    the total number of docs in the corpus
  - incrementTotalDocCount
```
public void incrementTotalDocCount()
```
    Description copied from interface: VocabCache
    
    Increment the doc count
    
    Specified by:
    
    incrementTotalDocCount in interface VocabCache<T extends SequenceElement>
  - incrementTotalDocCount
```
public void incrementTotalDocCount(int by)
```
    Description copied from interface: VocabCache
    
    Increment the doc count
    
    Specified by:
    
    incrementTotalDocCount in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    by - the number to increment by
  - tokens
```
public Collection<T> tokens()
```
    Description copied from interface: VocabCache
    
    All of the tokens in the cache, (not necessarily apart of the vocab)
    
    Specified by:
    
    tokens in interface VocabCache<T extends SequenceElement>
    
    Returns:
    
    the tokens for this cache
  - addToken
```
public void addToken(SequenceElement word)
```
    Description copied from interface: VocabCache
    
    Adds a token to the cache
    
    Specified by:
    
    addToken in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to add
  - tokenFor
```
public T tokenFor(String word)
```
    Description copied from interface: VocabCache
    
    Returns the token (again not necessarily in the vocab) for this word
    
    Specified by:
    
    tokenFor in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    word - the word to get the token for
    
    Returns:
    
    the vocab word for this token
  - hasToken
```
public boolean hasToken(String token)
```
    Description copied from interface: VocabCache
    
    Returns whether the cache contains this token or not
    
    Specified by:
    
    hasToken in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    token - the token to tes
    
    Returns:
    
    whether the token exists in the cache or not
  - importVocabulary
```
public void importVocabulary(VocabCache<T> vocabCache)
```
    Description copied from interface: VocabCache
    
    imports vocabulary
    
    Specified by:
    
    importVocabulary in interface VocabCache<T extends SequenceElement>
  - updateWordsOccurencies
```
public void updateWordsOccurencies()
```
    Description copied from interface: VocabCache
    
    Updates counters
    
    Specified by:
    
    updateWordsOccurencies in interface VocabCache<T extends SequenceElement>
  - removeElement
```
public void removeElement(String label)
```
    Description copied from interface: VocabCache
    
    Removes element with specified label from vocabulary Please note: Huffman index should be updated after element removal
    
    Specified by:
    
    removeElement in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    label - label of the element to be removed
  - removeElement
```
public void removeElement(T element)
```
    Description copied from interface: VocabCache
    
    Removes specified element from vocabulary Please note: Huffman index should be updated after element removal
    
    Specified by:
    
    removeElement in interface VocabCache<T extends SequenceElement>
    
    Parameters:
    
    element - SequenceElement to be removed
  - layerSize
```
public int layerSize()
```
    Description copied from interface: WeightLookupTable
    
    The layer size for the lookup table
    
    Specified by:
    
    layerSize in interface WeightLookupTable<T extends SequenceElement>
    
    Returns:
    
    the layer size for the lookup table
  - resetWeights
```
public void resetWeights(boolean reset)
```
    Description copied from interface: WeightLookupTable
    
    Clear out all weights regardless
    
    Specified by:
    
    resetWeights in interface WeightLookupTable<T extends SequenceElement>
  - plotVocab
```
public void plotVocab(Tsne tsne)
```
    Description copied from interface: WeightLookupTable
    
    Render the words via TSNE
    
    Specified by:
    
    plotVocab in interface WeightLookupTable<T extends SequenceElement>
    
    Parameters:
    
    tsne - the tsne to use
  - plotVocab
```
public void plotVocab()
```
    Description copied from interface: WeightLookupTable
    
    Render the words via tsne
    
    Specified by:
    
    plotVocab in interface WeightLookupTable<T extends SequenceElement>
  - putCode
```
public void putCode(int codeIndex,
                    org.nd4j.linalg.api.ndarray.INDArray code)
```
    Specified by:
    
    putCode in interface WeightLookupTable<T extends SequenceElement>
  - loadCodes
```
public org.nd4j.linalg.api.ndarray.INDArray loadCodes(int[] codes)
```
    Description copied from interface: WeightLookupTable
    
    Loads the co-occurrences for the given codes
    
    Specified by:
    
    loadCodes in interface WeightLookupTable<T extends SequenceElement>
    
    Parameters:
    
    codes - the codes to load
    
    Returns:
    
    an ndarray of code.length by layerSize
  - iterate
```
public void iterate(T w1,
                    T w2)
```
    Description copied from interface: WeightLookupTable
    
    Iterate on the given 2 vocab words
    
    Specified by:
    
    iterate in interface WeightLookupTable<T extends SequenceElement>
    
    Parameters:
    
    w1 - the first word to iterate on
    
    w2 - the second word to iterate on
  - iterateSample
```
public void iterateSample(T w1,
                          T w2,
                          AtomicLong nextRandom,
                          double alpha)
```
    Description copied from interface: WeightLookupTable
    
    Iterate on the given 2 vocab words
    
    Specified by:
    
    iterateSample in interface WeightLookupTable<T extends SequenceElement>
    
    Parameters:
    
    w1 - the first word to iterate on
    
    w2 - the second word to iterate on
    
    nextRandom - nextRandom for sampling
    
    alpha - the alpha to use for learning
  - putVector
```
public void putVector(String word,
                      org.nd4j.linalg.api.ndarray.INDArray vector)
```
    Description copied from interface: WeightLookupTable
    
    Inserts a word vector
    
    Specified by:
    
    putVector in interface WeightLookupTable<T extends SequenceElement>
    
    Parameters:
    
    word - the word to insert
    
    vector - the vector to insert
  - vector
```
public org.nd4j.linalg.api.ndarray.INDArray vector(String word)
```
    Specified by:
    
    vector in interface WeightLookupTable<T extends SequenceElement>
    
    Returns:
  - resetWeights
```
public void resetWeights()
```
    Description copied from interface: WeightLookupTable
    
    Reset the weights of the cache
    
    Specified by:
    
    resetWeights in interface WeightLookupTable<T extends SequenceElement>
  - setLearningRate
```
public void setLearningRate(double lr)
```
    Description copied from interface: WeightLookupTable
    
    Sets the learning rate
    
    Specified by:
    
    setLearningRate in interface WeightLookupTable<T extends SequenceElement>
  - vectors
```
public Iterator<org.nd4j.linalg.api.ndarray.INDArray> vectors()
```
    Description copied from interface: WeightLookupTable
    
    Iterates through all of the vectors in the cache
    
    Specified by:
    
    vectors in interface WeightLookupTable<T extends SequenceElement>
    
    Returns:
    
    an iterator for all vectors in the cache
  - getWeights
```
public org.nd4j.linalg.api.ndarray.INDArray getWeights()
```
    Specified by:
    
    getWeights in interface WeightLookupTable<T extends SequenceElement>
  - getVectorLength
```
public int getVectorLength()
```
    Description copied from interface: WeightLookupTable
    
    Returns outcome vector length
    
    Specified by:
    
    getVectorLength in interface WeightLookupTable<T extends SequenceElement>
    
    Returns:

Class JointStorage<T extends SequenceElement>

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.deeplearning4j.models.embeddings.WeightLookupTable

Field Detail

tableId

Constructor Detail

JointStorage

Method Detail

loadVocab

vocabExists

saveVocab

words

incrementWordCount

incrementWordCount

wordFrequency

containsWord

wordAtIndex

elementAtIndex

indexOf

vocabWords

totalWordOccurrences

wordFor

addWordToIndex

putVocabWord

numWords

docAppearedIn

incrementDocCount

setCountForDoc

totalNumberOfDocs

incrementTotalDocCount

incrementTotalDocCount

tokens

addToken

tokenFor

hasToken

importVocabulary

updateWordsOccurencies

removeElement

removeElement

layerSize

resetWeights

plotVocab

plotVocab

putCode

loadCodes

iterate

iterateSample

putVector

vector

resetWeights

setLearningRate

vectors

getWeights

getVectorLength