org.deeplearning4j.models.glove

Class AbstractCoOccurrences<T extends SequenceElement>

java.lang.Object

org.deeplearning4j.models.glove.AbstractCoOccurrences<T>

All Implemented Interfaces:

Serializable

public class AbstractCoOccurrences<T extends SequenceElement>
extends Object
implements Serializable

This class implements building cooccurrence map for abstract training corpus. However it's performance rather low, due to exsessive IO that happens in ShadowCopyThread PLEASE NOTE: Current implementation involves massive IO, and it should be rewritter as soon as ND4j gets sparse arrays support

Author:

[email protected]

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AbstractCoOccurrences.Builder<T extends SequenceElement>

Field Summary

Fields

Modifier and Type	Field and Description
protected ReentrantReadWriteLock	lock
protected static org.slf4j.Logger	logger
protected long	memory_threshold
protected SequenceIterator<T>	sequenceIterator
protected boolean	symmetric
protected File	targetFile
protected VocabCache<T>	vocabCache
protected int	windowSize
protected int	workers

Method Summary

Modifier and Type	Method and Description
void	fit()
double	getCoOccurrenceCount(T element1, T element2) This method returns cooccurrence distance weights for two SequenceElements
protected long	getMemoryFootprint() This method returns estimated memory footrpint, based on current CountMap content
protected long	getMemoryThreshold() This memory returns memory threshold, defined as 1/2 of memory allowed for allocation
Iterator<Pair<Pair<T,T>,Double>>	iterator() This method returns iterator with elements pairs and their weights.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

symmetric

protected boolean symmetric

windowSize

protected int windowSize

vocabCache

protected VocabCache<T extends SequenceElement> vocabCache

sequenceIterator

protected SequenceIterator<T extends SequenceElement> sequenceIterator

workers

protected int workers

targetFile

protected File targetFile

lock

protected ReentrantReadWriteLock lock

memory_threshold

protected long memory_threshold

logger

protected static final org.slf4j.Logger logger

Method Detail

getCoOccurrenceCount

public double getCoOccurrenceCount(@NonNull
                                   T element1,
                                   @NonNull
                                   T element2)

This method returns cooccurrence distance weights for two SequenceElements

Parameters:

element1 -

element2 -

Returns:

distance weight

getMemoryFootprint

protected long getMemoryFootprint()

This method returns estimated memory footrpint, based on current CountMap content

Returns:

getMemoryThreshold

protected long getMemoryThreshold()

This memory returns memory threshold, defined as 1/2 of memory allowed for allocation

Returns:

fit

public void fit()

iterator

public Iterator<Pair<Pair<T,T>,Double>> iterator()

This method returns iterator with elements pairs and their weights. Resulting iterator is safe to use in multi-threaded environment. Developer's note: thread safety on received iterator is delegated to PrefetchedSentenceIterator

Returns: