org.deeplearning4j.models.embeddings.loader

Class WordVectorSerializer

java.lang.Object

org.deeplearning4j.models.embeddings.loader.WordVectorSerializer

public class WordVectorSerializer
extends Object

This is utility class, providing various methods for WordVectors serialization

Author:

Adam Gibson, raver119

Method Summary

Modifier and Type	Method and Description
static WordVectors	fromPair(Pair<InMemoryLookupTable,VocabCache> pair) Load word vectors from the given pair
static WordVectors	fromTableAndVocab(WeightLookupTable table, VocabCache vocab) Load word vectors for the given vocab and table
static float	getFloat(byte[] b) Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java
static Word2Vec	loadFullModel(String path) This method loads full w2v model, previously saved with writeFullMethod call
static WordVectors	loadGoogleModel(File modelFile, boolean binary) Loads the google model
static WordVectors	loadGoogleModel(File modelFile, boolean binary, boolean lineBreaks) Loads the Google model.
static WordVectors	loadGoogleModelNonNormalized(File modelFile, boolean binary, boolean lineBreaks) Loads the Google model without normalization being applied.
static Pair<InMemoryLookupTable,VocabCache>	loadTxt(File vectorsFile) Loads an in memory cache from the given path (sets syn0 and the vocab)
static WordVectors	loadTxtVectors(File vectorsFile) Loads an in memory cache from the given path (sets syn0 and the vocab)
static WordVectors	loadTxtVectors(InputStream stream, boolean skipFirstLine) This method can be used to load previously saved model from InputStream (like a HDFS-stream)
static float	readFloat(InputStream is) Read a float from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java
static ParagraphVectors	readParagraphVectorsFromText(File file) Restores previously serialized ParagraphVectors model
static ParagraphVectors	readParagraphVectorsFromText(InputStream stream) Restores previously serialized ParagraphVectors model
static ParagraphVectors	readParagraphVectorsFromText(String path) Restores previously serialized ParagraphVectors model
static <T extends SequenceElement> SequenceVectors<T>	readSequenceVectors(SequenceElementFactory<T> factory, File file) This method loads previously saved SequenceVectors model from File
static <T extends SequenceElement> SequenceVectors<T>	readSequenceVectors(SequenceElementFactory<T> factory, InputStream stream) This method loads previously saved SequenceVectors model from InputStream
static String	readString(DataInputStream dis) Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java
static VocabCache<VocabWord>	readVocabCache(File file) This method reads vocab cache from provided file.
static VocabCache<VocabWord>	readVocabCache(InputStream stream) This method reads vocab cache from provided InputStream.
static void	writeFullModel(Word2Vec vec, String path) Saves full Word2Vec model in the way, that allows model updates without being rebuilt from scratches
static <T extends SequenceElement> void	writeSequenceVectors(SequenceVectors<T> vectors, SequenceElementFactory<T> factory, File file) This method saves specified SequenceVectors model to target file
static <T extends SequenceElement> void	writeSequenceVectors(SequenceVectors<T> vectors, SequenceElementFactory<T> factory, OutputStream stream) This method saves specified SequenceVectors model to target OutputStream
static <T extends SequenceElement> void	writeSequenceVectors(SequenceVectors<T> vectors, SequenceElementFactory<T> factory, String path) This method saves specified SequenceVectors model to target file path
static void	writeTsneFormat(Glove vec, org.nd4j.linalg.api.ndarray.INDArray tsne, File csv) Write the tsne format
static void	writeTsneFormat(Word2Vec vec, org.nd4j.linalg.api.ndarray.INDArray tsne, File csv) Write the tsne format
static void	writeVocabCache(VocabCache<VocabWord> vocabCache, File file) This method saves vocab cache to provided File.
static void	writeVocabCache(VocabCache<VocabWord> vocabCache, OutputStream stream) This method saves vocab cache to provided OutputStream.
static void	writeWordVectors(Glove vectors, File file) This method saves GloVe model to the given output stream.
static void	writeWordVectors(Glove vectors, OutputStream stream) This method saves GloVe model to the given OutputStream
static void	writeWordVectors(Glove vectors, String path) This method saves GloVe model to the given output stream.
static void	writeWordVectors(InMemoryLookupTable lookupTable, InMemoryLookupCache cache, String path) Writes the word vectors to the given path.
static void	writeWordVectors(ParagraphVectors vectors, File path) This method saves paragraph vectors to the given file.
static void	writeWordVectors(ParagraphVectors vectors, OutputStream stream) This method saves paragraph vectors to the given output stream.
static void	writeWordVectors(ParagraphVectors vectors, String path) This method saves paragraph vectors to the given path.
static <T extends SequenceElement> void	writeWordVectors(WeightLookupTable<T> lookupTable, File file) This mehod writes word vectors to the given file.
static <T extends SequenceElement> void	writeWordVectors(WeightLookupTable<T> lookupTable, OutputStream stream) This mehod writes word vectors to the given OutputStream.
static <T extends SequenceElement> void	writeWordVectors(WeightLookupTable<T> lookupTable, String path) This mehod writes word vectors to the given path.
static void	writeWordVectors(Word2Vec vec, BufferedWriter writer) Writes the word vectors to the given BufferedWriter.
static void	writeWordVectors(Word2Vec vec, File file) Writes the word vectors to the given path.
static void	writeWordVectors(Word2Vec vec, OutputStream outputStream) Writes the word vectors to the given OutputStream.
static void	writeWordVectors(Word2Vec vec, String path) Writes the word vectors to the given path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

loadGoogleModel

public static WordVectors loadGoogleModel(File modelFile,
                                          boolean binary)
                                   throws IOException

Loads the google model

Parameters:

modelFile - the path to the google model

binary - read from binary file format (if set to true) or from text file format.

Returns:

the loaded model

Throws:

IOException

loadGoogleModel

public static WordVectors loadGoogleModel(File modelFile,
                                          boolean binary,
                                          boolean lineBreaks)
                                   throws IOException

Loads the Google model.

Parameters:

modelFile - the input file

binary - read from binary or text file format

lineBreaks - if true, the input file is expected to terminate each line with a line break. This is typically the case for files created with recent versions of Word2Vec, but not for the downloadable model files.

Returns:

a Word2Vec object

Throws:

IOException

loadGoogleModelNonNormalized

public static WordVectors loadGoogleModelNonNormalized(File modelFile,
                                                       boolean binary,
                                                       boolean lineBreaks)
                                                throws IOException

Loads the Google model without normalization being applied. PLEASE NOTE: Use this method only if you understand why you need not-normalized model. In all other cases please use loadGoogleModel() instead.

Parameters:

modelFile -

binary -

lineBreaks -

Returns:

Throws:

IOException

readFloat

public static float readFloat(InputStream is)
                       throws IOException

Read a float from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java

Parameters:

is -

Returns:

Throws:

IOException

getFloat

public static float getFloat(byte[] b)

Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java

Parameters:

b -

Returns:

Throws:

IOException

readString

public static String readString(DataInputStream dis)
                         throws IOException

Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java

Parameters:

dis -

Returns:

Throws:

IOException

writeWordVectors

public static <T extends SequenceElement> void writeWordVectors(WeightLookupTable<T> lookupTable,
                                                                String path)
                                                         throws IOException

This mehod writes word vectors to the given path. Please note: this method doesn't load whole vocab/lookupTable into memory, so it's able to process large vocabularies served over network.

Type Parameters:

T -

Parameters:

lookupTable -

path -

Throws:

IOException

writeWordVectors

public static <T extends SequenceElement> void writeWordVectors(WeightLookupTable<T> lookupTable,
                                                                File file)
                                                         throws IOException

This mehod writes word vectors to the given file. Please note: this method doesn't load whole vocab/lookupTable into memory, so it's able to process large vocabularies served over network.

Type Parameters:

T -

Parameters:

lookupTable -

file -

Throws:

IOException

writeWordVectors

public static <T extends SequenceElement> void writeWordVectors(WeightLookupTable<T> lookupTable,
                                                                OutputStream stream)
                                                         throws IOException

This mehod writes word vectors to the given OutputStream. Please note: this method doesn't load whole vocab/lookupTable into memory, so it's able to process large vocabularies served over network.

Type Parameters:

T -

Parameters:

lookupTable -

stream -

Throws:

IOException

writeWordVectors

public static void writeWordVectors(@NonNull
                                    ParagraphVectors vectors,
                                    @NonNull
                                    File path)

This method saves paragraph vectors to the given file.

Parameters:

vectors -

path -

writeWordVectors

public static void writeWordVectors(@NonNull
                                    ParagraphVectors vectors,
                                    @NonNull
                                    String path)

This method saves paragraph vectors to the given path.

Parameters:

vectors -

path -

readParagraphVectorsFromText

public static ParagraphVectors readParagraphVectorsFromText(@NonNull
                                                            String path)

Restores previously serialized ParagraphVectors model

Parameters:

path - Path to file that contains previously serialized model

Returns:

readParagraphVectorsFromText

public static ParagraphVectors readParagraphVectorsFromText(@NonNull
                                                            File file)

Restores previously serialized ParagraphVectors model

Parameters:

file - File that contains previously serialized model

Returns:

readParagraphVectorsFromText

public static ParagraphVectors readParagraphVectorsFromText(@NonNull
                                                            InputStream stream)

Restores previously serialized ParagraphVectors model

Parameters:

stream - InputStream that contains previously serialized model

Returns:

writeWordVectors

public static void writeWordVectors(@NonNull
                                    Glove vectors,
                                    @NonNull
                                    File file)

This method saves GloVe model to the given output stream.

Parameters:

vectors - GloVe model to be saved

file - path where model should be saved to

writeWordVectors

public static void writeWordVectors(@NonNull
                                    Glove vectors,
                                    @NonNull
                                    String path)

This method saves GloVe model to the given output stream.

Parameters:

vectors - GloVe model to be saved

path - path where model should be saved to

writeWordVectors

public static void writeWordVectors(@NonNull
                                    Glove vectors,
                                    @NonNull
                                    OutputStream stream)

This method saves GloVe model to the given OutputStream

Parameters:

vectors - GloVe model to be saved

stream - OutputStream where model should be saved to

writeWordVectors

public static void writeWordVectors(ParagraphVectors vectors,
                                    OutputStream stream)

This method saves paragraph vectors to the given output stream.

Parameters:

vectors -

stream -

writeWordVectors

public static void writeWordVectors(InMemoryLookupTable lookupTable,
                                    InMemoryLookupCache cache,
                                    String path)
                             throws IOException

Writes the word vectors to the given path. Note that this assumes an in memory cache

Parameters:

lookupTable -

cache -

path - the path to write

Throws:

IOException

writeFullModel

public static void writeFullModel(@NonNull
                                  Word2Vec vec,
                                  @NonNull
                                  String path)

Saves full Word2Vec model in the way, that allows model updates without being rebuilt from scratches

Parameters:

vec - - The Word2Vec instance to be saved

path - - the path for json to be saved

loadFullModel

public static Word2Vec loadFullModel(@NonNull
                                     String path)
                              throws FileNotFoundException

This method loads full w2v model, previously saved with writeFullMethod call

Parameters:

path - - path to previously stored w2v json model

Returns:

- Word2Vec instance

Throws:

FileNotFoundException

writeWordVectors

public static void writeWordVectors(@NonNull
                                    Word2Vec vec,
                                    @NonNull
                                    String path)
                             throws IOException

Writes the word vectors to the given path. Note that this assumes an in memory cache

Parameters:

vec - the word2vec to write

path - the path to write

Throws:

IOException

writeWordVectors

public static void writeWordVectors(@NonNull
                                    Word2Vec vec,
                                    @NonNull
                                    File file)
                             throws IOException

Writes the word vectors to the given path. Note that this assumes an in memory cache

Parameters:

vec - the word2vec to write

file - the file to write

Throws:

IOException

writeWordVectors

public static void writeWordVectors(@NonNull
                                    Word2Vec vec,
                                    @NonNull
                                    OutputStream outputStream)
                             throws IOException

Writes the word vectors to the given OutputStream. Note that this assumes an in memory cache.

Parameters:

vec - the word2vec to write

outputStream - - OutputStream, where all data should be sent to the path to write

Throws:

IOException

writeWordVectors

public static void writeWordVectors(@NonNull
                                    Word2Vec vec,
                                    @NonNull
                                    BufferedWriter writer)
                             throws IOException

Writes the word vectors to the given BufferedWriter. Note that this assumes an in memory cache. BufferedWriter can be writer to local file, or hdfs file, or any compatible to java target.

Parameters:

vec - the word2vec to write

writer - - BufferedWriter, where all data should be written to the path to write

Throws:

IOException

fromTableAndVocab

public static WordVectors fromTableAndVocab(WeightLookupTable table,
                                            VocabCache vocab)

Load word vectors for the given vocab and table

Parameters:

table - the weights to use

vocab - the vocab to use

Returns:

wordvectors based on the given parameters

fromPair

public static WordVectors fromPair(Pair<InMemoryLookupTable,VocabCache> pair)

Load word vectors from the given pair

Parameters:

pair - the given pair

Returns:

a read only word vectors impl based on the given lookup table and vocab

loadTxtVectors

public static WordVectors loadTxtVectors(File vectorsFile)
                                  throws FileNotFoundException,
                                         UnsupportedEncodingException

Loads an in memory cache from the given path (sets syn0 and the vocab)

Parameters:

vectorsFile - the path of the file to load\

Returns:

Throws:

FileNotFoundException - if the file does not exist

UnsupportedEncodingException

loadTxt

public static Pair<InMemoryLookupTable,VocabCache> loadTxt(File vectorsFile)
                                                    throws FileNotFoundException,
                                                           UnsupportedEncodingException

Loads an in memory cache from the given path (sets syn0 and the vocab)

Parameters:

vectorsFile - the path of the file to load

Returns:

a Pair holding the lookup table and the vocab cache.

Throws:

FileNotFoundException - if the input file does not exist

UnsupportedEncodingException

loadTxtVectors

public static WordVectors loadTxtVectors(@NonNull
                                         InputStream stream,
                                         boolean skipFirstLine)
                                  throws IOException

This method can be used to load previously saved model from InputStream (like a HDFS-stream)

Parameters:

stream - InputStream that contains previously serialized model

skipFirstLine - Set this TRUE if first line contains csv header, FALSE otherwise

Returns:

Throws:

IOException

writeTsneFormat

public static void writeTsneFormat(Glove vec,
                                   org.nd4j.linalg.api.ndarray.INDArray tsne,
                                   File csv)
                            throws Exception

Write the tsne format

Parameters:

vec - the word vectors to use for labeling

tsne - the tsne array to write

csv - the file to use

Throws:

Exception

writeTsneFormat

public static void writeTsneFormat(Word2Vec vec,
                                   org.nd4j.linalg.api.ndarray.INDArray tsne,
                                   File csv)
                            throws Exception

Write the tsne format

Parameters:

vec - the word vectors to use for labeling

tsne - the tsne array to write

csv - the file to use

Throws:

Exception

writeSequenceVectors

public static <T extends SequenceElement> void writeSequenceVectors(@NonNull
                                                                    SequenceVectors<T> vectors,
                                                                    @NonNull
                                                                    SequenceElementFactory<T> factory,
                                                                    @NonNull
                                                                    String path)
                                                             throws IOException

This method saves specified SequenceVectors model to target file path

Type Parameters:

T -

Parameters:

vectors - SequenceVectors model

factory - SequenceElementFactory implementation for your objects

path - Target output file path

Throws:

IOException

writeSequenceVectors

public static <T extends SequenceElement> void writeSequenceVectors(@NonNull
                                                                    SequenceVectors<T> vectors,
                                                                    @NonNull
                                                                    SequenceElementFactory<T> factory,
                                                                    @NonNull
                                                                    File file)
                                                             throws IOException

This method saves specified SequenceVectors model to target file

Type Parameters:

T -

Parameters:

vectors - SequenceVectors model

factory - SequenceElementFactory implementation for your objects

file - Target output file

Throws:

IOException

writeSequenceVectors

public static <T extends SequenceElement> void writeSequenceVectors(@NonNull
                                                                    SequenceVectors<T> vectors,
                                                                    @NonNull
                                                                    SequenceElementFactory<T> factory,
                                                                    @NonNull
                                                                    OutputStream stream)
                                                             throws IOException

This method saves specified SequenceVectors model to target OutputStream

Type Parameters:

T -

Parameters:

vectors - SequenceVectors model

factory - SequenceElementFactory implementation for your objects

stream - Target output stream

Throws:

IOException

readSequenceVectors

public static <T extends SequenceElement> SequenceVectors<T> readSequenceVectors(@NonNull
                                                                                 SequenceElementFactory<T> factory,
                                                                                 @NonNull
                                                                                 File file)
                                                                          throws IOException

This method loads previously saved SequenceVectors model from File

Type Parameters:

T -

Parameters:

factory -

file -

Returns:

Throws:

IOException

readSequenceVectors

public static <T extends SequenceElement> SequenceVectors<T> readSequenceVectors(@NonNull
                                                                                 SequenceElementFactory<T> factory,
                                                                                 @NonNull
                                                                                 InputStream stream)
                                                                          throws IOException

This method loads previously saved SequenceVectors model from InputStream

Type Parameters:

T -

Parameters:

factory -

stream -

Returns:

Throws:

IOException

writeVocabCache

public static void writeVocabCache(@NonNull
                                   VocabCache<VocabWord> vocabCache,
                                   @NonNull
                                   File file)
                            throws IOException

This method saves vocab cache to provided File. Please note: it saves only vocab content, so it's suitable mostly for BagOfWords/TF-IDF vectorizers

Parameters:

vocabCache -

file -

Throws:

UnsupportedEncodingException

IOException

writeVocabCache

public static void writeVocabCache(@NonNull
                                   VocabCache<VocabWord> vocabCache,
                                   @NonNull
                                   OutputStream stream)
                            throws IOException

This method saves vocab cache to provided OutputStream. Please note: it saves only vocab content, so it's suitable mostly for BagOfWords/TF-IDF vectorizers

Parameters:

vocabCache -

stream -

Throws:

UnsupportedEncodingException

IOException

readVocabCache

public static VocabCache<VocabWord> readVocabCache(@NonNull
                                                   File file)
                                            throws IOException

This method reads vocab cache from provided file. Please note: it reads only vocab content, so it's suitable mostly for BagOfWords/TF-IDF vectorizers

Parameters:

file -

Returns:

Throws:

IOException

readVocabCache

public static VocabCache<VocabWord> readVocabCache(@NonNull
                                                   InputStream stream)
                                            throws IOException

This method reads vocab cache from provided InputStream. Please note: it reads only vocab content, so it's suitable mostly for BagOfWords/TF-IDF vectorizers

Parameters:

stream -

Returns:

Throws:

IOException