org.deeplearning4j.models.embeddings.loader

Class WordVectorSerializer

java.lang.Object

org.deeplearning4j.models.embeddings.loader.WordVectorSerializer

public class WordVectorSerializer
extends Object

Loads word 2 vec models

Author:

Adam Gibson

Constructor Summary

Constructors

Constructor and Description
WordVectorSerializer()

Method Summary

Modifier and Type	Method and Description
static WordVectors	fromPair(Pair<InMemoryLookupTable,VocabCache> pair) Load word vectors from the given pair
static WordVectors	fromTableAndVocab(WeightLookupTable table, VocabCache vocab) Load word vectors for the given vocab and table
static float	getFloat(byte[] b) Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java
static Word2Vec	loadFullModel(String path)
static WordVectors	loadGoogleModel(File modelFile, boolean binary) Loads the google model
static WordVectors	loadGoogleModel(File modelFile, boolean binary, boolean lineBreaks) Loads the Google model.
static Pair<InMemoryLookupTable,VocabCache>	loadTxt(File vectorsFile) Loads an in memory cache from the given path (sets syn0 and the vocab)
static WordVectors	loadTxtVectors(File vectorsFile) Loads an in memory cache from the given path (sets syn0 and the vocab)
static float	readFloat(InputStream is) Read a float from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java
static String	readString(DataInputStream dis) Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java
static void	writeFullModel(Word2Vec vec, String path) Saves full Word2Vec model in the way, that allows model updates without being rebuilt from scratches
static void	writeTsneFormat(Glove vec, org.nd4j.linalg.api.ndarray.INDArray tsne, File csv) Write the tsne format
static void	writeTsneFormat(Word2Vec vec, org.nd4j.linalg.api.ndarray.INDArray tsne, File csv) Write the tsne format
static void	writeWordVectors(InMemoryLookupTable lookupTable, InMemoryLookupCache cache, String path) Writes the word vectors to the given path.
static void	writeWordVectors(WordVectors vec, String path) Writes the word vectors to the given path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WordVectorSerializer

public WordVectorSerializer()

Method Detail

loadGoogleModel

public static WordVectors loadGoogleModel(File modelFile,
                                          boolean binary)
                                   throws IOException

Loads the google model

Parameters:

modelFile - the path to the google model

binary - read from binary file format (if set to true) or from text file format.

Returns:

the loaded model

Throws:

IOException

loadGoogleModel

public static WordVectors loadGoogleModel(File modelFile,
                                          boolean binary,
                                          boolean lineBreaks)
                                   throws IOException

Loads the Google model.

Parameters:

modelFile - the input file

binary - read from binary or text file format

lineBreaks - if true, the input file is expected to terminate each line with a line break. This is typically the case for files created with recent versions of Word2Vec, but not for the downloadable model files.

Returns:

a Word2Vec object

Throws:

IOException

readFloat

public static float readFloat(InputStream is)
                       throws IOException

Read a float from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java

Parameters:

is -

Returns:

Throws:

IOException

getFloat

public static float getFloat(byte[] b)

Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java

Parameters:

b -

Returns:

Throws:

IOException

readString

public static String readString(DataInputStream dis)
                         throws IOException

Read a string from a data input stream Credit to: https://github.com/NLPchina/Word2VEC_java/blob/master/src/com/ansj/vec/Word2VEC.java

Parameters:

dis -

Returns:

Throws:

IOException

writeWordVectors

public static void writeWordVectors(InMemoryLookupTable lookupTable,
                                    InMemoryLookupCache cache,
                                    String path)
                             throws IOException

Writes the word vectors to the given path. Note that this assumes an in memory cache

Parameters:

lookupTable -

cache -

path - the path to write

Throws:

IOException

writeFullModel

public static void writeFullModel(@NonNull
                                  Word2Vec vec,
                                  @NonNull
                                  String path)

Saves full Word2Vec model in the way, that allows model updates without being rebuilt from scratches

Parameters:

vec - - The Word2Vec instance to be saved

path - - the path for json to be saved

loadFullModel

public static Word2Vec loadFullModel(@NonNull
                                     String path)

Parameters:

path - - path to previously stored w2v json model

Returns:

- Word2Vec instance

writeWordVectors

public static void writeWordVectors(WordVectors vec,
                                    String path)
                             throws IOException

Writes the word vectors to the given path. Note that this assumes an in memory cache

Parameters:

vec - the word2vec to write

path - the path to write

Throws:

IOException

fromTableAndVocab

public static WordVectors fromTableAndVocab(WeightLookupTable table,
                                            VocabCache vocab)

Load word vectors for the given vocab and table

Parameters:

table - the weights to use

vocab - the vocab to use

Returns:

wordvectors based on the given parameters

fromPair

public static WordVectors fromPair(Pair<InMemoryLookupTable,VocabCache> pair)

Load word vectors from the given pair

Parameters:

pair - the given pair

Returns:

a read only word vectors impl based on the given lookup table and vocab

loadTxtVectors

public static WordVectors loadTxtVectors(File vectorsFile)
                                  throws FileNotFoundException

Loads an in memory cache from the given path (sets syn0 and the vocab)

Parameters:

vectorsFile - the path of the file to load\

Returns:

Throws:

FileNotFoundException - if the file does not exist

loadTxt

public static Pair<InMemoryLookupTable,VocabCache> loadTxt(File vectorsFile)
                                                    throws FileNotFoundException

Loads an in memory cache from the given path (sets syn0 and the vocab)

Parameters:

vectorsFile - the path of the file to load

Returns:

a Pair holding the lookup table and the vocab cache.

Throws:

FileNotFoundException - if the input file does not exist

writeTsneFormat

public static void writeTsneFormat(Glove vec,
                                   org.nd4j.linalg.api.ndarray.INDArray tsne,
                                   File csv)
                            throws Exception

Write the tsne format

Parameters:

vec - the word vectors to use for labeling

tsne - the tsne array to write

csv - the file to use

Throws:

Exception

writeTsneFormat

public static void writeTsneFormat(Word2Vec vec,
                                   org.nd4j.linalg.api.ndarray.INDArray tsne,
                                   File csv)
                            throws Exception

Write the tsne format

Parameters:

vec - the word vectors to use for labeling

tsne - the tsne array to write

csv - the file to use

Throws:

Exception