info.debatty.java.stringsimilarity

Class KShingling

java.lang.Object

info.debatty.java.stringsimilarity.KShingling

public class KShingling
extends Object

k-shingling is the operation of transforming a string (or text document) into a set of n-grams, which can be used to measure the similarity between two strings or documents. Generally speaking, a n-gram is any sequence of k tokens. We use here the definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive Datasets", Cambridge University Press: Multiple subsequent spaces are replaced by a single space, and a k-gram is a sequence of k characters.

Author:

Thibault Debatty http://www.debatty.info

Field Summary

Fields

Modifier and Type	Field and Description
protected int	k

Constructor Summary

Constructors

Constructor and Description
KShingling() k-shingling is the operation of transforming a string (or text document) into a set of n-grams, which can be used to measure the similarity between two strings or documents.
KShingling(int k)

Method Summary

Methods

Modifier and Type	Method and Description
protected int[]	getArrayProfile(String s) Compute and return the profile of s, as defined by Ukkonen "Approximate string-matching with q-grams and maximal matches".
int	getDimension() Return the number of different n-grams (k-shingles) found by this k-shingling instance.
int	getK()
StringProfile	getProfile(String s) Compute and returns the profile of string s The profiles of different strings can be used to compute cosine similarity or qgram distance.
StringSet	getSet(String s)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

k

protected int k

Constructor Detail

KShingling

public KShingling()

k-shingling is the operation of transforming a string (or text document) into a set of n-grams, which can be used to measure the similarity between two strings or documents. Generally speaking, a k-gram is any sequence of k tokens. We use here the definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive Datasets", Cambridge University Press: Multiple subsequent spaces are replaced by a single space, and a k-gram is a sequence of k characters. Default value of k is 5 (recommended for emails). A good rule of thumb is to imagine that there are only 20 characters and estimate the number of k-shingles as 20^k. For large documents, such as research articles, k = 9 is considered a safe choice.

KShingling

public KShingling(int k)

Method Detail

getK

public int getK()

getArrayProfile

protected int[] getArrayProfile(String s)

Compute and return the profile of s, as defined by Ukkonen "Approximate string-matching with q-grams and maximal matches". https://www.cs.helsinki.fi/u/ukkonen/TCS92.pdf The profile is the number of occurrences of k-shingles, and is used to compute q-gram similarity, Jaccard index, etc. E.g. if s = ABCAB and k =2 The KShingling object will store the dictionary of n-grams: {AB BC CA} and the profile will be [2 1 1] Attention: the space requirement of a single profile can be as large as 20^k x 4Bytes (sizeof(int)) Computation cost is O(n)

Parameters:

s -

Returns:

the profile of this string as an array of integers

getProfile

public StringProfile getProfile(String s)

Compute and returns the profile of string s The profiles of different strings can be used to compute cosine similarity or qgram distance.

Parameters:

s -

Returns:

the profile of this string, as a StringProfile object

getSet

public StringSet getSet(String s)

getDimension

public int getDimension()

Return the number of different n-grams (k-shingles) found by this k-shingling instance.

Returns:

the total number of different n-grams found by this k-shingling instance