info.debatty.java.stringsimilarity

Class KShingling

java.lang.Object

info.debatty.java.stringsimilarity.KShingling

public class KShingling
extends Object

k-shingling is the operation of transforming a string (or text document) into a set of n-grams, which can be used to measure the similarity between two strings or documents. Generally speaking, a n-gram is any sequence of k tokens. We use here the definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive Datasets", Cambridge University Press: Multiple subsequent spaces are replaced by a single space, and a k-gram is a sequence of k characters.

Author:

Thibault Debatty http://www.debatty.info

Field Summary

Fields

Modifier and Type	Field and Description
protected int	k

Constructor Summary

Constructors

Constructor and Description
KShingling() k-shingling is the operation of transforming a string (or text document) into a set of n-grams, which can be used to measure the similarity between two strings or documents.
KShingling(int k)

Method Summary

Methods

Modifier and Type	Method and Description
int	getK()
StringProfile	getProfile(String s) Compute and returns the profile of string s The profiles of different strings can be used to compute cosine similarity or qgram distance.
StringSet	getSet(String s)
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

k

protected int k

Constructor Detail

KShingling

public KShingling()

k-shingling is the operation of transforming a string (or text document) into a set of n-grams, which can be used to measure the similarity between two strings or documents. Generally speaking, a k-gram is any sequence of k tokens. We use here the definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive Datasets", Cambridge University Press: Multiple subsequent spaces are replaced by a single space, and a k-gram is a sequence of k characters. Default value of k is 5 (recommended for emails). A good rule of thumb is to imagine that there are only 20 characters and estimate the number of k-shingles as 20^k. For large documents, such as research articles, k = 9 is considered a safe choice.

KShingling

public KShingling(int k)

Method Detail

main

public static void main(String[] args)

getK

public int getK()

getProfile

public StringProfile getProfile(String s)

Compute and returns the profile of string s The profiles of different strings can be used to compute cosine similarity or qgram distance.

Parameters:

s -

Returns:

getSet

public StringSet getSet(String s)