LuceneRDD

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. lazy val Config: Config

   

   Definition Classes

   Configurable

5. val IndexAnalyzerConfigName: Option[String]

   

   Attributes

   protected

   Definition Classes

   AnalyzerConfigurable

6. val LuceneSimilarityConfigValue: Option[String]

   

   Attributes

   protected

   Definition Classes

   SimilarityConfigurable

7. val QueryAnalyzerConfigName: Option[String]

   

   Attributes

   protected

   Definition Classes

   AnalyzerConfigurable

8. def apply(dataFrame: DataFrame): LuceneRDD[Row]

   

   Constructor with default index, query analyzers and Lucene similarity

   Constructor with default index, query analyzers and Lucene similarity

   dataFrame

   Input DataFrame

9. def apply(dataFrame: DataFrame, indexAnalyzer: String, queryAnalyzer: String, similarity: String): LuceneRDD[Row]

   

10. def apply(dataFrame: DataFrame, indexAnalyzer: String, queryAnalyzer: String, similarity: String, indexAnalyzerPerField: Map[String, String], queryAnalyzerPerField: Map[String, String]): LuceneRDD[Row]

    

    Instantiate a LuceneRDD from a DataFrame

    Instantiate a LuceneRDD from a DataFrame

    dataFrame

    Spark DataFrame

11. def apply[T](elems: Iterable[T])(implicit arg0: ClassTag[T], sc: SparkContext, conv: (T) ⇒ Document): LuceneRDD[T]

    

12. def apply[T](elems: RDD[T])(implicit arg0: ClassTag[T], conv: (T) ⇒ Document): LuceneRDD[T]

    

13. def apply[T](elems: Iterable[T], indexAnalyzer: String, queryAnalyzer: String, similarity: String, indexAnalyzerPerField: Map[String, String], queryAnalyzerPerField: Map[String, String])(implicit arg0: ClassTag[T], sc: SparkContext, conv: (T) ⇒ Document): LuceneRDD[T]

    

    Instantiate a LuceneRDD with an iterable

    Instantiate a LuceneRDD with an iterable

    T

    Input type

    elems

    Elements to index

    indexAnalyzer

    Index analyzer name

    queryAnalyzer

    Query analyzer name

    similarity

    Lucene scoring similarity, i.e., BM25 or TF-IDF

    indexAnalyzerPerField

    Lucene Analyzer per field (indexing time), default empty

    queryAnalyzerPerField

    Lucene Analyzer per field (query time), default empty

    sc

    Spark Context

14. def apply[T](elems: RDD[T], indexAnalyzer: String, queryAnalyzer: String, similarity: String, indexAnalyzerPerField: Map[String, String], queryAnalyzerPerField: Map[String, String])(implicit arg0: ClassTag[T], conv: (T) ⇒ Document): LuceneRDD[T]

    

    Instantiate a LuceneRDD given an RDD[T]

    Instantiate a LuceneRDD given an RDD[T]

    T

    Generic type

    elems

    RDD of type T

    indexAnalyzer

    Index analyzer name

    queryAnalyzer

    Query analyzer name

    similarity

    Lucene scoring similarity, i.e., BM25 or TF-IDF

    indexAnalyzerPerField

    Lucene Analyzer per field (indexing time), default empty

    queryAnalyzerPerField

    Lucene Analyzer per field (query time), default empty

15. final def asInstanceOf[T0]: T0

    

    Definition Classes

    Any

16. def blockDedup(entities: DataFrame, rowToQuery: (Row) ⇒ Query, blockingColumns: Array[String], topK: Int = 3, luceneRDDParams: LuceneRDDParams = LuceneRDDParams()): RDD[(Row, Array[Row])]

    

    Deduplication via blocking

    Deduplication via blocking

    entities

    Entities DataFrame to deduplicate

    rowToQuery

    Function that maps Row to Lucene Query

    blockingColumns

    Columns on which exact match is required

    topK

    Number of top-K query results

    luceneRDDParams

    Parameters for index-time and query-time analysis

17. def blockEntityLinkage(queries: DataFrame, entities: DataFrame, rowToQuery: (Row) ⇒ Query, queryPartColumns: Array[String], entityPartColumns: Array[String], topK: Int = 3, luceneRDDParams: LuceneRDDParams = LuceneRDDParams()): RDD[(Row, Array[Row])]

    

    Entity linkage between two DataFrame by blocking / filtering on one or more columns.

    Entity linkage between two DataFrame by blocking / filtering on one or more columns.

    queries

    Queries / entities to be linked with @corpus

    entities

    DataFrame of entities to be linked with queries parameter

    rowToQuery

    Function[Row, Query] that converts Row to a Lucene Query

    queryPartColumns

    List of query columns for HashPartitioner

    entityPartColumns

    List of entity columns for HashPartitioner

    topK

    Number of linked results

    luceneRDDParams

    Parameters for index and query time analysis

    returns

    Returns top-k linked results as RDD of Tuple2 where _1 is query and _2 is top-k linked results as SparkScoreDoc.

18. def clone(): AnyRef

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

19. final def eq(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

20. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

21. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

22. def getAnalyzer(analyzerName: Option[String]): Analyzer

    

    Attributes

    protected

    Definition Classes

    AnalyzerConfigurable

23. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

24. def getOrElseClassic(): String

    

    Attributes

    protected

    Definition Classes

    SimilarityConfigurable

25. def getOrElseEn(analyzerName: Option[String]): String

    

    Get the configured analyzers or fallback to English

    Get the configured analyzers or fallback to English

    Attributes

    protected

    Definition Classes

    AnalyzerConfigurable

26. def getSimilarity(similarityName: Option[String]): Similarity

    

    Attributes

    protected

    Definition Classes

    SimilarityConfigurable

27. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

28. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

29. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

30. def isTraceEnabled(): Boolean

    

    Attributes

    protected

    Definition Classes

    Logging

31. def log: Logger

    

    Attributes

    protected

    Definition Classes

    Logging

32. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

33. def logDebug(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

34. def logError(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

35. def logError(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

36. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

37. def logInfo(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

38. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

39. def logTrace(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

40. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

41. def logWarning(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

42. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

43. final def notify(): Unit

    

    Definition Classes

    AnyRef

44. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

45. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

46. def toString(): String

    

    Definition Classes

    AnyRef → Any

47. def version(): Map[String, Any]

    

    Return project information, i.e., version number, build time etc

    Return project information, i.e., version number, build time etc

    Definition Classes

    Versionable

48. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

49. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

50. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )