TapasForQuestionAnswering is an implementation of TaPas - a BERT-based model specifically designed for answering questions about tabular data. It takes TABLE and DOCUMENT annotations as input and tries to answer the questions in the document by using the data from the table. The model is based in BertForQuestionAnswering and shares all its parameters with it.

Pretrained models can be loaded with pretrained of the companion object:

val tapas = TapasForQuestionAnswering.pretrained()
  .setInputCols(Array("document_question", "table"))
  .setOutputCol("answer")

The default model is "table_qa_tapas_base_finetuned_wtq", if no name is provided.

For available pretrained models please see the Models Hub.

Example

import spark.implicits._
import com.johnsnowlabs.nlp.base._
import com.johnsnowlabs.nlp.annotator._
import org.apache.spark.ml.Pipeline

 val questions =
   """
    |Who earns 100,000,000?
    |Who has more money?
    |How old are they?
    |""".stripMargin.trim

 val jsonData =
   """
    |{
    | "header": ["name", "money", "age"],
    | "rows": [
    |   ["Donald Trump", "$100,000,000", "75"],
    |   ["Elon Musk", "$20,000,000,000,000", "55"]
    | ]
    |}
    |""".stripMargin.trim

 val data = Seq((jsonData, questions))
  .toDF("json_table", "questions")
  .repartition(1)

val docAssembler = new MultiDocumentAssembler()
  .setInputCols("json_table", "questions")
  .setOutputCols("document_table", "document_questions")

val sentenceDetector = SentenceDetectorDLModel
  .pretrained()
  .setInputCols(Array("document_questions"))
  .setOutputCol("question")

val tableAssembler = new TableAssembler()
  .setInputFormat("json")
  .setInputCols(Array("document_table"))
  .setOutputCol("table")

val tapas = TapasForQuestionAnswering
  .pretrained()
  .setInputCols(Array("question", "table"))
  .setOutputCol("answer")

val pipeline = new Pipeline()
  .setStages(
    Array(
      docAssembler,
      sentenceDetector,
      tableAssembler,
       tapas))

val pipelineModel = pipeline.fit(data)
val result = pipeline.fit(data).transform(data)

result
  .selectExpr("explode(answer) as answer")
  .selectExpr(
    "answer.metadata.question",
    "answer.result")

+-----------------------+----------------------------------------+
|question               |result                                  |
+-----------------------+----------------------------------------+
|Who earns 100,000,000? |Donald Trump                            |
|Who has more money?    |Elon Musk                               |
|How much they all earn?|COUNT($100,000,000, $20,000,000,000,000)|
|How old are they?      |AVERAGE(75, 55)                         |
+-----------------------+----------------------------------------+