Instance Constructors

1.  new TextReader(titleLengthSize: Int = 50, storeContent: Boolean = false, blockSplit: String = BLOCK_SPLIT_PATTERN, groupBrokenParagraphs: Boolean = false, paragraphSplit: String = DOUBLE_PARAGRAPH_PATTERN, shortLineWordThreshold: Int = 5, maxLineCount: Int = 2000, threshold: Double = 0.1)
   titleLengthSize

   Maximum character length used to determine if a text block qualifies as a title during parsing. The default value is 50.

   storeContent

   Timeout value in seconds for reading remote HTML resources. Applied when fetching content from URLs. By default, it is set to false.

   groupBrokenParagraphs

   Whether to merge fragmented lines into coherent paragraphs using heuristics based on line length and structure. By default, it is set to false.

   paragraphSplit

   Regex pattern used to detect paragraph boundaries when grouping broken paragraphs. Default value is set to double paragraph pattern.

   shortLineWordThreshold

   Maximum word count for a line to be considered 'short' during broken paragraph grouping. Default value is 5.

   maxLineCount

   Maximum number of lines to evaluate when estimating paragraph layout characteristics. Default value is 2000.

   threshold

   Threshold ratio of empty lines used to decide between new line-based or broken-paragraph grouping. Default value is 0.1.

   Example

   val filePath = "./txt-files/simple-text.txt"
   val textReader = new TextReader()
   val textDf = textReader.txt(filePath)

   textDf.show()
   +--------------------+--------------------+
   |                path|                 txt|
   +--------------------+--------------------+
   |file:/content/txt...|[{Title, BIG DATA...|
   +--------------------+--------------------+
   
   textDf.printSchema()
   root
    |-- path: string (nullable = true)
    |-- txt: array (nullable = true)
    |    |-- element: struct (containsNull = true)
    |    |    |-- elementType: string (nullable = true)
    |    |    |-- content: string (nullable = true)
    |    |    |-- metadata: map (nullable = true)
    |    |    |    |-- key: string
    |    |    |    |-- value: string (valueContainsNull = true)

   For more examples please refer to this notebook.