sealed abstract case class AnalyzerPipe[F[_]](readerF: Reader => Resource[F, TokenGetter])(implicit F: Async[F])
AnalyzerPipe provides methods to tokenize a possibly very long Stream[F, String] or Stream[F, Byte], such as from a file. When possible, prefer starting with a Stream[F, Byte] and use tokenizeBytes.
Emits a string for every token, as determined by the Analyzer, in the input stream.
Emits a string for every token, as determined by the Analyzer, in the input stream. A space is inserted between each element in the input stream to avoid accidentally combining words. See tokenizeStringsRaw to avoid this behaviour.
Emits a string for every token, as determined by the Analyzer, in the input stream.
Emits a string for every token, as determined by the Analyzer, in the input stream. Becareful, the end of one string will be joined with the beginning of the next in the Analyzer. See tokenizeStrings to automatically intersperse spaces.