csexp/csexp.tokenize/StatefulTokenizer

StatefulTokenizer

object StatefulTokenizer
Companion
class
class Object
trait Matchable
class Any

Value members

Concrete methods

def apply(tokens: Vector[(Int, SToken)]): StatefulTokenizer

Create a StatefulTokenizer from an immediable token stream.

Create a StatefulTokenizer from an immediable token stream.

def apply(inputStream: InputStream): StatefulTokenizer

Create a StatefulTokenizer from an input stream.

Create a StatefulTokenizer from an input stream.