Packages

package audio

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. trait ReadWav2Vec2ForAudioTensorflowModel extends ReadTensorflowModel
  2. trait ReadablePretrainedWav2Vec2ForAudioModel extends ParamsAndFeaturesReadable[Wav2Vec2ForCTC] with HasPretrained[Wav2Vec2ForCTC]
  3. class Wav2Vec2ForCTC extends AnnotatorModel[Wav2Vec2ForCTC] with HasBatchedAnnotateAudio[Wav2Vec2ForCTC] with HasAudioFeatureProperties with WriteTensorflowModel

    Wav2Vec2 Model with a language modeling head on top for Connectionist Temporal Classification (CTC).

    Wav2Vec2 Model with a language modeling head on top for Connectionist Temporal Classification (CTC). Wav2Vec2 was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli.

    The annotator takes audio files and transcribes it as text. The audio needs to be provided pre-processed an array of floats.

    Note that this annotator is currently not supported on Apple Silicon processors such as the M1. This is due to the processor not supporting instructions for XLA.

    Pretrained models can be loaded with pretrained of the companion object:

    val speechToText = Wav2Vec2ForCTC.pretrained()
  .setInputCols("audio_assembler")
  .setOutputCol("text")

    The default model is "asr_wav2vec2_base_960h", if no name is provided.

    For available pretrained models please see the Models Hub.

    To see which models are compatible and how to import them see https://github.com/JohnSnowLabs/spark-nlp/discussions/5669 and to see more extended examples, see Wav2Vec2ForCTCTestSpec.

    Example

    import spark.implicits._
import com.johnsnowlabs.nlp.base._
import com.johnsnowlabs.nlp.annotators._
import com.johnsnowlabs.nlp.annotators.audio.Wav2Vec2ForCTC
import org.apache.spark.ml.Pipeline

val audioAssembler: AudioAssembler = new AudioAssembler()
  .setInputCol("audio_content")
  .setOutputCol("audio_assembler")

val speechToText: Wav2Vec2ForCTC = Wav2Vec2ForCTC
  .pretrained()
  .setInputCols("audio_assembler")
  .setOutputCol("text")

val pipeline: Pipeline = new Pipeline().setStages(Array(audioAssembler, speechToText))

val bufferedSource =
  scala.io.Source.fromFile("src/test/resources/audio/csv/audi_floats.csv")

val rawFloats = bufferedSource
  .getLines()
  .map(_.split(",").head.trim.toFloat)
  .toArray
bufferedSource.close

val processedAudioFloats = Seq(rawFloats).toDF("audio_content")

val result = pipeline.fit(processedAudioFloats).transform(processedAudioFloats)
result.select("text.result").show(truncate = false)
+------------------------------------------------------------------------------------------+
|result                                                                                    |
+------------------------------------------------------------------------------------------+
|[MISTER QUILTER IS THE APOSTLE OF THE MIDLE CLASES AND WE ARE GLAD TO WELCOME HIS GOSPEL ]|
+------------------------------------------------------------------------------------------+

Value Members

  1. object Wav2Vec2ForCTC extends ReadablePretrainedWav2Vec2ForAudioModel with ReadWav2Vec2ForAudioTensorflowModel with Serializable

    This is the companion object of Wav2Vec2ForCTC.

    This is the companion object of Wav2Vec2ForCTC. Please refer to that class for the documentation.

Ungrouped