Packages

object impl

Attributes
protected[kafka]
Source
KafkaClient.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. impl
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. sealed trait PartitionPublishConnection[F[_]] extends AnyRef
  2. sealed trait Publisher[F[_]] extends AnyRef

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. val consumerBrokerId: @@[Int, Broker]
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. def fetchBrokerConnection[F[_]](brokerConnection: (BrokerAddress, Option[FiniteDuration]) => Pipe[F, RequestMessage, ResponseMessage], version: protocol.kafka.ProtocolVersion.Value, clientId: String)(address: BrokerAddress, readTimeout: FiniteDuration)(implicit arg0: Async[F]): Pipe[F, FetchRequest, (FetchRequest, FetchResponse)]

    Augments connection to broker to FetchRequest/FetchResponse pattern.

    Augments connection to broker to FetchRequest/FetchResponse pattern.

    Apart of supplying fetch fith proper details, this echoes original request with every fetch

    brokerConnection

    Connection to broker

    version

    protocol version

    clientId

    Id of client

    address

    Address of broker.

  10. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. def leaderFor[F[_]](requestMeta: (BrokerAddress, MetadataRequest) => F[MetadataResponse], seed: Seq[BrokerAddress])(topicId: @@[String, TopicName], partition: @@[Int, PartitionId])(implicit arg0: Sync[F]): F[Option[BrokerAddress]]

    Queries all supplied seeds for first leader and then returns that leader.

    Queries all supplied seeds for first leader and then returns that leader. Returns None if no seed replied with leader for that partition

    requestMeta

    A function that requests signle metadata

    seed

    A seed of brokers

    topicId

    Id of topic

    partition

    Id of partition

  14. def leadersDiscrete[F[_]](metaRequestConnection: (BrokerAddress) => Pipe[F, MetadataRequest, MetadataResponse], seed: Seq[BrokerAddress], delay: FiniteDuration, topics: Vector[@@[String, TopicName]])(implicit arg0: Async[F], arg1: Logger[F]): Stream[F, Map[(@@[String, TopicName], @@[Int, PartitionId]), BrokerAddress]]

    Creates discrete signal of leaders that is queried from periodical query of metadata from brokers.

    Creates discrete signal of leaders that is queried from periodical query of metadata from brokers. This will query supplied seeds in order given and then with first seed that succeeds this will compile map of metadata that is emitted.

    While this stream is consumed, this will keep connection with very first broker that have answered this.

    If there is no broker available to server metadata request, this will fail as NoBrokerAvailable

    If the broker from which metadata are queried will fail, this will try next broker in supplied seed.

    metaRequestConnection

    connection to create against the given broker

    seed

    Seed of ensemble to use to query metadata from

    delay

    Delay to refresh new metadata from last known good broker

    topics

    If nonempty, filters topic for which the metadata are queried

  15. def messagesFromResult(protocol: protocol.kafka.ProtocolVersion.Value, result: PartitionFetchResult): Vector[TopicMessage]

    Because result of fetch can retrieve messages in compressed and nested forms, This decomposes result to simple vector by traversing through the nested message results.

    Because result of fetch can retrieve messages in compressed and nested forms, This decomposes result to simple vector by traversing through the nested message results.

    result

    Result from teh fetch

  16. def metadataConnection[F[_]](brokerConnection: (BrokerAddress) => Pipe[F, RequestMessage, ResponseMessage], version: protocol.kafka.ProtocolVersion.Value, clientId: String)(address: BrokerAddress)(implicit arg0: Sync[F]): Pipe[F, MetadataRequest, MetadataResponse]

    Creates connection that allows to submit offset Requests.

  17. def mkClient[F[_]](ensemble: Set[BrokerAddress], publishConnection: (@@[String, TopicName], @@[Int, PartitionId]) => F[PartitionPublishConnection[F]], fetchMetadata: (BrokerAddress, MetadataRequest) => F[MetadataResponse], fetchConnection: (BrokerAddress, FiniteDuration) => Pipe[F, FetchRequest, (FetchRequest, FetchResponse)], offsetConnection: (BrokerAddress) => Pipe[F, OffsetsRequest, OffsetResponse], metaRequestConnection: (BrokerAddress) => Pipe[F, MetadataRequest, MetadataResponse], queryOffsetTimeout: FiniteDuration, protocol: protocol.kafka.ProtocolVersion.Value)(implicit arg0: Logger[F], arg1: Async[F]): F[(KafkaClient[F], F[Unit])]

    Creates a client and F that cleans up lients resources.

    Creates a client and F that cleans up lients resources.

    ensemble

    Initial kafka clients to connect to

    fetchMetadata

    A function fo fetch metadata from client specified provided address and signal of state.

  18. def mkPublishers[F[_]](createPublisher: (@@[String, TopicName], @@[Int, PartitionId]) => F[PartitionPublishConnection[F]])(implicit arg0: Async[F]): F[Publisher[F]]

    Produces a publisher that for every publishes partition-topic will spawn PartitionPublishConnection.

    Produces a publisher that for every publishes partition-topic will spawn PartitionPublishConnection. That connection is handling then all publish requests for given partition. Connections are cached are re-used on next publish.

    createPublisher

    Function to create single publish connection to given partition.

  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  22. def offsetConnection[F[_]](brokerConnection: (BrokerAddress) => Pipe[F, RequestMessage, ResponseMessage], version: protocol.kafka.ProtocolVersion.Value, clientId: String)(address: BrokerAddress)(implicit arg0: Async[F]): Pipe[F, OffsetsRequest, OffsetResponse]

    Creates connection that allows to submit offset Requests.

  23. def publishLeaderConnection[F[_]](connection: (BrokerAddress) => Pipe[F, RequestMessage, ResponseMessage], protocol: protocol.kafka.ProtocolVersion.Value, clientId: String, getLeaderFor: (@@[String, TopicName], @@[Int, PartitionId]) => F[Option[BrokerAddress]], getLeaderDelay: FiniteDuration, topicId: @@[String, TopicName], partition: @@[Int, PartitionId], publishFailMaxAttempts: Int = 3)(implicit arg0: Logger[F], arg1: Async[F]): F[PartitionPublishConnection[F]]

    With every leader for each topic and partition active this keeps connection open.

    With every leader for each topic and partition active this keeps connection open. Connection is open once the topic and partition will get first produce request to serve.

    connection

    Function handling connection to Kafka Broker

    protocol

    Protocol

    clientId

    Id of the client

    getLeaderFor

    Returns a leader for supplied topic and partition

    getLeaderDelay

    Wait that much time to retry for new leader if leader is not known

    topicId

    Id of the topic

    partition

    Id of the partition

    publishFailMaxAttempts

    Max number of attempts when publish to a topic fails, this is usually caused by kafka leader change.

  24. def queryOffsetRange[F[_]](getLeader: (@@[String, TopicName], @@[Int, PartitionId]) => F[Option[BrokerAddress]], brokerOffsetConnection: (BrokerAddress) => Pipe[F, OffsetsRequest, OffsetResponse], maxTimeForQuery: FiniteDuration)(topicId: @@[String, TopicName], partition: @@[Int, PartitionId])(implicit arg0: Async[F]): F[(@@[Long, Offset], @@[Long, Offset])]

    Queries offsets for given topic and partition.

    Queries offsets for given topic and partition. Returns offset of first message kept (head) and offset of next message that will arrive to topic. When numbers are equal, then the topic does not include any messages at all.

    getLeader

    Queries leader for the partition supplied

    brokerOffsetConnection

    A function to create connection to broker to send // receive OffsetRequests

    topicId

    Id of the topic

    partition

    Id of the partition

  25. def requestReplyBroker[F[_], I <: Request, O <: Response](f: (BrokerAddress) => Pipe[F, RequestMessage, ResponseMessage], protocol: protocol.kafka.ProtocolVersion.Value, clientId: String)(address: BrokerAddress, input: I)(implicit arg0: Async[F], arg1: Typeable[O]): F[O]

    Request // reply communication to broker.

    Request // reply communication to broker. This sends one message I and expect one result O

  26. def subscribePartition[F[_]](topicId: @@[String, TopicName], partition: @@[Int, PartitionId], firstOffset: @@[Long, Offset], prefetch: Boolean, minChunkByteSize: Int, maxChunkByteSize: Int, maxWaitTime: FiniteDuration, protocol: protocol.kafka.ProtocolVersion.Value, fetchConnection: (BrokerAddress, FiniteDuration) => Pipe[F, FetchRequest, (FetchRequest, FetchResponse)], getLeader: (@@[String, TopicName], @@[Int, PartitionId]) => F[Option[BrokerAddress]], queryOffsetRange: (@@[String, TopicName], @@[Int, PartitionId]) => F[(@@[Long, Offset], @@[Long, Offset])], leaderFailureTimeout: FiniteDuration, leaderFailureMaxAttempts: Int)(implicit arg0: Async[F], arg1: Logger[F]): Stream[F, TopicMessage]

    Subscribes to given partition and topic starting offet supplied.

    Subscribes to given partition and topic starting offet supplied. Each subscription creates single connection to isr.

    topicId

    Id of the topic

    partition

    Partition id

    firstOffset

    Offset from where to start (including this one). -1 designated start with very first message published (tail)

    getLeader

    Function to query for available leader

    queryOffsetRange

    Queries range of offset kept for given topic. First is head (oldest message offset) second is tail (offset of the message not yet in topic)

  27. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  28. def toString(): String
    Definition Classes
    AnyRef → Any
  29. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  30. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  31. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AnyRef

Inherited from Any

Ungrouped