Packages

package root

Definition Classes: root

package com

Definition Classes: root

package github

Definition Classes: com

package garyaiki

Definition Classes: github

package dendrites

Utility functions for case classes and Properties files

ccToMap

Case class fields to map of field names and values

val kvMap = ccToMap(cc)
 kvMap foreach {
  case (key, value) => gRecord.put(key, value)
}

isElementEqual

Is case class field at index a specified type

system.actorOf(props) ! GetCustomerAccountBalances(2, Set(Checking, Savings, MoneyMarket))
  receiveOne(2.seconds) match {
   case result: IndexedSeq[Product] ⇒ {
    assert(isElementEqual(result(0), 0, Checking))
    assert(isElementEqual(result(1), 0, Savings))
    assert(isElementEqual(result(2), 0, MoneyMarket))
   }
   case result ⇒ assert(false, s"Expect 3 AccountTypes, got $result")
  }

loadProperties

Load Properties file from classpath

val prop: Properties = loadProperties("kafkaProducer.properties")

Definition Classes: garyaiki

package aggregator

Aggregate functions

mean

Mean of sequence of Numeric

val doubles = List(1.0, 2.1, 3.2)
val m: Either[String,Double] = mean(doubles)

Definition Classes: dendrites

package algebird

Aggregation functions for Twitter Algebird.

Algebird provides implicit implementations of common types which are imported here. Class extraction methods in com.github.garyaiki.dendrites can be used to extract a field from case classes and tuples.

AveragedValue

Average a Sequence of values com.github.garyaiki.dendrites.algebird.AveragedSpec

val bigDecimals: Seq[BigDecimal]
val avg0 = avg(bigDecimals)

Average a sequence of AveragedValues

val bigDecimals2: Seq[BigDecimal]
val avg1 = avg(bigDecimals2)
val avgs = Vector[AveragedValue](avg0, avg1)
val avgSum = sumAverageValues(avgs)

BloomFilter

Fast find if a word is in a dictionary OSX and Linux have dictionaries you can use to create BloomFilters com.github.garyaiki.dendrites.fixtures.SysProcessUtils for Paths to properNames, connectives, words and functions to read them com.github.garyaiki.dendrites.algebird.fixtures.BloomFilterBuilder for creation of BloomFilters for these dictionaries and select test words for them.

Create a BloomFilter for OSX words dictionary com.github.garyaiki.dendrites.algebird.BloomFilterSpec

val falsePositivepProb: Double = 0.01
val words = readWords(wordsPath)
val wordsBF = createBF(words, fpProb)

Is word in BloomFilter

val falsePositivepProb: Double = 0.01
val word = "path"
val inDict = wordsBF.contains(word).isTrue

Is false positive rate acceptable

val falsePositivepProb: Double = 0.01
val wordsFalseWords: IndexedSeq[String]
val falsePositives = for {
i <- wordsFalseWords
if wordsBF.contains(i).isTrue
} yield i
val acceptable = falsePositives.size < words.size * fpProb

CountMinSketch

Test data is IP addresses repeated a random number of times com.github.garyaiki.dendrites.algebird.CountMinSketchSpec Estimate total number of elements seen so far com.github.garyaiki.dendrites.fixtures.InetAddressesBuilder

val addrs = inetAddresses(ipRange)
val longZips = inetToLongZip(addrs)
val longs = testLongs(longZips)
implicit val m = createCMSMonoid[Long]()
val cms = createCountMinSketch(longs)
val estimatedCount = cms.totalCount

Estimate count of elements with the same value as the one selected

val estFreq = cms.frequency(longZips(5))

Sum a Sequence of CountMinSketch then estimate combined total number of elements

val cms1 = createCountMinSketch(longs)
val cmss = Vector(cms, cms1)
val cmsSum = sumCountMinSketch(cmss)
val estimatedCount = cmsSum.totalCount

From a Sequence of CountMinSketch estimate count of elements with the indexed same value

val estFreq = cmsSum.frequency(longZips(5))

DecayedValue

Test data is a sine wave with a value for each of 360 degrees with a corresponding time value. The idea is a rising and falling value over a year com.github.garyaiki.dendrites.fixtures.TrigUtils

Moving average from the initial value to specified index com.github.garyaiki.dendrites.algebird.DecayedValueSpec

val sines = genSineWave(100, 0 to 360)
val days = Range.Double(0.0, 361.0, 1.0)
val sinesZip = sines.zip(days)
val decayedValues = toDecayedValues(sinesZip, 10.0, None)
val avgAt90 = decayedValues(90).average(10.0)

Moving average from specified index to specified index

val avg80to90 = decayedValues(90).averageFrom(10.0, 80.0, 90.0)

HyperLogLog

Create a HLL from a sequence of Int com.github.garyaiki.dendrites.algebird.HyperLogLogSpec

implicit val ag = HyperLogLogAggregator(12)
val ints: Seq[Int]
val hll = createHLL(ints)

Create a sequence of HLL

val ints2: Seq[Int]
val hll2 = createHLL(ints2)
val hlls = Vector(hll, hll2)

Create a HLL from a sequence of Long

val longs: Seq[Long]
val hll = createHLL(longs)

Create a sequence of HLL

val longs2: Seq[Long]
val hll2 = createHLL(longs2)
val hlls = Vector(hll, hll2)

Sum a Sequence of HLL and estimate total size

val sum = hlls.reduce(_ + _)
val size = sum.estimatedSize

Create a sequence of Approximate HHL approximate Map a sequence of HLL to a sequence of Approximate

val hlls = Vector(hll, hll2)
val approxs = mapHLL2Approximate(hlls)

Sum a Sequence of Approximate and estimate total size

val sum = approxs.reduce(_ + _)

QTree

Build QTree from a Sequence com.github.garyaiki.dendrites.algebird.fixtures.QTreeBuilder

val level = 5
implicit val qtBDSemigroup = new QTreeSemigroup[BigDecimal](level)
val qtBD = buildQTree[BigDecimal](bigDecimals)

Get its InterQuartileMean com.github.garyaiki.dendrites.algebird.QTreeSpec

val iqm = qtBD.interQuartileMean
// iqm._1 lower bound
// iqm._2 upper bound

Sum a Sequence of QTrees to a QTree

val qTrees = Vector(qtBD, qtBD2)
val sumQTree = sumQTrees(qTrees)

Functor

Map elements of a sequence.

def wrapMax[Int](x: Int) = Max(x)
val wm = SeqFunctor.map[Int, Max[Int]](List(1,2,3,4))(wrapMax)

val bigDecimals: Seq[BigDecimal]
val negBigDecimals = SeqFunctor.map[BigDecimal, BigDecimal](bigDecimals)(negate)
val invertedBigDecimals = SeqFunctor.map[BigDecimal, BigDecimal](bigDecimals)(inverse)

Map the mapped elements of a sequence: f() andThen g().

val bigDecimals: Seq[BigDecimal]
val invertedNegBigDecimals = andThen[BigDecimal, BigDecimal, BigDecimal](ap)( inverse)( negate)

Max Min

For Sequence types that have a Semigroup, Monoid and Ordering Get Max element of a sequence. com.github.garyaiki.dendrites.algebird.MaxSpec

val iqm = qtBD.interQuartileMean

val bigDecimals: Seq[BigDecimal]
val max = max(bigDecimals)
val optBigDecs: [Option[BigDecimal]]
val max2 = max(optBigDecs.flatten)
val eithBigInts = Seq[Either[String, BigInt]]
val max3 = max(filterRight(eithBigInts)

Get Min element of a sequence. com.github.garyaiki.dendrites.algebird.MinSpec

val bigDecimals: Seq[BigDecimal]
val min = min(bigDecimals)
val optBigDecs: [Option[BigDecimal]]
val min2 = min(optBigDecs.flatten)
val eithBigInts = Seq[Either[String, BigInt]]
val min3 = min(filterRight(eithBigInts)

Definition Classes: dendrites

package avro

Avro serializer/deserializer functions

Load Avro Schema from file

val schema = loadSchema(filename)

Serialize case class to bytearray

val bytes = ccToByteArray(schema, GetAccountBalances(1L))
val record = new ProducerRecord[String, Array[Byte]](topic, key, bytes)
val rm: RecordMetadata = producer.send(record).get()

Map bytearray to Avro GenericRecord

new GraphStageLogic(shape) {
setHandler(in, new InHandler {
  override def onPush(): Unit = {
    val bytes = grab(in)
    val record = byteArrayToGenericRecord(schema, bytes)
    push(out, f(record))
  }
})

Definition Classes: dendrites

package avro4s

Extend Avro4sOps to serialize/deserialize case class

object Avro4sShoppingCartCmd  extends Avro4sOps[ShoppingCartCmd] {
  implicit val schemaFor = SchemaFor[ShoppingCartCmd]
  implicit val toRecord = ToRecord[ShoppingCartCmd]
  implicit val fromRecord = FromRecord[ShoppingCartCmd]

implement case class serializer

def toBytes(caseClass: ShoppingCartCmd): Array[Byte] = {
  val baos = new ByteArrayOutputStream()
  val output = AvroOutputStream.binary[ShoppingCartCmd](baos)
  output.write(caseClass)
  output.close()
  baos.toByteArray
}

implement deserializer

def toCaseClass(bytes: Array[Byte]): ShoppingCartCmd = {
  val in = new ByteArrayInputStream(bytes)
  val input = AvroInputStream.binary[ShoppingCartCmd](in)
  val result = input.iterator.toSeq
  result(0)
}

Definition Classes: dendrites

package cassandra

Common functions for Cassandra Java Driver

Create single node Cassandra Cluster.

val config = ConfigFactory.load()
val ipAddress = config.getString("dendrites.cassandra.ipAddress")
val cluster = createCluster(ipAddress)

Create Cluster with multiple host nodes and a RetryPolicy

val addresses = myConfig.getInetAddresses()
val retryPolicy = new LoggingRetryPolicy(DefaultRetryPolicy.INSTANCE)
cluster = createCluster(addresses, retryPolicy)

Log cluster's metadata.

logMetadata(cluster)

Bind Cassandra's QueryLogger to cluster

registerQueryLogger(cluster)

Create a simple LoadBalancing policy

val myConfig = PlaylistSongConfig
val lbp = createLoadBalancingPolicy(myConfig.localDataCenter)

Initialize LoadBalancingPolicy

initLoadBalancingPolicy(cluster, lbp)

Connect to Cassandra. Return a Session which is thread safe and may last app's lifetime

session = connect(cluster)

Create a Keyspace

val schema = myConfig.keySpace
val strategy = myConfig.replicationStrategy
val createSchemaRS = createSchema(session, schema, strategy, 3)

Create a PreparedStatement to return all rows of a table

val plPreStmt = selectAll(session, schema, Playlists.table)

Asynchronously execute a BoundStatement

val selAllRS = executeBoundStmt(session, new BoundStatement(plPreStmt))

Get every row in a result set

val allRows = getAllRows(selAllRS)

Drop schema

dropSchema(session, schema)

Asynchronously close Session and Cluster. Turns Cassandra's Java Futures into Scala Futures

close(session, cluster)

Definition Classes: dendrites

package concurrent

Functions for concurrency

Transform a Java Guava ListenableFuture into a Scala Future

val sessCloseF = cassandraSession.closeAsync()
val scalaSessF = listenableFutureToScala[Unit](sessCloseF.asInstanceOf[ListenableFuture[Unit]])
scalaSessF onComplete {
  case Success(x) => logger.debug("session closed")
  case Failure(t) => logger.error(t, "session closed failed {}", t.getMessage())
}

Calculate exponential backoff delay Constant params can be passed to first argument list on startup

val min = config getInt("dendrites.timer.min-backoff")
val minDuration = FiniteDuration(min, MILLISECONDS)
val max = config getInt("dendrites.timer.max-backoff")
val maxDuration = FiniteDuration(max, MILLISECONDS)
val randomFactor = config getDouble("dendrites.timer.randomFactor")
val curriedDelay = calculateDelay(minDuration, maxDuration, randomFactor) _
//Second arg list can be curried
val curriedDelay = consumerConfig.curriedDelay
val duration = curriedDelay(retries)
if(duration < maxDuration) {
  waitForTimer = true
  scheduleOnce(None, duration)
} else {
  failStage(e) // too many retries
}

Definition Classes: dendrites

package cqrs

Definition Classes: dendrites

package examples

Definition Classes: dendrites

package account

Functions for AccountType, the case objects that distinguish them and AccountBallances, their optional list of account ids and balances

Filter for AccountType

if(isAccountType(MoneyMarket) == true)

Filter for desired balances types

val cb = CheckingAccountBalances[BigDecimal](Some(List((3,3000.3), (33,3300.33), (333,3.33))))
if(isAccountBalances(cb) == true)

Filter for desired types in an Option[List[(Long, A)]]

val filtered = accountBalances.filter(isAccBalances)

Extract the 'A' balance values from Option[List[(Long, A)]]

def extractBalancesFlow[A]: Flow[Seq[AnyRef], Seq[A], NotUsed] =
    Flow[Seq[AnyRef]].map(extractBalancesVals[A])

Extract balances from Seq[AccountBalances]

def exf: Flow[Seq[AnyRef], Seq[BigDecimal], NotUsed] = Flow[Seq[AnyRef]].map(extractBalancesVals[BigDecimal])

package cqrs

package filters

Filter and extractor functions for sequences, Either, Option, tuples and case classes

Filter Sequence of different types of case classes

val mixCaseClasses = keyBigDecimal ++ keyBigInt ++ keyBoolean ++ keyDouble ++ keyFloat
val filtered = filterProducts(mixCaseClasses, fieldFilter, isType[BigDecimal])

Filter by type

if(filtered.forall(isType[BigDecimal]))

Filter by Option[type]

val filtered = filterProducts(mixCaseClasses, fieldFilter, isOptionType[Int])

Filter by Either where Left is a String or Right is the type

val filtered = filterProducts(mixCaseClasses, fieldFilter, isEitherStringRight[BigInt])

Extract a case class field by index

val eithBigDecs = extractElementByIndex[Either[String, BigDecimal]](keyEithBigDec, 1)

Filter Right values of Either

def collectRightFlow[A, B]: Flow[Seq[Either[A, B]], Seq[B], NotUsed] =
      Flow[Seq[Either[A, B]]].collect(PartialFunction(filterRight))

Definition Classes: dendrites

package http

Provides Class to create an HostConnectionPool.

Provides Class to create an HostConnectionPool. Also functions to create requests and handle response

Get Config, ip address, port as tuple3

val hostConfig = getHostConfig("dendrites.checking-balances.http.interface","my.http.port")

Append path to host URL

val baseURL = configBaseUrl("my.http.path", hostConfig)

Create StringBuilder with URL including path

val url = createUrl(scheme, ipDomain, port, path)

Create StringBuilder with request path i.e. "?", "key=value" for all fields in case class

val balanceQuery = GetAccountBalances(goodId)
val checkingPath = "/account/balances/checking/"
val balancesQuery = caseClassToGetQuery(balanceQuery)()
val q = checkingPath ++ balancesQuery

Definition Classes: dendrites
See also: Config API

package kafka

Provides Classes to add fields to KafkaConsumer and KafkaProducer.

Provides Classes to add fields to KafkaConsumer and KafkaProducer. Also Factories for KafkaConsumer, KafkaProducer from their Properties files

Create KafkaConsumer with properties

val consumer = createConsumer[Key, Value]("kafkaConsumer.properties")

Create KafkaProducer with properties

val producer = createProducer[Key, Value]("kafkaProducer.properties")

Definition Classes: dendrites
See also: Config API

package reflection

Reflection utils for logging and debugging

Log type information

val result = results.toIndexedSeq
log.debug("result:{}", weakParamInfo(result))

Definition Classes: dendrites
See also: TypeTags and Manifests

package stream

Akka Stream Flows

Map sequence of Option to sequence of values

val (pub, sub) = TestSource.probe[Seq[Option[BigInt]]]
  .via(flattenFlow)
  .via(maxFlow)
  .toMat(TestSink.probe[BigInt])(Keep.both)
  .run()

Map sequence of Either to sequence of values

val (pub, sub) = TestSource.probe[Seq[Either[String, Double]]]
  .via(collectRightFlow)
  .via(maxFlow)
  .toMat(TestSink.probe[Double])(Keep.both)
  .run()

Flow accepts tuple3 from zip stage, wraps tuple3LeftRight which maps 3 Eithers to Sequence of Lefts and a Sequence of Rights

def zipper = ZipWith((in0: Either[String, AnyRef],
  in1: Either[String, AnyRef],
  in2: Either[String, AnyRef]) => (in0, in1, in2))
val flowGraph = GraphDSL.create() { implicit builder =>
  val zip = builder.add(zipper)
  val fgLR = builder.add(leftRightFlow)