Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package scalatest

    ScalaTest's main traits, classes, and other members, including members supporting ScalaTest's DSL for the Scala interpreter.

    ScalaTest's main traits, classes, and other members, including members supporting ScalaTest's DSL for the Scala interpreter.

    Definition Classes
    org
  • package concurrent

    Classes, traits, and objects related to testing asynchronous and multi-threaded behavior.

    Classes, traits, and objects related to testing asynchronous and multi-threaded behavior.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • package enablers

    Classes, traits, and objects for typeclasses that enable ScalaTest's DSLs.

    Classes, traits, and objects for typeclasses that enable ScalaTest's DSLs.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • package events

    Classes for events sent to the org.scalatest.Reporter to report the results of running tests.

    Classes for events sent to the org.scalatest.Reporter to report the results of running tests.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • package exceptions

    Classes and traits for exceptions thrown by ScalaTest.

    Classes and traits for exceptions thrown by ScalaTest.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • package fixture

    Classes and traits supporting ScalaTest's "fixture" style traits, which allow you to pass fixture objects into tests.

    Classes and traits supporting ScalaTest's "fixture" style traits, which allow you to pass fixture objects into tests.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • package prop

    Scalatest support for Property-based testing.

    Scalatest support for Property-based testing.

    Introduction to Property-based Testing

    In traditional unit testing, you write tests that describe precisely what the test will do: create these objects, wire them together, call these functions, assert on the results, and so on. It is clear and deterministic, but also limited, because it only covers the exact situations you think to test. In most cases, it is not feasible to test all of the possible combinations of data that might arise in real-world use.

    Property-based testing works the other way around. You describe properties -- rules that you expect your classes to live by -- and describe how to test those properties. The test system then generates relatively large amounts of synthetic data (with an emphasis on edge cases that tend to make things break), so that you can see if the properties hold true in these situations.

    As a result, property-based testing is scientific in the purest sense: you are stating a hypothesis about how things should work (the property), and the system is trying to falsify that hypothesis. If the tests pass, that doesn't prove the property holds, but it at least gives you some confidence that you are probably correct.

    Property-based testing is deliberately a bit random: while the edge cases get tried upfront, the system also usually generates a number of random values to try out. This makes things a bit non-deterministic -- each run will be tried with somewhat different data. To make it easier to debug, and to build regression tests, the system provides tools to re-run a failed test with precisely the same data.

    Background

    TODO: Bill should insert a brief section on QuickCheck, ScalaCheck, etc, and how this system is similar and different.

    Using Property Checks

    In order to use the tools described here, you should import this package:

    import org.scalatest._
import org.scalatest.prop._

    This library is designed to work well with the types defined in Scalactic, and some functions take types such as PosZInt as parameters. So it can also be helpful to import those with:

    import org.scalactic.anyvals._

    In order to call forAll, the function that actually performs property checks, you will need to either extend or import GeneratorDrivenPropertyChecks, like this:

    class DocExamples extends FlatSpec with Matchers with GeneratorDrivenPropertyChecks {

    There's nothing special about FlatSpec, though -- you may use any of ScalaTest's styles with property checks. GeneratorDrivenPropertyChecks extends CommonGenerators, so it also provides access to the many utilities found there.

    What Does a Property Look Like?

    Let's check a simple property of Strings -- that if you concatenate a String to itself, its length will be doubled:

    "Strings" should "have the correct length when doubled" in {
  forAll { (s: String) =>
    val s2 = s * 2
    s2.length should equal (s.length * 2)
  }
}

    (Note that the examples here are all using the FlatSpec style, but will work the same way with any of ScalaTest's styles.)

    As the name of the tests suggests, the property we are testing is the length of a String that has been doubled.

    The test begins with forAll. This is usually the way you'll want to begin property checks, and that line can be read as, "For all Strings, the following should be true".

    The test harness will generate a number of Strings, with various contents and lengths. For each one, we compute s * 2. (* is a function on String, which appends the String to itself as many times as you specify.) And then we check that the length of the doubled String is twice the length of the original one.

    Using Specific Generators

    Let's try a more general version of this test, multiplying arbitrary Strings by arbitrary multipliers:

    "Strings" should "have the correct length when multiplied" in {
  forAll { (s: String, n: PosZInt) =>
    val s2 = s * n.value
    s2.length should equal (s.length * n.value)
  }
}

    Again, you can read the first line of the test as "For all Strings, and all non-negative Integers, the following should be true". (PosZInt is a type defined in Scalactic, which can be any positive integer, including zero. It is appropriate to use here, since multiplying a String by a negative number doesn't make sense.)

    This intuitively makes sense, but when we try to run it, we get a JVM Out of Memory error! Why? Because the test system tries to test with the "edge cases" first, and one of the more important edge cases is Int.MaxValue. It is trying to multiply a String by that, which is far larger than the memory of even a big computer, and crashing.

    So we want to constrain our test to sane values of n, so that it doesn't crash. We can do this by using more specific Generators.

    When we write a forAll test like the above, ScalaTest has to generate the values to be tested -- the semi-random Strings, Ints and other types that you are testing. It does this by calling on an implicit Generator for the desired type. The Generator generates values to test, starting with the edge cases and then moving on to randomly-selected values.

    ScalaTest has built-in Generators for many major types, including String and PosZInt, but these Generators are generic: they will try any value, including values that can break your test, as shown above. But it also provides tools to let you be more specific.

    Here is the fixed version of the above test:

    "Strings" should "have the correct length when multiplied" in {
  forAll(strings, posZIntsBetween(0, 1000))
  { (s: String, n: PosZInt) =>
    val s2 = s * n.value
    s2.length should equal (s.length * n.value)
  }
}

    This is using a variant of forAll, which lets you specify the Generators to use instead of just picking the implicit one. CommonGenerators.strings is the built-in Generator for Strings, the same one you were getting implicitly. (The other built-ins can be found in CommonGenerators. They are mixed into GeneratorDrivenPropertyChecks, so they are readily available.)

    But CommonGenerators.posZIntsBetween is a function that creates a Generator that selects from the given values. In this case, it will create a Generator that only creates numbers from 0 to 1000 -- small enough to not blow up our computer's memory. If you try this test, this runs correctly.

    The moral of the story is that, while using the built-in Generators is very convenient, and works most of the time, you should think about the data you are trying to test, and pick or create a more-specific Generator when the test calls for it.

    CommonGenerators contains many functions that are helpful in common cases. In particular:

    • xxsBetween (where xxs might be Int, Long, Float or most other significant numeric types) gives you a value of the desired type in the given range, as in the posZIntsBetween() example above.
    • CommonGenerators.specificValue and CommonGenerators.specificValues create Generators that produce either one specific value every time, or one of several values randomly. This is useful for enumerations and types that behave like enumerations.
    • CommonGenerators.evenly and CommonGenerators.frequency create higher-level Generators that call other Generators, either more or less equally or with a distribution you define.

    Testing Your Own Types

    Testing the built-in types isn't very interesting, though. Usually, you have your own types that you want to check the properties of. So let's build up an example piece by piece.

    Say you have this simple type:

    sealed trait Shape {
  def area: Double
}
case class Rectangle(width: Int, height: Int) extends Shape {
  require(width > 0)
  require(height > 0)
  def area: Double = width * height
}

    Let's confirm a nice straightforward property that is surely true: that the area is greater than zero:

    "Rectangles" should "have a positive area" in {
   forAll { (w: PosInt, h: PosInt) =>
     val rect = Rectangle(w, h)
     rect.area should be > 0.0
   }
 }

    Note that, even though our class takes ordinary Ints as parameters (and checks the values at runtime), it is actually easier to generate the legal values using Scalactic's PosInt type.

    This should work, right? Actually, it doesn't -- if we run it a few times, we quickly hit an error!

    [info] Rectangles
[info] - should have a positive area *** FAILED ***
[info]   GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
[info]    (DocExamples.scala:42)
[info]     Falsified after 2 successful property evaluations.
[info]     Location: (DocExamples.scala:42)
[info]     Occurred when passed generated values (
[info]       None = PosInt(399455539),
[info]       None = PosInt(703518968)
[info]     )
[info]     Init Seed: 1568878346200

    TODO: fix the above error to reflect the better errors we should get when we merge in the code being forward-ported from 3.0.5.

    Looking at it, we can see that the numbers being used are pretty large. What happens when we multiply them together?

    scala> 399455539 * 703518968
res0: Int = -2046258840

    We're hitting an Int overflow problem here: the numbers are too big to multiply together and still get an Int. So we have to fix our area function:

    case class Rectangle(width: Int, height: Int) extends Shape {
  require(width > 0)
  require(height > 0)
  def area: Double = width.toLong * height.toLong
}

    Now, when we run our property check, it consistently passes. Excellent -- we've caught a bug, because ScalaTest tried sufficiently large numbers.

    Composing Your Own Generators

    Doing things as shown above works, but having to generate the parameters and construct a Rectangle every time is a nuisance. What we really want is to create our own Generator that just hands us Rectangles, the same way we can do for PosInt. Fortunately, this is easy.

    Generators can be composed in for comprehensions. So we can create our own Generator for Rectangle like this:

    implicit val rectGenerator = for {
  w <- posInts
  h <- posInts
}
  yield Rectangle(w, h)

    Taking that line by line:

    w <- posInts

    CommonGenerators.posInts is the built-in Generator for positive Ints. So this line puts a randomly-generated positive Int in w, and

    h <- posInts

    this line puts another one in h. Finally, this line:

    yield Rectangle(w, h)

    combines w and h to make a Rectangle.

    That's pretty much all you need in order to build any normal case class -- just build it out of the Generators for the type of each field. (And if the fields are complex data structures themselves, build Generators for them the same way, until you are just using primitives.)

    Now, our property check becomes simpler:

    "Generated Rectangles" should "have a positive area" in {
   forAll { (rect: Rectangle) =>
     rect.area should be > 0.0
   }
 }

    That's about as close to plain English as we can reasonably hope for!

    Filtering Values with whenever()

    Sometimes, not all of your generated values make sense for the property you want to check -- you know (via external information) that some of these values will never come up. In cases like this, you can create a custom Generator that only creates the values you do want, but it's often easier to just use Whenever.whenever. (Whenever is mixed into GeneratorDrivenPropertyChecks, so this is available when you need it.)

    The Whenever.whenever function can be used inside of GeneratorDrivenPropertyChecks.forAll. It says that only the filtered values should be used, and anything else should be discarded. For example, look at this property:

    "Fractions" should "get smaller when squared" in {
  forAll { (n: Float) =>
    whenever(n > 0 && n < 1) {
      (n * n) should be < n
    }
  }
}

    We are testing a property of numbers less than 1, so we filter away everything that is not the numbers we want. This property check succeeds, because we've screened out the values that would make it fail.

    Discard Limits

    You shouldn't push Whenever.whenever too far, though. This system is all about trying random data, but if too much of the random data simply isn't usable, you can't get valid answers, and the system tracks that.

    For example, consider this apparently-reasonable test:

    "Space Chars" should "not also be letters" in {
  forAll { (c: Char) =>
    whenever (c.isSpaceChar) {
      assert(!c.isLetter)
    }
  }
}

    Although the property is true, this test will fail with an error like this:

    [info] Lowercase Chars
[info] - should upper-case correctly *** FAILED ***
[info]   Gave up after 0 successful property evaluations. 49 evaluations were discarded.
[info]   Init Seed: 1568855247784

    Because the vast majority of Chars are not spaces, nearly all of the generated values are being discarded. As a result, the system gives up after a while. In cases like this, you usually should write a custom Generator instead.

    The proportion of how many discards to permit, relative to the number of successful checks, is configuration-controllable. See GeneratorDrivenPropertyChecks for more details.

    Randomization

    The point of Generator is to create pseudo-random values for checking properties. But it turns out to be very inconvenient if those values are actually random -- that would mean that, when a property check fails occasionally, you have no good way to invoke that specific set of circumstances again for debugging. We want "randomness", but we also want it to be deterministic, and reproducible when you need it.

    To support this, all "randomness" in ScalaTest's property checking system uses the Randomizer class. You start by creating a Randomizer using an initial seed value, and call that to get your "random" value. Each call to a Randomizer function returns a new Randomizer, which you should use to fetch the next value.

    GeneratorDrivenPropertyChecks.forAll uses Randomizer under the hood: each time you run a forAll-based test, it will automatically create a new Randomizer, which by default is seeded based on the current system time. You can override this, as discussed below.

    Since Randomizer is actually deterministic (the "random" values are unobvious, but will always be the same given the same initial seed), this means that re-running a test with the same seed will produce the same values.

    If you need random data for your own Generators and property checks, you should use Randomizer in the same way; that way, your tests will also be re-runnable, when needed for debugging.

    Debugging, and Re-running a Failed Property Check

    In Testing Your Own Types above, we found to our surprise that the property check failed with this error:

    [info] Rectangles
[info] - should have a positive area *** FAILED ***
[info]   GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
[info]    (DocExamples.scala:42)
[info]     Falsified after 2 successful property evaluations.
[info]     Location: (DocExamples.scala:42)
[info]     Occurred when passed generated values (
[info]       None = PosInt(399455539),
[info]       None = PosInt(703518968)
[info]     )
[info]     Init Seed: 1568878346200

    There must be a bug here -- but once we've fixed it, how can we make sure that we are re-testing exactly the same case that failed?

    This is where the pseudo-random nature of Randomizer comes in, and why it is so important to use it consistently. So long as all of our "random" data comes from that, then all we need to do is re-run with the same seed.

    That's why the Init Seed shown in the message above is crucial. We can re-use that seed -- and therefore get exactly the same "random" data -- by using the -S flag to ScalaTest.

    So you can run this command in sbt to re-run exactly the same property check:

    testOnly *DocExamples -- -z "have a positive area" -S 1568878346200

    Taking that apart:

    • testOnly *DocExamples says that we only want to run suites whose paths end with DocExamples
    • -z "have a positive area" says to only run tests whose names include that string.
    • -S 1568878346200 says to run all tests with a "random" seed of 1568878346200

    By combining these flags, you can re-run exactly the property check you need, with the right random seed to make sure you are re-creating the failed test. You should get exactly the same failure over and over until you fix the bug, and then you can confirm your fix with confidence.

    Configuration

    In general, forAll() works well out of the box. But you can tune several configuration parameters when needed. See GeneratorDrivenPropertyChecks for info on how to set configuration parameters for your test.

    Table-Driven Properties

    Sometimes, you want something in between traditional hard-coded unit tests and Generator-driven, randomized tests. Instead, you sometimes want to check your properties against a specific set of inputs.

    (This is particularly useful for regression tests, when you have found certain inputs that have caused problems in the past, and want to make sure that they get consistently re-tested.)

    ScalaTest supports these, by mixing in TableDrivenPropertyChecks. See the documentation for that class for the full details.

    Definition Classes
    scalatest
  • package tagobjects

    Singleton-object versions of ScalaTest's built-in tags.

    Singleton-object versions of ScalaTest's built-in tags.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • package tags
    Definition Classes
    scalatest
  • package time

    Classes, traits, and objects for ScalaTest's time DSL.

    Classes, traits, and objects for ScalaTest's time DSL.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • package tools

    Tools for running ScalaTest.

    Tools for running ScalaTest.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest
  • Framework
  • Runner
  • ScalaTestAntTask
  • ScalaTestFramework
  • package verbs

    Classes and traits that support ScalaTest DSLs.

    Classes and traits that support ScalaTest DSLs.

    This package is released as part of the scalatest-core module.

    Definition Classes
    scalatest

package tools

Tools for running ScalaTest.

This package is released as part of the scalatest-core module.

Source
package.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. tools
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. class Framework extends sbt.testing.Framework

    This class is ScalaTest's implementation of the new Framework API that is supported in sbt 0.13.

    This class is ScalaTest's implementation of the new Framework API that is supported in sbt 0.13.

    To use ScalaTest in sbt, you should add ScalaTest as dependency in your sbt build file, the following shows an example for using ScalaTest 2.0 with Scala 2.10.x project:

    "org.scalatest" % "scalatest_2.10" % "2.0" % "test"

    To pass argument to ScalaTest from sbt, you can use testOptions:

    testOptions in Test += Tests.Argument("-h", "target/html")  // Use HtmlReporter

    If you are using multiple testing frameworks, you can pass arguments specific to ScalaTest only:

    testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-h", "target/html") // Use HtmlReporter

    Supported arguments

    Integration in sbt 0.13 supports same argument format as Runner, except the following arguments:

    • -R -- runpath is not supported because test path and discovery is handled by sbt
    • -s -- suite is not supported because sbt's test-only serves the similar purpose
    • -A -- again is not supported because sbt's test-quick serves the similar purpose
    • -j -- junit is not supported because in sbt different test framework should be supported by its corresponding Framework implementation
    • -b -- testng is not supported because in sbt different test framework should be supported by its corresponding Framework implementation
    • -P -- concurrent/parallel is not supported because parallel execution is controlled by sbt.
    • -q is not supported because test discovery should be handled by sbt, and sbt's test-only or test filter serves the similar purpose
    • -T is not supported because correct ordering of text output is handled by sbt
    • -g is not supported because current Graphic Reporter implementation works differently than standard reporter
    New Features of New Framework API

    New Framework API supports a number of new features that ScalaTest has utilized to support a better testing experience in sbt. The followings are summary of new features supported by the new Framework API:

    • Specified behavior of single instance of Runner per project run (non-fork), and a new done method
    • API to return nested tasks
    • API to support test execution in fork mode
    • Selector API to selectively run tests
    • Added new Ignored, Canceled and Pending status
    • Added sbt Tagging support
    Specified behavior of single instance of Runner per project run (non-fork), and a new done method

    In new Framework API, it is now a specified behavior that Framework's runner method will be called to get a Runner instance once per project run. Arguments will be passed when calling Framework's runner and this gives ScalaTest a good place to perform setup tasks, such as initializing Reporters.

    There's also a new done on Runner interface, which in turns provide a good spot for ScalaTest to perform cleanup tasks, such as disposing the Reporters. HtmlReporter depends on this behavior to generate its index.html. In addition, done can return framework-specific summary text for sbt to render at the end of the project run, which allows ScalaTest to return its own summary text.

    API to return nested Suites as sbt Tasks

    In new Framework API, a new concept of Task was introduced. A Task has an execute method that can return more Tasks for execution. ScalaTest does not utilize this feature, it always return empty array for sub-tasks.

    API to support test execution in fork mode

    Forking was added to sbt since version 0.12, you can find documentation for forking support in sbt at Forking in sbt.

    Although forking is already available in sbt since 0.12, there's no support in old Framework API, until it is added in new Framework API that is supported in sbt 0.13. With API provided with new Framework API, ScalaTest creates real Reporters in the main process, and uses SocketReporter in forked process to send events back to the main process, and get processed by real Reporters at the main process. All of this is transparent to any custom Reporter implementation, as only one instance of the custom Reporter will be created to process the events, regardless of whether the tests run in same or forked process.

    Selector API to selectively run tests

    New Framework API includes a set of comprehensive API to select tests for execution. Though new Framework API supports fine-grained test selection, current sbt's test-only and test-quick supports up to suite level selection only, or SuiteSelector as defined in new Framework API. This Framework implementation already supports SuiteSelector, NestedSuiteSelector, TestSelector and NestedTestSelector, which should work once future sbt version supports them.

    Added new Ignored, Canceled and Pending status

    Status Ignored, Canceled and Pending are added to new Framework API, and they match perfectly with ScalaTest's ignored tests (now reported as Ignored instead of Skipped), as well as canceled and pending tests newly added in ScalaTest 2.0.

    Added sbt Tagging support

    Sbt supports task tagging, but has no support in old Framework API for test frameworks to integrate it. New Framework API supports it, and you can now use the following annotations to annotate your suite for sbt built-in resource tags:

    They will be mapped to corresponding resource tag CPU, Disk and Network in sbt.

    You can also define custom tag, which you'll need to write it as Java annotation:

    import java.lang.annotation.Target;
import java.lang.annotation.Retention;
import org.scalatest.TagAnnotation;

@TagAnnotation("custom")
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE})
public @interface Custom {}

    which will be translated to Tags.Tag("custom") in sbt.

  2. class ScalaTestAntTask extends Task

    An ant task to run ScalaTest.

    An ant task to run ScalaTest. Instructions on how to specify various options are below. See the main documentation for object Runner class for a description of what each of the options does.

    To use the ScalaTest ant task, you must first define it in your ant file using taskdef. Here's an example:

     <path id="scalatest.classpath">
   <pathelement location="${lib}/scalatest.jar"/>
   <pathelement location="${lib}/scala-library.jar"/>
   <-- scala-actors.jar needed only for ScalaTest <= 1.9.1 on Scala >= 2.10.0 -->
   <pathelement location="${lib}/scala-actors.jar"/>
 </path>

 <target name="main" depends="dist">
   <taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestAntTask">
     <classpath refid="scalatest.classpath"/>
   </taskdef>

   <scalatest ...
 </target>

    Note that you only need the scala-actors.jar if you are using ScalaTest version 1.9.1 or earlier with Scala 2.10 or later. Once defined, you use the task by specifying information in a scalatest element:

      <scalatest ...>
    ...
  </scalatest>

    You can place key value pairs into the config map using nested <config> elements, like this:

      <scalatest>
    <config name="dbname" value="testdb"/>
    <config name="server" value="192.168.1.188"/>

    You can specify a runpath using either a runpath attribute and/or nested <runpath> elements, using standard ant path notation:

      <scalatest runpath="serviceuitest-1.1beta4.jar:myjini">

    or

      <scalatest>
    <runpath>
      <pathelement location="serviceuitest-1.1beta4.jar"/>
      <pathelement location="myjini"/>
    </runpath>

    To add a URL to your runpath, use a <runpathurl> element (since ant paths don't support URLs):

      <scalatest>
    <runpathurl url="http://foo.com/bar.jar"/>

    You can specify reporters using nested <reporter> elements, where the type attribute must be one of the following:

    • graphic
    • file
    • memory
    • junitxml
    • html
    • stdout
    • stderr
    • reporterclass

    Each may include a config attribute to specify the reporter configuration. Types file, memory, junitxml, html, and reporterclass require additional attributes (the css attribute is optional for the html reporter):

      <scalatest>
    <reporter type="stdout" config="FD"/>
    <reporter type="file" filename="test.out"/>
    <reporter type="memory" filename="target/memory.out"/>
    <reporter type="junitxml" directory="target"/>
    <reporter type="html" directory="target" css="src/main/html/mystylesheet.css"/>
    <reporter type="reporterclass" classname="my.ReporterClass"/>

    Specify tags to include and/or exclude using <tagsToInclude> and <tagsToExclude> elements, like this:

      <scalatest>
    <tagsToInclude>
        CheckinTests
        FunctionalTests
    </tagsToInclude>

    <tagsToExclude>
        SlowTests
        NetworkTests
    </tagsToExclude>

    Tags to include or exclude can also be specified using attributes tagsToInclude and tagsToExclude, with arguments specified as whitespace- delimited lists.

    To specify suites to run, use either a suite attribute or nested <suite> elements:

      <scalatest suite="com.artima.serviceuitest.ServiceUITestkit">

    or

      <scalatest>
    <suite classname="com.artima.serviceuitest.ServiceUITestkit"/>

    To specify tests to run, use nested <test> elements with either a 'name' or 'substring' attribute:

      <scalatest>
    <test name="hello test"/>
    <test substring="hello"/>

    To specify suites using members-only or wildcard package names, use either the membersonly or wildcard attributes, or nested <membersonly> or <wildcard> elements:

      <scalatest membersonly="com.artima.serviceuitest">

    or

      <scalatest wildcard="com.artima.joker">

    or

      <scalatest>
    <membersonly package="com.artima.serviceuitest"/>
    <wildcard package="com.artima.joker"/>

    Use attribute suffixes="[pipe-delimited list of suffixes]" to specify that only classes whose names end in one of the specified suffixes should be included in discovery searches for Suites to test. This can be used to improve discovery time or to limit the scope of a test. E.g.:

      <scalatest suffixes="Spec|Suite">

    Use attribute testsfile="[file name]" or nested <testsfile> elements to specify files containing a list of tests to be run. This is used to rerun failed/canceled tests listed in files written by the memory reporter. E.g.:

      <scalatest testsfile="target/memory.out">

    or

      <scalatest>
    <testsfile filename="target/memory.out"/>

    Use attribute parallel="true" to specify parallel execution of suites. (If the parallel attribute is left out or set to false, suites will be executed sequentially by one thread.) When parallel is true, you can include an optional sortSuites attribute to request that events be sorted on-the-fly so that events for the same suite are reported together, with a timeout, (e.g., sortSuites="true"), and an optional numthreads attribute to specify the number of threads to be created in thread pool (e.g., numthreads="10").

    Use attribute haltonfailure="true" to cause ant to fail the build if there's a test failure.

    Use attribute fork="true" to cause ant to run the tests in a separate process.

    When fork is true, attribute maxmemory may be used to specify the maximum memory size that will be passed to the forked jvm.  For example, the following setting will cause "-Xmx1280M" to be passed to the java command used to run the tests.

      <scalatest maxmemory="1280M">

    When fork is true, nested <jvmarg> elements may be used to pass additional arguments to the forked jvm. For example, if you are running into 'PermGen space' memory errors, you could add the following jvmarg to bump up the JVM's MaxPermSize value:

      <jvmarg value="-XX:MaxPermSize=128m"/>

  3. class ScalaTestFramework extends scalatools.testing.Framework

    Class that makes ScalaTest tests visible to SBT (prior to version 0.13).

    Class that makes ScalaTest tests visible to SBT (prior to version 0.13).

    To use ScalaTest in SBT, you should add ScalaTest as dependency in your SBT build file, the following shows an example for using ScalaTest 2.0 with Scala 2.10.x project:

    "org.scalatest" % "scalatest_2.10" % "2.0" % "test"

    To pass argument to ScalaTest from SBT, you can use testOptions:

    testOptions in Test += Tests.Argument("-u", "target/junit")  // Use JUnitXmlReporter

    If you are using multiple testing frameworks, you can pass arguments specific to ScalaTest only:

    testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/junit") // Use JUnitXmlReporter

    Supported arguments

    Integration in SBT 0.13 supports same argument format as Runner, except the following arguments:

    • -R -- runpath is not supported because test path and discovery is handled by SBT
    • -s -- suite is not supported because SBT's test-only serves the similar purpose
    • -A -- again is not supported because SBT's test-quick serves the similar purpose
    • -j -- junit is not supported because in SBT different test framework should be supported by its corresponding Framework implementation
    • -b -- testng is not supported because in SBT different test framework should be supported by its corresponding Framework implementation
    • -P -- concurrent/parallel is not supported because parallel execution is controlled by SBT.
    • -q is not supported because test discovery should be handled by SBT, and SBT's test-only or test filter serves the similar purpose
    • -T is not supported because correct ordering of text output is handled by SBT
    • -g is not supported because current Graphic Reporter implementation works differently than standard reporter

    It is highly recommended to upgrade to SBT 0.13 to enjoy the best of ScalaTest 2.0 SBT integration. Due to limitations in old Framework API (prior to SBT 0.13), it is hard to support ScalaTest features in the most efficient way. One example is the nested suites, where in old Framework API they has to be executed sequentially, while new Framework API (included in SBT 0.13) the concept of nested Task has enabled parallel execution of ScalaTest's nested suites.

Value Members

  1. object Runner

    Application that runs a suite of tests.

    Application that runs a suite of tests.

    Note: this application offers the full range of ScalaTest features via command line arguments described below. If you just want to run a suite of tests from the command line and see results on the standard output, you may prefer to use ScalaTest's simple runner.

    The basic form of a Runner invocation is:

    scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]

    The arguments Runner accepts are described in the following table:

    argumentdescriptionexample
    -Dkey=valuedefines a key/value pair for the config map-DmaxConnections=100
    -R <runpath elements>the specifies the runpath from which tests classes will be
    discovered and loaded (Note: only one -R allowed)    		Unix: -R target/classes:target/generated/classes
    Windows: -R target\classes;target\generated\classes
    -n <tag name>specifies a tag to include (Note: only one tag name allowed per -n)-n UnitTests -n FastTests
    -l <tag name>specifies a tag to exclude (Note: only one tag name allowed per -l)-l SlowTests -l PerfTests
    -P[CS][integer thread count]specifies a parallel run, with optional cached thread pool, suite sorting and thread count
    (Note: only one -P allowed)    		-P, -PS, -PS8, -P8, -PC or -PCS. Negative or zero number of thread count is not allowed.
    -s <suite class name>specifies a suite class to run-s com.company.project.StackSpec
    -m <members-only package>requests that suites that are direct members of the specified package
    be discovered and run    		-m com.company.project
    -w <wildcard package>requests that suites that are members of the specified package or its subpackages
    be discovered and run    		-w com.company.project
    -q <suffixes>specify suffixes to discover-q Spec -q Suite
    -Qdiscover only classes whose names end with Spec or Suite
    (or other suffixes specified by -q)    		-Q
    -j <JUnit class name>instantiate and run a JUnit test class-j StackTestClass
    -b <TestNG XML file>run TestNG tests using the specified TestNG XML file-b testng.xml
    -F <span scale factor>a factor by which to scale time spans
    (Note: only one -F is allowed)    		-F 10 or -F 2.5
    -T <sorting timeout>specifies a integer timeout (in seconds) for sorting the events of
    parallel runs back into sequential order    		-T 5
    -i <suite ID>specifies a suite to run by ID (Note: must follow -s,
    and is intended to be used primarily by tools such as IDEs.)    		-i com.company.project.FileSpec-file1.txt
    -t <test name>select the test with the specified name-t "An empty Stack should complain when popped"
    -z <test name substring>select tests whose names include the specified substring-z "popped"
    -g[NCXEHLOPQMD]select the graphical reporter-g
    -f[NCXEHLOPQMDWSFU] <filename>select the file reporter-f output.txt
    -u <directory name>select the JUnit XML reporter-u target/junitxmldir
    -h <directory name> [-Y <css file name>]select the HTML reporter, optionally including the specified CSS file-h target/htmldir -Y src/main/html/customStyles.css
    -vprint the ScalaTest version-v or, also -version
    -o[NCXEHLOPQMDWSFU]select the standard output reporter-o
    -e[NCXEHLOPQMDWSFU]select the standard error reporter-e
    -C[NCXEHLOPQMD] <reporter class>select a custom reporter-C com.company.project.BarReporter
    -M <file name>memorize failed and canceled tests in a file, so they can be rerun with -A (again)-M rerun.txt
    -A <file name>used in conjunction with -M (momento) to select previously failed
    and canceled tests to rerun again    		-A rerun.txt
    -W <delay> <period>requests notifications of slowpoke tests, tests that have been running
    longer than delay seconds, every period seconds.    		-W 60 60

    The simplest way to start Runner is to specify the directory containing your compiled tests as the sole element of the runpath, for example:

    scala -classpath scalatest-<version>.jar org.scalatest.tools.Runner -R compiled_tests

    Given the previous command, Runner will discover and execute all Suites in the compiled_tests directory and its subdirectories, and show results in graphical user interface (GUI).

    Executing suites

    Each -s argument must be followed by one and only one fully qualified class name. The class must either extend Suite and have a public, no-arg constructor, or be annotated by a valid WrapWith annotation.

    Specifying the config map

    A config map contains pairs consisting of a string key and a value that may be of any type. (Keys that start with "org.scalatest." are reserved for ScalaTest. Configuration values that are themselves strings may be specified on the Runner command line. Each configuration pair is denoted with a "-D", followed immediately by the key string, an "=", and the value string. For example:

    -Ddbname=testdb -Dserver=192.168.1.188

    Specifying a runpath

    A runpath is the list of filenames, directory paths, and/or URLs that Runner uses to load classes for the running test. If runpath is specified, Runner creates a custom class loader to load classes available on the runpath. The graphical user interface reloads the test classes anew for each run by creating and using a new instance of the custom class loader for each run. The classes that comprise the test may also be made available on the classpath, in which case no runpath need be specified.

    The runpath is specified with the -R option. The -R must be followed by a space, a double quote ("), a white-space-separated list of paths and URLs, and a double quote. If specifying only one element in the runpath, you can leave off the double quotes, which only serve to combine a white-space separated list of strings into one command line argument. If you have path elements that themselves have a space in them, you must place a backslash (\) in front of the space. Here's an example:

    -R "serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar target/class\ files"

    Specifying reporters

    Reporters can be specified on the command line in any of the following ways:

    • -g[configs...] - causes display of a graphical user interface that allows tests to be run and results to be investigated
    • -f[configs...] <filename> - causes test results to be written to the named file
    • -u <directory> - causes test results to be written to junit-style xml files in the named directory
    • -h <directory> [-Y <CSS file>] - causes test results to be written to HTML files in the named directory, optionally included the specified CSS file
    • -a <number of files to archive> - causes specified number of old summary and durations files to be archived (in summaries/ and durations/ subdirectories) for dashboard reporter (default is two)
    • -o[configs...] - causes test results to be written to the standard output
    • -e[configs...] - causes test results to be written to the standard error
    • -k <host> <port> - causes test results to be written to socket in the named host and port number, using XML format
    • -K <host> <port> - causes test results to be written to socket in the named host and port number, using Java object binary format
    • -C[configs...] <reporterclass> - causes test results to be reported to an instance of the specified fully qualified Reporter class name

    The [configs...] parameter, which is used to configure reporters, is described in the next section.

    The -C option causes the reporter specified in <reporterclass> to be instantiated. Each reporter class specified with a -C option must be public, implement org.scalatest.Reporter, and have a public no-arg constructor. Reporter classes must be specified with fully qualified names. The specified reporter classes may be deployed on the classpath. If a runpath is specified with the -R option, the specified reporter classes may also be loaded from the runpath. All specified reporter classes will be loaded and instantiated via their no-arg constructor.

    For example, to run a suite named MySuite from the mydir directory using two reporters, the graphical reporter and a file reporter writing to a file named "test.out", you would type:

    java -jar scalatest.jar -R mydir -g -f test.out -s MySuite

    The -g, -o, or -e options can appear at most once each in any single command line. Multiple appearances of -f and -C result in multiple reporters unless the specified <filename> or <reporterclass> is repeated. If any of -g, -o, -e, <filename> or <reporterclass> are repeated on the command line, the Runner will print an error message and not run the tests.

    Runner adds the reporters specified on the command line to a dispatch reporter, which will dispatch each method invocation to each contained reporter. Runner will pass the dispatch reporter to executed suites. As a result, every specified reporter will receive every report generated by the running suite of tests. If no reporters are specified, a graphical runner will be displayed that provides a graphical report of executed suites.

    Configuring reporters

    Each reporter option on the command line can include configuration characters. Configuration characters are specified immediately following the -g, -o, -e, -f, or -C. The following configuration characters, which cause reports to be dropped, are valid for any reporter:

    • N - drop TestStarting events
    • C - drop TestSucceeded events
    • X - drop TestIgnored events
    • E - drop TestPending events
    • H - drop SuiteStarting events
    • L - drop SuiteCompleted events
    • O - drop InfoProvided events
    • P - drop ScopeOpened events
    • Q - drop ScopeClosed events
    • R - drop ScopePending events
    • M - drop MarkupProvided events

    A dropped event will not be delivered to the reporter at all. So the reporter will not know about it and therefore not present information about the event in its report. For example, if you specify -oN, the standard output reporter will never receive any TestStarting events and will therefore never report them. The purpose of these configuration parameters is to allow users to selectively remove events they find add clutter to the report without providing essential information.

    The following three reporter configuration parameters may additionally be used on standard output (-o), standard error (-e), and file (-f) reporters:

    • W - without color
    • D - show all durations
    • S - show short stack traces
    • F - show full stack traces
    • U - unformatted mode
    • I - show reminder of failed and canceled tests without stack traces
    • T - show reminder of failed and canceled tests with short stack traces
    • G - show reminder of failed and canceled tests with full stack traces
    • K - exclude TestCanceled events from reminder

    If you specify a W, D, S, F, U, R, T, G, or K for any reporter other than standard output, standard error, or file reporters, Runner will complain with an error message and not perform the run.

    Configuring a standard output, error, or file reporter with D will cause that reporter to print a duration for each test and suite. When running in the default mode, a duration will only be printed for the entire run.

    Configuring a standard output, error, or file reporter with F will cause that reporter to print full stack traces for all exceptions, including TestFailedExceptions. Every TestFailedException contains a stack depth of the line of test code that failed so that users won't need to search through a stack trace to find it. When running in the default, mode, these reporters will only show full stack traces when other exceptions are thrown, such as an exception thrown by production code. When a TestFailedException is thrown in default mode, only the source filename and line number of the line of test code that caused the test to fail are printed along with the error message, not the full stack trace.

    The 'U' unformatted configuration removes some formatting from the output and adds verbosity. The purpose of unformatted (or, "ugly") mode is to facilitate debugging of parallel runs. If you have tests that fail or hang during parallel runs, but succeed when run sequentially, unformatted mode can help. In unformatted mode, you can see exactly what is happening when it is happening. Rather than attempting to make the output look as pretty and human-readable as possible, unformatted mode will just print out verbose information about each event as it arrives, helping you track down the problem you are trying to debug.

    By default, a standard output, error, or file reporter inserts ansi escape codes into the output printed to change and later reset terminal colors. Information printed as a result of run starting, completed, and stopped events is printed in cyan. Information printed as a result of ignored or pending test events is shown in yellow. Information printed as a result of test failed, suite aborted, or run aborted events is printed in red. All other information is printed in green. The purpose of these colors is to facilitate speedy reading of the output, especially the finding of failed tests, which can get lost in a sea of passing tests. Configuring a standard output, error, or file reporter into without-color mode (W) will turn off this behavior. No ansi codes will be inserted.

    The R, T, and G options enable "reminders" of failed and, optionally, canceled tests to be printed at the end of the summary. This minimizes or eliminates the need to search and scroll backwards to find out what tests failed or were canceled. For large test suites, the actual failure message could have scrolled off the top of the buffer, making it otherwise impossible to see what failed. You can configure the detail level of the stack trace for regular reports of failed and canceled tests independently from that of reminders. To set the detail level for regular reports, use S for short stack traces, F for full stack traces, or nothing for the default of no stack trace. To set the detail level for reminder reports, use T for reminders with short stack traces, G for reminders with full stack traces in reminders, or R for reminders with no stack traces. If you wish to exclude reminders of canceled tests, i.e., only see reminders of failed tests, specify K along with one of R, T, or G, as in "-oRK".

    For example, to run a suite using two reporters, the graphical reporter configured to present every reported event and a standard error reporter configured to present everything but test starting, test succeeded, test ignored, test pending, suite starting, suite completed, and info provided events, you would type:

    scala -classpath scalatest-<version>.jar -R mydir -g -eNDXEHLO -s MySuite

    Note that no white space is allowed between the reporter option and the initial configuration parameters. So "-e NDXEHLO" will not work, "-eNDXEHLO" will work.

    Specifying tags to include and exclude

    You can specify tag names of tests to include or exclude from a run. To specify tags to include, use -n followed by a white-space-separated list of tag names to include, surrounded by double quotes. (The double quotes are not needed if specifying just one tag.) Similarly, to specify tags to exclude, use -l followed by a white-space-separated list of tag names to exclude, surrounded by double quotes. (As before, the double quotes are not needed if specifying just one tag.) If tags to include is not specified, then all tests except those mentioned in the tags to exclude (and in the org.scalatest.Ignore tag), will be executed. (In other words, the absence of a -n option is like a wildcard, indicating all tests be included.) If tags to include is specified, then only those tests whose tags are mentioned in the argument following -n and not mentioned in the tags to exclude, will be executed. For more information on test tags, see the documentation for Suite. Here are some examples:

    • -n CheckinTests
    • -n FunctionalTests -l org.scalatest.tags.Slow
    • -n "CheckinTests FunctionalTests" -l "org.scalatest.tags.Slow org.scalatest.tags.Network"

    Specifying suffixes to discover

    You can specify suffixes of Suite names to discover. To specify suffixes to discover, use -q followed by a vertical-bar-separated list of suffixes to discover, surrounded by double quotes. (The double quotes are not needed if specifying just one suffix.) Or you can specify them individually using multiple -q's. If suffixes to discover is not specified, then all suffixes are considered. If suffixes is specified, then only those Suites whose class names end in one of the specified suffixes will be considered during discovery. Here are some examples:

    • -q Spec
    • -q "Spec|Suite"
    • -q Spec -q Suite

    Option -Q can be used to specify a default set of suffixes "Spec|Suite". If you specify both -Q and -q, you'll get Spec and Suite in addition to the other suffix or suffixes you specify with -q.

    Specifying suffixes can speed up the discovery process because class files with names not ending the specified suffixes can be immediately disqualified, without needing to load and inspect them to see if they either extend Suite and declare a public, no-arg constructor, or are annotated with WrapWith.

    Executing Suites in parallel

    With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is useful to be able to run tests in parallel. If you include -P on the command line, Runner will pass a Distributor to the Suites you specify with -s. Runner will set up a thread pool to execute any Suites passed to the Distributor's put method in parallel. Trait Suite's implementation of runNestedSuites will place any nested Suites into this Distributor. Thus, if you have a Suite of tests that must be executed sequentially, you should override runNestedSuites as described in the documentation for Distributor.

    The -P option may optionally be appended with a number (e.g. "-P10" -- no intervening space) to specify the number of threads to be created in the thread pool. If no number (or 0) is specified, the number of threads will be decided based on the number of processors available.

    Specifying Suites

    Suites are specified on the command line with a -s followed by the fully qualified name of a Suite subclass, as in:

    -s com.artima.serviceuitest.ServiceUITestkit

    Each specified suite class must be public, a subclass of org.scalatest.Suite, and contain a public no-arg constructor. Suite classes must be specified with fully qualified names. The specified Suite classes may be loaded from the classpath. If a runpath is specified with the -R option, specified Suite classes may also be loaded from the runpath. All specified Suite classes will be loaded and instantiated via their no-arg constructor.

    The runner will invoke execute on each instantiated org.scalatest.Suite, passing in the dispatch reporter to each execute method.

    Runner is intended to be used from the command line. It is included in org.scalatest package as a convenience for the user. If this package is incorporated into tools, such as IDEs, which take over the role of runner, object org.scalatest.tools.Runner may be excluded from that implementation of the package. All other public types declared in package org.scalatest.tools.Runner should be included in any such usage, however, so client software can count on them being available.

    Specifying "members-only" and "wildcard" Suite paths

    If you specify Suite path names with -m or -w, Runner will automatically discover and execute accessible Suites in the runpath that are either a member of (in the case of -m) or enclosed by (in the case of -w) the specified path. As used in this context, a path is a portion of a fully qualified name. For example, the fully qualifed name com.example.webapp.MySuite contains paths com, com.example, and com.example.webapp. The fully qualifed name com.example.webapp.MyObject.NestedSuite contains paths com, com.example, com.example.webapp, and com.example.webapp.MyObject. An accessible Suite is a public class that extends org.scalatest.Suite and defines a public no-arg constructor. Note that Suites defined inside classes and traits do not have no-arg constructors, and therefore won't be discovered. Suites defined inside singleton objects, however, do get a no-arg constructor by default, thus they can be discovered.

    For example, if you specify -m com.example.webapp on the command line, and you've placed com.example.webapp.RedSuite and com.example.webapp.BlueSuite on the runpath, then Runner will instantiate and execute both of those Suites. The difference between -m and -w is that for -m, only Suites that are direct members of the named path will be discovered. For -w, any Suites whose fully qualified name begins with the specified path will be discovered. Thus, if com.example.webapp.controllers.GreenSuite exists on the runpath, invoking Runner with -w com.example.webapp will cause GreenSuite to be discovered, because its fully qualifed name begins with "com.example.webapp". But if you invoke Runner with -m com.example.webapp, GreenSuite will not be discovered because it is directly a member of com.example.webapp.controllers, not com.example.webapp.

    If you specify no -s, -m, or -w arguments on the command line to Runner, it will discover and execute all accessible Suites in the runpath.

    Selecting suites and tests

    Runner accepts three arguments that facilitate selecting suites and tests: -i, -t, and -z. The -i option enables a suite to be selected by suite ID. This argument is intended to allow tools such as IDEs or build tools to rerun specific tests or suites from information included in the results of a previous run. A -i must follow a -s that specifies a class with a public, no-arg constructor. The -i parameter can be used, for example, to rerun a nested suite that declares no zero-arg constructor, which was created by containing suite that does declare a no-arg constructor. In this case, -s would be used to specify the class ScalaTest can instantiate directly, the containing suite that has a public, no-arg constructor, and -i would be used to select the desired nested suite. One important use case for -i is to enable such a nested suite that aborted during the previous run to be rerun.

    The -t argument allows a test to be selected by its (complete) test name. Like -i, the -t argument is primarily intented to be used by tools such as IDEs or build tools, to rerun selected tests based on information obtained from the results of a previous run. For example, -t could be used to rerun a test that failed in the previous run. The -t argument can be used directly by users, but because descriptive test names are usually rather long, the -z argument (described next), will usually be a more practical choice for users. If a -t follows either -s or -i, then it only applies to the suite identified. If it is specified independent of a -s or -i, then discovery is performed to find all Suites containing the test name.

    The -z option allows tests to be selected by a simplified wildcard: any test whose name includes the substring specified after -z will be selected. For example, -z popped would select tests named "An empty stack should complain when popped" and "A non-empty stack should return the last-pushed value when popped, but not "An empty stack should be empty". In short, -z popped would select any tests whose name includes the substring "popped", and not select any tests whose names don't include "popped". This simplified approach to test name wildcards, which was suggested by Mathias Doenitz, works around the difficulty of finding an actual wildcard character that will work reliably on different operating systems. Like -t, if -z follows -s or -i, then it only applies to the Suite specified. Otherwise discovery is performed to find all Suites containing test names that include the substring.

    Specifying a span scale factor

    If you specify a integer or floating point span scale factor with -F, trait ScaledTimeSpans trait will return the specified value from its implementation of spanScaleFactor. This allows you to tune the "patience" of a run (how long to wait for asynchronous operations) from the command line. For more information, see the documentation for trait ScaledTimeSpans.

    Specifying TestNG XML config file paths

    If you specify one or more file paths with -b (b for Beust, the last name of TestNG's creator), Runner will create a org.scalatest.testng.TestNGWrapperSuite, passing in a List of the specified paths. When executed, the TestNGWrapperSuite will create one TestNG instance and pass each specified file path to it for running. If you include -b arguments, you must include TestNG's jar file on the class path or runpath. The -b argument will enable you to run existing TestNG tests, including tests written in Java, as part of a ScalaTest run. You need not use -b to run suites written in Scala that extend TestNGSuite. You can simply run such suites with -s, -m, or -w parameters.

    Specifying JUnit tests

    JUnit tests, including ones written in Java, may be run by specifying -j classname, where the classname is a valid JUnit class such as a TestCase, TestSuite, or a class implementing a static suite() method returning a TestSuite.

    To use this option you must include a JUnit jar file on your classpath.

    Memorizing and rerunning failed and canceled tests

    You can memorize failed and canceled tests using -M:

    -M failed-canceled.txt

    All failed and canceled tests will be memorized in failed-canceled.txt, to rerun them again, you use -A:

    -A failed-canceled.txt

    Slowpoke notifications

    You can request to recieve periodic notifications of slowpokes, tests that have been running longer than a given amount of time, specified in seconds by the first integer after -W, the delay. You specify the period between slowpoke notifications in seconds with the second integer after -W, the period. Thus to receive notifications very minute of tests that have been running longer than two minutes, you'd use:

    -W 120 60

    Slowpoke notifications will be sent via AlertProvided events. The standard out reporter, for example, will report such notifications like:

    *** Test still running after 2 minutes, 13 seconds: suite name: ExampleSpec, test name: An egg timer should take 10 minutes.

Inherited from AnyRef

Inherited from Any

Ungrouped