case class Approval(message: String) extends AssignmentEvaluation with Product with Serializable

Approval of an assignment.

message: the message shown with the approval (I think the worker can see this?)

case class Assignment[Response](hitTypeId: String, hitId: String, assignmentId: String, workerId: String, acceptTime: Long, submitTime: Long, response: Response, feedback: String) extends Product with Serializable

Represents a single annotator's response to a HIT.

Similar to the "Assignment" type from the MTurk API, except that it only corresponds to a *finished* annotation, which may have been accepted or rejected. Also is well-typed for the purposes of an experiment, parametrized over the type of Response you want.

Another bit of post-processing: this accommodates for a "feedback" field separate from the actual response, to make working with response data easier, instead of having to embed the feedback in the response field.

Response: the desired data representation for annotators' responses
hitId: the ID of the HIT this assignment was for
assignmentId: the unique ID given to this assignment on MTurk
workerId: the ID of the worker who did this assignment
acceptTime: the time (millis from epoch) when the worker accepted the HIT
submitTime: the time (millis from epoch) when the worker submitted the HIT
response: the worker's response to the HIT

sealed trait AssignmentEvaluation extends AnyRef

Represents an accept/reject decision for an assignment.

class FileSystemHITDataService extends HITDataService with StrictLogging

Implementation of HITDataService against the file system.

Implementation of HITDataService against the file system. Works for most purposes; would be better to have an implementation against a database for a really big project.

case class HIT[Prompt](hitTypeId: String, hitId: String, prompt: Prompt, creationTime: Long) extends Product with Serializable

Represents a HIT that has already been uploaded to MTurk.

Represents a HIT that has already been uploaded to MTurk. It may not still be on MTurk, but it should still be saved to disk.

Parameterized over a desired data representation for "prompts", which are transformed by a TaskSpecification into questions for the MTurk interface.

Prompt: the data representation of a question
hitId: the unique ID assigned to this HIT by MTurk
prompt: the data used to create the question shown to workers
creationTime: the time (millis from epoch) that the HIT was created

trait HITDataService extends AnyRef

API for services that store the data returned by workers across HITs on MTurk.

API for services that store the data returned by workers across HITs on MTurk. Semantics are append-only; you can save HITs and Assignments and get them later, but cannot delete any data.

Not expected to be thread-safe in general. TODO: implement this using the interpreter pattern instead.

case class HITInfo[Prompt, Response](hit: HIT[Prompt], assignments: List[Assignment[Response]]) extends Product with Serializable

Data structure to hold all data relevant to a particular HIT.

Data structure to hold all data relevant to a particular HIT. TODO: should include rejected assignments as well.

case class HITType(title: String, description: String, keywords: String, reward: Double, autoApprovalDelay: Long, assignmentDuration: Long, qualRequirements: Array[QualificationRequirement] = ...) extends Product with Serializable

Represents all of the fields necessary to produce a HIT type.

HIT Types define classes of HITs that are grouped together in the Mechanical Turk interface. A worker does not so much choose to work on an individual HIT as an individual HIT Type. After they complete one HIT, they will automatically be shown another of the same HIT Type, and if they so choose, they can automatically accept that next HIT.

From the requester's perspective, many operations with the RequesterService (such as searching and finding reviewable HITs) will be desired to apply only to HITs of a particular HIT type. We do this using the HIT Type ID of the HIT type, computed by Amazon and returned by the register() method on a HIT Type object. This gives us a way to segregate our interactions with the API between different tasks, allowing for running two different kinds of tasks in parallel (for example, a Q/A gathering task and validation task).

The HIT Type ID of a HIT is determined (automatically, by Amazon) on the basis of a set of values: its title, description, keyword, rewards, auto-approval delay, assignment duration, qualification requirements, assignment review policy, and HIT review policy. We don't bother with assignment review policies or HIT review policies, which are MTurk's way of automating assignment review; instead, we just review them here in the code. The rest of these fields are left for an instance to specify. If you wish tasks to be segregated / run separately, make sure their HIT Types differ in at least one of those fields.

title: the title of the HIT type; one of the first things workers see when browsing
description: a longer (but still short) description of the task, displayed when a worker clicks on the HIT group while browsing
keywords: comma-separated keywords, helping workers find the task through search
reward: the payment, in dollars, to a worker for having an assignment approved
autoApprovalDelay: the time, in seconds, before MTurk will automatically approve an assignment if you do not approve/reject it manually
assignmentDuration: The time, in seconds, allowed for a worker to complete a single assignment

class InMemoryHITDataService extends HITDataService

Simple in-memory implementation of HITDataService for use in testing and sample tasks.

case class Rejection(message: String) extends AssignmentEvaluation with Product with Serializable

Rejection of an assignment.

The message parameter should be used to give an explanation of why the work was rejected. I also like to pair it with advice on how to avoid another rejection.

message: the message shown to the worker with the rejection

Packages

spacro

package spacro

Type Members

Value Members

Ungrouped