Approval of an assignment.
Represents a single annotator's response to a HIT.
Represents a single annotator's response to a HIT.
Similar to the "Assignment" type from the MTurk API, except that it only corresponds to a *finished* annotation, which may have been accepted or rejected. Also is well-typed for the purposes of an experiment, parametrized over the type of Response you want.
Another bit of post-processing: this accommodates for a "feedback" field separate from the actual response, to make working with response data easier, instead of having to embed the feedback in the response field.
the desired data representation for annotators' responses
the ID of the HIT this assignment was for
the unique ID given to this assignment on MTurk
the ID of the worker who did this assignment
the time (millis from epoch) when the worker accepted the HIT
the time (millis from epoch) when the worker submitted the HIT
the worker's response to the HIT
Represents an accept/reject decision for an assignment.
Implementation of HITDataService against the file system.
Implementation of HITDataService against the file system. Works for most purposes; would be better to have an implementation against a database for a really big project.
Represents a HIT that has already been uploaded to MTurk.
Represents a HIT that has already been uploaded to MTurk. It may not still be on MTurk, but it should still be saved to disk.
Parameterized over a desired data representation for "prompts", which are transformed by a TaskSpecification into questions for the MTurk interface.
the data representation of a question
the unique ID assigned to this HIT by MTurk
the data used to create the question shown to workers
the time (millis from epoch) that the HIT was created
API for services that store the data returned by workers across HITs on MTurk.
API for services that store the data returned by workers across HITs on MTurk. Semantics are append-only; you can save HITs and Assignments and get them later, but cannot delete any data.
Not expected to be thread-safe in general. TODO: implement this using the interpreter pattern instead.
Data structure to hold all data relevant to a particular HIT.
Data structure to hold all data relevant to a particular HIT. TODO: should include rejected assignments as well.
Represents all of the fields necessary to produce a HIT type.
Represents all of the fields necessary to produce a HIT type.
HIT Types define classes of HITs that are grouped together in the Mechanical Turk interface. A worker does not so much choose to work on an individual HIT as an individual HIT Type. After they complete one HIT, they will automatically be shown another of the same HIT Type, and if they so choose, they can automatically accept that next HIT.
From the requester's perspective, many operations with the RequesterService (such as searching and finding reviewable HITs) will be desired to apply only to HITs of a particular HIT type. We do this using the HIT Type ID of the HIT type, computed by Amazon and returned by the register() method on a HIT Type object. This gives us a way to segregate our interactions with the API between different tasks, allowing for running two different kinds of tasks in parallel (for example, a Q/A gathering task and validation task).
The HIT Type ID of a HIT is determined (automatically, by Amazon) on the basis of a set of values: its title, description, keyword, rewards, auto-approval delay, assignment duration, qualification requirements, assignment review policy, and HIT review policy. We don't bother with assignment review policies or HIT review policies, which are MTurk's way of automating assignment review; instead, we just review them here in the code. The rest of these fields are left for an instance to specify. If you wish tasks to be segregated / run separately, make sure their HIT Types differ in at least one of those fields.
the title of the HIT type; one of the first things workers see when browsing
a longer (but still short) description of the task, displayed when a worker clicks on the HIT group while browsing
comma-separated keywords, helping workers find the task through search
the payment, in dollars, to a worker for having an assignment approved
the time, in seconds, before MTurk will automatically approve an assignment if you do not approve/reject it manually
The time, in seconds, allowed for a worker to complete a single assignment
Simple in-memory implementation of HITDataService for use in testing and sample tasks.
Rejection of an assignment.
Rejection of an assignment.
The message parameter should be used to give an explanation of why the work was rejected. I also like to pair it with advice on how to avoid another rejection.
the message shown to the worker with the rejection
Provides classes for managing tasks on Mechanical Turk.
Utility classes, methods, and extension methods for spacro.
Approval of an assignment.
the message shown with the approval (I think the worker can see this?)