Project

General

Profile

Documentation » Documentation 2024-1 »

AMTurk 2024-1

Back to Table of Contents

Overview

This document aims to instruct how to set up a GIFT course to be used within Amazon Mechanical Turk.

How Mechanical Turk Workers Get Paid

Mechanical Turk workers will only get paid if you approve their submission (or if the approval window expires). This puts the responsibility of verifying if their "rewards code" is valid on the Requester (e.g. YOU). GIFT approaches this by forcing the workers to submit their Mechanical Turk ID as a survey response, then instructing them at the end of the course to use this ID as the Mechanical Turk reward code. We verify that the submitted ID in Mechanical Turk is a person that successfully completed the course by comparing the ID to the course survey response. If the IDs match, then you can be sure that the person reached the end of the course and deserves their reward. If you have an ID in Mechanical Turk but not as a survey response, then that user did not complete the course and should be rejected.

Capturing the Mechanical Turk Worker ID

When creating your GIFT course, you will want to insert the Mechanical Turk survey at the beginning and the end of the course to capture the worker's Mechanical Turk ID. The reason to have the survey in the beginning of the course is to be able to match the experiment user with their Mechanical Turk ID. This way you can track their progress through the course, even if they don't finish it. You'll also want to capture their ID at the end of the course as a quick indicator that they progressed through the entire course and deserve their reward.

Insert Mechanical Turk Survey into the Course

  1. Drag and drop a Survey/Test course object into the course flow to create your first Mechanical Turk survey.
  2. After naming it, select "Select Existing" from the right panel.
  3. Click on the "Public Surveys" tab and in the search bar type "Turk". This should filter it down to only the survey you want.
  4. Select the "Mechanical Turk Survey" and click the "Select" button.
    Note: If you cannot find the survey, follow the steps in Creating a Mechanical Turk Survey From Scratch.
  5. The survey will be pre-populated. If you desire, you can reword the question as you see fit to match your needs. When finished you can close the survey by clicking the 'X' in the top right corner and selecting "Save First, Then Exit".
    Note: You might receive a warning about enabling editing due to dependencies on the survey, this is normal and you can click "Enable Editing" to proceed.

Repeat these steps to create your second Mechanical Turk survey. Make sure to position them so that the "meat" of the course is between them (see image).

Creating a Mechanical Turk Survey From Scratch

If you could not find the Mechanical Turk Survey in Public Surveys, then you might be on an older version of GIFT. You can replicate the survey by following these steps:

  1. Drag and drop a Survey/Test course object into the course flow to create the Mechanical Turk survey.
  2. Name the survey, and select "Create New" from the right panel.
  3. Select "Collect Learner Information (Not Actionable)".
  4. Click "Add Survey Item" at the bottom of the window and select a "Free Response" survey item.
  5. Enter your question text where prompted.
    An example of this text could be: "Please enter your Amazon Mechanical Turk ID:"
  6. In the Question Properties panel do the following actions:
    • Check "Required"
    • Enter a Help Message
      An example of this text could be: "You can always find your Worker ID on the top left corner of the Amazon Mechanical Turk website. Clicking on the ID will automatically copy it to your clipboard so it’s ready for pasting."
    • Enter the "Tag": "Mechanical Turk ID"
  7. When finished you can close the survey by clicking the 'X' in the top right corner and selecting "Save First, Then Exit".

Publish Your Course as an Experiment

After you have built and saved your course you will need to publish it as an experiment to allow the Mechanical Turk workers to access and run the course.

  1. Navigate to the Publish tool by clicking "Publish Courses" at the top of the window.
  2. Click "Publish Course".
  3. Make sure "Experiment" is selected (this is the default).
  4. Enter a "Published Course Name". This should be informative as to what the course is. If you want, you can have it be the same name as the course you just created.
  5. The "Description" is optional and allows you to provide more detail about the course.
  6. Select your course from the list of available courses. You can use the search box to filter the list down to the one you are looking for.
  7. Click "Publish Course".
  8. A dialog will appear with your generated Experiment URL. This is the URL you will use as the Mechanical Turk "Survey Link".

View Results / Export Report

In order to view the results of the Experiment (see who has taken the course), you can export a Report. This report will show you the submitted answers to the Mechanical Turk surveys you inserted in your course earlier. This is how you will determine if a Mechanical Turk worker deserves to be approved for the monetary reward.

  1. Navigate to the Publish tool by clicking "Publish Courses" at the top of the window.
  2. Click the Experiment you want to build the report for to expand it and see more information (Number of attempts, build report, the Experiment URL, etc...).
  3. Click "Pause and Build Report".
    NOTE: While the Experiment is paused, it cannot be accessed by Mechanical Turk workers!
  4. In "Frequently Reported Event Types", check "Survey responses".
  5. Check "Merge each participant's events into a single row". This should help make it easier to compare the before and after Mechanical Turk survey responses.
    There is a lot of data gathered during a course execution. For this purpose we only want to check the Mechanical Turk survey responses, but feel free to explore the other data available.
  6. Click "Create Report".
  7. The report will be built and downloaded. Open the csv file in a program like Microsoft Excel for easier readability.
  8. Each row represents a single Mechanical Turk worker's attempt. Find the two columns that start with "Mechanical_Turk_ID_...".
    • If both columns have values, then the Mechanical Turk worker reach the beginning and ending survey, verifying that they completed the entire course.
    • If only one column has a value, then the Mechanical Turk worker started the course, but quit before finishing it.
    • If neither column has a value, then the Mechanical Turk worker started the course, but quit almost immediately.