Are These Items Different Improved

ver. 1.0.1

last updated: 04/08/08

Update 04/08/08 (ver. 1.0.1):Bug fixed. When you use keyboard shortcut to make your selection with the auto-submit off, you can now hit 'enter' to submit the hit.

This script has been downloaded times.

Description:

Provides general cleanup and streamlining to the "Are these items different" HITs. Allows keyboard shortcut for radio buttons and provides option to auto-submit.

Screenshot of HIT with script installed.

On the standard Items HITs, the two items and their descriptions are arranged vertically with the radio buttons at the bottom. This requires a fair amount of scrolling for this quick-decision HIT. This script streamlines the HIT for quicker comparison and submission.

The Layout: The radio buttons are moved to the top of the HIT, and the items are placed side-by-side.

The blue box that describes the HIT has been streamlined to be as short as we could make it. (We removed the qualification information, and put the rest of the text on one line.)

The instructions are hidden using the show/hide scriptlet. Amazon has 6-8 different versions of the instructions, so it will take viewing quite a few HITs before you get all the instructions accepted into the routine. Once you've reviewed the instructions, click "Don't show me about this change again."

View of the instructions warning.

Selecting radio buttons: User can choose the radio button with a keyboard shortcut. The default shortcuts are: "s" selects the "These items are exact duplicates" button, and the "x" selects "These items are different." radio button. You can customize the keyboard shortcuts under the "User Script Commands" menu, using "Change hot keys." To change the hot keys, separate your two selections by a space. The first selection is the "exact duplicates", and the second is the "different." You can choose letters, numbers, and include ctrl and alt with them by adding a "+." So, for example, if you would like y for same and n for different, you type: "y n" Or, if you'd like ctrl+y and alt+n, you type:"ctrl+y alt+n". If you input an incorrect combination, the input dialog box will reappear.

Auto-submit option: The ultimate in speed. User can choose to have the HIT automatically submitted upon pressing "s" or "x", or the user-defined keyboard shortcuts. Note: this is an advanced feature. You must be very confident in your selection abilities to use this feature successfully. When I first gave it a try, I made several errors before getting used to the selection process.

To turn the auto-submit on, right-click on the Greasemonkey icon, then choose the "User Script Commands" menu option, and choose "Toggle auto submit." The state of the auto-submit toggle is shown at the top of the HIT.

Note: When the auto-submit is on, manually selecting the radio button with a mouse click does not automatically submit the HIT.

Other Notes:

If you're really looking to speed up your Turking, open two tabs in one window with active Items HITs running in each window, with the "Automatically accept next HIT" selected. While you are working on one HIT, the other is loading. With this script and the auto-submit on, you should be able to type, e.g.: x, ctrl+PageDown, x, ctrl+PageDown, s, ctrl+PageDown, etc. Both ctrl+tab and ctrl+PageDown move to the next tab in Firefox. Lickety split!

Instructions:

You must have Firefox, with the Greasemonkey extension installed.

User can choose to have selection automatically submitted via the "User Script Commands", choosing "Toggle auto submit." The state of the auto submit is stated at the top of the HIT.

Keyboard shortcuts can be changed via the "Change hot keys" menu option. The two selections should be separated by a space. You can choose any letter, number or include ctrl+ or alt+. Your new selection will be printed immediately after the radio button.

---

Disclaimer:
Our scripts are provided "as-is". We always aim to provide a well-tested and useful script that aids in your Turking and causes no adverse effects. Given the huge variety of configurations on which our scripts might be used we can never guarantee that something won't go wrong. We take no responsibility for any inconvenience, increased rejection rate, blocking by a requester, loss of income or damage or any other problem that use of our scripts might cause. We recommend that you only use HIT-specific scripts on HITs that you're very familiar with. When you use HIT-specific scripts, treat it as if you were starting a new type of HIT with a new Requester - try doing a few, then wait to be sure that they're getting accepted.

Show/Hide Scriptlet

The show/hide scriptlet is a small piece of script that we use in many HIT-specific Greasemonkey scripts. Because it is embedded, it is not a script that can be downloaded independently.

Description:

This scriptlet hides the instructions in a HIT. The instructions can be viewed by clicking on "show instructions." If the instructions change, a warning will appear, saying "Instructions appear to have changed. Click link above to view instructions." After the user views the changed instructions, they can click "Don't warn me about this change again" to accept the new set of instructions. After clicking this link, the changed-instructions warning won't appear again, unless the instructions change another time.



Other notes:

We don't think instructions should ever be completely removed from a HIT. A worker needs the ability to review the instructions as necessary. However, after doing hundreds, if not thousands of HITs, a worker does not need to scroll through identical instructions hit after hit.

On first install, the user will receive the changed-instructions warning, and will have to click "Don't warn me about this change again."

GeorgeTag script update #3

We've released version 1.2 of GeorgeTag Image Tagging Improved.

Here's the latest improvement:

You can customize the keyboard shortcut! Don't like the Alt+w, or it doesn't work for you? Choose ctrl+k, or alt+z, or even ctrl+alt+q. Pretty much whatever combination you find most convenient.

From the User Script Commands menu in Greasemonkey, choose "Image Tagging change skip key." In the dialog box, you can type in your choice. The format is like the examples above... you can type ctrl or alt or both, then the "+" key, then a letter or number. (Any character that requires the shift key won't work, but capital letters are ok.) If you input an invalid choice, the dialog box will reappear. The new choice appears in the header of the HIT, and the default remains alt+w.

Hopefully this improvement helps those of you who have non-English set-ups, or even the left-handed folks.

Open HITs Warning

ver. 1.0

last updated: 04/06/08

This script has been downloaded times.

Description:
Colors the "HITs" tab at the top of each mTurk page a discreet but noticeable shade of red when you have a HIT assigned to you. This provides a visual notification to help you avoid letting HITs expire.

Screenshot of mTurk with the script installed when a HIT is assigned.

This script checks whether you have an active HITs assigned to you (i.e. HITs which you've accepted but have not yet completed). If you do have active HITs assigned, it changes the color of the "HITs" tab at the top of the mTurk page to a discreet shade of red to provide a visual notification that you have work that needs completing. The tab will appear red no matter which mTurk page you are viewing.

It's all too easy to forget that you've accepted a HIT and allow it to expire as a result, hurting your qualifications score. With this script installed it will be much harder to forget!

Other Notes:
The tab color will only update when a page is refreshed or a new page opened - so if you have two mTurk windows open and accept a HIT in one, the "HITs" tab in the other won't turn red unless you refresh or open a new page in that window. This script will not warn you about HITs that have already expired (it's too late to do anything about them anyway!).

Instructions:
You must have Firefox, with the Greasemonkey extension installed. There are no options to set.

---

Disclaimer:
Our scripts are provided "as-is". We always aim to provide a well-tested and useful script that aids in your Turking and causes no adverse effects. Given the huge variety of configurations on which our scripts might be used we can never guarantee that something won't go wrong. We take no responsibility for any inconvenience, increased rejection rate, blocking by a requester, loss of income or damage or any other problem that use of our scripts might cause. We recommend that you only use HIT-specific scripts on HITs that you're very familiar with. When you use HIT-specific scripts, treat it as if you were starting a new type of HIT with a new Requester - try doing a few, then wait to be sure that they're getting accepted.

Requesters: Getting High Quality Results, Part 2

As a follow-up to my original post, Requesters: Getting High Quality Results, there are several more strategies Requesters can use to keep their quality high. You might consider these to be advanced strategies.

9. Use groups of HITs to "grade" workers:
   Rather than using a short qualification test, Requesters can use HITs to grade workers. To do this, you can run several batches of HITs at various times of the day, and analyze the returned data carefully. Then, assign the top workers a qualification value, and only allow workers above that threshold to work on your future HIT groups. Amazon Media Content recently tried this method for their advanced HITs. The downside of this grading method is that you only catch those workers who happened to find your HITs when you posted them. In addition, you might not have a large enough group of workers to draw on, and over time you can have dwindling numbers of workers completing your HITs. To rectify these problems, you will occasionally have to post open HITs to re-grade workers.
   
10. Use sliding grade scale:
    This is like using a dynamic qualification test. After setting a grade (either via a qualification test, or via an assigned value), Requesters can assign qualification points for every approved response, and take away points for every rejection. This way a worker is motivated to give the highest-quality responses. CastingWords transcription HITs work this way. Don't forget to tell workers this is your methodology!
    
11. Include/exclude countries
    Besides qualifying workers based on their approval/rejection stats, Requesters can restrict or allow workers based on their location. Using a locale qualification is a blunt instrument, but sometimes it's the only instrument available.
    
    At least one Requester has previously mentioned that upon analysis, they noticed poor quality work returned from particular countries, perhaps due to language issues. Unfortunately, there is no Amazon "system qualification" for language. When a worker looks at their locale qualification, it states:

    The Location Qualification represents the location you specified with your mailing address. Some HITs may only be available to residents of particular countries, states, provinces or cities.

    Unfortunately, there is no "OR" operator when you list locale codes. This means that, as stated in the AWS mTurk Requester documentation, you cannot, for example, allow workers from US or GB or AU or NZ or CA. The best you can do is to exclude workers from a list of countries. What's unfortunate yet again, is that Amazon restricts the number of qualifications for a HIT group to a maximum of 10. (See this document and search for "QualificationRequirement".)
    
    Establishing HITs based on the location of the worker is fraught with political problems. Workers frequently complain on the message boards at Turker Nation when a Requester only allows workers from a particular country (especially the US). Using a locale qualification to effectively impose a language qualification will inevitably unfairly block some works who speak the language fluently. Be aware that some workers will be annoyed by this, but you might not have any other way to apply a fluency test.
    

As a Requester, you are well within your rights to include and exclude workers who don't meet your criteria to help you achieve the highest quality results. Nowhere in the terms of service does it state that you have to be "fair" in excluding workers. In fact, Amazon won't arbitrate in any Requester-Worker disputes, as stated in the Participation Agreement.

However, the more restrictions and complicated hoops you place in your process, the less likely a worker will work on those HITs. You should also be prepared to respond to many inquiries any time you exclude any group of people. If it appears that you aren't being fair, you could also get a bad reputation, which will scare off other workers.

Although you may have to test out different strategies, when implemented correctly these 11 strategies can help Requesters achieve a high quality return on investment. Workers who achieve the "elite" qualifications take pride in their status, and might well put your HIT groups at the top of their to-do list.