GeorgeTag script update #2

We have the latest and greatest version of the Georgetag script for you, ver. 1.1. This version corrects two bugs and adds two great new features (no more scrolling, and minimized clicking!):

1. Bug fix 1: If you happened to have first clicked in the "Enter ALL text found in image" text field before selecting a radio button, then selected the "no text" radio button, the box turned red and warned you that you needed to type at least one character. Now once "no text" is selected, the box does not give this warning.


2. Bug fix 2: If you had already selected "no text" then changed your mind and wanted to add text, previously you had to manually erase the "no text." Now, if you click "Contains Text", it will clear the text box, but only if it previously said "no text."


3. New Feature 1: Refocusing of text boxes when radio buttons are checked. When you click the "Contains Text" radio button, the cursor is automatically placed in the "Enter ALL text found in image" field. When you select "no text", the cursor is automatically placed in the "tags" text box. No more clicking around! Select and start typing! (Don't forget, the TAB will jump to the next selection as well.)


4. New Feature 2: Click ALT+w to jump to the next photo. (No more scrolling!) The ALT+w jump iterates through the photos, and once you get to the end, it will put you back at the beginning. Note: the first ALT+w takes you to the second image (since the first was already in view). However, if you forget and scroll the page yourself, the script will only jump to what it thinks is the next image. For example, say you do the first two images using ALT+w, then keep going to the 4th by manually scrolling, the next ALT+w will take you to the 3rd image. It's hard to describe, so you will have to play around with the feature.
   

If you can't wait to get your hands on the latest, greatest version 1.1, head over to the GeorgeTag Image Tagging Improved post, and install the new version. (You shouldn't need to un-install the old script, Greasemonkey should overwrite the old one.)

GeorgeTag script update

Georgetag has uploaded new Image Tagging HITs, with changed title and instructions. This caused ver. 1.0 of our script to break.

We have uploaded ver. 1.0.1 to the blog.

Visit GeorgeTag Image Tagging Improved for the latest version of the script. You shouldn't need to un-install the old script; Greasemonkey should overwrite it for you.

Requesters: Getting High Quality Results

Sounds like TagCow has had some great publicity and response to their enterprise! This means more work, but a few snags as well. Georgetag posted some questions to Turker Nation. These are problems that most Requesters will encounter, but particularly those with high-volume HITs. Although my response is particularly about the Image Tagging set, the tactics outlined below are general enough to be used by any Requester.

Quote:

Garbage tags (intentional
and unintentional):
We filter out meaningless words (the, it, an, a, with) but we are getting some totally irrelevant tags
Vulgarity (jokesters are hijacking the program)"
It looks like MTurk is doing some filtering and we are doing some filtering as well. (Can anyone confirm that?)
...
Incomplete taggings:
We have gotten some images tagged with "boy" where there is more that could obviously be said about the photo, like "boy", "playing", "trains"

There are several things you can do to keep the quality high. Here is a list of tactics implemented by other Requesters on mTurk. I'm certainly not suggesting you use all these ideas, but one or two might work well.

1. Have a good, representative list of examples, and a good description.
   I know folks over at Turker Nation have already mentioned this, but your description is very vague. We aren't certain if you want us to make a list of everything we see in the picture, or just keep it as simple as possible. A separate webpage with lots of examples will go very far. We can then emulate these examples.


2. Warn workers what response will be rejected, and what behavior will get them banned.
   A clear (but not overly-dramatic) warning might just be enough to scare off Roboform-type workers.


3. Require a minimum approval rating for workers.
   Some recent HITs by Amazon's Media Content, and the information extraction group had the approval rating greater than 90%. (Smart Travel Media also has this threshold.) This sounds about right to me. After doing 40k HITs, mine is 99.8%. In the forums, even those who do HITs with high rejections (like the items HITs) still seem to have above 90%. This will prevent some workers who are continually trying to "game" the system.


4. Offer a bonus for high-quality work.
   Georgetag is already offering up a volume bonus based on approvals, which is excellent! Few requesters do this, and more should. Once you get a good verification workflow established, you could track workers' responses and reward those that have submitted the highest quality and volume. Money talks.


5. Include "gotchas."
   Include some pictures that should have an obvious response, like a flower, bird, etc., and particularly ones with a word to include. Then you can start to weed out or ban workers who miss these images. The "are these items different" has a qualification set up and a large set of gotchas. When you miss one, your qualification goes down by 200 points. Basically, after getting 3 wrong, you get timed out for some length of time.


6. Ban very bad workers.
   And ban them quickly. If you haven't already implemented it, check if a single worker is giving you the same response (or few responses) over and over again. I wouldn't be surprised if workers are bypassing the "Enter ALL text found in image" step. Luckily this can be automated. And don't forget to give them some rejections if they replicate their responses beyond some acceptable level.


7. Set up verification HITs.
   After getting all the tags for a given image, you could then create a HIT where the worker verifies that the tags are relevant, and could let you know if any are vulgar or meaningless. An image with a row of checkboxes would be ideal, where you select any tags that are bad, and a comment field to let you know about anything unusual. Hopefully then you can get a great set of tags and you can identify the bad workers using another method.


8. Use a qualification test.
   This one might be a pain to grade, but you could have 5 images that the worker has to successfully tag before being able to do your HITs. Some qualifications even have a quiz about the purpose of the HIT and whether an example is appropriate or not. The quiz-style could be automatically graded.

Implementing 1, 2, and 3 is dead-easy. Making a nice page with a list of good and bad examples will go a long way to fixing some of your problems. I think some workers might be inadvertently giving you poor tags due to lack of understanding.

Politically, using any of the tactics 2-8 can be slightly tricky. (Item 1 should be done by all Requesters. Don't forget, the "description" field cannot be seen by workers once they are in the HIT.) You don't want to scare off your best workers, nor stop people from trying your HITs. Don't be too threatening, or too strict. Workers get very upset if they feel wrongly slighted, and will happily share with all on the Turker Nation forums.

To end on a positive note:
You'll find that most workers really do want to give you exactly the high-quality response that you require. When paid well, we are eager to perfect our responses, and love having discussions and feedback. Continue a good dialog, and you will have a group of willing, quality workers in no time!

GeorgeTag Image Tagging Improved

ver. 1.2.2

last updated: 06/25/08

Update 06/25/08 (ver 1.2.2): Fixed to conform to Georgetag's new name TagCow.

Update 4/8/08 (ver. 1.2): Added capability for user-defined skip image keyboard shortcut, through a "User Script Commands" Greasemonkey option.

Update 4/7/08 (ver. 1.1.2): Fixed to work with new Georgetag group, titled "Image Tagging - Describe what you see. Earn a volume BONUS! Click here to see how."

Update 4/4/08 (ver. 1.1): Fixes 2 bugs and adds 2 features. Click here to see new features.

Update 4/3/08 (ver. 1.0.1): Requester changed the instructions and title of the HIT. New version works with both the old and the new instructions.

This script has been downloaded times.

Description:

Does 5 things to the "Image Tagging" HITs by georgetag:

1. Places the 3 input fields next to each image (rather than below)
   
2. When "Does Not Contain Text" radio button is selected, the "text found in image" field is automatically filled in with "no text"
   
3. All images can be scaled based on a user-supplied percentage, to optimize the layout on your screen
   
4. Once text/no text radio button is selected, cursor is placed in appropriate text box.
   
5. Scroll through images using a ALT+w, or a user-defined keyboard shortcut.

Screenshot of HIT with script installed. Click on image for full-size version.

The requester georgetag, has uploaded a massive dump of image tagging HITs. They are offering a bonus based on the number of approved HITs, so we wanted to simply speed up the throughput of working on these HITs.

This script increases the efficiency of the Turker workflow by rearranging the layout and removing the reduncancy of the "no text" radio button/text field.

1. Rearranging the layout of the hit: Currently in these HITs, each image and its associated 3 input fields are are aligned vertically. We have created a 2x5 table, so that the 3 fields for a given image appear to the right of the image. This arrangement not only reduces the amount of scrolling, but it also allows you to see the tag text field and the image together on the screen.
   
2. Auto-filling of "no text": These Image Tagging HITs have a redundancy input, which is a time-waster for workers. Even when you click the radio button for "Does Not Contain Text", you must also fill out the second text field with "no text." This script automatically fills in the "text found in image" field with "no text" when the "Does Not Contain Text" radio button is clicked.
   
3. Resizing of Images:If you don't like the standard size of the images, you can resize all the images by a given percentage. I find that 75% makes the height of the image the same as the input boxes, but you might have a different preference. To set the scale, right click on the Greasemonkey face in the bottom toolbar, Select "User Script Commands", you will see a text box pop out that says "Image Tagging image size (currently 100%)". If you click that text, a pop-up window appears, and you can enter the image scale as a percentage. For example, to scale all images at 50%, enter "50" in this box. The image scale will take effect the next HIT you view.
   
4. New Feature 1: Refocusing of text boxes when radio buttons are checked. When you click the "Contains Text" radio button, the cursor is automatically placed in the "Enter ALL text found in image" field. When you select "no text", the cursor is automatically placed in the "tags" text box. No more clicking around! Select and start typing! (Don't forget, the TAB will jump to the next selection as well.)
   
5. New Feature 2: Click ALT+w to jump to the next photo. (No more scrolling!) The ALT+w jump iterates through the photos, and once you get to the end, it will put you back at the beginning. Note: the first ALT+w takes you to the second image (since the first was already in view). However, if you forget and scroll the page yourself, the script will only jump to what it thinks is the next image. For example, say you do the first two images using ALT+w, then keep going to the 4th by manually scrolling, the next ALT+w will take you to the 3rd image. It's hard to describe, so you will have to play around with the feature.
   
6. User-defined skip image keyboard shortcut. You can change the keyboard shortcut to be a more convenient combination. See Instructions below for directions.

Other important notes:

The Image Zoom add-on is a helpful extension for these HITs. If you set a small image scale, you can use Image Zoom (on right-click) to zoom in and out of each image for inspection.

Please note that this script was made on-the-fly, and will cease working if the Requester changes the HIT. In addition, we have only tested the script on Firefox ver. 2.0.0.13 and Greasemonkey 0.7.20080121.0. If you find bugs, please let us know, and we'll try to sort out what's going on, time permitting.

So you are aware, we have already had HITs approved by the requester for which we used this script.

Instructions:

You must have Firefox, with the Greasemonkey extension installed.

Image scale can be set via the Greasemonkey "User Script Commands." The image scale defaults at 100%.

You can change the skip image keyboard shortcut. From the User Script Commands menu in Greasemonkey, choose "Image Tagging change skip key." In the dialog box, you can type in your choice. The format is "ctrl" or "alt" or both ("ctrl+alt"), then the "+" key, then a letter or number. (Any character that requires the shift key won't work, but capital letters are ok.) Examples are: ctrl+k, or alt+z, or ctrl+alt+q. If you input an invalid choice, the dialog box will reappear. The new choice appears in the header of the HIT, and the default remains alt+w.

---

Disclaimer:
Our scripts are provided "as-is". We always aim to provide a well-tested and useful script that aids in your Turking and causes no adverse effects. Given the huge variety of configurations on which our scripts might be used we can never guarantee that something won't go wrong. We take no responsibility for any inconvenience, increased rejection rate, blocking by a requester, loss of income or damage or any other problem that use of our scripts might cause. We recommend that you only use HIT-specific scripts on HITs that you're very familiar with. When you use HIT-specific scripts, treat it as if you were starting a new type of HIT with a new Requester - try doing a few, then wait to be sure that they're getting accepted.

mTurk is for the dogs

Oh dear! The latest post at the AWS blog finally reveals in print how Amazon feels about the Workers on mTurk. (Something we have known all along! Where's that Worker mTurk API?) The guys in the development section have developed a DCI - The Dog Computer Interface.

They train the office dog, Rufus, to do mTurk hits. And what would a dog do with his reward? They are using dog biscuits instead of the Amazon Payments system, clearly violating the Terms of Service. No wonder our rejection stats have been down the tubes on the items HITs! I am so furious!

mTurk is for poor, starving Workers, period. I highly doubt the ultra-cute Rhodesian Ridgeback would have the troubles I have begging for food. I can't even get strangers to scratch behind my ears! Leave those penny bear HITs for me.

...and their DCI post was published on April 1. As is this one. ;-)

Although, the funniest jokes are ones that have a hint of truth in them.