Requesters: Getting High Quality Results, Part 2

As a follow-up to my original post, Requesters: Getting High Quality Results, there are several more strategies Requesters can use to keep their quality high. You might consider these to be advanced strategies.

9. Use groups of HITs to "grade" workers:
   Rather than using a short qualification test, Requesters can use HITs to grade workers. To do this, you can run several batches of HITs at various times of the day, and analyze the returned data carefully. Then, assign the top workers a qualification value, and only allow workers above that threshold to work on your future HIT groups. Amazon Media Content recently tried this method for their advanced HITs. The downside of this grading method is that you only catch those workers who happened to find your HITs when you posted them. In addition, you might not have a large enough group of workers to draw on, and over time you can have dwindling numbers of workers completing your HITs. To rectify these problems, you will occasionally have to post open HITs to re-grade workers.
   
10. Use sliding grade scale:
    This is like using a dynamic qualification test. After setting a grade (either via a qualification test, or via an assigned value), Requesters can assign qualification points for every approved response, and take away points for every rejection. This way a worker is motivated to give the highest-quality responses. CastingWords transcription HITs work this way. Don't forget to tell workers this is your methodology!
    
11. Include/exclude countries
    Besides qualifying workers based on their approval/rejection stats, Requesters can restrict or allow workers based on their location. Using a locale qualification is a blunt instrument, but sometimes it's the only instrument available.
    
    At least one Requester has previously mentioned that upon analysis, they noticed poor quality work returned from particular countries, perhaps due to language issues. Unfortunately, there is no Amazon "system qualification" for language. When a worker looks at their locale qualification, it states:

    The Location Qualification represents the location you specified with your mailing address. Some HITs may only be available to residents of particular countries, states, provinces or cities.

    Unfortunately, there is no "OR" operator when you list locale codes. This means that, as stated in the AWS mTurk Requester documentation, you cannot, for example, allow workers from US or GB or AU or NZ or CA. The best you can do is to exclude workers from a list of countries. What's unfortunate yet again, is that Amazon restricts the number of qualifications for a HIT group to a maximum of 10. (See this document and search for "QualificationRequirement".)
    
    Establishing HITs based on the location of the worker is fraught with political problems. Workers frequently complain on the message boards at Turker Nation when a Requester only allows workers from a particular country (especially the US). Using a locale qualification to effectively impose a language qualification will inevitably unfairly block some works who speak the language fluently. Be aware that some workers will be annoyed by this, but you might not have any other way to apply a fluency test.
    

As a Requester, you are well within your rights to include and exclude workers who don't meet your criteria to help you achieve the highest quality results. Nowhere in the terms of service does it state that you have to be "fair" in excluding workers. In fact, Amazon won't arbitrate in any Requester-Worker disputes, as stated in the Participation Agreement.

However, the more restrictions and complicated hoops you place in your process, the less likely a worker will work on those HITs. You should also be prepared to respond to many inquiries any time you exclude any group of people. If it appears that you aren't being fair, you could also get a bad reputation, which will scare off other workers.

Although you may have to test out different strategies, when implemented correctly these 11 strategies can help Requesters achieve a high quality return on investment. Workers who achieve the "elite" qualifications take pride in their status, and might well put your HIT groups at the top of their to-do list.