How do you ensure quality?

Our entire platform was built to execute tasks at scale, but with quality as the #1 goal. Here are some ways we’ve integrated quality.

First, any judgment (aka any task done through the Squad human task interfaces) is done by a known contractor that has passed a series of ‘Capability’ tests:

1. Before starting out as a contractor on the Squad app, each applicant must pass our general skills test (about 1 out of 10 applicants pass this phase)
2. Each accepted contractor has a profile and reputation in our system that relates back to every task they’ve ever done. We understand their skills, experience, education, task history, and quality scores. We then group contractors according to various skills.
3. Each contractor demonstrates skills that are needed for a specific task (for example: Moderation skills). We only open up specific tasks to pools of contractors that are accepted in particular skill groups (for example: Moderation, American Culture, Calling Tasks, etc)
4. Finally, each contractor must pass a Task Specific Training also known as a Qualifier. Once they demonstrate they know the rules and have passed sample tests, we allow them to access the task set.

Second, since all tasks are completed through our human task interfaces, Squad has the ability to measure and track all actions (clicks, time spent on actions, right and wrong answers, etc). As such, we are able to measure ‘Intent’ down to the tiniest detail. Here are some of the tools we employ to ensure quality.

1. We leverage several QA techniques depending on cost and quality considerations including Sampling Maker Checker formats, Multiple Judgments, Gold Standard Data, and other QA techniques.
2. Contractors only get paid for tasks that are correct, as validated by multiple types of Quality Assurance checks as mentioned in point 1. Also, we are able to measure timing. So for example, if a task should take an average person 8 seconds, if a contractor does it in 1 second, then we raise a flag in our system and ensure that that task is reviewed by our team. If any abuse is found, we remove that contractor from our system, they do not get paid, and the task is put back into the system. Fairness and accuracy at the heart of the system.

Finally, whenever possible we leverage artificial intelligence to accomplish work. Each time we do that, we receive a confidence score as an output. Every time the confidence score falls below an allowable threshold, we use human(s) to judge those results to ensure accuracy, and further retrain the system to improve accuracy the next time around. This is called using a Human-in-the-Loop. At the end it is the best balance of Humans + AI together to accomplish the highest quality output while still accounting for other success factors like cost, flexibility, and speed.