Tidal join the rising tide

For our project we create a platform that recursively divides the work until workers are handed bite sized tasks. Many ideas concerning worker retention, hierarchies, and motivation were brought up in “The Future of Crowd Work”. Our goal is to adapt many of these ideas into the current platform designed by Turkomatic, which is already organized in a hierarchical fashion, with “branch” workers breaking up and delegating tasks to other workers further down the subtree. Let us call this hierarchy a “task-tree”, where workers who recursively divide up tasks are called “branchers”, workers who finish a task are “leafers”, and workers who recombine completed tasks are “sappers”.

This method of “divide and conquer” work-flow is will implement a novel form of 1-level supervision where branch-workers must approve the any actions of their branch-designated children. Essentially, as the tree grows each new node must be approved by its parent before being released as a new task. This means that branch workers must remain on the site for a while until their children are ready to be approved. However, this will not be a problem because while each brancher waits, they are casually monitoring the tasks and can perform other HITs.

On the other hand, sappers use the data submitted by the original brancher, as well as see the original input and output requests that were made for this level in the tree. Sappers can give feedback on the quality of the work from both leafers and branchers. Sappers are penalized if their work is rejected further up the tree by sappers closer to the root node. Currently this is the only way to get feedback for sappers since no other worker class can observe their work. This method encourages branching workers to create easily followed instructions, improves the decision making of the entire tree, and helps make more uniformly structured solutions.

The question raised in the “future of crowd work” as to whether or not workers should choose jobs, or platforms should push jobs. We aim to mix the two options by allowing users to choose which jobs they prefer (either branching, leafing, or sapping), but we will rank workers within each field so that we can give the most important jobs to the best workers. For example, when starting a tree we will offer the root node to the best branching workers we have available. Turkomatic noted that they saw great improvements in their task-tree when the first few branches were made by professionals or workers with great care (and also immediate degradation in quality with a lazy brancher early on).

The ranks of the workers will be determined on a “point” based system, which improves the feedback and rewards for branch-workers. Table 1 shows a loose idea of what some categories might be that help determine the points each worker can score per HIT.

This system also leaves a great amount of room for including AI practices and machine learning algorithms that could take advantage of other information to help us guide workers to where they will be most efficient in their chosen field, and how much to reward them

Design Implementation Overview

Workers choose whether they want be branchers, sappers, or leafers before they are automatically given tasks for their chosen field. Fig1 shows the intended interconnect diagram of our system.

HITs are automatically chosen for workers who will “preview” the HIT before deciding whether they want to accept it or not.

As workers accumulate more and more points they will be automatically given higher priority HITs (e.g. branch nodes that are closer to the root), which should improve the overall performance of future task-trees.

Branch workers can only branch a maximum number of children (TBD, likely 5). Branch workers will say whether their child tasks are branches or leaves upon creation.

Branchers have to approve the tasks that their child processes attempt to spawn. Branchers have to write stubs for each function that they request and while they’re waiting to supervise they will write test cases. This added work will give branchers a good reason to stay around long enough to supervise their children.

If we use something like JS/python then we can test compiling in the browser. The leafer can get the test case and use it to test their stuff before they submit it to the sapper (we have to tell them though that the test cases might not be correct and they can submit even though it doesn’t pass the brancher’s test).

Sappers can rate branchers on how well the test cases helped them debug. This could lead to a “checks and balances” type system between brancher/sapper/leafer.

Payment: BASE_PAY + (score_for_this_hit / max_score_per_hit) * score_bonus

Sappers are recombining the programs at each node back up the tree to make sure everything compiles and actually does what it say it does. If they encounter a problem then they identify which child gave a bad output and we will respawn that subtree. We will do some penalization on that faulty subtree.

If a leaf gets starved out then we increase the pay over time to some max amount. Then if nobody still takes it until a max timer runs out we then give it back out at a branch task (the root of a new tree, because their supervisors left already). Branchers who approve tasks that get starved out will get negative points.