A Mechanical Turk-Integrated Development Environment system is disclosed. An integrated development environment (IDE) can include one or more interfaces capable of communicating with a mechanical turk engine. As a developer creates applications within the IDE, the developer can use the IDE to submit one or more requests to the mechanical turk engine. The engine constructs a mechanical turk project based on the requests and provides project tasks to workers. The results of the tasks can then be compiled and integrated back into the developer's application via the IDE. An example use includes constructing large domain specific data sets that can be applied to spoken dialog interfaces.