NEXT: A system to easily connect crowdsourcing and adaptive data collection

1. Introduction

1. Problem statement
   1. Crowdsourcing is data collection from many humans
   2. Collecting of crowdsourced data can be expensive
   3. Especially with simple tasks with objective quality goal
   4. Adaptive sampling algorithms study exactly this. Use all previous responses.
   5. Objectively better results (show graph)
   6. ...but hard to connect with crowdsourcing (show data flow)
   7. Even more difficult to allow practical use (accessible thru cloud, practical use by collab)
2. NEXT
   1. UW Madison has a solution which answers the challenges presented by crowdsourcing + adaptive sampling.
   2. Adaptive sampling algorithm delivers promise of higher productivity in the class of problems NEXT is optimized to solve.
   3. Cloud based, user configurable tool.
   4. Has been effectively used by collaborators in a variety of disciplines.
   5. Implemented in Python, works at scale, REST API
   6. Below, will describe
      1. Example problem NEXT can solve
      2. The interfaces that NEXT provides
      3. System goals and features
3. Example problem
   1. What does TNY want?
   2. What does NEXT present?
   3. How does sampling alg work?
4. NEXT - System created by UW Madison to ……
   1. Machine Learning Background
      1. How do adaptive sampling algorithms work?
      2. More ML vocabulary
   2. System goals
      1. wide experimentalist use,
      2. simple algorithm development
      3. experiment monitoring tools
   3. NEXT Architecture and Usage
      1. Terminology
         1. Have block diagram
         2. Users, participants, experiment, query, interface
      2. Interfaces
         1. What interfaces does it provide? Show pictures.
         2. Where are applications are these interfaces?
         3. What are some theoretic guarantees?
      3. Experiment monitoring
         1. Results
         2. Experiment info (date launched, etc)
         3. Timing information
      4. Experimentalist use
         1. Running NEXT
         2. Experiment launch
         3. Experiment save/restore
      5. Algorithm development
         1. Required functions
         2. Arguments and returns for each function
         3. Database access
         4. Example
5. Conclusion
   1. Problems like ….. Can be solved using adaptative sampling techniques such as implemented in NEXT.
   2. NEXT is available to many applications but has particular applicability to crowdsourced data that benefits from adaptive sampling in refinement.
   3. NEXT is user configurable and experiments can be monitored in process.
   4. Shown system that connects
   5. Easy to use in collaboration
   6. In practice, works at scale.