Artificial Intelligence and Work: AAAI 2019 Fall Symposium

A two and one-half day symposium was held 7–9 November 2019 in Washington, DC, USA, to discuss and plan how AI researchers will contribute to research on human work with artificial intelligence. The symposium schedule and papers are available (NB. you must be logged in to the website to see the papers).

Draft Symposium Schedule

Day 1

9:00 Welcome and overview
9:30 Opening talks
11:00 Panel
12:30 Small group discussions of research questions over lunch
2:00 Panel
3:30 Small group project discussions
6:00 Symposium reception

Discussions. The heart of the symposium will be small group discussions on research gaps and plans for future research. Example topics include public acceptance, organizational change management, legal issues, security, education and careers. Attendees will be asked to discuss what is known and what needs to be learned about these topics. Each group will have one or more of the panelists as a convenor and will select one or more members to take notes capturing the discussion. Written products of the groups (e.g., brainstorming notes) will also be captured. On the first days, groups will refine a list of possible transdisciplinary research problems, starting with those that have been identified in other RCN workshops and adding to them. Convenors will be provided with a structured agenda to guide the process of ideating, clustering and ranking research ideas. The lists will be shared as posters at the end of the day, to allow participants to see connections or even switch groups for the second day.

Day 2

9:00 Panel
11:00 Small group project discussions
12:30 Research round tables over lunch
2:00 Overview of NSF activities
2:30 Small group project discussions
4:00 Small group report back and plenary preparation
6:00 Symposium plenary session

Discussions. The breakout groups will continue work on the second day. Discussions on the second day will focus on ways to advance the research (e.g., build shared language and share resources needed for these problems), with a particular focus on what AI researchers have to contribute and where other disciplines can inform AI research. In the afternoon, the work of the breakout groups will be discussed in a plenary session of the symposium before being presented in summary in the overall plenary sessions.

Day 3

9:00 Talk
AI for a Generative Economy: intelligent systems and work in a just and sustainable future
Ron Eglash, Lionel Robert, Audrey Bennett, Kwame Robinson, Michael Lachney and William Babbitt
9:30 Small group discussions of next steps
11:00 Small group discussions of next steps
11:30 Report on plans for next steps and call to arms
12:15 Wrap up and end

Closing discussion. The symposium will close with a group design activity about what additional resources (e.g., website, shared datasets, research protocols, impact cases) participants think would be useful in supporting interdisciplinary convergence research and skills development on the topic of work with AI in the coming years. We will encourage participants to reflect on the potential for developing collaboration spaces to support research as well as what they might be able to contribute directly.

Wrap up. The final activity of the symposium will be a short evaluation of the activities and reflections on what was learned.

Symposium Theme

The symposium theme is Artificial Intelligence and Work. AI technologies are characterized by increased autonomy: an ability to learn and an ability to interact with other systems and with humans. While AI-enabled systems do not possess the generalized intelligence that humans have, they are increasingly capable of performing tasks that have traditionally been the sole purview of humans. For example, machines can now recognize images or speech in particular domains with an ability approaching or surpassing that of humans [e.g., 3]. These abilities enable them to perform useful work, sometimes more accurately than humans, with greater speed and at less cost. (For the purposes of this symposium, we adopt the definition of work from a recent NSF solicitation, NSF 19-541, “mental or physical activity to achieve tangible benefit such as income, profit, or community welfare”). The potential impacts on human work and employment are immense, calling for a coordinated research response. While the topic is touched on in many venues, there is as yet no regular conference on the topic and AAAI Symposia have not yet addressed the topic of AI and human work specifically.

Much of the rhetoric about work and AI focuses on people being put out of work by automation. But this perspective is too simplistic: tasks that can be automated rarely stand in isolation [2] and are defined by context. For example, consider an automated system that diagnoses skin cancer [3]. To be practicable, a system needs to fit the complex work of a medical practice. Someone must order the imaging, image the correct body part using the right lighting, explain the diagnosis to the patient, family members or other doctors (in varied and appropriate ways), bill insurance companies, monitor ongoing performance, anticipate potential malpractice suits and so on. All this surrounding work must adapt to an automated dermatologist and vice versa.

While AI technology is advancing quickly, the human side of the relationship—comprising people, organizations, legal frameworks and social values, to name a few—is evolving much more slowly. The result is a potential mismatch between the technologies and the organizational and individual contexts of use that risks unexpected or undesired consequences. Undesired consequences include deskilling that weakens the ability of workers to manage the system or adapt to future jobs, the creation of overly fragile systems and automation surprises, when machines behave in unexpected ways, e.g., learning anti-social behaviours.

As a result, understanding how AI systems will be used and affect human work is a pressing challenge. How we design organizations, how we design education and how we build systems will all be impacted. The likelihood of this transformed future leads to a series of questions that will anchor the proposed symposium. To name a few:

  • How are contemporary AI systems being deployed and what effects on human work are we seeing?
  • What can be done in the near future to more effectively design and deploy AI, while at the same time improving the knowledge and satisfaction of human workers?
  • Given what we know about work, what are some untapped opportunities or open challenges for AI-based systems?
  • How will technologies currently on the horizon affect work and what are reasonable anticipatory strategies, in particular, related to training and education?
  • What are possible second- and third-order effects on work of AI, both current and on the horizon, and how might these be anticipated and mitigated?
  • How do impacts differ across settings, industries, socio-economic status and geography?

Aim and Goals

To foster our goal of convergence we will bring together researchers who share an interest in work and a deep knowledge of the potentials and limitations of AI. An AAAI symposium will help ensure that the AI research community perspective is fully represented in the activities of the RCN and conversely, to enrich discussions in the AI community with perspectives from other disciplines. The symposium will provide a forum to advance three specific goals.

First, we seek to frame important research problems around work and AI. Roco et al. suggested that, “using forecasting, early signs of change, scenario setting, and other approaches, it is possible to establish a credible vision for what is desired in the longer term for a knowledge and technology field” [4:149].

Second, we want to develop a more integrated language base about these problems and issues that can span disciplines. As the NSF notes, “as disciplines interact, the knowledge, theories, methods, data, research communities and languages are increasingly intermingled or integrated” [1].

Third, we want to define needs for supporting technologies and resources that can promote convergence in the community of scholars interested in this topic.

In achieving these goals, we acknowledge the need for sustained interaction with industrial and policy partners who are an important audience for the research. In particular, we anticipate that the location in Washington, DC will enable attendance by government employees.

Participants. The maximum number of participants will be 40, to balance the desire for a high level of interaction with the desire for diversity among participants. Ideal participants for this symposium will be big picture thinkers interested in addressing the simultaneous consideration of technical, individual, group, organizational and societal issues.

Products. One product of the symposium will be a white paper based on the breakout groups summarizing what we know, priorities for needed research and possible research projects. A second outcome will be an article in a practitioner journal related to one of the topics in the workshop, co-authored by the a subset of the organizers and symposium participants. For example, work by one of the investigators and his collaborators, based on a case study of automation, was recently published in Communications of the ACM [5]. A third possibility is a journal special issue on the topic. Finally, a key success metric will be the formation of productive connections among attendees supporting interdisciplinary research and convergence.


[1] National Science Foundation. (2017). Convergence Research at NSF. Available from

[2] Chui, M., Manyika, J., & Miremadi, M. (2015, November). Four fundamentals of workplace automation. McKinsey Quarterly, 1–9. Available from: functions/digital-mckinsey/our-insights/four-fundamentals-of-workplace-automation

[3] Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639), 115–118. doi: 10.1038/nature21056

[4] Roco, M. C., Bainbridge, W. S., Tonn, B., & Whitesides, G. (Eds.). (2013). Convergence of Knowledge, Technology, and Society: Beyond Convergence of Nano-Bio-Info-Cognitive Technologies. World Technology Evaluation Center, Inc. Available from

[5] Seidel, S., Berente, N., Lindberg, A., Nickerson, J. V., Lyytinen, K. (2019). Autonomous tools and design work: A triple-loop approach to human-machine learning. Communications of the ACM, 62(1), 50-57, doi: 10.1145/3210753

Thursday, November 7, 2019 - 08:00 to Saturday, November 9, 2019 - 15:00

Could Machine Learning Be a General-Purpose Technology? Evidence from Online Job Postings

Bledi Taska, Avi Goldfarb and Florenta Teodoridis

There has been a great deal of speculation that machine learning might be a general-purpose technology; however, the commercial application of machine learning is relatively new and general-purpose technologies are typically identified with the benefit of many years of hindsight. It is useful to have an early sense of whether machine learning is a general-purpose technology in order to understand whether distinct complementary business practices and the help of third- party service providers will be needed in order to benefit from the technology. In this paper, we provide an approach to identifying a general-purpose technology before it has widely diffused. Using data from online job postings, we compare machine learning to eight other emerging technologies in terms of breadth of industries with job postings, the importance and breadth of research roles, and the costs of innovation in organizational practices. Our results show that ML is particularly likely to be a general-purpose technology, suggesting that firms adopting ML should be patient, should expect to implement changes to organizational processes, and should recognize that their industries are likely to change as a result. In contrast, firms adopting other technologies should look for more direct and tangible benefits.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Middle Class Jobs Amid Highly Automated Systems?

Laura Cesafsky, Erik Stayton and Melissa Cefkin

An operator sits in front of a giant, curved monitor on an otherwise Spartan white desk. With mouse and keyboard, she interacts remotely with an autonomous vehicle (AV) out on the roadway that needs, and has ‘called for,’ her help. The AV ‘wants’ to go around an obstacle—a double-parked delivery vehicle—that impedes its progress, but it is not sure if it should execute an overtake maneuver. The young woman examines the scene, clicks a series of buttons and, in response to her input, the car cautiously edges out, crosses the double yellow line, and drives around the obstruction to continue on its journey.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Algorithmic Management and Algorithmic Competencies: Understanding and Appropriating Algorithms in Gig work

Mohammad Hossein Jarrahi

Data-driven algorithms now enable digital labor platforms to automatically manage transactions between thousands of gig workers and service recipients. Recent research on algorithmic management outlines information asymmetries, which make it difficult for gig workers to gain control over their work due a lack of understanding how algorithms on digital labor platforms make important decisions such as assigning work and evaluating workers. By building on an empirical study of Upwork users, I make it clear that users are not passive recipients of algorithmic management. I explain how workers make sense of different automated fea- tures of the Upwork platform, developing a literacy for understanding and working with algorithms. I also highlight the ways through which workers may use this knowledge of algorithms to work around or manipulate them to retain some professional autonomy while working through the platform.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Skill shifts and skill rebundling as machines take on cognitive tasks

Shiyan Zhang, Michael Zur Muehlen and Jeffrey V. Nickerson

Artificial intelligence (AI) is playing an increasingly important role in the workplace, potentially disrupting the nature of many occupations. While currently, the use cases of AI seem to be focusing on specific industries and sectors, we believe AI has the capacity of impacting tasks and jobs across broader occupation groups and changing the future of work because it is a general technology.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Augmenting human intelligence in infrastructural and creative work

Joel Chan

With new waves of progress in AI across diverse fields of planning, problem solving, and natural language processing, the question of how AI might change the present and future of work is a pressing one. Within this larger question, I am interested in discussing how we can design AI to augment (not replace) human intelli- gence in infrastructural and creative work. This ques- tion manifests in two sets of issues that intersect with my own work, which I would be keen to share to stimu- late discussion: 1) supporting classification work while retaining nuance, 2) exploring how ”errors” in AI sys- tems might be harnessed for creative work (rather than designed out entirely).


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Modeling the Future of Workforce Development in the Age of AI

Ellen Scully-Russ and Farhana Faruqe

It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

AI Tools to Augment Fluid Intelligence

Lydia Chilton

The most important problems of our day involve interdisciplinary challenges. No one person has all the required expertise and this creates gaps in knowledge, communication barriers, misidentifying the true nature of problems, and the risks of interrupting existing workflows. Part of the future of work will involve building the AI tools that do the low-level tasks, so that we augment peoples ability to work at the high level of putting pieces together in new ways and adapting to the environment. In psychology, this is referred to as fluid intelligence - and it is valuable in solving problems within fields and across fields. In my own work, I have built systems that show how computation and AI can support people’s fluid intelligence in the entire iterative design workflow. There is always a backup approach for when AI fails, and the AI is giving constant feedback to the people running the workflow to control its direction. These systems are a concrete instance of how AI can do low-level tasks in a workflow and thus augment our ability to think fluidly and adapt.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Powering Collective Action with Artificial Intelligence

Saiph Savage

State of the art systems fall short at helping people to execute ambitious goals. For instance,
creating new infrastructure to support the indoor navigation of the blind. The main challenge is
that citizens have heterogeneous knowledge. As a consequence, they usually lack the skills to
complete the work needed to reach the collective goals. To address this problem, I introduce a
novel architecture conformed of Intelligent Civic Blocks. My Civic Blocks use artificial intelligence
to: 1) help citizens to build at execution time the skills they need to complete the work of a
collective effort; 2) empower citizens to become entrepreneurs and find new solutions to
collective problems. In this talk, I will present case-studies showcasing how my architecture can
lead to collective action in a range of areas. My research enables anyone to orchestrate
collective action to create their envisioned societies.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

On Automating Conversations

Ting-Hao Huang

From 2016 to 2018, we developed and deployed Chorus, a system that blends real-time human computation with artificial intelligence (AI) and has real-world, open conversations with users. We took a top-down approach that started with a working crowd-powered system, Chorus, and then created a framework, Evorus, that enables Chorus to automate itself over time. Over our two-year deployment, more than 420 users talked with Chorus, having over 2,200 conversation sessions. This line of work demonstrated how a crowd-powered conversational assistant can be automated over time, and more importantly, how such a system can be deployed to talk with real users to help them with their everyday tasks. This position paper discusses two sets of challenges that we explored during the development and deployment of Chorus and Evorus: the challenges that come from being an “agent” and those that arise from the subset of conversations that are more difficult to automate.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Solving AI’s last-mile problem with crowd-augmented expert work

Kurt Luther

Visual search tasks, such as identifying an unknownperson or location in a photo,are a crucial element of many forms of investigative work, from academic research, to journalism, to law enforcement. WhileAI techniques likecomputer visioncan often quickly and accurately narrow down a large search space of thousands of possibilities to a shortlist of promising candidates, they usually cannot select the correct match(es) among those, a challenge known as the last-mile problem.We have developedan approach called crowd-augmented expert workto leverage the complementary strengths of human intelligence to solve the last-mile problem. We reporton case studiesdeveloping and deploying two visual search tools, GroundTruth and Photo Sleuth, to illustrate this approach.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

AI for a Generative Economy: the role of intelligent systems in sustaining unalienated labor, environment, and society

Ron Eglash, Lionel Robert, Audrey Bennett, Kwame Robinson, Michael Lachney and William Babbitt

Extractive economies pull value from a system without restoring it. Unsustainable extraction of ecological value includes over-fishing, clear-cut logging, etc. Extraction of labor value is similarly objectionable: assembly line jobs for example increase the likelyhood of cardiovascular disease, depression, suicide and other problems. Extraction of social value--vacuuming up online personal information, commodification of the public sphere, and so on-- constitutes a third form. But all three domains--ecological value, labor value, and social value--can thrive in unalienated forms if we can create a future of work that replaces extraction with generative cycles. AI is a key technology in developing these alternative economic forms. This paper describes some initial experiments with African, African American, and Native American artisans who were willing to experiment with the introduction of computational enhancements to their work. Following our report on these initial results, we map out a vision for how AI could scale up labor that sustains “heritage algorithms”, ecologically situated value chains and other hybrid forms that prevent value alienation while flourishing from its robust circulation.


It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.