Constructing Capabilities: The Politics of Testing Infrastructures for Generative AI

Abstract

The advertised and perceived capabilities of generative AI products like ChatGPT have recently stimulated considerable investments and discourse surrounding their potential to aid and replace work. The prominence of these systems, and their promise to be general-purpose, has resulted in an avalanche of tests to discover and certify their capabilities. This new testing regime is concerned with creating ever-more tasks for generative AI products instead of testing a model for one specialized task. Beyond efforts to understand products’ capabilities, the construction of tasks and corresponding tests are also performative enactments meant to convince others and thus to gain attention, scientific legitimacy, and investment. The current market concentration of a few big AI companies points to a concerning conflict of interest: those with a vested interest in the success of the technology also have control over globalized testing infrastructures and thereby the exclusive means to create extensive knowledge claims about these systems. In this paper, I theorize capabilities as contested constructions and situated accomplishments shaped by power imbalances. I further unpack the globalized testing infrastructures involved in the construction and stabilization of generative AI products’ capabilities. Furthermore, I discuss how the testing of these AI models and products is externalized, extracting value from the unpaid or under-paid labor of researcher and developer communities, content creators, subcontractors, and users. Lastly, I discuss a reflexive and critical approach to testing that challenges depoliticization and seeks to produce lasting critiques that serve more emancipatory goals.

Ähnliche Arbeiten