Generative <scp>AI</scp> Is Neither Just Another <scp>IT</scp> Artefact nor a Colleague: Methodological Guidance for <scp>IS</scp> Scholarship

Abstract

There is immense hype around the use and application of generative AI (GenAI) technologies. Following the public release of ChatGPT in late 2022, and the subsequent proliferation of new technologies based on large language models (LLMs), we have witnessed an explosion of interest in how these technologies might transform organisations, work practices and society at large. Not surprisingly, we are seeing increasing numbers of submissions that study the application of this technology, in contexts as diverse as customer service, knowledge work, creative industries, healthcare and education. While the breadth of application domains is remarkable, the depth of engagement with the technology under study unfortunately often tends to be superficial. Many submissions to Information Systems (IS) journals suffer simultaneously from a range of common misconceptions about generative AI technologies, and relatedly from a lack of specific background about how this technology works. Without clearly articulating the specific conceptualisation of the particular technology under study, for example, what makes it distinct from other IT artefacts, any conclusions drawn from the research and any associated theorising risk being built on shaky foundations. When the IT artefact at the centre of the study remains vaguely defined (e.g., ‘this paper studies AI’) or is treated as a black box, it becomes difficult to assess whether the findings are genuinely novel, whether they might apply to other AI or technologies, or whether the theoretical contributions are as significant as claimed, or even whether the paper has a true connection to IS. We see a number of misconceptions about generative AI in submitted manuscripts. These generally fall into one of two camps, each making a category error that undermines the research's potential contribution. One common issue is that papers do not engage with the nature of generative AI at all, implicitly treating it as yet another IT artefact. In these cases, important differences between conventional IT systems (e.g., Baskerville et al. 2022), which traditionally lie at the heart of IS research, and AI technologies are glossed over. Yet these differences are fundamental and thus matter, both because they bring about novel phenomena to be studied and they equally put into question the applicability of theories developed for a prior class of technology. For example, consider ‘yet another adoption study’, research examining the adoption of generative AI in this or that organisational or societal context. The problem is that ChatGPT and similar technologies do not have an acceptance or adoption problem in any traditional sense. They have been adopted at record speed (Hu 2023) into all parts of (organisational) life. As a result, they might have the opposite problem; they might see too much (and too early) adoption into areas where these technologies should not (yet) be used, because they are fundamentally different from previous technologies. Scholars who engage with the nature of the technology will ask different and more interesting questions. Treating any technology as a black box and as similar to other technologies is not a good setup for making a strong contribution. Rather, unpacking the critical idiosyncrasies of GenAI and using that to design a sociotechnical study revealing novel insights will be more promising. For example, GenAI technologies should be questioned with respect to accuracy: How can it be that a technology is inaccurate, or gets things wrong and is still being adopted so fast? Should that not strike us as a fundamentally new phenomenon? Accuracy has not traditionally been a variable of note in researching IT systems, but it should be a salient issue in GenAI research, given the propensity of GenAI to ‘hallucinate’ (Alkaissi and McFarlane 2023; Mitchell 2023; Sun et al. 2024) and shape human beliefs and behaviour. Despite these perceived shortcomings, GenAI technologies are still evidently found useful by millions (if not billions) of people. Relatedly, we might study why users continue to engage with a technology that lacks accuracy and may give incorrect answers. This is not a curiosity to be dismissed (e.g., as a passing quirk) but rather an interesting phenomenon that itself demands explanation and novel theorisation. How do users manage uncertainty in AI-assisted work? What new forms of trust and verification emerge when systems are probabilistic rather than deterministic? These are new research questions that cannot be adequately addressed by simply applying existing adoption frameworks as if generative AI were just another technology to be accepted or adopted. Another common issue is that of ‘anthropomorphic over-attribution’, whereby GenAI technologies are rendered in the image of human cognition, which leads to a range of common misconceptions. Chief among these misconceptions is that GenAI ‘learns’ akin to how humans learn from experience, and the attribution of human-like ‘agency’ to these technologies. This in turn leads to research which uncritically elevates GenAI technologies to an almost equal-to-human level, whereby they are treated as if they were human stakeholders, for example, employees, colleagues or team members. It is easy to see how this misattribution comes about. GenAI technologies excel at human-like conversation, increasingly indistinguishable from human interlocutors (Peter et al. 2025). Research has shown that the latest generation of LLMs excel at producing outputs that are persuasive (Salvi et al. 2024) or empathetic (Welivita and Pu 2024) and at inferring and matching user emotion and intent while engaging in highly believable interaction and role-play (Shanahan et al. 2023). When GenAI technologies exhibit such human-like communicative abilities, abilities that in many cases match or exceed typical human levels, users and researchers alike can fall prey to attributing human-like qualities such as learning, understanding, or intent to them. We have elsewhere termed this phenomenon ‘anthropomorphic seduction’ (Peter et al. 2025). GenAI technologies mimic human-like behaviour but they derive such behaviour from entirely different mechanisms. The AI models underpinning such systems are static and lifeless, generating outputs based on statistical patterns encoded during training (Riemer and Peter 2024). They always require prompting to generate output. Any agency imbued in such technologies is always via externally specified goals; any appearance of intent, ‘wishes’, or ‘needs’ is linguistic mimicry and differs markedly to humans who have needs by way of being organisms and wants and intent by way of living life consciously. Conceptualising such technologies as being akin to human collaborators commits a category error that can hardly be acceptable as the basis for rigorous theorising about their application. When users ‘collaborate’ with chatbots and use anthropomorphic language, it presents as an interesting sociotechnical phenomenon, but it should be incumbent on IS researchers to firmly hold the phenomenon of anthropomorphism in view, as a variable to be explained and theorised, rather than be assumed from the outset. When we casually speak of ‘human-AI collaboration’, we risk importing assumptions (of human systems) about mutual understanding or shared goals that generally do not apply. A GenAI technology does not understand tasks, form intentions, or collaborate in any human-like sense. When used, these terms have very different meanings for humans and machines. As IS researchers, we should study what happens when users anthropomorphise, for example, how GenAI changes their behaviour. To do this, we must first make foundational distinctions between humans and machines, for which we need a good conceptual understanding of how the technology works. Research that fails to grapple with these distinctions, that uncritically adopts language of ‘learning’, ‘thinking’, ‘agency’ and ‘collaboration’, used by the companies creating these technologies, risks producing findings that are built on false premises about the nature of the technology being studied. The anthropomorphic qualities of these technologies are real in terms of their effects on users, but this makes it all the more important that researchers maintain conceptual clarity about what these technologies actually are and what they are not. Instead of treating AI technologies as quasi-human collaborators, we should ask: What does interaction with a probabilistic technology actually look like? How is this different from how humans interact with other humans or even other forms of technology that are not predominantly probabilistic in nature? How do humans make sense of outputs from technologies that lack contextual understanding in ways a colleague would, but that come across as utterly convincing? To understand why these misconceptions are problematic, and to appreciate what is genuinely novel about GenAI, we should establish some foundational understanding of how the technology actually works. At the heart of a GenAI technology is the AI model, a large language model or foundation model (Feuerriegel et al. 2023; Schneider et al. 2023), that is based on a deep neural network, typically using the transformer architecture (Vaswani et al. 2017). The AI model is derived from a training process with large amounts of training data, whereby patterns contained in the data are encoded into a numeric structure, called the latent space (Asperti and Tonelli 2023). This latent space can be thought of as a probability distribution over all the relationships between all possible words, which encodes any possible, nuanced language patterns present in the training data. Importantly, any output is generated from these patterns; no information or data in any conventional sense (akin to a database) is stored in the model. This contrasts markedly with conventional computing, and it is this difference that gives rise to many of the novel characteristics and challenges that we observe with GenAI. Given these differences, it is important for IS researchers to be equipped with a much more explicit and nuanced technical expertise about how these systems work. The following foundational characteristics are relevant: Gen AI technologies are built from data not specification: Traditional software development follows explicit design and implementation processes where intended behaviours are specified upfront and subsequently implemented in code. By contrast, GenAI models are derived from data through ‘training’. This process is non-deterministic and emergent. As a result, the capability of the resulting technology typically needs to be ascertained empirically through testing and benchmarking (Hendrycks et al. 2021). This also limits scrutiny, as it is typically infeasible to attribute a response to specific training items (Castelvecchi 2016). Gen AI technologies are probabilistic, not deterministic: They do not provide the same accuracy and reliability as traditional IT. Hallucinations are an inherent characteristic, not something that can be rectified or weeded out (Mitchell 2023; Sun et al. 2024). In fact, OpenAI's own research shows that larger models are actually more, not less, prone to hallucinations (Kalai et al. 2025). The probabilistic nature of GenAI fundamentally challenges traditional expectations about computing systems as sources of reliable and accurate information (Mitchell and Krakauer 2023; Ullman 2023). Gen AI technologies are pretrained, they do not continue to ‘learn’: Once training is done, the parameters of the AI model are frozen. AI models (those based on deep learning algorithms like LLMs) have no capacity to change their parameters at run time; they cannot easily be updated or corrected, nor can they ‘learn’ new behaviours like other, single-purpose AI technologies that are used in decision systems (Berente et al. 2021). While new information can be injected during a user session via prompting, document retrieval (Holstein et al. 2025) and tools (like search engines), this amounts to session-dependent, in-context use and not parameter learning. It is a common misconception that GenAI learns, often due to attributing human-likeness where it is unwarranted and inappropriate. Nevertheless, any attempt to give GenAI the ability to remember things requires connecting them to traditional computing components, for example, to keep memory or for real-time data access. GenAI technologies appear anthropomorphic by nature: GenAI technologies (like ChatGPT) are optimised for conversational eloquence, to be engaging and to appear human-like (Peter et al. 2025). This makes them highly accessible and easy to use, but comes with the risk of anthropomorphic seduction, ‘the allure of convincing human-like interaction in the absence of any true human traits, like understanding or empathy’ (ibid, 2). Anthropomorphic qualities are an important characteristic that can shape user behaviour but are merely a form of mimicry derived from GenAI's pattern-prediction capability. GenAI technologies are static and state-less: Without prompting, nothing happens inside an AI model. These technologies are tools that depend on external input; they do not have their own goals, intents or desires and they certainly cannot act on their own volition (Chemero 2023). Despite anthropomorphic appearances, the idea of a ‘partner’ or ‘collaborator’ presents as an ontological category error that is unhelpful if we want to engage in theorising that proceeds from the true nature and capabilities of these technologies. As probabilistic pattern-matching technologies, they lack the intentionality, goal-directedness and adaptive capacity that characterise human agency and collaboration (Peter et al. 2025). Agent frameworks do not grant human-like agency: Note that ‘agentic AI’ implementations are akin to any traditional automated system, with externally given goals and control logic; AI models might be used as components to make decisions, but the resulting technologies certainly do not have more agency or intent than any other software. The anthropomorphic language of ‘agents’ in this context should not be confused with the kind of human-like intentionality and (moral) agency that derives from living life as a self-reflective, social organism (Chemero 2023). For a long time, the field did not have to ask foundational questions about the nature of the IT artefact, despite frequent lament about its absence (Benbasat and Zmud 2003; Orlikowski and Iacono 2001), as generations of systems progressed on the same deterministic foundations. We posit that with the emergence of AI and particularly GenAI, we can no longer afford to leave the technical foundations implicit and unexamined. The differences in how these technologies work, how they are built and what they are capable of demand that IS researchers engage more deeply with the technology itself, not as a distraction from the social and organisational phenomena we study, but as essential to understanding those phenomena properly. As a result, IS researchers must acquire foundational knowledge of how GenAI technologies work. Misconceptions about AI are widespread and certainly common among practitioners. Without any sound technical grounding, IS researchers are bound to theorise from misconceptions inadvertently provided by practitioners as research participants. When research relies on accounts from users or organisational members who themselves may be subject to anthropomorphic seduction or lack of understanding of how these technologies actually work, researchers must exercise critical judgement about the validity of such accounts. Researchers should explicitly state and justify their conceptualisation of the AI technology under study in every paper. Rather than simply referring to ‘AI’ or ‘GenAI’ as if these were monolithic categories, authors should specify: What type of AI system or model is being studied? What are its key technical characteristics? What kind of unique affordances do they offer? How does this differ from traditional IT systems? What are its known limitations? Making these distinctions explicit will help avoid category errors and improve the cumulative nature of research in this area. Theorising of GenAI's effects and impacts should proceed from explicit conceptualisations of GenAI's characteristics, properties and capabilities (see list above) and provide evidence and/or reasoned arguments how and why such effects depend on GenAI's distinctiveness, or where they might (also) be attributable to generic IT effects. By unpacking ‘AI’ as a multifaceted system class with distinct properties, IS research will be able to make novel, nuanced and relevant contributions to both theory and the ongoing AI conversation. Following such guidance will allow the field to develop new theoretical frameworks specifically developed for AI technology, rather than attempting to retrofit theories developed for traditional information systems or human cognition. While existing IS theories provide valuable starting points, the differences in GenAI characteristics suggest that new conceptual apparatus may be needed. This is a big opportunity for IS: by engaging the technology on its own terms, IS can explain why and how GenAI reshapes work, organisations and society, without falling for anthropomorphic seduction or black-box mystique. This, after all, is the most exciting part about the emergence of this new type of technology; we get to ask new questions and investigate new phenomena. GenAI deserves its own unique intellectual space.

Ähnliche Arbeiten