On the role of AI in my research

AI is upon is, and although I would probably be ok if it wasn’t around, I have been (and still am, to a certain extent) tempted to use it in my research. So here I’m gonna articulate some of my thoughts on AI. This isn’t written to convince anyone, or even to convince myself. Just to lay out all my thoughts and take stock of my preconceptions, disapointments, hopes and desires, etc.

Also, I’m gonna use AI, LLM and whatever other brand names and marketing buzzwords interchangably here. Draw whatever conclusions you want about that.

I see AI as being potentially useful in a few productive activities I regularly engage in:

• Transcribing spoken words into written text

  Transcription is a significant component of processing interview data, and this can be extremely slow work. It’s a lot easier to edit a transcript produced through a computer algorithm rather than start from scratch. I used trint, otter and other similar tools before all the AI hype, and more recently I’ve been using whisper to transcribe voice notes that I record while I’m waiting for the bus or drifting off to sleep. I’m not really sure how they’re much different, to be honest. Is AI just a rebrand of natural language processing in these contexts? Either way, I will most certainly be using some automatic transcrion tool in my research.

• Summarizing, breaking down and simplifying complex bundles of ideas

  I do a lot of reading, and it can be hard to get through everything on my list. I therefore make lots of compromises and refrain from reading some things because I just can’t make enough time to get through everything. I imagine that AI can help summarize some key points across a whole corpus of articles on my to-read pile, and I may try it out once I have time to figure out the right tooling for the job. However, I do gain a lot of value from the process of reading. Specifically, as a scholar of scientific practice, I’m interested in the language and rhetoric authors use to describe and situate their methods and findings, and I’m not sure if automatic summary tools can capture and communicate this nuance in ways that I want.

• Generating code snippets for data processing and visualization

  This is arguably the most productive potential application I can imagine. Specifically, I’m thinking about using this to generate R code that processes and visualizies data according to imagined outcomes. This is directly relevant to a project I’m working on where I’ve already finished the workflows for scraping and processing the data, I have the questions I want to ask of it, but I don’t have the practical know-how to generate the code that will allow me to address them. ggplot is just so dense to me, and stitching together code snippets from stack exchange is a major pain in the ass that produces a horrible abomination of code that would not pass the muster of any rigorous code review. What’s more, those queries to search stack exchange are already half-formed AI prompts! At least an AI would generate some harmony in the code, and I might learn something by having a tidy and consistent template.

I’m more ambivalent and critical about using AI in these contexts where it’s been really hyped:

• Any form of writing, including generating emails and abstracts

  For me, writing is a creative process and a way of unerstanding. It’s a mechanism through which I come to learn about something. The experience of drafting and revising a document is crucial to my research process. This is especially important for honing my position as a scholar at the intersection of various disciplinary communities, who have distinct language and modes of communication.

• Querying for truth claims

  To be clear, the idea that knowledge can be total, absolute and disembodied is deeply flawed, and the popular reception of AI as a neutral observer and reporter of nature makes me sad. That being said, I’m still ambivalent about the potential for specialized, home-grown LLMs as means of parsing, sorting through and obtaining greater value from under-used resources. There are patterns in even the messiest and least formal documents we create, and even if we can’t draw information from these documents, LLMs may be useful to help us reflect on the circumstances of their creation. I keep thinking about Shawn Graham’s twitter bots in this context (which were not based on AI, but whatever), which attempted to spit out segments of artificial reports and fieldwork drama, which real archaeologists often related and resonded to. These responses were interesting to me, often expressed as collective fascination, titilation or disgust, and reminiscient of the apprehension one might experience when hearing your own voice played back while standing at the opposite end of a long hallway. Reacting to distortions of your own experience from very different perspectives can be a really powerful reflexive exercise.

• As a brainstorming tool, or as a rubber duck

  I’ve heard about people using AI chatbots as agents to bounce their ideas off of. Kind of like eliza, but for productive work. While I think it’s intriguing, I don’t know where I’d start. Also, drawing up the prompt and figuring out how to ask the right questions may already be enough to get the ideas flowing. I think I already do this in some ways by drafting little ephemeral notes, usually directed toward a specific person or imaginary audience while anticipating their feedback. It also somehow seems like a perverse way to de-socialize work, and in a world where students and postdocs feel increasingly isolated, I’d much rather solicit and provide feedback among peers. This has been the foundation of some of my most solid friendships and professional partnerships, and should be encouraged.

I also have some previously-unstated opinions in relation to some common critiques of AI:

• Process versus product

  AI seems to be really good at devising formulaic outputs. That is, it’s good at getting things to look like things whose shapes are already well-defined. This can be valuable in various use cases, like writing emails according to a template or translating texts between languages. I could imagine it being really helpful for those who are coming into a field where certain skills are taken for granted, such as learning how to write “proper” academic emails as a student who is not fluent in english. Imagine being up against a deadline for a job application, while also being knee-deep in unpaid work to get your name out there; an LLM could be a godsend. So I don’t discount easy outputs as inherently bad. A standard output for one is a week-long struggle for another, so I think this distinction between product and process is a false and misleading dichotomy.

• Bad instructions

  Sometimes I find it really hard to believe that people could earnestly follow whatever an AI tells them. But I think we’re getting to the point of urban mythmaking, similar to the older wariness about following your GPS into a lake. There’s a story behind every warning sign, even if it’s a projection of what you think might happen if you disregard it.

• “Intelligence”

  One weird thing about AI branding is the smushing together of some unified idea of what constitutes “intelligence”. We’ve already been through this with “smart” gadgets, which have always just been ways to capture consumer products under a platforms proprietary injected plastic molds and information protocols. AI is literally just a way to sell you a new version of the smart gadget you threw out last year.

• Truthiness, i.e., AI’s ability to sound authoritative while also making false claims

  I cringe at any retort to a screenshot of AI giving a wrong definition of a thing. Accuracy of responses should come secondary to critique of the notion that all forms of knowledge can be presented in terms of absolute, disembodied and universally truths. For example, when people ridicule AI’s inability to identify the capitols of various nation states, I see missed opportunities to challenge the value of any answer that anyone might provide. True subversion would be to reject or re-frame the question and the simplicity with which it is addressed.

  One another related note, I see a lot of weird parallels between myths about truth claims made by AI and by practitioners of qualitative data analysis (QDA) — and, as a qualitative researcher, this is obviously a bit unsettling. Specifically, in both QDA and AI, there is no actual attempt to make absolute truth claims, but the focus is rather on attempting to identify and draw out meaningful elements of elicitations in a corpus, and to trace patterns between them. In my current opinion, the key difference lies in positionality. Any QDA researcher who laim that their cases are representative of all experiences will be laughed out of the room. Meanwhile, AI is lauded for the claims made by their creators that it can derive unambiguous and concrete knowledge from inherently situated and biased data sources. Humility is key while contributing to collective knowledge bases, and AI risks changing the dynamic away from deriving greater value from constructive discourse and toward a system where the loudest voice in the room wins.

• Climate change

  AI uses a lot of energy, and is therefore said to be wasteful. However I think there are certain wasteful components of AI. For instance, generative models that spit out a full sentence to wrap around the answer to a question don’t have to do all that extra work. Also, not everyone is reliant on fossil fuels, and the critique that AI is necessarily bad for the environment is laden with a thick American accent (as is the case with so many of the loudest opinions on the internet).

  That being said, there are enormous problems with resource allocation in AI, and I’m not trying to dismiss all concerns. I see these concerns as relating to the distribution of power and wealth in society at large, and AI is one aspect of this. Sometimes I wonder if comparisons can be made between using AI in selective research contexts and eating a burger or a banana, which each have their own environmental costs. But thinking in this way is a bit of a trap.

I also see that rhetoric, including anxieties about AI, differs in the various communities I participate in:

• In digital-x, where x = {archaeology | humanities | librarianship | whatever}

  There’s a lot of experimentation going on. Honestly, I don’t know much about it and I tend to scroll past any discussion about AI applications in archaeology that appears in my feed. Part of me sees it as a passing trend, but it could be better framed as a wild frontier, as is the case with many other things in digital archaeology. People are still in the process of taming the landscape, to make it better suit their needs, and maybe I’ll join in once the settlement is established. But I’m not personally motivated by the dynamism of the current landscape, at least in this particular domain.

• Epidemiology, biostats, public health

  I’m still too new in this community to really make sense of this yet. I’ll continue to watch and learn and listen.

• Broader social science and humanities, as well as libraries, archives and museums

  Critique tends to follow broader, more abstract, and more common-sense lines of thought. In my view, much of this does not really account for the material problems and imperfections in which the social sciences and humanities operate. AI is a lifeline for many people in an overworked, overburdened, under-resourced and hyper-competitive environment, and tut-tutting around how other people use AI sometimes comes across as tone-deaf and disrespectful. Some criticisms of AI being used in real, practical circumstances make me second guess critics’ supposed commitments to improving the social experience of research. The fundamental problem is inequitable access to financial and material resources, and AI’s prevalence is a major symptom of, or — depending on your perspective — resolution to that. People’s who recognize this have no choice but to post broader and more abstract criticisms, which come across as somewhat hollow when disconnected from real and tangible experiences.

• Senior faculty

  Probably the most ambivalent of all communities are senior faculty, who want AI to be useful and will test the waters without fully committing. Which is fine and very prudent, and honestly I identify most with this perspective, despite my position as a lowly postdoc.

• Grad students

  I engage with many grad students. I share my workspace with grad students and encounter them constantly in my day to day neighbourhood forays, where I overhear and sometimes participate in conversations about AI. In my new work environment (Epidemiology, Biostatistics and Occupational Health), the grad students who I engage with have a relatively positive perception of AI. They seem to find greater value in the ability to automate complex processes, using it as a black box of sorts, with predictable and abstracted inputs and outputs, which they see as especially helpful for coding. Outside of this space I’m encountering way more diversity of thought on AI, and I’m not quite sure how to group these viewpoints to structure a proper reaction. I think this in fact contributes to the multitude of perspectives, since no one really cares that much one way or the other to really have a strong opinion (though I sense an overwhelming dissatisfaction when it comes to AI in consumer contexts; this post is largely about productive uses of AI in research and pedagogy).

  I was also told about students learning RStats by just having AI generate their code. The person who pointed this out to me related this to the growing misconception that to learn stats you first need to learn how to code. This in turn relates to the sense that to learn how to do RStats, you just need to memorize a series of steps and copy the text from the slides into the IDE. So, in the end, AI reveals the inadequacy of the teaching mechanisms for programming and stats classes, similarly to how AI has revealed the inadequacy of essay-writing as a pedagogical technique.

  On the other hand, some students are concerned about dulling their skills, or even not being able to take advantage of opportunities to learn new skills, due to the temptation to automate these tasks. Some upper-year PhD students are glad that they were trained in the fundamentals prior to the AI hype wave. This makes me wonder how students are determining what skills they think they need to know how to do on their own and what is worth running through an LLM. Does it basically operate as a bullshit sensor, where you can smell from a distance that the work is just gonna be tedium and irrelevant? Or is it more out of practical necessity, where you’re stretched so thin that you simply have to rely on these tools to achieve anything meaningful, almost as a mechanism for salvaging one’s work from the claws of austerity? In either case, this points to PhD programs’ inadequacy to match students’ needs and desires, and overwhelming amount of administravia or (seemingly) irrelevant work that students are made to do, which get in the way of their true interests.

Maybe I’ll have more to share some other time.