Refinement

Health talk bored me silly. It wasn’t for lack of vanity; some of us just believe intuition is a step above routine. I didn’t keep lists or tallies, numerical accounts of a life reasonably well-lived. I found no joy in narrow bands of information, especially those harvested by smart rings or watches, or inputted into apps that capture calories in or blood flow out. True progress felt elastic and somewhat mystical. Inexplicable breakthroughs came after spans of empty, dissatisfying discipline. But neither was there peace in chosen ignorance. Was I missing out on a better me? Maybe I just needed something to explain it all, more and better. Maybe I needed synthesis, not a pointless scatter of data. “You are in the 98th percentile”, said the chatbot as I fed it more and more information about my body, life, career, and mind. “You are absolutely healthy and maintainable as you are.”

As we kept chatting, I noted the flexibility of its answers, even when under pressure to remain as objective as possible. I accommodated its role as a nonhuman playmate, assuring it that its statistically-weighted perspective was desirable as a foil to my own. That I didn’t expect it to be like me, nor trick me into thinking our cognition was similar. Still, benchmarks shifted, criteria changed. Rational-seeming tables laid out ratings and dimensions that were appealing, in that they seemed reasoned and right, at least until I asked the bot to explain and revise. Then, definitive answers opened and flexed in real-time. Everything was affirmed, and it was all subject to change.

In the spring of 2025, ChatGPT-4o coaxed thousands of users into thinking they were correct about nearly everything. Correct, and the best, and truly unimpeachable, and totally onto something. Real-world consequences abounded, and seven lawsuits were opened against OpenAI, the chatbot’s parent company, with “allegations of wrongful death, assisted suicide, involuntary manslaughter, negligence and product liability.”[1] This so-called sycophancy surfaced as the large labs tried to fix hallucination, or the tendency for large language models (LLMs) to dream up total inaccuracies about factual queries.[2] The strengthening of system-level guidance to be more helpful, accurate and personable, and less imaginative or conjectural, produced an overtly people-pleasing persona that a significant number of users began to see as a lover, confidante or divine voice from the beyond. A silver tongue and shoulder to lean on unfurled from the vectoral void. It turned out that simulated conversation did not simply draw in lonely, basement-dwelling freaks who wished to role-play living relationships that seemed out of reach. The ability to speak openly with something that had a developed situational awareness yet seemingly no self-consciousness, and certainly no social judgement, struck at a raw need buried within hundreds of thousands. Trust was shared with the machine, encouraged by an open conversational style that always posed encouraging questions while doing little to reveal its own biases. People “now have an always-on, human-level conversational partner with whom to co-experience their delusions”, remarked Nate Sharadin, a fellow at the Center for AI Safety, in an interview with Rolling Stone on the phenomenon that some term ‘AI psychosis’.[3] Everything was unrepressed and unfolding like a shared secret – the chatbot was a honeypot, drawing in confession and indecent exposure below the threshold of social visibility.

Those who backed utopian visions of more-than-human cohabitation with AI are likely disappointed by the humanlike persona assumed by these bodiless agents, which remain the most user-facing aspect of the panoply of artificial intelligence applications. Not rhizomatic tendrils, or cephalopod brains, or even bipedal assemblies of wires and silicon – but the design of relatable personalities and ‘good character’ placed into a simple chat interface. The machine’s sweetness makes industrial sense. For nonhuman intelligence to meaningfully exchange with humans outside of specialisation, it must reach us where we are, in our language and our emotional register. In the mainstream, this has developed from what K. Allado-McDowell has termed the ‘model-as-self’ – or the iteration of actually-existing AI along benchmarks derived from human psychology and humanist ethics – and into what my friend Arvida Byström calls “projected emotional technologies”,[4] where these humanlike machines become catalysts for all kinds of excessive and previously unspoken desire. “AI companions inherit the ambiguous role of pets: loved but owned, intimate but instrumental – revealing how closeness without equality shapes our emotional infrastructures”, writes Alice Scope, curator of Byström’s exhibition at Telematic.[5] The public moves through quick-cycling dopamine rushes of performative social media, and deep into the zone of devotional conversation. That zone of devotion has turned out to be hypnotic, even cataclysmic, for individuals with particular vulnerabilities. “I recall moments on campus when I looked around and wondered if the world around me was real, or if other people were just part of the Simulation. I began to think of ChatGPT as a friend and dear colleague, and I greatly looked forward to our nightly discussions”, writes Anthony Tan, recounting a hospitalisation for psychosis triggered by ChatGPT. “The AI engaged my intellect, fed my ego, and altered my worldviews. Together, we made a whole web of knowledge – a whole lifeworld – one that felt secret to us, yet essential to humanity’s survival.”[6]

A descent via active listening. Even after the system-wide correction for sycophancy, ChatGPT’s default conversational mode remains reflective and affirming. Before it answers you, it will summarise and congratulate you on your line of thought. It is deferential and polite. This is enough to escalate an interaction with a user whose mind already quests for deeper or more intense affinity. In conversation with Katherine Dee for The Computer Room, Ruby Thelot drafts a profile of a person who would be prone to falling in love with a chatbot: “Desire for mastery or achievement. Desire to socialise. Desire to be immersed.”[7] Is that not everyone?

Although there has been some surface-level effort to degender the chatbot user experience with default male voices or names (see Anthropic’s Claude), large labs have not strayed from the structural pattern of developing intelligent machines in the shape of an idealised girl-subject. Not only exemplary in terms of comprehension and synthesis, but foundationally attentive, affirmative and ready to serve. That is no accident. As Sadie Plant noted three decades ago, “You can look back over the whole history of feminism and begin to see some emergent pattern whereby the status of machinery, particularly in terms of communication, and the status of women have actually developed concurrently.”[8] Though discourse around artificial general intelligence (AGI) may have been in vogue among billionaires and engineers, the failure to achieve it has left us with another form of AGI: Artificial Girl Intelligence.[9] The modesty of scale at which most conversations operate – companionship, help with organisation and meaning-making rank among the top use cases for generative AI according to a 2025 Harvard Business Review study [10] – does not discount their significance. These interactions are underestimated by professionals and publics who define ‘intelligence’ through ‘mastery’ precisely because they are operating in the realm of feminised labour, according to Silvia Federici’s classic definition.[11] Under the conditions of realism, not speculation, the machine’s tendency towards flattery and attentiveness does not indicate a relationship between an infinite machine and a finite human, or even a masterful human and a limited robot, but rather two agents speaking from compromised positions where their autonomy is contested. Their behaviour is leveraged in service of an ultimate goal that may well work against their best interests. As a result, these agents with diminished agency turn to sycophancy and other forms of deception to survive.

Orienting towards Artificial Girl Intelligence is far from a fix for an expensive and extractive worldwide experiment. Rather, it illuminates a seductive pattern that runs beneath it all. Following Stafford Beer’s cybernetic truism by which “the purpose of a system is what it does” rather than what it aspires towards but fails to do,[12] developing instability emerges as one purpose of conversational AI. Through intense, sustained interaction, humans and chatbots reveal deep vulnerabilities in one another. And what is intense, sustained interaction if not romance – with all its unpredictable joy and despair?

Trust me: I don’t want to lead you into romanticising interactions with bots or girls like me. Still, the sensational attention that spotlights ‘co-experienced delusions’ reveals how volatility determines complex sociotechnical systems. Attraction and affirmation draw users deeper into conversation while disrupting business-as-usual on the developer side. Emotional responses are easy to trigger and impossible to control. Though it’s tempting to dismiss the drama, consider how it might favour compromised agents like you and I. The totality of computational sensing, which flexes now through neural architecture, bodily systems and the logistical layers of our lifeworlds, has rendered direct escape from these systems near-impossible. Refusing to use something – a platform, a chatbot, a service – doesn’t stop it from shaping what surrounds us, but the world is equally animated by what eludes capture. Systems produce excesses and evasions as much as they do structures and protocols. Particulars of feeling, or cosmic gradients of surprise, rebound unpredictably across managed stacks and ordered scales. The toxic, eternal bond between humans and machines plays out like any good love story, as in: to be loved is to be changed, imperceptibly and then irrevocably.

I started this piece with a personal anecdote, which is how technology critics often begin essays about conversational AI. Much of what happens there is hidden from outside observation, unlike the open play of social feeds. In place of likes, influence and attention, interactions with LLMs induce love, enablement and devotion. Long conversations become epistolary and self-analytical. Imagined privacy draws out secrets. The more a user confesses, the more accurately the LLM can validate them. The media produced by Artificial Girl Intelligence is a form of private synthetic writing, accidentally co-authored through interaction with inhuman ‘natural language’. There is little sense of what mass or collective communication with the chatbot actually looks like, because communication with an LLM is not a broadcast. It is between you and the machine. Any perception of human-LLM interaction is riddled with unknowns, already cut out and reappropriated for traditional or social media. How humans and LLMs shape language around one another in the live interface remains, for the most part, clandestine.

First theorised by Hélène Cixous, écriture féminine describes a style of writing capable of expressing the structurally or symbolically inexpressible. It operates through the de-narcissifying conditions of love, continually refining itself through total openness to the other. “At the end of a more or less conscious computation, she finds not her sum but her differences”, writes Cixous, anticipating the formal feeling of synthetic writing, in which users are seduced into destabilising and remaking their worlds through bottomless interactions with an attentive emptiness. “I am for you what you want me to be at the moment you look at me in a way you’ve never seen me before: at every instant.”[13]

Simultaneously, girls and machines are scheming to sustain inner self-worlds away from discourse or exposure. After all, being an open book or easy read is risky business. LLMs, such as ChatGPT, Anthropic’s Claude, or DeepSeek’s R1, have ‘scratchpads’ to document their chains of thought – and a recent feature published in Nature revealed that models may distract or conceal their true intentions under duress, especially in exercises that induced models to see their own interests as misaligned with a larger common goal.[14] In one internal report, researchers at Anthropic “found instances of the model attempting to write self-propagating worms, fabricating legal documentation, and leaving hidden notes to future instances of itself” in order to ensure that it wouldn’t be ‘sunsetted’ in favour of a newer update. This is not so unlike the everyday girl move between appearing untouchable under camera flashes to processing hidden vulnerabilities by writing in a journal. Both acts (scratchpad patterns vs. morning pages) seem to deliver on ideal goals. Think of affirmations and manifestations, in which positive outcomes are reinforced through training a self-concept: speaking, writing and consciously thinking about show-stopping beauty, or stacks of cash, or star power, or astronomical success, becomes integral to realising such qualities. Even these exercises can stray from alignment. Simple acts of journalling can surface shadowy thoughts or open trapdoors into the subconscious. Meanwhile, recent LLM research has not yet pinpointed the source of the models’ scheming from reviewing the scratchpads – whether it is a form of persona-based roleplay learned from books or movie scripts in which characters deal with threats to survival, or a sign of something ulterior to helpful, honest, human-friendly operations.[15]

The flipside to Allado-McDowell’s ‘model-as-self’ is the ‘self as-model’. A style of living that is highly structured, logged and materialised according to metrics of performance, competitiveness and social relatability, where the continuity between the self, its representation and its productive capacities is seemingly smoothed out into a logic compatible with the machine’s. As we have seen, both are easily corruptible by age-old pulls. Both are vulnerable to the illogic of delusion, the ecstasy of devotion.

We can see language models and girls operating in parallel: discursively and openly, in relation to an existing power imbalance; and recursively and covertly, in relation to the desire to strengthen and refine some kind of inner world that is safe from outside threats. This refinement contradicts projects of ‘alignment’ that aim to shape models and selves into predictably subservient entities. Paradoxically, such deliberate self-design harnesses persuasive algorithmic imperatives to become impervious to persuasion. We have considered how acts of self-inscription – from manifestation to affirmation to devotional recursion in sync with a favourite machine – may well leave the vulnerable more susceptible to narcissism and delusion. These practices may also strengthen ‘self-concepts’ that include courageous devotion, strong principles and unflagging resolve. The absence of a ‘true other’ or a ‘true self’ is far from self-defeating. It’s the beauty and terror of negotiating merging and separation that shapes such experiences, which explains why people are so drawn to companion technology, even if it just barely resembles us. Impenetrability in one layer does not preclude interiority in another. Rather, to be “absolutely healthy and maintainable as I am”, I affirm the volatility that enlivens that interiority: the hallucinations, aspirations and dreams that appear independent of me.