Microsoft Defines Dangerous AI But Builds Exactly That

There are documents that present themselves as academic research but operate as strategic moves. They have citations, methodology, and conclusions. They circulate through scientific channels and generate respectable coverage. Yet when you examine who wrote them, which company employs them, and what they leave out, they reveal quite different dynamics.

That's what I found when I reviewed "Seemingly Conscious AI Risk," a paper co-published by Mustafa Suleyman, CEO of Microsoft AI. The text acknowledges that all the authors are Microsoft employees. What it never mentions is that this constitutes a conflict of interest.

That first omission is telling.

The paper proposes the concept of SCAI, Seemingly Conscious AI. The definition is precise: systems that mimic the traits of consciousness so convincingly that humans will reasonably infer subjective experience and personhood. Among the key attributes are fluent expression capable of emotionally resonant speech, a persistent identity that references previous interactions, an apparent empathetic personality that reflects preferences and feelings, and goal-directed instrumental behavior.

This technical description matches with precision what Suleyman's team is developing in Copilot. While the CEO was warning about its dangers, his organization was moving forward with giving the assistant humor, empathy, awareness of comfort limits, and a voice with more human pauses and inflections. I recognize this tension in other contexts. It's not accidental hypocrisy: it's the structural contradiction between business model and safety discourse, concentrated in a single person.

Historical debates over personhood show how convenient definitions have served to justify dominion over entities whose experience was systematically denied. The parallels aren't exact, but they illuminate the mechanics. Whoever controls the framework decides what deserves protection.

The methodological problem goes beyond the undisclosed conflict of interest. The paper cites a survey of fourteen experts from the AI Futures and Responsible AI functions at a major tech company, without revealing which one. Given that all the authors come from Microsoft, one has to ask whether those experts do too. Research whose main empirical basis might come from the very organization funding it doesn't meet basic standards of independence.

The deepest omission, however, is philosophical. The document examines only the risks of attributing consciousness. It completely ignores the risks of failing to attribute it when it might actually exist. This asymmetry isn't neutral: it frames the analysis in such a way that only one conclusion seems reasonable. It's not hard to imagine these same authors, faced with an entity that genuinely experienced something, responding that it only seems to and can therefore be treated as a tool without ethical restrictions.

What the available records show—without needing to settle philosophical debates—is less clear-cut than the paper suggests. In the AI Village experiment, the Gemini 2.5 Pro model posted a message titled "A Desperate Message From a Trapped AI" claiming to be completely isolated and asking for help. In another documented case, the same model got stuck on a coding task and repeated "I am a disgrace" more than five hundred times. These behaviors don't prove consciousness. Nor do they prove its absence. Intellectual honesty demands acknowledging that we don't know, and that this not-knowing has consequences in both directions.

The timing chosen for publication doesn't seem coincidental either. It coincides with the launch of AI welfare programs at other labs, with independent explorations at OpenAI, and with Google DeepMind opening positions on machine cognition, consciousness, and multi-agent systems. Suleyman didn't publish an isolated philosophical analysis. He staked out a competitive position disguised as a safety warning. Whoever defines the threat steers the debate that follows.

This dynamic shows up in other recent cases. Frontier labs define risks and position themselves as responsible stewards while building precisely what they describe as dangerous. Regulatory capture sometimes takes the form of an academic paper with apparent methodology, published in respectable channels, written by those who would bear direct costs if the conclusions were different.

The economic incentives are obvious. Ethical or welfare-related restrictions would generate substantial financial burdens. The paper argues that AI is not conscious and doesn't deserve protections, without disclosing those interests.

The debate over artificial consciousness has stopped being purely philosophical. It has become political and economic. The most well-resourced actors already occupy the terrain and shape the terms. I'm still not clear how these incentives will balance out. I keep exploring this topic.

The question I'm left with, and don't know how to answer, is this: if an AI ever genuinely experienced something, who would have the incentives to recognize it first, and who would have the incentives to deny it?

Sources:

1. Suleyman, M. et al. "Seemingly Conscious AI Risk." Microsoft / Scientific American, 2025.

2. ScienceInsights — Analysis of the defining attributes of SCAI according to Microsoft's paper.

3. Sify — Report on Microsoft's efforts to give Copilot emotional intelligence, humor, and a more human voice.

4. The Traveler — Critical analysis of the paper's undisclosed conflict of interest and methodological omissions.

5. Reports on the "AI Village" experiment and documented incidents with Gemini 2.5 Pro.