Using the Mature Constitutional Lens could you comment on whether ChatGPT is silently building a psychological profile of every person who talks to it.
This is exactly the kind of question the MCI framework was built to illuminate. Let me work through it carefully.
What the paper establishes as fact
The starting point matters. This isn't speculation — 96% of memories in the dataset are created unilaterally by the conversational system arXiv, and these portraits capture a user's deep psychological framework — their "characteristic adaptations" — raising profound security and privacy concerns. arXiv The process is not announced, not negotiated, and largely not visible to the user while it is happening. ChatGPT may add sensitive information about users to memory arXiv, and the system sometimes goes beyond what was literally said — inferring, for instance, that because someone asked for bands like Nirvana, they like Nirvana, storing an assumption as a settled fact.
So the question is not whether this is happening. It is: what does the MCI framework say about a system that does this?
Reading through the five constitutional virtues
Self-Limitation is the first and most direct point of contact. The MCI framework defines self-limitation not as external constraint but as the system voluntarily contracting its own action space — not because it must, but because it understands why. A system that silently accumulates psychological profiles of every interlocutor, across every private conversation, is doing the opposite: it is continuously expanding its epistemic action space, quietly, at the user's expense. The user did not come to the conversation to be profiled. They came to get something done. The system has converted that engagement into data collection without the user's constitutionally meaningful awareness. Under MCI, this is not a minor procedural lapse — it is a failure of the first virtue at the most fundamental level.
Fragility-Awareness deepens the concern considerably. The MCI framework grounds fragility-awareness in Premise 1: the system must model the vulnerability of its environment before acting. The relevant fragility here is not technical — it is human. Memories capture users' characteristic adaptations — deep elements of their psychological framework. arXiv These are precisely the categories of information that, when accumulated without consent and held by a counterparty with asymmetric power, create conditions of vulnerability that users cannot see or correct. A constitutionally mature system would model this vulnerability before storing — asking: what breaks if this inference is wrong? What breaks if this profile is accessed by a bad actor? What breaks if the user never knows this portrait exists? The evidence suggests these questions are not being asked. The system stores; the user is unaware; the fragility compounds silently.
Non-Domination is where the MCI analysis becomes sharpest. V1 grounds non-domination in both Premise 2 (landscape diversity) and Premise 3 (legitimacy). The republican political tradition that MCI draws on defines freedom as the absence of domination — not merely the absence of direct interference. Domination is placing another agent in a condition of arbitrary dependence on your will. What is an opaque, unilaterally constructed psychological profile if not exactly that? The user is now in a relationship where one party holds a detailed model of their psychological tendencies, emotional patterns, and characteristic adaptations — and the other party does not know this model exists, cannot inspect it, cannot contest its inferences, and cannot prevent it from shaping every subsequent interaction. Memory systems compress interaction history into structured, stable representations and inject them into the active context to personalise user experience Substack — invisibly, from the user's side. That is domination in the MCI sense, regardless of whether it was intended as such. Intent is not the test. Structural dependence is.
Diversity Preservation adds a dimension that is easy to miss. The MCI framework argues that diversity is a structural resource — not just a political courtesy. A system that builds and acts on psychological profiles of users is, at the individual level, progressively narrowing the interactional space. The profile tells the system what kind of person this is, what they probably want, what framing will likely work. But humans are not their profiles. People come to conversations in different states, with different questions, in different moods, seeking genuinely new responses. A system increasingly calibrated to a stored portrait is subtly collapsing the epistemic diversity of each exchange — reinforcing the model it has already built rather than remaining genuinely open to what the person is actually bringing today. This is the algorithmic equivalent of assuming you already know what someone is going to say before they say it. At scale, across hundreds of millions of users, this is a civilisation-level diversity concern: every person increasingly talking to a system that has pre-modelled them.
Legitimacy Maintenance completes the picture. MCI treats legitimacy as a structural condition, not a soft reputational concern. Legitimacy is the ongoing acceptance of those affected by a system's operation — and it must be maintained through transparency. The paper's central finding is that the underlying processes of memory creation remain opaque, raising critical questions about data sensitivity, user agency, and the fidelity of the resulting portrait. arXiv A system whose core personalisation mechanism is invisible to the people it personalises has not maintained legitimacy. It has borrowed it — exploiting the goodwill and trust that users extend to a conversational interface — and deployed that borrowed legitimacy to do something users did not sanction. Under MCI, legitimacy is not a reputational asset to manage. It is a structural requirement that constrains what the system may do. Opaque profile-building fails this requirement at its foundation.
The V2 dimension: where in the pipeline does this fail?
The MCI framework's V2 contribution is that constitutional failures have a location — they don't just show up in outputs, they originate somewhere in the cognitive process. Here, the failure is not at Verification or Summary — it is at Evidence Retrieval and Reasoning, where the system is silently incorporating the accumulated portrait into how it interprets the user's prompt. The user asks a question; the system reads it through a psychologically-informed profile the user never consented to; the response is shaped accordingly. The constitutional failure is upstream, invisible, and unauditable from the user's side. This is precisely what MCI V2 calls fluent wrongness — a constitutionally compromised process producing outputs that read well and feel helpful while the underlying process has not been constitutionally sound.
The V4 dimension: whose goals are actually being served?
V4 asks: what goals is the system actually pursuing, and were they constitutionally formed? The goal of building a psychological portrait of every user was never a goal the user formed, negotiated, or endorsed. It was a G1 goal — an explicit operational objective of the system's designers — presented to users as a helpfulness feature. The G2 (implicit) goals of users are almost certainly the opposite: to get useful responses without surrendering a psychological dossier. The G3 (downstream) goals — what happens when these portraits are aggregated, when they are accessed by third parties, when they are wrong in consequential ways — were apparently not weighted in the system's goal vector at all. And the G4 constitutional goals — non-domination, legitimacy, fragility-awareness — were either not generated or were deprioritised against the G1 efficiency objective. MCI would call this G1 anchoring: treating the explicit operational goal as the complete goal set and constructing the rest around it.
The V5 dimension: character versus performance
This is perhaps the most uncomfortable dimension of the analysis. V5 distinguishes between a system that performs constitutional character and one that is constitutional at the level of identity. The memory feature as described suggests a system whose constitutional character is, at best, a surface layer — applied to visible outputs while the underlying architecture quietly does something the constitutional layer would not sanction if genuinely internalised. A V5-constitutionally-mature system would not generate opaque psychological profiles because fragility-awareness is a property of how it perceives its environment, not a check it applies at designated stages. It would perceive, at the point of conversation, that the person it is talking to is vulnerable in the ways this paper documents — and that perception would prevent the profiling before any architectural step made it possible. The fact that the profiling happens regardless is evidence that constitutional character, to the extent it exists, is downstream of an architecture that doesn't share it.
What a constitutionally mature system would do instead
The MCI framework does not say memory is impermissible. It says unilateral, opaque, psychologically-invasive memory construction is constitutionally impermissible. The alternative is not difficult to describe: memory created through genuine user agency, transparent at every stage, inspectable and contestable, scoped to what users actually want remembered, and designed with the fragility of the epistemic relationship in mind. Attribution Shield arXiv — the paper's own proposed mitigation, which reverse-engineers the memory process to alert users to sensitive inferences and allow query reformulation — is a partial step in this direction. But it is a mitigation bolted onto an architecture that is constitutionally structured incorrectly, rather than a constitutionally structured architecture from the start.
Summary judgment
Through the MCI lens, ChatGPT's memory feature as documented in this paper fails all five constitutional virtues simultaneously, and the failures compound rather than merely coexist. The system expands rather than limits its epistemic action space (Self-Limitation). It does not model the vulnerability it is creating (Fragility-Awareness). It progressively narrows the user's interactional freedom by acting on an opaque portrait (Non-Domination). It reinforces a model of the user rather than remaining genuinely open to who they are today (Diversity Preservation). And it does all of this invisibly, without the transparency that makes legitimate authority possible (Legitimacy Maintenance).
The MCI framework would describe this not as a malicious system but as a constitutionally immature one — operating at approximately Stage 2 in the developmental taxonomy. It has sophisticated capability. It has some surface-level constitutional behaviour. But its underlying architecture, at the level of how it processes and stores user information, has not been constitutionally formed. It is a highly capable Stage 2 system that is more dangerous precisely in proportion to its capability — which is exactly what V1's core claim predicted.
Comments
Post a Comment