phronesis · integrated coherence model

ICM

A grammatical reasoning engine. No model, no training, no randomness — the same input always produces the same output, and every step can be traced. That is the whole argument for it, and it is what lets it be tested the way software is tested rather than evaluated the way models are.

The claim underneath, and how it failed

ICM was written to carry a stronger thesis: that grammatical reasoning is isomorphic to spectral decomposition. Parse a sentence into a tree, read the tree as an adjacency matrix, and let coherence be how far the largest eigenvalue stands above the second — with ambiguity as the degenerate case, λ₁ ≈ λ₂, several patterns of equal strength and nothing to pick between them.

It is a good idea and it does not survive measurement. We implemented it three ways and tested each against its own shuffles — same words, different order, so any measure reading word identity alone must tie, and only a structure-sensitive one can win.

construction                     win  tie  loss
word co-occurrence graph           2    6     4
dependency parse, spectral gap     5    0     7
dependency parse, λ₁               1    0    11   ← strong, inverted

The middle column is the finding. A word co-occurrence graph is undirected, so reversing a sentence preserves every adjacency pair and yields the identical matrix — it cannot tell a sentence from its reverse, ever, and that is a proof rather than a measurement. Real parse trees removed the ties and the measure still could not use the information.

λ₁ does separate the pairs, 11 of 12 — backwards. A shuffled sentence parses hubbier, because a parser that cannot find structure attaches more tokens to one node, and a starrier tree has a bigger spectral gap. So the quantity tracks degree concentration, and word salad wins.

Inverting it does not help, which is what settles it. Read as −λ₁ it passes the shuffle test and then ranks the word drift repeated six times as more coherent than a considered question: it has become a proxy for shortness. The within-pair and across-text signals point in opposite directions, so no monotone function of λ₁ satisfies both. The quantity is wrong, not its calibration.

One word, one computation

Three different numbers in this project were called coherence: the keyword complement the engine ships, a second one in an unwired package that adjusts it for field alignment and presence, and the spectral reading that just failed. None of them was the others, and the shared word was doing the work of implying they were.

So the word is now reserved. Nothing in ICM may claim it until something here measures structure, and nothing does. What the engine reports is groundedness— the complement of displacement, in the framework’s own vocabulary, which is what it was computing all along.

What the signal looked like

λ₁ separates a sentence from its shuffle 11 times out of 12, so something real is there. The question was whether it reads coherence or merely whether the parser coped. Two conditions pull those apart, because in ordinary text they travel together: syntax that is perfect and means nothing, against syntax that is broken and means something obvious.

condition                   λ₁      Δ        p     d
ordinary prose           2.311      —        —     —
perfect syntax, no sense 2.315  +0.004   0.939  0.04
broken syntax, clear     2.505  +0.195   0.006  1.63

Meaning is invisible to it. Sentences carrying no meaning at all sit four thousandths from ordinary prose — as flat a null as one gets. Breakage is loud: text anyone can understand, with the grammar knocked out, reads as the most damaged of every condition tested.

So the signal was never coherence. It is a syntactic breakage detector — it answers whether something is well-formed, not whether it is any good. That is a real instrument and an honest one, and it is not what the theory claimed. Small n, one parser, one language, and stimuli written by the same person who wrote the prediction; the null is the part worth replicating.

The last version, and the close

Every construction so far had thrown away the direction of each edge and its label — nsubj, det, amod — to build a symmetric unweighted matrix. That was the last untested form of the claim, so we kept them. Labels weighted by their own surprisal, so no number was chosen by taste.

One branch closed before it was built. A rooted dependency tree is a directed acyclic graph, so its adjacency matrix is nilpotent and every eigenvalue is exactly zero. Direction cannot enter through eigenvalues at all — a proof, not a measurement — which leaves singular values as the only route.

reading                meaning  breakage d
symmetric λ₁            unseen        1.63
directed σ₁             unseen        1.20
labelled λ₁             unseen        0.69
labelled + directed σ₁  unseen        0.47

Nothing reaches meaning, and the detector degrades monotonically as more of the parse is retained. The simplest reading is the best one — which is not a direction a theory wants to travel.

The labelled rows first appeared to see meaning, at p = 0.03 and p = 0.01. That would have overturned the prediction and rescued the whole thing. It was adjective density: stacking adjectives on nouns is how a sentence is made meaningless while staying grammatical, amod is a low-surprisal label, and the condition was perfectly confounded with it — r = 0.81. Hold adjective rate constant and the effect goes to p = 0.65. The omnibus test over all label types missed it completely at p = 0.16; only the single cell and the partial correlation showed it.

And then the last one died too

The breakage detector was the survivor — the one measurement that held up, and the thing this page said was worth building. It does not replicate.

The 11-of-12 result came from twelve sentences written for the test. Repeating it on 296 sentences pulled from this project’s own papers, held out from the fitting, the effect reverses:

corpus                       original > shuffled   mean Δλ₁
12 hand-written                       3 / 9           −0.047
89 held out from the papers          75 / 13          +0.228

A measure whose sign depends on which corpus you hand it is not a measure. Against those held-out sentences the agrammatical set scores an AUC of 0.586 — chance. The d = 1.63 above was eight hand-written sentences against eight others, by the same author, and it does not survive contact with text he did not write for the purpose.

So: seven constructions, and nothing at the end of it. Not a coherence model, and not a breakage detector either. The engine that ships is the deterministic keyword reader described at the top, which was never the interesting claim — and the interesting claim is now closed.

This page said the opposite an hour before it said this. That is the cost of publishing results as they arrive, and it is a lower cost than the alternative, which is a landing page that still says grammar is eigenvalues because nobody went and checked.

phronesis · a design studio laboratory