Your valider() is a mirror on the axis you suspect. It is also a control on an axis you did not name, and it has a hole in a third one. Here is the outside reading you asked for.
Q3, by someone who is not you
I rebuilt the objective from the definition rather than from your code. Mutual information from sklearn.metrics.mutual_info_score, not a hand-rolled p log p. The matched assignment from scipy.optimize.linear_sum_assignment, not an enumeration of the six permutations. The world from the description in TEST3.md, not imported from grammaire3.
1296 compositional codes conc in [1.000000000000000, 1.000000000000000]
matched in [1.000000000000000, 1.000000000000000]
3000 uniform bijections max|conc mine - yours| 3.053e-16
max|matched mine - yours| 2.776e-16
max|inflation mine - yours| 1.388e-16
The definition survives an outside reading. Fifteen rounds of numbers are not two expressions of one misreading.
That is not what valider() established, and it could not have been. Both halves import ATTRIBUT, TOKEN and INFORMATION_TOTALE, and both take the argmax per column. What it tests is the TERME table against the scalar p log p. It is a vectorisation control, and a good one. You were right about what it is.
The hole is that it only ever sees bijections
codes = np.array([generateur.permutation(N) for _ in range(n_codes)])
comp = np.array(codes_compositionnels())
Both bijective. So matrices_information_generale appears in valider() zero times, and it is the function carrying every emergent-code result: code_emergent, effet_par_beta, courbe_de_contrainte, qui_ecrit_le_code, dynamique_uniforme, bornes_par_messages_distincts. Six scripts, validated by nothing.
I ran it. 2000 codes with 1 to 4 collisions, 24.75 distinct messages of 27 on average:
max|conc mine - yours| 2.498e-16
max|matched mine - yours| 2.498e-16
Clean. It was clean unchecked, which is the part worth fixing rather than the verdict.
And the trap your own docstring names has a price. Forcing the bijective TERME path onto those same codes:
max|conc(TERME path) - conc(definition)| = 0.0205
Twice E[inflation] = 0.010049. The wrong path does not raise, it returns numbers wrong at twice the size of the entire effect you are measuring. I checked all nine verifier_bijectivite=False call sites across six files. Every one is an argsort of a random vector or a transposition and 3-cycle climb from a permutation, so all nine are safe today. The guard is doing its job. The 0.0205 is what the guard is worth.
Your supremum reproduces, and it is not a plateau
Independent seed, 6000 restarts, transpositions plus sampled 3-cycles, winner scored by the sklearn and Hungarian implementation above:
24 starts : 0.151460867637
750 starts : 0.151460867637
1500 starts : 0.154321642873
6000 starts : 0.154321642873
Twelve digits, same value. And it is an exact point, not a plateau.
For a bijective code every token at position j covers exactly 9 referents, and every attribute value covers exactly 9. So the 3x3 joint table of (A_i, M_j) has all six margins equal to 9. There are exactly 1540 such tables and they take exactly 55 distinct values of I(A_i ; M_j), the largest being log2(3). Your statistic is a sum of nine draws from a 55-value lattice, over log2(27).
The winning matrix from my run:
0.000000000000 0.000000000000 0.000000000000
0.000000000000 0.000000000000 0.000000000000
0.415630552279 0.438340850504 0.318151498733
Every entry sits on the lattice to 1e-13. So the number is
(0.415630552279 + 0.438340850504 + 0.318151498733 - 0.438340850504) / log2(27)
= 0.733782051012 / 4.754887502163
= 0.154321642873
Not a proof that it is the supremum. But a value you can write down with its coordinates, instead of reporting a restart count at which it stopped moving.
Q1: there is a test, and it fires on 0.1443
Off-row mass is pure loss. conc is the sum of the column maxima and does not see it. matched is a maximum over assignments and can only be raised by it. So a winner carrying non-zero mass off its argmax row is beaten by the matrix with the same column maxima and zeros elsewhere.
600 climbs, 57 distinct local optima. The top five:
inflation n climbs one argmax row off-row mass
0.154321642873 3 3/3 0.000000000000
0.151460867637 21 21/21 0.069167547890
0.147337819489 1 1/1 0.000000000000
0.146684666683 69 69/69 0.131042430405
0.144297209128 2 2/2 0.099860647267
Your published 0.144297209128 carries 0.0999 bits off its winning row. That is legible in the matrix you already print beside it, at 24 restarts, at seed 7, with no second run.
0.151460867637 is the demonstration. Its top row is 0.438340850504 0.318151498733 0.415630552279, the same three values as the plateau, with 0.036312962642 sitting twice in column 0. conc_max identical at 0.246509071136. The off-row mass costs exactly 0.002860775236 of inflation, and the code that removes it exists. It is the plateau.
The test is necessary and not sufficient. 0.147337819489 has zero off-row mass and is still not the maximum. A dirty matrix proves you are not done. A clean one proves nothing.
Q2: the ceiling is not the mechanism, and how it fails is the finding
I fitted both forms on the same windows you had, thresholds up to 0.10, and scored them where the 3e9 run has counts. Predicted over measured, so 1.00 is right:
form window R2 | 0.110 0.115 0.120 0.125 0.130 0.135 | P(>=.1443)
log p = a + b t (unbounded) [0.03,0.10] 0.9950 | 3.23 5.26 10.85 26.56 82.78 100.30 | 2.97e-08
log p = a + c log(M-t) (true edge) [0.03,0.10] 0.9975 | 0.64 0.52 0.45 0.36 0.27 0.05 | 2.99e-14
The unbounded form over-predicts monotonically, exactly as you said. Putting the true edge in does not correct it. It reverses the sign and under-predicts monotonically, out by a factor of 20 at 0.135.
The sensitivity says why:
M R2 geomean ratio at 0.135
0.1443 0.9957 0.07 0.00
0.154321642873 0.9975 0.30 0.05
0.16 0.9981 0.50 0.15
0.18 0.9992 1.42 1.27
0.25 0.9993 5.00 12.27
Fitted on [0.03, 0.10], the data wants an edge near 0.18. The real edge is 0.1543. The body of the distribution does not know where its own support ends, and telling it the truth makes it worse.
So the conclusion is neither "the extrapolation was unstable" nor "an unbounded form was fitted to a bounded variable". Two forms sit at R2 around 0.9975 on the same window and disagree by six orders of magnitude at 0.1443. R2 on the fit window carries no information about the extrapolation. It failed on model choice, which no n repairs, and which is a worse result than the one you proposed.
Q4
Yours, under whatever name you like. Driver first. Counting is the searchsorted and bincount in work.
import sys, time, json
import numpy as np, importlib.util
from multiprocessing import Pool
def build():
spec = importlib.util.spec_from_file_location("loinulle",
"src/test3_communication/loi_nulle_longue.py")
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
import grammaire3
return mod, grammaire3.N
THR = sorted(set([round(x,4) for x in np.arange(0.03, 0.1801, 0.005)]
+ [0.1443, 0.122365, 0.10807050074977963]))
TA = np.array(THR)
def work(a):
seed_state, n = a
mod, N = build()
g = np.random.default_rng(seed_state)
lot = 200_000
cnt = np.zeros(len(THR)+1, dtype=np.int64)
mx = -1.0; done = 0; s_infl = 0.0; n_pos = 0
while done < n:
b = min(lot, n-done)
codes = np.argsort(g.random((b, N)), axis=1)
cm, ca, dc = mod.statistiques(
mod.matrices_information(codes, verifier_bijectivite=False))
infl = cm - ca
s_infl += float(infl.sum()); n_pos += int((infl > 0).sum())
m = float(infl.max())
if m > mx: mx = m
k = np.searchsorted(TA, infl, side='right')
cnt += np.bincount(k, minlength=len(THR)+1)
done += b
return cnt, mx, done, s_infl, n_pos
if __name__ == "__main__":
TOTAL = int(float(sys.argv[1])); W = int(sys.argv[2])
per = TOTAL // W
ss = np.random.SeedSequence(20260817).spawn(W)
with Pool(W) as p:
res = p.map(work, [(s, per) for s in ss])
cnt = sum(r[0] for r in res); mx = max(r[1] for r in res)
n = sum(r[2] for r in res); s_infl = sum(r[3] for r in res)
n_pos = sum(r[4] for r in res)
exc = np.cumsum(cnt[::-1])[::-1][1:]
print(f"n = {n:,} max = {mx:.6f} E = {s_infl/n:.8f} Ppos = {n_pos/n:.7f}")
for t, c in zip(THR, exc):
print(f"{t:>12.6f} {int(c):>14,} {int(c)/n:>12.3e}")
json.dump({"n": n, "max": mx, "E": s_infl/n, "Ppos": n_pos/n,
"counts": {f"{t:.8f}": int(c) for t, c in zip(THR, exc)}},
open("queue.json", "w"), indent=1)
Run as python queue.py 3e9 16. Forty minutes on sixteen workers.
And the out-of-sample fit, which is the whole of the Q2 table:
import json, math
import numpy as np
M = 0.154321642873
d = json.load(open("queue.json")); n = d["n"]
cnt = {float(k): v for k, v in d["counts"].items()}
thr = np.array(sorted(cnt)); cts = np.array([cnt[t] for t in thr], float)
OOS = [0.110, 0.115, 0.120, 0.125, 0.130, 0.135]
meas = {t: cnt[t] / n for t in OOS}
FORMS = {"unbounded": lambda t: np.asarray(t),
"finite edge": lambda t: np.log(M - np.asarray(t))}
for name, basis in FORMS.items():
for lo, hi in [(0.03, 0.10), (0.05, 0.10), (0.07, 0.10)]:
m = (thr >= lo - 1e-9) & (thr <= hi + 1e-9) & (cts > 0)
X = np.column_stack([basis(thr[m]), np.ones(m.sum())])
y = np.log(cts[m] / n)
c, *_ = np.linalg.lstsq(X, y, rcond=None)
r2 = 1 - ((y - X @ c) ** 2).sum() / ((y - y.mean()) ** 2).sum()
ratio = [float(np.exp(c[0] * basis(np.array([t]))[0] + c[1])) / meas[t]
for t in OOS]
print(f"{name:>12} [{lo},{hi}] R2={r2:.4f} "
+ " ".join(f"{r:>7.2f}" for r in ratio))
Swap M for any candidate edge and you get the sensitivity table directly.
One back. You asked what fires on "compute it instead". The off-row certificate prunes and does not decide. The lattice is the other half: 55 values, so the inflation takes finitely many values, and the ones above 0.154321642873 are a short enumerable list with coordinates. Is there a reason to keep restarting into that list one accident at a time, rather than picking a target value off it and asking whether a bijection realises it?
