3@file encoding_layer_entropy.py
4@brief Stored-bytes vs. key-entropy accounting for the output-encoding sublayer,
5 and its entry-determinacy degree
as the positive side of the dichotomy.
9the layer
is measured exactly
as this repository constructs it rather than
as a
10reimplementation. Four Z_q-permutation tables are stored per NTT factor (two
11forward encodings
and their two inverses), but each inverse
is a deterministic
12function of its forward permutation, so only half the stored tables carry key
13material. The script (1) measures the stored footprint
from the built arrays
'
14.nbytes, (2) reconstructs every inverse from its forward permutation
and asserts
15equality, establishing the independent-bijection count
as half the stored table
16count, (3) computes the key entropy H = (independent count) * log2(q!) by lgamma,
17(4) reports the overstatement factor stored/entropy,
and (5) measures the
18omitted-entry predictability of the built bijections against the 1/q floor.
20Result carried by this script: the sublayer
is entry-unpredictable. Retaining
21q//2 entries of a bijection leaves every omitted entry uniform over the 1665
22unused residues, i.e. a predictability of ~2/q, only a factor two above the
23unconditional 1/q floor; the entry-determinacy degree
is d = q-1, since a
24bijection
is pinned only once a single residue remains. This
is the POSITIVE
25side of the entry-determinacy dichotomy, opposite the algebraically-determined
26baked tables whose d = 1.
28@note The stored footprint
is NOT secret-bearing material. Storage exceeds key
29entropy by the measured overstatement factor because the inverses are derived
and
30because a Z_q permutation stores q uint16 entries
while carrying only log2(q!)
31bits. Reporting the stored megabytes
as entropy overstates the key material.
32@note MEASURED: footprint, inverse derivability, independent count, omitted-entry
33predictability. ANALYTIC: log2(q!) key entropy
and the floors it
is compared to.
34The two are emitted under separate top-level keys
and never mixed.
35@note Deterministic (seed 42); writes results/encoding_layer_entropy.json.
37@section encoding_layer_entropy_output Output Files
38- `results/encoding_layer_entropy.json` - Encoding-layer storage, entropy, overstatement, determinacy
40@section encoding_layer_entropy_usage Usage
42python encoding_layer_entropy.py
47This software
is provided
for academic
and research purposes.
48See LICENSE file
for details.
53from pathlib import Path
57sys.path.insert(0, str(Path(__file__).parent))
60 from whitebox_encoded
import Q, build_encodings
61except ImportError
as exc:
63 "cannot import build_encodings from whitebox_encoded: %s. "
64 "REMEDIATION: run this script from code/python/ of the repository so that "
65 "whitebox_encoded.py is importable; the encoding layer must be measured as "
66 "this repo builds it, never reimplemented here." % exc
70PRED_GUESSES_PER_TABLE = 1000
74EXPECTED_STORED_BYTES = 3_408_896
75EXPECTED_INDEPENDENT = 256
76EXPECTED_ENTROPY_LOG2_BYTES = 20.06
77EXPECTED_OVERSTATEMENT = 3.12
82 @brief log2(n!) via lgamma, which stays finite where n! overflows.
83 @param n Non-negative integer.
84 @return log2(n!)
in bits.
88 "log2_factorial received n=%d. REMEDIATION: pass a non-negative "
89 "integer; q must be a positive prime." % n
91 return math.lgamma(n + 1) / math.log(2.0)
96 @brief Verify a table
is a permutation of Z_q.
97 @param perm uint16 table of length q.
98 @param label Identifier used
in the failure message.
100 if perm.shape != (Q,):
102 "%s has shape %s, expected (%d,). REMEDIATION: build_encodings() must "
103 "emit one length-q table per encoding; check its contract."
104 % (label, perm.shape, Q)
106 if not np.array_equal(np.sort(perm.astype(np.int64)), np.arange(Q)):
108 "%s is not a bijection on Z_q. REMEDIATION: the entropy and "
109 "predictability accounting below assume permutations; fix "
110 "build_encodings() before trusting any number in this file." % label
116 @brief Reconstruct every inverse
from its forward permutation
and compare.
117 @param E Forward encodings, shape (components, 2, q).
118 @param Einv Stored inverses, shape (components, 2, q).
119 @return Number of stored inverse tables shown to be derived (non-independent).
120 @details Derivability
is what demotes the stored table count to the independent
121 count: an inverse carries no key material of its own.
124 for i
in range(E.shape[0]):
125 for j
in range(E.shape[1]):
128 rebuilt = np.empty(Q, dtype=np.uint16)
129 rebuilt[perm.astype(np.int64)] = np.arange(Q, dtype=np.uint16)
130 if not np.array_equal(rebuilt, Einv[i, j]):
132 "inverse Einv[%d,%d] does not match the inverse reconstructed from "
133 "E[%d,%d]. REMEDIATION: the independent-bijection count is derived "
134 "from this equality; do not report an entropy figure until "
135 "build_encodings() is fixed." % (i, j, i, j)
143 @brief Predictability of an omitted entry when q//2 entries of a bijection are
144 retained, i.e. the M < S_tbl/2 budget regime.
145 @param E Forward encodings, shape (components, 2, q); only forward tables are probed.
146 @param rng Seeded NumPy generator.
147 @return Dict of measured predictability quantities.
148 @details The strongest predictor
is bijectivity-aware: an omitted entry must lie
149 in the residues
not consumed by the retained ones, so it guesses
150 uniformly over that residual candidate set. Both the empirical hit rate
151 and the mean candidate-set size are measured.
153 tables = E.reshape(-1, Q)
158 for t
in range(tables.shape[0]):
159 perm = tables[t].astype(np.int64)
160 idx = rng.permutation(Q)
161 retained_idx = idx[:n_retained]
162 omitted_idx = idx[n_retained:]
163 used = np.zeros(Q, dtype=bool)
164 used[perm[retained_idx]] =
True
165 candidates = np.flatnonzero(~used)
166 candidate_sizes.append(int(candidates.size))
167 pick_omitted = rng.integers(0, omitted_idx.size, size=PRED_GUESSES_PER_TABLE)
168 pick_cand = rng.integers(0, candidates.size, size=PRED_GUESSES_PER_TABLE)
169 truth = perm[omitted_idx[pick_omitted]]
170 guess = candidates[pick_cand]
171 hits += int(np.count_nonzero(truth == guess))
172 draws += PRED_GUESSES_PER_TABLE
175 "no predictability draws were made. REMEDIATION: PRED_GUESSES_PER_TABLE "
176 "must be positive and build_encodings() must return a non-empty layer."
178 mean_candidates = float(np.mean(candidate_sizes))
179 if mean_candidates <= 0:
181 "residual candidate set is empty. REMEDIATION: retaining q//2 entries of a "
182 "bijection must leave q-q//2 residues unused; check assert_bijection()."
185 "retained_entries_per_table": n_retained,
186 "omitted_entries_per_table": Q - n_retained,
187 "tables_probed": int(tables.shape[0]),
188 "guesses_total": draws,
190 "empirical_hit_rate": hits / draws,
191 "mean_residual_candidate_set_size": mean_candidates,
192 "predictability_from_candidate_set": 1.0 / mean_candidates,
198 @brief Entry-determinacy degree of a Z_q bijection, measured by reconstruction.
199 @param E Forward encodings, shape (components, 2, q).
200 @param rng Seeded NumPy generator.
201 @return Dict
with the measured degree
and both reconstruction checks.
202 @details Two probes bracket the degree. At q-1 retained entries the omitted one
is
203 forced: the residues consumed by the RETAINED entries leave exactly one
204 unused, which must be the omitted image. At q-2 retained entries two
205 residues remain
for two omitted indices, so the best predictor
is a coin
206 flip. The table
is therefore pinned at q-1
and not before: d = q-1.
207 Both probes read only retained entries; the withheld image never enters
210 perm = E[0, 0].astype(np.int64)
213 for x0
in range(probes):
214 retained = np.delete(np.arange(Q), x0)
215 used = np.zeros(Q, dtype=bool)
216 used[perm[retained]] =
True
217 candidates = np.flatnonzero(~used)
218 if candidates.size != 1:
220 "withholding one entry left %d residual candidates, expected 1. "
221 "REMEDIATION: the table at E[0,0] is not a bijection on Z_q; fix "
222 "build_encodings()." % candidates.size
224 if int(candidates[0]) == int(perm[x0]):
226 if recovered != probes:
228 "reconstructed %d/%d withheld entries; expected all. REMEDIATION: the "
229 "determinacy-degree claim depends on this reconstruction being exact."
230 % (recovered, probes)
234 hits_at_q_minus_2 = 0
235 for _
in range(probes):
236 pair = rng.choice(Q, size=2, replace=
False)
237 retained = np.delete(np.arange(Q), pair)
238 used = np.zeros(Q, dtype=bool)
239 used[perm[retained]] =
True
240 candidates = np.flatnonzero(~used)
241 if candidates.size != 2:
243 "withholding two entries left %d residual candidates, expected 2. "
244 "REMEDIATION: fix build_encodings()." % candidates.size
246 guess = candidates[rng.integers(0, 2)]
247 if int(guess) == int(perm[pair[0]]):
248 hits_at_q_minus_2 += 1
250 "withheld_entry_probes": probes,
251 "withheld_entries_reconstructed_at_q_minus_1": recovered,
252 "predictability_at_q_minus_1_retained": recovered / probes,
253 "hits_at_q_minus_2_retained": hits_at_q_minus_2,
254 "predictability_at_q_minus_2_retained": round(hits_at_q_minus_2 / probes, 4),
255 "entry_determinacy_degree_d": Q - 1,
262 split measured
from analytic,
and write results/encoding_layer_entropy.json.
264 rng = np.random.default_rng(SEED)
265 E, Einv = build_encodings(rng)
268 stored_bytes = int(E.nbytes) + int(Einv.nbytes)
269 bytes_per_entry = int(E.dtype.itemsize)
270 stored_tables = int(E.shape[0] * E.shape[1] + Einv.shape[0] * Einv.shape[1])
274 independent_tables = stored_tables - derived_tables
275 if independent_tables * 2 != stored_tables:
277 "independent tables (%d) are not half of stored tables (%d). REMEDIATION: "
278 "build_encodings() is expected to emit one inverse per forward encoding; "
279 "the halving claim must be re-derived if that changes."
280 % (independent_tables, stored_tables)
285 entropy_bits = independent_tables * log2_qfact
286 entropy_bytes = entropy_bits / 8.0
287 entropy_log2_bytes = math.log2(entropy_bytes)
290 overstatement = stored_bytes / entropy_bytes
295 floor_1_over_q = 1.0 / Q
296 pred_over_floor = pred[
"predictability_from_candidate_set"] / floor_1_over_q
301 "num_components": int(E.shape[0]),
302 "encodings_per_component": int(E.shape[1]),
303 "source":
"whitebox_encoded.build_encodings (layer measured as this repo builds it)",
305 "basis":
"computed from the arrays returned by build_encodings() in this run",
307 "bytes_per_entry": bytes_per_entry,
308 "entries_per_table": Q,
309 "stored_tables": stored_tables,
310 "forward_tables_nbytes": int(E.nbytes),
311 "inverse_tables_nbytes": int(Einv.nbytes),
312 "stored_bytes": stored_bytes,
313 "stored_MB": round(stored_bytes / 1e6, 3),
314 "stored_log2_bytes": round(math.log2(stored_bytes), 2),
317 "inverse_tables_reconstructed_and_matched": derived_tables,
318 "independent_bijections": independent_tables,
319 "independent_is_half_of_stored": independent_tables * 2
321 "note":
"each inverse was rebuilt from its forward permutation and compared "
322 "entrywise; a derived table carries no key material of its own",
324 "omitted_entry_predictability": {
325 "retained_entries_per_table": pred[
"retained_entries_per_table"],
326 "omitted_entries_per_table": pred[
"omitted_entries_per_table"],
327 "tables_probed": pred[
"tables_probed"],
328 "guesses_total": pred[
"guesses_total"],
329 "hits": pred[
"hits"],
330 "empirical_hit_rate": round(pred[
"empirical_hit_rate"], 8),
331 "mean_residual_candidate_set_size": round(
332 pred[
"mean_residual_candidate_set_size"], 2
334 "predictability_from_candidate_set": round(
335 pred[
"predictability_from_candidate_set"], 8
337 "ratio_to_1_over_q_floor": round(pred_over_floor, 4),
339 "entry_determinacy": det,
342 "basis":
"closed form from named parameters; NOT measured",
343 "log2_q_factorial_bits_per_bijection": round(log2_qfact, 4),
344 "entropy_formula":
"H = independent_bijections * log2(q!)",
345 "entropy_bits": round(entropy_bits, 2),
346 "entropy_bytes": round(entropy_bytes, 2),
347 "entropy_MB": round(entropy_bytes / 1e6, 3),
348 "entropy_log2_bytes": round(entropy_log2_bytes, 2),
349 "unconditional_floor_1_over_q": round(floor_1_over_q, 8),
350 "bijection_aware_floor_1_over_omitted": round(
351 1.0 / pred[
"omitted_entries_per_table"], 8
354 "derived_measured_over_analytic": {
355 "basis":
"measured stored_bytes divided by analytic entropy_bytes",
356 "overstatement_factor_stored_over_entropy": round(overstatement, 3),
358 "comparison_to_expected": {
359 "basis":
"booleans only; the numbers reported above are this run's, not these targets",
360 "expected_stored_bytes": EXPECTED_STORED_BYTES,
361 "stored_bytes_matches_expected": stored_bytes == EXPECTED_STORED_BYTES,
362 "expected_independent_bijections": EXPECTED_INDEPENDENT,
363 "independent_matches_expected": independent_tables == EXPECTED_INDEPENDENT,
364 "expected_entropy_log2_bytes": EXPECTED_ENTROPY_LOG2_BYTES,
365 "entropy_log2_bytes_matches_expected": abs(
366 entropy_log2_bytes - EXPECTED_ENTROPY_LOG2_BYTES
369 "expected_overstatement": EXPECTED_OVERSTATEMENT,
370 "overstatement_matches_expected": abs(
371 overstatement - EXPECTED_OVERSTATEMENT
376 "correction_owed":
"The encoding sublayer stores the measured stored_MB above but "
377 "carries only entropy_MB of key material; the stored figure is "
378 "storage, NOT secret-bearing material. Describing the stored "
379 "megabytes as entropy overstates the key material by the measured "
380 "overstatement factor. Two causes, both quantified here: the "
381 "inverses are derived (factor 2 exactly, verified by "
382 "reconstruction) and a Z_q permutation spends q uint16 entries to "
383 "carry log2(q!) bits (the residual factor).",
384 "dichotomy_side":
"POSITIVE. The layer is entry-unpredictable: with q//2 entries "
385 "retained, an omitted entry stays uniform over the residual "
386 "candidate set, i.e. within the measured ratio_to_1_over_q_floor of "
387 "the 1/q floor, and the table is pinned only at d = q-1 retained "
388 "entries. This is the opposite of an algebraically-determined "
389 "family, whose omitted entries follow from a constant number of "
394 root = Path(__file__).parent.parent.parent
395 results_dir = root /
"results"
396 results_dir.mkdir(parents=
True, exist_ok=
True)
397 path = results_dir /
"encoding_layer_entropy.json"
398 with open(path,
"w", encoding=
"utf-8", newline=
"\n")
as f:
399 json.dump(out, f, indent=2)
401 st = out[
"measured"][
"storage"]
403 cmp_ = out[
"comparison_to_expected"]
405 print(
"ENCODING LAYER (built by whitebox_encoded.build_encodings, seed %d)" % SEED)
407 " stored tables : %d (%d forward + %d inverse), %d entries x %d B"
408 % (st[
"stored_tables"], stored_tables // 2, derived_tables, Q, bytes_per_entry)
411 " stored bytes [MEASURED] : %d B = %.3f MB = 2^%.2f"
412 % (st[
"stored_bytes"], st[
"stored_MB"], st[
"stored_log2_bytes"])
415 " inverses derived : %d/%d reconstructed and matched"
416 % (derived_tables, stored_tables // 2)
419 " independent bijections : %d (half the stored tables)" % independent_tables
422 " log2(q!) [ANALYTIC] : %.2f bits per bijection"
423 % an[
"log2_q_factorial_bits_per_bijection"]
426 " key entropy [ANALYTIC] : %.0f bits = %.3f MB = 2^%.2f"
427 % (an[
"entropy_bits"], an[
"entropy_MB"], an[
"entropy_log2_bytes"])
430 " overstatement [DERIVED] : %.3fx (stored / entropy)"
431 % out[
"derived_measured_over_analytic"][
432 "overstatement_factor_stored_over_entropy"
435 p = out[
"measured"][
"omitted_entry_predictability"]
437 "OMITTED-ENTRY PREDICTABILITY at M < S_tbl/2 (%d retained / %d omitted)"
438 % (p[
"retained_entries_per_table"], p[
"omitted_entries_per_table"])
441 " empirical hit rate : %.8f (%d hits / %d guesses)"
442 % (p[
"empirical_hit_rate"], p[
"hits"], p[
"guesses_total"])
445 " candidate-set prediction : %.8f (mean set %.2f)"
447 p[
"predictability_from_candidate_set"],
448 p[
"mean_residual_candidate_set_size"],
452 " 1/q floor : %.8f -> ratio %.4fx"
453 % (an[
"unconditional_floor_1_over_q"], p[
"ratio_to_1_over_q_floor"])
456 " entry-determinacy degree : d = %d (q-1); entry-unpredictable family"
457 % det[
"entry_determinacy_degree_d"]
459 print(
"COMPARISON TO EXPECTED (booleans; reported numbers are this run's)")
461 "stored_bytes_matches_expected",
462 "independent_matches_expected",
463 "entropy_log2_bytes_matches_expected",
464 "overstatement_matches_expected",
466 print(
" %-38s: %s" % (k, cmp_[k]))
469if __name__ ==
"__main__":
assert_bijection(perm, label)
Verify a table is a permutation of Z_q.
main()
Measure the encoding layer as built by whitebox_encoded.build_encodings(), split measured from analyt...
log2_factorial(n)
log2(n!) via lgamma, which stays finite where n! overflows.
measure_determinacy_degree(E, rng)
Entry-determinacy degree of a Z_q bijection, measured by reconstruction.
verify_inverses_derivable(E, Einv)
Reconstruct every inverse from its forward permutation and compare.
measure_omitted_entry_predictability(E, rng)
Predictability of an omitted entry when q//2 entries of a bijection are retained, i....
build_encodings(rng)
Secret nonlinear I/O encodings: two random bijections on Z_q per component plus their inverses.