PQ-WB-KEM: White-Box ML-KEM-768 1.0
PQ-WB-KEM: Toward a White-Box Construction of ML-KEM-768 with Arithmetic Masking for M2M Communications - Mathematics (MDPI)
Loading...
Searching...
No Matches
encoding_layer_entropy.py
Go to the documentation of this file.
1# -*- coding: utf-8 -*-
2"""
3@file encoding_layer_entropy.py
4@brief Stored-bytes vs. key-entropy accounting for the output-encoding sublayer,
5 and its entry-determinacy degree as the positive side of the dichotomy.
6@ingroup core
7
8@details The encoding sublayer is built by whitebox_encoded.build_encodings(), so
9the layer is measured exactly as this repository constructs it rather than as a
10reimplementation. Four Z_q-permutation tables are stored per NTT factor (two
11forward encodings and their two inverses), but each inverse is a deterministic
12function of its forward permutation, so only half the stored tables carry key
13material. The script (1) measures the stored footprint from the built arrays'
14.nbytes, (2) reconstructs every inverse from its forward permutation and asserts
15equality, establishing the independent-bijection count as half the stored table
16count, (3) computes the key entropy H = (independent count) * log2(q!) by lgamma,
17(4) reports the overstatement factor stored/entropy, and (5) measures the
18omitted-entry predictability of the built bijections against the 1/q floor.
19
20Result carried by this script: the sublayer is entry-unpredictable. Retaining
21q//2 entries of a bijection leaves every omitted entry uniform over the 1665
22unused residues, i.e. a predictability of ~2/q, only a factor two above the
23unconditional 1/q floor; the entry-determinacy degree is d = q-1, since a
24bijection is pinned only once a single residue remains. This is the POSITIVE
25side of the entry-determinacy dichotomy, opposite the algebraically-determined
26baked tables whose d = 1.
27
28@note The stored footprint is NOT secret-bearing material. Storage exceeds key
29entropy by the measured overstatement factor because the inverses are derived and
30because a Z_q permutation stores q uint16 entries while carrying only log2(q!)
31bits. Reporting the stored megabytes as entropy overstates the key material.
32@note MEASURED: footprint, inverse derivability, independent count, omitted-entry
33predictability. ANALYTIC: log2(q!) key entropy and the floors it is compared to.
34The two are emitted under separate top-level keys and never mixed.
35@note Deterministic (seed 42); writes results/encoding_layer_entropy.json.
36
37@section encoding_layer_entropy_output Output Files
38- `results/encoding_layer_entropy.json` - Encoding-layer storage, entropy, overstatement, determinacy
39
40@section encoding_layer_entropy_usage Usage
41@code{.sh}
42python encoding_layer_entropy.py
43@endcode
44
45@author Ugur Coruh
46@copyright
47This software is provided for academic and research purposes.
48See LICENSE file for details.
49"""
50import json
51import math
52import sys
53from pathlib import Path
54
55import numpy as np
56
57sys.path.insert(0, str(Path(__file__).parent))
58
59try:
60 from whitebox_encoded import Q, build_encodings
61except ImportError as exc:
62 raise RuntimeError(
63 "cannot import build_encodings from whitebox_encoded: %s. "
64 "REMEDIATION: run this script from code/python/ of the repository so that "
65 "whitebox_encoded.py is importable; the encoding layer must be measured as "
66 "this repo builds it, never reimplemented here." % exc
67 )
68
69SEED = 42
70PRED_GUESSES_PER_TABLE = 1000 # Monte-Carlo draws per bijection for the hit rate
71
72# Values claimed by prior work / the papers under correction. Used ONLY as
73# comparison targets: every emitted number is produced by this run.
74EXPECTED_STORED_BYTES = 3_408_896
75EXPECTED_INDEPENDENT = 256
76EXPECTED_ENTROPY_LOG2_BYTES = 20.06
77EXPECTED_OVERSTATEMENT = 3.12
78
79
81 """
82 @brief log2(n!) via lgamma, which stays finite where n! overflows.
83 @param n Non-negative integer.
84 @return log2(n!) in bits.
85 """
86 if n < 0:
87 raise RuntimeError(
88 "log2_factorial received n=%d. REMEDIATION: pass a non-negative "
89 "integer; q must be a positive prime." % n
90 )
91 return math.lgamma(n + 1) / math.log(2.0)
92
93
94def assert_bijection(perm, label):
95 """
96 @brief Verify a table is a permutation of Z_q.
97 @param perm uint16 table of length q.
98 @param label Identifier used in the failure message.
99 """
100 if perm.shape != (Q,):
101 raise RuntimeError(
102 "%s has shape %s, expected (%d,). REMEDIATION: build_encodings() must "
103 "emit one length-q table per encoding; check its contract."
104 % (label, perm.shape, Q)
105 )
106 if not np.array_equal(np.sort(perm.astype(np.int64)), np.arange(Q)):
107 raise RuntimeError(
108 "%s is not a bijection on Z_q. REMEDIATION: the entropy and "
109 "predictability accounting below assume permutations; fix "
110 "build_encodings() before trusting any number in this file." % label
111 )
112
113
115 """
116 @brief Reconstruct every inverse from its forward permutation and compare.
117 @param E Forward encodings, shape (components, 2, q).
118 @param Einv Stored inverses, shape (components, 2, q).
119 @return Number of stored inverse tables shown to be derived (non-independent).
120 @details Derivability is what demotes the stored table count to the independent
121 count: an inverse carries no key material of its own.
122 """
123 derived = 0
124 for i in range(E.shape[0]):
125 for j in range(E.shape[1]):
126 perm = E[i, j]
127 assert_bijection(perm, "E[%d,%d]" % (i, j))
128 rebuilt = np.empty(Q, dtype=np.uint16)
129 rebuilt[perm.astype(np.int64)] = np.arange(Q, dtype=np.uint16)
130 if not np.array_equal(rebuilt, Einv[i, j]):
131 raise RuntimeError(
132 "inverse Einv[%d,%d] does not match the inverse reconstructed from "
133 "E[%d,%d]. REMEDIATION: the independent-bijection count is derived "
134 "from this equality; do not report an entropy figure until "
135 "build_encodings() is fixed." % (i, j, i, j)
136 )
137 derived += 1
138 return derived
139
140
142 """
143 @brief Predictability of an omitted entry when q//2 entries of a bijection are
144 retained, i.e. the M < S_tbl/2 budget regime.
145 @param E Forward encodings, shape (components, 2, q); only forward tables are probed.
146 @param rng Seeded NumPy generator.
147 @return Dict of measured predictability quantities.
148 @details The strongest predictor is bijectivity-aware: an omitted entry must lie
149 in the residues not consumed by the retained ones, so it guesses
150 uniformly over that residual candidate set. Both the empirical hit rate
151 and the mean candidate-set size are measured.
152 """
153 tables = E.reshape(-1, Q)
154 n_retained = Q // 2
155 hits = 0
156 draws = 0
157 candidate_sizes = []
158 for t in range(tables.shape[0]):
159 perm = tables[t].astype(np.int64)
160 idx = rng.permutation(Q)
161 retained_idx = idx[:n_retained]
162 omitted_idx = idx[n_retained:]
163 used = np.zeros(Q, dtype=bool)
164 used[perm[retained_idx]] = True
165 candidates = np.flatnonzero(~used) # residues still available
166 candidate_sizes.append(int(candidates.size))
167 pick_omitted = rng.integers(0, omitted_idx.size, size=PRED_GUESSES_PER_TABLE)
168 pick_cand = rng.integers(0, candidates.size, size=PRED_GUESSES_PER_TABLE)
169 truth = perm[omitted_idx[pick_omitted]]
170 guess = candidates[pick_cand]
171 hits += int(np.count_nonzero(truth == guess))
172 draws += PRED_GUESSES_PER_TABLE
173 if draws == 0:
174 raise RuntimeError(
175 "no predictability draws were made. REMEDIATION: PRED_GUESSES_PER_TABLE "
176 "must be positive and build_encodings() must return a non-empty layer."
177 )
178 mean_candidates = float(np.mean(candidate_sizes))
179 if mean_candidates <= 0:
180 raise RuntimeError(
181 "residual candidate set is empty. REMEDIATION: retaining q//2 entries of a "
182 "bijection must leave q-q//2 residues unused; check assert_bijection()."
183 )
184 return {
185 "retained_entries_per_table": n_retained,
186 "omitted_entries_per_table": Q - n_retained,
187 "tables_probed": int(tables.shape[0]),
188 "guesses_total": draws,
189 "hits": hits,
190 "empirical_hit_rate": hits / draws,
191 "mean_residual_candidate_set_size": mean_candidates,
192 "predictability_from_candidate_set": 1.0 / mean_candidates,
193 }
194
195
197 """
198 @brief Entry-determinacy degree of a Z_q bijection, measured by reconstruction.
199 @param E Forward encodings, shape (components, 2, q).
200 @param rng Seeded NumPy generator.
201 @return Dict with the measured degree and both reconstruction checks.
202 @details Two probes bracket the degree. At q-1 retained entries the omitted one is
203 forced: the residues consumed by the RETAINED entries leave exactly one
204 unused, which must be the omitted image. At q-2 retained entries two
205 residues remain for two omitted indices, so the best predictor is a coin
206 flip. The table is therefore pinned at q-1 and not before: d = q-1.
207 Both probes read only retained entries; the withheld image never enters
208 the reconstruction.
209 """
210 perm = E[0, 0].astype(np.int64)
211 probes = min(64, Q)
212 recovered = 0
213 for x0 in range(probes):
214 retained = np.delete(np.arange(Q), x0) # everything except the target
215 used = np.zeros(Q, dtype=bool)
216 used[perm[retained]] = True # images of RETAINED entries only
217 candidates = np.flatnonzero(~used)
218 if candidates.size != 1:
219 raise RuntimeError(
220 "withholding one entry left %d residual candidates, expected 1. "
221 "REMEDIATION: the table at E[0,0] is not a bijection on Z_q; fix "
222 "build_encodings()." % candidates.size
223 )
224 if int(candidates[0]) == int(perm[x0]):
225 recovered += 1
226 if recovered != probes:
227 raise RuntimeError(
228 "reconstructed %d/%d withheld entries; expected all. REMEDIATION: the "
229 "determinacy-degree claim depends on this reconstruction being exact."
230 % (recovered, probes)
231 )
232
233 # q-2 retained: the degree is not below q-1, so the pair stays a coin flip.
234 hits_at_q_minus_2 = 0
235 for _ in range(probes):
236 pair = rng.choice(Q, size=2, replace=False)
237 retained = np.delete(np.arange(Q), pair)
238 used = np.zeros(Q, dtype=bool)
239 used[perm[retained]] = True
240 candidates = np.flatnonzero(~used)
241 if candidates.size != 2:
242 raise RuntimeError(
243 "withholding two entries left %d residual candidates, expected 2. "
244 "REMEDIATION: fix build_encodings()." % candidates.size
245 )
246 guess = candidates[rng.integers(0, 2)] # best available: pick one of the two
247 if int(guess) == int(perm[pair[0]]):
248 hits_at_q_minus_2 += 1
249 return {
250 "withheld_entry_probes": probes,
251 "withheld_entries_reconstructed_at_q_minus_1": recovered,
252 "predictability_at_q_minus_1_retained": recovered / probes,
253 "hits_at_q_minus_2_retained": hits_at_q_minus_2,
254 "predictability_at_q_minus_2_retained": round(hits_at_q_minus_2 / probes, 4),
255 "entry_determinacy_degree_d": Q - 1,
256 }
257
258
259def main():
260 """
261 @brief Measure the encoding layer as built by whitebox_encoded.build_encodings(),
262 split measured from analytic, and write results/encoding_layer_entropy.json.
263 """
264 rng = np.random.default_rng(SEED)
265 E, Einv = build_encodings(rng)
266
267 # (a) stored footprint, straight off the built arrays
268 stored_bytes = int(E.nbytes) + int(Einv.nbytes)
269 bytes_per_entry = int(E.dtype.itemsize)
270 stored_tables = int(E.shape[0] * E.shape[1] + Einv.shape[0] * Einv.shape[1])
271
272 # (b) inverses are derived -> independent count is half the stored tables
273 derived_tables = verify_inverses_derivable(E, Einv)
274 independent_tables = stored_tables - derived_tables
275 if independent_tables * 2 != stored_tables:
276 raise RuntimeError(
277 "independent tables (%d) are not half of stored tables (%d). REMEDIATION: "
278 "build_encodings() is expected to emit one inverse per forward encoding; "
279 "the halving claim must be re-derived if that changes."
280 % (independent_tables, stored_tables)
281 )
282
283 # (c) key entropy of the independent bijections (analytic: log2(q!) per bijection)
284 log2_qfact = log2_factorial(Q)
285 entropy_bits = independent_tables * log2_qfact
286 entropy_bytes = entropy_bits / 8.0
287 entropy_log2_bytes = math.log2(entropy_bytes)
288
289 # (d) storage overstates key material by this factor
290 overstatement = stored_bytes / entropy_bytes
291
292 # (e) an omitted entry stays near the 1/q floor at the M < S_tbl/2 budget
294 det = measure_determinacy_degree(E, rng)
295 floor_1_over_q = 1.0 / Q
296 pred_over_floor = pred["predictability_from_candidate_set"] / floor_1_over_q
297
298 out = {
299 "seed": SEED,
300 "q": Q,
301 "num_components": int(E.shape[0]),
302 "encodings_per_component": int(E.shape[1]),
303 "source": "whitebox_encoded.build_encodings (layer measured as this repo builds it)",
304 "measured": {
305 "basis": "computed from the arrays returned by build_encodings() in this run",
306 "storage": {
307 "bytes_per_entry": bytes_per_entry,
308 "entries_per_table": Q,
309 "stored_tables": stored_tables,
310 "forward_tables_nbytes": int(E.nbytes),
311 "inverse_tables_nbytes": int(Einv.nbytes),
312 "stored_bytes": stored_bytes,
313 "stored_MB": round(stored_bytes / 1e6, 3),
314 "stored_log2_bytes": round(math.log2(stored_bytes), 2),
315 },
316 "independence": {
317 "inverse_tables_reconstructed_and_matched": derived_tables,
318 "independent_bijections": independent_tables,
319 "independent_is_half_of_stored": independent_tables * 2
320 == stored_tables,
321 "note": "each inverse was rebuilt from its forward permutation and compared "
322 "entrywise; a derived table carries no key material of its own",
323 },
324 "omitted_entry_predictability": {
325 "retained_entries_per_table": pred["retained_entries_per_table"],
326 "omitted_entries_per_table": pred["omitted_entries_per_table"],
327 "tables_probed": pred["tables_probed"],
328 "guesses_total": pred["guesses_total"],
329 "hits": pred["hits"],
330 "empirical_hit_rate": round(pred["empirical_hit_rate"], 8),
331 "mean_residual_candidate_set_size": round(
332 pred["mean_residual_candidate_set_size"], 2
333 ),
334 "predictability_from_candidate_set": round(
335 pred["predictability_from_candidate_set"], 8
336 ),
337 "ratio_to_1_over_q_floor": round(pred_over_floor, 4),
338 },
339 "entry_determinacy": det,
340 },
341 "analytic": {
342 "basis": "closed form from named parameters; NOT measured",
343 "log2_q_factorial_bits_per_bijection": round(log2_qfact, 4),
344 "entropy_formula": "H = independent_bijections * log2(q!)",
345 "entropy_bits": round(entropy_bits, 2),
346 "entropy_bytes": round(entropy_bytes, 2),
347 "entropy_MB": round(entropy_bytes / 1e6, 3),
348 "entropy_log2_bytes": round(entropy_log2_bytes, 2),
349 "unconditional_floor_1_over_q": round(floor_1_over_q, 8),
350 "bijection_aware_floor_1_over_omitted": round(
351 1.0 / pred["omitted_entries_per_table"], 8
352 ),
353 },
354 "derived_measured_over_analytic": {
355 "basis": "measured stored_bytes divided by analytic entropy_bytes",
356 "overstatement_factor_stored_over_entropy": round(overstatement, 3),
357 },
358 "comparison_to_expected": {
359 "basis": "booleans only; the numbers reported above are this run's, not these targets",
360 "expected_stored_bytes": EXPECTED_STORED_BYTES,
361 "stored_bytes_matches_expected": stored_bytes == EXPECTED_STORED_BYTES,
362 "expected_independent_bijections": EXPECTED_INDEPENDENT,
363 "independent_matches_expected": independent_tables == EXPECTED_INDEPENDENT,
364 "expected_entropy_log2_bytes": EXPECTED_ENTROPY_LOG2_BYTES,
365 "entropy_log2_bytes_matches_expected": abs(
366 entropy_log2_bytes - EXPECTED_ENTROPY_LOG2_BYTES
367 )
368 < 0.01,
369 "expected_overstatement": EXPECTED_OVERSTATEMENT,
370 "overstatement_matches_expected": abs(
371 overstatement - EXPECTED_OVERSTATEMENT
372 )
373 < 0.01,
374 },
375 "interpretation": {
376 "correction_owed": "The encoding sublayer stores the measured stored_MB above but "
377 "carries only entropy_MB of key material; the stored figure is "
378 "storage, NOT secret-bearing material. Describing the stored "
379 "megabytes as entropy overstates the key material by the measured "
380 "overstatement factor. Two causes, both quantified here: the "
381 "inverses are derived (factor 2 exactly, verified by "
382 "reconstruction) and a Z_q permutation spends q uint16 entries to "
383 "carry log2(q!) bits (the residual factor).",
384 "dichotomy_side": "POSITIVE. The layer is entry-unpredictable: with q//2 entries "
385 "retained, an omitted entry stays uniform over the residual "
386 "candidate set, i.e. within the measured ratio_to_1_over_q_floor of "
387 "the 1/q floor, and the table is pinned only at d = q-1 retained "
388 "entries. This is the opposite of an algebraically-determined "
389 "family, whose omitted entries follow from a constant number of "
390 "retained ones.",
391 },
392 }
393
394 root = Path(__file__).parent.parent.parent
395 results_dir = root / "results"
396 results_dir.mkdir(parents=True, exist_ok=True)
397 path = results_dir / "encoding_layer_entropy.json"
398 with open(path, "w", encoding="utf-8", newline="\n") as f:
399 json.dump(out, f, indent=2)
400
401 st = out["measured"]["storage"]
402 an = out["analytic"]
403 cmp_ = out["comparison_to_expected"]
404 print("wrote", path)
405 print("ENCODING LAYER (built by whitebox_encoded.build_encodings, seed %d)" % SEED)
406 print(
407 " stored tables : %d (%d forward + %d inverse), %d entries x %d B"
408 % (st["stored_tables"], stored_tables // 2, derived_tables, Q, bytes_per_entry)
409 )
410 print(
411 " stored bytes [MEASURED] : %d B = %.3f MB = 2^%.2f"
412 % (st["stored_bytes"], st["stored_MB"], st["stored_log2_bytes"])
413 )
414 print(
415 " inverses derived : %d/%d reconstructed and matched"
416 % (derived_tables, stored_tables // 2)
417 )
418 print(
419 " independent bijections : %d (half the stored tables)" % independent_tables
420 )
421 print(
422 " log2(q!) [ANALYTIC] : %.2f bits per bijection"
423 % an["log2_q_factorial_bits_per_bijection"]
424 )
425 print(
426 " key entropy [ANALYTIC] : %.0f bits = %.3f MB = 2^%.2f"
427 % (an["entropy_bits"], an["entropy_MB"], an["entropy_log2_bytes"])
428 )
429 print(
430 " overstatement [DERIVED] : %.3fx (stored / entropy)"
431 % out["derived_measured_over_analytic"][
432 "overstatement_factor_stored_over_entropy"
433 ]
434 )
435 p = out["measured"]["omitted_entry_predictability"]
436 print(
437 "OMITTED-ENTRY PREDICTABILITY at M < S_tbl/2 (%d retained / %d omitted)"
438 % (p["retained_entries_per_table"], p["omitted_entries_per_table"])
439 )
440 print(
441 " empirical hit rate : %.8f (%d hits / %d guesses)"
442 % (p["empirical_hit_rate"], p["hits"], p["guesses_total"])
443 )
444 print(
445 " candidate-set prediction : %.8f (mean set %.2f)"
446 % (
447 p["predictability_from_candidate_set"],
448 p["mean_residual_candidate_set_size"],
449 )
450 )
451 print(
452 " 1/q floor : %.8f -> ratio %.4fx"
453 % (an["unconditional_floor_1_over_q"], p["ratio_to_1_over_q_floor"])
454 )
455 print(
456 " entry-determinacy degree : d = %d (q-1); entry-unpredictable family"
457 % det["entry_determinacy_degree_d"]
458 )
459 print("COMPARISON TO EXPECTED (booleans; reported numbers are this run's)")
460 for k in (
461 "stored_bytes_matches_expected",
462 "independent_matches_expected",
463 "entropy_log2_bytes_matches_expected",
464 "overstatement_matches_expected",
465 ):
466 print(" %-38s: %s" % (k, cmp_[k]))
467
468
469if __name__ == "__main__":
470 main()
assert_bijection(perm, label)
Verify a table is a permutation of Z_q.
main()
Measure the encoding layer as built by whitebox_encoded.build_encodings(), split measured from analyt...
log2_factorial(n)
log2(n!) via lgamma, which stays finite where n! overflows.
measure_determinacy_degree(E, rng)
Entry-determinacy degree of a Z_q bijection, measured by reconstruction.
verify_inverses_derivable(E, Einv)
Reconstruct every inverse from its forward permutation and compare.
measure_omitted_entry_predictability(E, rng)
Predictability of an omitted entry when q//2 entries of a bijection are retained, i....
build_encodings(rng)
Secret nonlinear I/O encodings: two random bijections on Z_q per component plus their inverses.