PQ-WB-KEM: White-Box ML-KEM-768 1.0
PQ-WB-KEM: Toward a White-Box Construction of ML-KEM-768 with Arithmetic Masking for M2M Communications - Mathematics (MDPI)
Loading...
Searching...
No Matches
whitebox_reference.py
Go to the documentation of this file.
1# -*- coding: utf-8 -*-
2"""
3@file whitebox_reference.py
4@brief Reference PQ-WB-KEM base multiplication on the real NTT-domain secret:
5build white-box tables (two constructions), verify correctness against the
6reference base multiply, measure serialized table size and masking overhead.
7@ingroup core
8
9@details The ML-KEM base multiply operates in the NTT domain: the stored secret
10is shat = NTT(s), whose coordinates are full-range over Z_q (a CBD-small s in the
11coefficient domain has a full-range NTT image). Per degree-2 factor
12Z_q[X]/(X^2 - gamma_i), with gamma_i = zeta^(2i+1):
13 p0 = shat0*c0 + gamma_i*shat1*c1, p1 = shat0*c1 + shat1*c0.
14Two lookup-only constructions realise this over full-range operands:
15 A (shared): one q x q multiply table M[x][y] = x*y mod q; the secret enters
16 as a full-range index and the gamma twiddle is a second lookup. Table is
17 secret-independent and shared across all components and all k polynomials.
18 B (baked): per-component q-entry tables with the fixed secret baked in,
19 h0_i[x]=shat0_i*x, h1_i[x]=shat1_i*x, h2_i[x]=gamma_i*shat1_i*x, indexed by
20 one full-range ciphertext coordinate. Smaller for small module rank k.
21Outputs p0,p1 are masked with three arithmetic shares each. Deterministic
22(seed 42); results in results/whitebox_reference.json.
23@note The masking-overhead ratio is wall-clock; wallclock_ci.py aggregates N=20
24 runs (mean +/- 95% CI) per environment for robust reporting.
25@note Reference NumPy/Python measurement on the simulation host, not a deployed
26embedded binary; the deployed footprint is measured (base tables here, plus the
27encoding sublayer and coefficient shuffle), only the latency stays a projection.
28
29@section whitebox_reference_output Output Files
30- `results/whitebox_reference.json` - Reference base multiply, self-test, sizes
31
32@section whitebox_reference_usage Usage
33@code{.sh}
34python whitebox_reference.py
35@endcode
36
37@author Ugur Coruh
38@copyright
39This software is provided for academic and research purposes.
40See LICENSE file for details.
41"""
42import json
43import os
44import time
45import numpy as np
46
47Q = 3329
48N = 256
49K = 3 # ML-KEM-768 module rank
50NUM_COMPONENTS = N // 2 # 128 degree-2 NTT factors
51ZETA = 17 # primitive 256th root of unity mod q
52SEED = 42
53
54
56 """
57 @brief Powers of zeta mod q for the NTT.
58 @return Array of 256 zeta powers.
59 """
60 z = np.zeros(256, dtype=np.int64)
61 z[0] = 1
62 for i in range(1, 256):
63 z[i] = (z[i - 1] * ZETA) % Q
64 return z
65
66
67ZETA_POWERS = _zeta_powers()
68
69
70def _bitrev7(k):
71 """
72 @brief Bit-reversal of a 7-bit index (FIPS-203 NTT twiddle order).
73 @param k Index in [0, 128).
74 @return Bit-reversed 7-bit index.
75 """
76 return int(format(k & 0x7F, "07b")[::-1], 2)
77
78
79def ntt_forward(poly):
80 """
81 @brief Forward NTT: coefficient domain to NTT domain.
82 @param poly Length-256 coefficient vector.
83 @return NTT-domain coefficient vector (full-range over Z_q).
84 """
85 r = poly.copy().astype(np.int64)
86 k, length = 1, 128
87 while length >= 2:
88 for start in range(0, 256, 2 * length):
89 zeta = ZETA_POWERS[_bitrev7(k)]
90 k += 1
91 for j in range(start, start + length):
92 t = (zeta * r[j + length]) % Q
93 r[j + length] = (r[j] - t) % Q
94 r[j] = (r[j] + t) % Q
95 length //= 2
96 return r
97
98
99GAMMAS = np.array(
100 [pow(ZETA, 2 * _bitrev7(i) + 1, Q) for i in range(NUM_COMPONENTS)], dtype=np.int64
101)
102
103
104CBD_RANGE = {(-2) % Q, (-1) % Q, 0, 1, 2} # {-2..2} reduced mod q
105
106ETA = 2 # ML-KEM-768 secret distribution parameter (FIPS 203)
107
108
109def sample_cbd(rng, size, eta=ETA):
110 """
111 @brief Draw from the centered binomial distribution CBD_eta of FIPS 203.
112 @param rng Seeded numpy Generator.
113 @param size Number of coefficients.
114 @param eta Distribution parameter; 2 for the ML-KEM-768 secret.
115 @return Integer array over {-eta..eta}, distributed as sum(a_i) - sum(b_i).
116 @note Not uniform over the support: CBD(2) puts 6/16 on 0, 4/16 on each of +/-1 and
117 1/16 on each of +/-2. Sampling uniformly over {-2..2} would share the support but not
118 the distribution, and the manuscript states the secret is drawn from the CBD.
119 """
120 a = rng.integers(0, 2, size=(size, eta)).sum(axis=1)
121 b = rng.integers(0, 2, size=(size, eta)).sum(axis=1)
122 return a - b
123
124
125def cbd_range_occupancy(shat_polys):
126 """
127 @brief Count NTT-domain secret coordinates landing in the coefficient-domain
128 CBD range {-2..2} mod q.
129 @param shat_polys List of NTT-domain secret polynomials.
130 @return (coordinates, in_range_count).
131 """
132 coords = np.concatenate([h for h in shat_polys])
133 return int(coords.size), sum(1 for v in coords if int(v) in CBD_RANGE)
134
135
137 """
138 @brief Occupancy rate over many independent secrets, so the reference-sample
139 count is not mistaken for the underlying rate.
140 @param n_seeds Number of independent secrets sampled.
141 @return (coordinates, in_range_count) totals across all seeds.
142 """
143 tot = hit = 0
144 for sd in range(n_seeds):
145 rng = np.random.default_rng(sd)
146 shat = [ntt_forward(sample_cbd(rng, N) % Q) % Q for _ in range(K)]
147 c, h = cbd_range_occupancy(shat)
148 tot += c
149 hit += h
150 return tot, hit
151
152
153def base_mul_direct(sh, uh, i):
154 """
155 @brief Reference base multiply for degree-2 factor i (direct arithmetic).
156 @param sh NTT-domain secret polynomial (length 256, full-range).
157 @param uh NTT-domain ciphertext polynomial (length 256).
158 @param i Component index in [0, 128).
159 @return (p0, p1) product coordinates mod q.
160 """
161 s0, s1 = int(sh[2 * i]), int(sh[2 * i + 1])
162 c0, c1 = int(uh[2 * i]), int(uh[2 * i + 1])
163 g = int(GAMMAS[i])
164 return (s0 * c0 + g * s1 * c1) % Q, (s0 * c1 + s1 * c0) % Q
165
166
167def _negacyclic(f, g):
168 """
169 @brief Schoolbook multiply in Z_q[X]/(X^256+1); independent correctness oracle.
170 @param f First length-256 coefficient vector.
171 @param g Second length-256 coefficient vector.
172 @return Length-256 product coefficient vector.
173 """
174 h = np.zeros(N, dtype=np.int64)
175 for a in range(N):
176 fa = int(f[a])
177 if not fa:
178 continue
179 for b in range(N):
180 c = a + b
181 if c < N:
182 h[c] = (h[c] + fa * int(g[b])) % Q
183 else:
184 h[c - N] = (h[c - N] - fa * int(g[b])) % Q
185 return h
186
187
188def _poly_mod_x2(f, gamma):
189 """
190 @brief Reduce a length-256 polynomial modulo (X^2 - gamma) over Z_q.
191 @param f Coefficient vector.
192 @param gamma Component twiddle constant.
193 @return (a0, a1) reduced coordinates.
194 """
195 a0 = a1 = 0
196 gm = 1
197 for m in range(NUM_COMPONENTS):
198 a0 = (a0 + int(f[2 * m]) * gm) % Q
199 a1 = (a1 + int(f[2 * m + 1]) * gm) % Q
200 gm = (gm * gamma) % Q
201 return a0, a1
202
203
205 """
206 @brief Construction A table: one shared full multiply table M[x][y]=x*y mod q.
207 @return uint16 array of shape (q, q); secret-independent.
208 """
209 x = np.arange(Q, dtype=np.int64)
210 return (np.multiply.outer(x, x) % Q).astype(np.uint16)
211
212
213def base_mul_shared(M, sh, uh, i):
214 """
215 @brief Construction A base multiply via shared-table lookups (full-range secret).
216 @param M Shared multiply table.
217 @param sh NTT-domain secret; @param uh NTT-domain ciphertext; @param i Component.
218 @return (p0, p1) mod q.
219 """
220 s0, s1 = int(sh[2 * i]), int(sh[2 * i + 1])
221 c0, c1 = int(uh[2 * i]), int(uh[2 * i + 1])
222 p0 = (int(M[s0, c0]) + int(M[int(GAMMAS[i]), int(M[s1, c1])])) % Q
223 p1 = (int(M[s0, c1]) + int(M[s1, c0])) % Q
224 return p0, p1
225
226
227def build_baked_tables(sh):
228 """
229 @brief Construction B tables for one secret polynomial: per-component q-entry
230 tables with the fixed secret baked in.
231 @param sh NTT-domain secret polynomial (full-range).
232 @return list of 128 (h0, h1, h2) uint16 triples.
233 """
234 x = np.arange(Q, dtype=np.int64)
235 tabs = []
236 for i in range(NUM_COMPONENTS):
237 s0, s1 = int(sh[2 * i]), int(sh[2 * i + 1])
238 h0 = ((s0 * x) % Q).astype(np.uint16)
239 h1 = ((s1 * x) % Q).astype(np.uint16)
240 h2 = ((int(GAMMAS[i]) * s1 * x) % Q).astype(np.uint16)
241 tabs.append((h0, h1, h2))
242 return tabs
243
244
245def base_mul_baked(tabs, uh, i):
246 """
247 @brief Construction B base multiply via baked per-component lookups.
248 @param tabs Baked tables for the secret polynomial; @param uh Ciphertext; @param i Component.
249 @return (p0, p1) mod q.
250 """
251 c0, c1 = int(uh[2 * i]), int(uh[2 * i + 1])
252 h0, h1, h2 = tabs[i]
253 p0 = (int(h0[c0]) + int(h2[c1])) % Q
254 p1 = (int(h0[c1]) + int(h1[c0])) % Q
255 return p0, p1
256
257
258def three_share(v, rng):
259 """
260 @brief Split a value into three arithmetic shares mod q.
261 @param v Value; @param rng NumPy Generator.
262 @return (a, b, c) with a+b+c == v mod q.
263 """
264 a = int(rng.integers(0, Q))
265 b = int(rng.integers(0, Q))
266 return a, b, (v - a - b) % Q
267
268
269def measure_overhead(iters=100000, seed=SEED):
270 """
271 @brief Construction-B protected-primitive overhead (baked table lookup plus
272 3-share masking versus direct arithmetic) on the real full-range
273 NTT-domain secret, for wall-clock CI aggregation (wallclock_ci.py).
274 @param iters Number of base multiplications timed per variant.
275 @param seed Deterministic seed.
276 @return Overhead ratio (protected / direct).
277 """
278 rng = np.random.default_rng(seed)
279 shat = ntt_forward(sample_cbd(rng, N) % Q) % Q
280 baked = build_baked_tables(shat)
281 idx = rng.integers(0, NUM_COMPONENTS, size=iters)
282 uu = rng.integers(0, Q, size=(iters, N))
283 t0 = time.perf_counter()
284 for kk in range(iters):
285 base_mul_direct(shat, uu[kk], int(idx[kk]))
286 t_direct = time.perf_counter() - t0
287 t0 = time.perf_counter()
288 for kk in range(iters):
289 pt = base_mul_baked(baked, uu[kk], int(idx[kk]))
290 r1 = (kk * 40503) % Q
291 r2 = (kk * 2654435761) % Q
292 _ = (r1, r2, (pt[0] - r1 - r2) % Q)
293 r3 = (kk * 40503 + 7) % Q
294 r4 = (kk * 2654435761 + 3) % Q
295 _ = (r3, r4, (pt[1] - r3 - r4) % Q)
296 t_b = time.perf_counter() - t0
297 return t_b / t_direct
298
299
300def main():
301 """
302 @brief Build both constructions on the real NTT-domain secret, check
303 correctness, measure sizes and masking overhead, write JSON.
304 """
305 rng = np.random.default_rng(SEED)
306
307 # Real ML-KEM-768 secret: k CBD polynomials, stored in NTT domain (full-range)
308 s_polys = [sample_cbd(rng, N) % Q for _ in range(K)]
309 shat = [ntt_forward(s) % Q for s in s_polys]
310 shat_lo = min(int(h.min()) for h in shat)
311 shat_hi = max(int(h.max()) for h in shat)
312
313 # Occupancy of the five-value CBD range: the sampled rate, not an asserted zero.
314 ref_coords, ref_hits = cbd_range_occupancy(shat)
315 ms_seeds = 200
316 ms_coords, ms_hits = measure_cbd_range_occupancy(ms_seeds)
317
319 baked = [build_baked_tables(shat[j]) for j in range(K)]
320
321 # Correctness: both constructions vs the reference base multiply, on the real
322 # full-range secret, plus the masked recombination, over random ciphertexts.
323 trials = 5000
324 okA = okB = True
325 for _ in range(trials):
326 j = int(rng.integers(0, K))
327 i = int(rng.integers(0, NUM_COMPONENTS))
328 uh = rng.integers(0, Q, size=N)
329 want = base_mul_direct(shat[j], uh, i)
330 a = base_mul_shared(M, shat[j], uh, i)
331 b = base_mul_baked(baked[j], uh, i)
332 p0 = three_share(want[0], rng)
333 p1 = three_share(want[1], rng)
334 if a != want or b != want or (sum(p0) % Q, sum(p1) % Q) != want:
335 okA = a == want
336 okB = b == want
337 break
338 assert okA and okB, "CORRECTNESS FAILED on full-range NTT-domain secret"
339
340 # End-to-end correctness against an independent oracle: the NTT-domain base
341 # multiply must reproduce the schoolbook negacyclic product in Z_q[X]/(X^256+1),
342 # reduced per degree-2 factor. This certifies a valid ML-KEM transform rather
343 # than re-expressing the same bilinear form two ways.
344 s_chk = rng.integers(0, Q, size=N)
345 c_chk = rng.integers(0, Q, size=N)
346 ns, nc = ntt_forward(s_chk) % Q, ntt_forward(c_chk) % Q
347 h_true = _negacyclic(s_chk, c_chk)
348 vec_ok = all(
349 base_mul_direct(ns, nc, i) == _poly_mod_x2(h_true, int(GAMMAS[i]))
350 for i in range(NUM_COMPONENTS)
351 )
352 assert vec_ok, "END-TO-END NTT correctness failed vs schoolbook negacyclic product"
353
354 # Footprints (measured, serialized)
355 sizeA = M.astype(np.uint16).tobytes().__len__()
356 per_comp_baked = NUM_COMPONENTS * 3 * Q * 2
357 sizeB_total = per_comp_baked * K
358
359 # Overhead: masked table path vs direct arithmetic (both constructions)
360 iters = 100000
361 idx = rng.integers(0, NUM_COMPONENTS, size=iters)
362 uu = rng.integers(0, Q, size=(iters, N))
363 sh0 = shat[0]
364
365 t0 = time.perf_counter()
366 for kk in range(iters):
367 base_mul_direct(sh0, uu[kk], int(idx[kk]))
368 t_direct = time.perf_counter() - t0
369
370 t0 = time.perf_counter()
371 for kk in range(iters):
372 pt = base_mul_shared(M, sh0, uu[kk], int(idx[kk]))
373 r1 = (kk * 40503) % Q
374 r2 = (kk * 2654435761) % Q
375 _ = (r1, r2, (pt[0] - r1 - r2) % Q)
376 r3 = (kk * 40503 + 7) % Q
377 r4 = (kk * 2654435761 + 3) % Q
378 _ = (r3, r4, (pt[1] - r3 - r4) % Q)
379 t_A = time.perf_counter() - t0
380
381 bk0 = baked[0]
382 t0 = time.perf_counter()
383 for kk in range(iters):
384 pt = base_mul_baked(bk0, uu[kk], int(idx[kk]))
385 r1 = (kk * 40503) % Q
386 r2 = (kk * 2654435761) % Q
387 _ = (r1, r2, (pt[0] - r1 - r2) % Q)
388 r3 = (kk * 40503 + 7) % Q
389 r4 = (kk * 2654435761 + 3) % Q
390 _ = (r3, r4, (pt[1] - r3 - r4) % Q)
391 t_B = time.perf_counter() - t0
392
393 out = {
394 "seed": SEED,
395 "q": Q,
396 "n": N,
397 "k": K,
398 "num_components": NUM_COMPONENTS,
399 "secret_domain": {
400 "coefficient_domain": "CBD, s in {-2..2}",
401 "ntt_domain_operand": "shat = NTT(s), full-range over Z_q",
402 "shat_range": [shat_lo, shat_hi],
403 "cbd_range_occupancy": {
404 "basis": "measured",
405 "definition": "fraction of NTT-domain secret coordinates falling in the "
406 "five-value coefficient-domain range {-2..2} reduced mod q",
407 "reference_sample": {
408 "seed": SEED,
409 "k": K,
410 "coordinates": ref_coords,
411 "in_cbd_range": ref_hits,
412 "percent": round(100.0 * ref_hits / ref_coords, 4),
413 },
414 "multi_seed": {
415 "seeds": ms_seeds,
416 "coordinates": ms_coords,
417 "in_cbd_range": ms_hits,
418 "percent": round(100.0 * ms_hits / ms_coords, 4),
419 },
420 "uniform_prediction_percent": round(100.0 * len(CBD_RANGE) / Q, 4),
421 "note": "The rate is NOT zero, though this reference sample happens to contain "
422 "none -- which is exactly why one sample cannot settle it. The measured rate tracks the 5/q "
423 "rate expected of a coordinate ranging over all of Z_q, which is the "
424 "point: the NTT image inherits no coefficient-domain smallness. A rate "
425 "of exactly 0 on a finite sample is a property of that sample, not of "
426 "the distribution.",
427 },
428 "note": "The base multiply operand is the full-range NTT-domain secret; "
429 "coefficient-domain smallness does not carry into the NTT domain.",
430 },
431 "correctness_trials": trials,
432 "correctness_passed": bool(okA and okB),
433 "wholevector_crosscheck_passed": bool(vec_ok),
434 "constructions": {
435 "A_shared_full_table": {
436 "description": "One shared q x q multiply table; secret enters as a "
437 "full-range index; secret-independent, shared across all "
438 "components and all k polynomials.",
439 "total_bytes": sizeA,
440 "total_MB": round(sizeA / 1e6, 2),
441 "overhead_factor": round(t_A / t_direct, 2),
442 "overhead_factor_machine_dependent": True,
443 "basis": "measured (table built and serialized; overhead on simulation host)",
444 },
445 "B_baked_per_component": {
446 "description": "Per-component q-entry tables with the fixed secret baked "
447 "in (h0,h1,h2), indexed by one full-range ciphertext "
448 "coordinate; scales with module rank k.",
449 "per_component_bytes": NUM_COMPONENTS * 3 * Q * 2 // NUM_COMPONENTS,
450 "per_secret_poly_bytes": per_comp_baked,
451 "per_secret_poly_MB": round(per_comp_baked / 1e6, 2),
452 "total_bytes": sizeB_total,
453 "total_MB": round(sizeB_total / 1e6, 2),
454 "overhead_factor": round(t_B / t_direct, 2),
455 "overhead_factor_machine_dependent": True,
456 "basis": "measured (tables built and serialized; overhead on simulation host)",
457 },
458 },
459 "determinism": {
460 "footprints_deterministic": True,
461 "machine_dependent_fields": [
462 "constructions.A_shared_full_table.overhead_factor",
463 "constructions.B_baked_per_component.overhead_factor",
464 ],
465 "note": "All size/footprint fields are seeded (seed 42) and byte-identical "
466 "run-to-run; the overhead_factor fields are wall-clock ratios measured "
467 "on the simulation host and vary run-to-run. run_all.py flags this "
468 "experiment machine_dependent=False for its deterministic footprints; "
469 "the byte-identical claim excludes the two overhead_factor timings.",
470 },
471 "note": "Reference NumPy/Python measurement on the simulation host. Both "
472 "constructions operate on the real full-range NTT-domain secret and are "
473 "verified against the reference base multiply. These are the measured base "
474 "tables; the full deployed footprint is likewise measured (encoding sublayer "
475 "in whitebox_encoded.py, coefficient shuffle in obfuscation_storage.py), and "
476 "only the deployment latency remains an analytical projection.",
477 }
478 script_dir = os.path.dirname(os.path.abspath(__file__))
479 results_dir = os.path.join(os.path.dirname(os.path.dirname(script_dir)), "results")
480 os.makedirs(results_dir, exist_ok=True)
481 with open(
482 os.path.join(results_dir, "whitebox_reference.json"),
483 "w",
484 encoding="utf-8",
485 newline="\n",
486 ) as f:
487 json.dump(out, f, indent=2)
488
489 print(
490 "CORRECTNESS: PASSED (%d trials, both constructions, full-range secret)"
491 % trials
492 )
493 print("shat range: [%d, %d] (full-range NTT-domain secret)" % (shat_lo, shat_hi))
494 print(
495 "CBD-range occupancy: %d/%d = %.4f%% (reference sample), %.4f%% over %d seeds, 5/q = %.4f%%"
496 % (
497 ref_hits,
498 ref_coords,
499 100.0 * ref_hits / ref_coords,
500 100.0 * ms_hits / ms_coords,
501 ms_seeds,
502 100.0 * len(CBD_RANGE) / Q,
503 )
504 )
505 print(
506 "A shared full table : %.2f MB, overhead %.2fx"
507 % (out["constructions"]["A_shared_full_table"]["total_MB"], t_A / t_direct)
508 )
509 print(
510 "B baked per-comp : %.2f MB (k=%d), overhead %.2fx"
511 % (out["constructions"]["B_baked_per_component"]["total_MB"], K, t_B / t_direct)
512 )
513
514
515if __name__ == "__main__":
516 main()
_bitrev7(k)
Bit-reversal of a 7-bit index (FIPS-203 NTT twiddle order).
measure_overhead(iters=100000, seed=SEED)
Construction-B protected-primitive overhead (baked table lookup plus 3-share masking versus direct ar...
build_shared_table()
Construction A table: one shared full multiply table M[x][y]=x*y mod q.
_negacyclic(f, g)
Schoolbook multiply in Z_q[X]/(X^256+1); independent correctness oracle.
_poly_mod_x2(f, gamma)
Reduce a length-256 polynomial modulo (X^2 - gamma) over Z_q.
_zeta_powers()
Powers of zeta mod q for the NTT.
main()
Build both constructions on the real NTT-domain secret, check correctness, measure sizes and masking ...
sample_cbd(rng, size, eta=ETA)
Draw from the centered binomial distribution CBD_eta of FIPS 203.
measure_cbd_range_occupancy(n_seeds=200)
Occupancy rate over many independent secrets, so the reference-sample count is not mistaken for the u...
three_share(v, rng)
Split a value into three arithmetic shares mod q.
cbd_range_occupancy(shat_polys)
Count NTT-domain secret coordinates landing in the coefficient-domain CBD range {-2....