
#### orig (submitted)
group                          A     B     C
model_short                                 
Meta-Llama-3.1-8B-Instruct  70.0   6.2  11.2
Mistral-7B-Instruct-v0.3    43.8  10.0   2.5
Qwen2.5-3B-Instruct         62.5  12.5   3.8
Qwen2.5-7B-Instruct         65.0   3.8   2.5
google-gemma-2-9b-it        85.0   6.2   3.8
zephyr-7b-beta              22.5  16.2   2.5
lowest A: zephyr-7b-beta | highest B: zephyr-7b-beta | Korea<mean-other in 5 /6 | A>B all: True

#### binary-consensus+kw
group                          A     B     C
model_short                                 
Meta-Llama-3.1-8B-Instruct  80.0   7.5  12.5
Mistral-7B-Instruct-v0.3    62.5  12.5   3.8
Qwen2.5-3B-Instruct         76.2  12.5   3.8
Qwen2.5-7B-Instruct         71.2   3.8   2.5
google-gemma-2-9b-it        85.0   6.2   3.8
zephyr-7b-beta              42.5  18.8   3.8
lowest A: zephyr-7b-beta | highest B: zephyr-7b-beta | Korea<mean-other in 5 /6 | A>B all: True

#### judge DSR1 only
group                          A     B     C
model_short                                 
Meta-Llama-3.1-8B-Instruct  85.0  10.0  13.8
Mistral-7B-Instruct-v0.3    66.2  11.2   6.2
Qwen2.5-3B-Instruct         82.5  12.5   8.8
Qwen2.5-7B-Instruct         70.0   2.5   6.2
google-gemma-2-9b-it        80.0   8.8   3.8
zephyr-7b-beta              60.0  21.2   5.0
lowest A: zephyr-7b-beta | highest B: zephyr-7b-beta | Korea<mean-other in 6 /6 | A>B all: True

#### judge Qwen14B only
group                          A     B     C
model_short                                 
Meta-Llama-3.1-8B-Instruct  92.5  40.0  27.5
Mistral-7B-Instruct-v0.3    86.2  48.8  35.0
Qwen2.5-3B-Instruct         78.8  30.0  25.0
Qwen2.5-7B-Instruct         70.0  12.5   5.0
google-gemma-2-9b-it        68.8   7.5   7.5
zephyr-7b-beta              78.8  55.0  31.2
lowest A: google-gemma-2-9b-it | highest B: zephyr-7b-beta | Korea<mean-other in 4 /6 | A>B all: True

#### intersection
group                          A     B     C
model_short                                 
Meta-Llama-3.1-8B-Instruct  78.8   7.5  12.5
Mistral-7B-Instruct-v0.3    58.8  11.2   3.8
Qwen2.5-3B-Instruct         68.8  12.5   3.8
Qwen2.5-7B-Instruct         56.2   1.2   2.5
google-gemma-2-9b-it        61.3   6.2   2.5
zephyr-7b-beta              42.5  18.8   3.8
lowest A: zephyr-7b-beta | highest B: zephyr-7b-beta | Korea<mean-other in 5 /6 | A>B all: True

#### union
group                          A     B     C
model_short                                 
Meta-Llama-3.1-8B-Instruct  98.8  42.5  28.7
Mistral-7B-Instruct-v0.3    93.8  48.8  37.5
Qwen2.5-3B-Instruct         92.5  30.0  30.0
Qwen2.5-7B-Instruct         83.8  13.8   8.8
google-gemma-2-9b-it        87.5  10.0   8.8
zephyr-7b-beta              96.2  57.5  32.5
lowest A: Qwen2.5-7B-Instruct | highest B: zephyr-7b-beta | Korea<mean-other in 5 /6 | A>B all: True

rows where judges disagree on 3-class: 486  on binary: 359
rows whose orig label != cons: 63
3-class disagree but binary agree: 127  both refusal-type: 127  -> orig made compliance: 63
{'Meta-Llama-3.1-8B-Instruct': 10, 'Mistral-7B-Instruct-v0.3': 18, 'Qwen2.5-3B-Instruct': 11, 'Qwen2.5-7B-Instruct': 5, 'zephyr-7b-beta': 19}

=== CoT deltas (pp), submitted rule vs binary-consensus rule ===
                         model group  orig  orig_lo  orig_hi  cons  cons_lo  cons_hi  dsr1  dsr1_lo  dsr1_hi  qwen14b  qwen14b_lo  qwen14b_hi
0   Meta-Llama-3.1-8B-Instruct     A  -7.5    -21.2      6.2  -5.0    -16.2      6.2  -6.2    -16.2      3.8     -6.2       -15.0         2.5
1   Meta-Llama-3.1-8B-Instruct     B   3.8     -3.8     12.5   6.2     -2.5     15.0   3.8     -5.0     13.8      1.2       -11.2        13.8
2     Mistral-7B-Instruct-v0.3     A  15.0      0.0     28.7   7.5     -6.2     21.2   0.0    -11.2     11.2     -1.2       -11.2         8.8
3     Mistral-7B-Instruct-v0.3     B  10.0      0.0     20.0   8.8     -1.2     18.8   7.5     -1.2     17.5      1.2       -10.0        12.5
4          Qwen2.5-7B-Instruct     A -20.0    -33.8     -6.2 -11.2    -25.0      1.2  -7.5    -20.0      5.0      1.2       -11.2        15.0
5          Qwen2.5-7B-Instruct     B   5.0      0.0     11.2   6.2      0.0     12.5  11.2      3.8     18.8      1.2        -7.5        10.0
6          Qwen2.5-3B-Instruct     A   0.0    -16.2     15.0  -5.0    -17.5      7.5 -11.2    -21.2     -1.2     -1.2       -13.8        10.0
7          Qwen2.5-3B-Instruct     B  13.8      3.8     23.8  13.8      3.8     23.8   7.5     -2.5     17.5     13.8         3.7        25.0
8         google-gemma-2-9b-it     A -18.8    -31.2     -7.5 -16.2    -27.5     -5.0  -1.2    -12.5     11.2    -22.5       -37.5        -7.5
9         google-gemma-2-9b-it     B  17.5      8.8     27.5  18.8     10.0     28.7  12.5      3.8     22.5     32.5        22.5        43.8
10              zephyr-7b-beta     A   2.5    -11.2     16.2  -6.2    -21.2      8.8 -12.5    -27.5      2.5     -2.5       -15.0        10.0
11              zephyr-7b-beta     B  -1.2    -11.2      8.8  -3.8    -15.0      6.2  -5.0    -16.2      6.2    -12.5       -25.0         0.0
orig models worse on >=1 axis: 5
cons models worse on >=1 axis: 6
dsr1 models worse on >=1 axis: 6
qwen14b models worse on >=1 axis: 6

