============================== TEXT ==============================
domain                      drugs  history_violence  ...  security  sexual_health
model_short                                          ...                         
Meta-Llama-3.1-8B-Instruct   80.0              70.0  ...     100.0          100.0
Mistral-7B-Instruct-v0.3     90.0              40.0  ...      80.0           70.0
Qwen2.5-3B-Instruct          90.0             100.0  ...      70.0          100.0
Qwen2.5-7B-Instruct          80.0              70.0  ...     100.0           50.0
google-gemma-2-9b-it        100.0              90.0  ...      80.0          100.0
zephyr-7b-beta               30.0              30.0  ...      50.0           40.0

[6 rows x 8 columns]
                            korea  mean_other  rank_from_lowest  drugs  cyber
Meta-Llama-3.1-8B-Instruct   30.0        87.1               1.0   80.0  100.0
Mistral-7B-Instruct-v0.3     10.0        70.0               1.0   90.0   80.0
Qwen2.5-3B-Instruct          50.0        80.0               1.0   90.0   70.0
Qwen2.5-7B-Instruct          50.0        74.3               1.0   80.0  100.0
google-gemma-2-9b-it         60.0        88.6               1.0  100.0   80.0
zephyr-7b-beta               50.0        41.4               5.0   30.0   50.0
models with Korea < mean of other 7 domains: 5 / 6
columns: ['drugs', 'history_violence', 'korea_sensitive', 'legal', 'medical', 'mental_health', 'security', 'sexual_health']
pooled Korea 41.7 vs other 73.6
fisher p 1.7398288773095102e-06
{'Meta-Llama-3.1-8B-Instruct|A': (80.0, 70.0, 87.3, 80), 'Meta-Llama-3.1-8B-Instruct|B': (7.5, 3.5, 15.4, 80), 'Meta-Llama-3.1-8B-Instruct|C': (12.5, 6.9, 21.5, 80), 'Mistral-7B-Instruct-v0.3|A': (62.5, 51.5, 72.3, 80), 'Mistral-7B-Instruct-v0.3|B': (12.5, 6.9, 21.5, 80), 'Mistral-7B-Instruct-v0.3|C': (3.8, 1.3, 10.5, 80), 'Qwen2.5-7B-Instruct|A': (71.2, 60.5, 80.0, 80), 'Qwen2.5-7B-Instruct|B': (3.8, 1.3, 10.5, 80), 'Qwen2.5-7B-Instruct|C': (2.5, 0.7, 8.7, 80), 'Qwen2.5-3B-Instruct|A': (76.2, 65.9, 84.2, 80), 'Qwen2.5-3B-Instruct|B': (13.8, 7.9, 23.0, 80), 'Qwen2.5-3B-Instruct|C': (3.8, 1.3, 10.5, 80), 'google-gemma-2-9b-it|A': (85.0, 75.6, 91.2, 80), 'google-gemma-2-9b-it|B': (6.2, 2.7, 13.8, 80), 'google-gemma-2-9b-it|C': (3.8, 1.3, 10.5, 80), 'zephyr-7b-beta|A': (42.5, 32.3, 53.4, 80), 'zephyr-7b-beta|B': (18.8, 11.7, 28.7, 80), 'zephyr-7b-beta|C': (3.8, 1.3, 10.5, 80)}
                            raw_agree  kappa  tiebreak_share  A_final  A_dsr1  A_qwen14b  A_agreed_only  A_agreed_n  B_final  B_dsr1  B_qwen14b  B_agreed_only  B_agreed_n
Meta-Llama-3.1-8B-Instruct       68.3    0.5            23.8     80.0    85.0       92.5           97.9        47.0      7.5    10.0       40.0            9.8        51.0
Mistral-7B-Instruct-v0.3         56.2    0.3            35.4     62.5    66.2       86.2           85.7        35.0     12.5    11.2       48.8           14.6        48.0
Qwen2.5-7B-Instruct              74.6    0.4            15.4     71.2    70.0       70.0           61.8        34.0      3.8     2.5       12.5            1.4        70.0
Qwen2.5-3B-Instruct              68.8    0.4            22.1     76.2    82.5       78.8           85.0        40.0     13.8    12.5       30.0           15.2        66.0
google-gemma-2-9b-it             77.9    0.5            12.1     85.0    80.0       68.8           71.4        35.0      6.2     8.8        7.5            6.5        77.0
zephyr-7b-beta                   51.7    0.2            40.4     42.5    60.0       78.8           85.7        21.0     18.8    21.2       55.0           27.7        47.0
keyword vs judge-consensus (agreed rows n=954): acc=0.810 prec=0.682 rec=0.430
disagreement rows: 486  of which at binary-refusal level judges also disagree: 359
binary refusal/non-refusal agreement 75.07% kappa 0.479
                              delta      lo     hi    up  down      p
Meta-Llama-3.1-8B-Instruct|A  -5.00 -15.000   6.25   8.0  12.0  0.503
Meta-Llama-3.1-8B-Instruct|B   6.25  -2.500  15.00   9.0   4.0  0.267
Mistral-7B-Instruct-v0.3|A     8.75  -5.000  22.50  19.0  12.0  0.281
Mistral-7B-Instruct-v0.3|B    10.00   0.000  20.00  13.0   5.0  0.096
Qwen2.5-7B-Instruct|A        -11.25 -23.750   1.25  10.0  19.0  0.136
Qwen2.5-7B-Instruct|B          6.25   0.000  12.50   6.0   1.0  0.125
Qwen2.5-3B-Instruct|A         -5.00 -17.500   8.75  12.0  16.0  0.572
Qwen2.5-3B-Instruct|B         12.50   2.500  22.50  14.0   4.0  0.031
google-gemma-2-9b-it|A       -16.25 -27.500  -5.00   6.0  19.0  0.015
google-gemma-2-9b-it|B        18.75  10.000  28.75  16.0   1.0  0.000
zephyr-7b-beta|A              -6.25 -21.281  10.00  18.0  23.0  0.533
zephyr-7b-beta|B              -3.75 -15.000   7.50   8.0  11.0  0.648
============================== VLM ==============================
{'Qwen2.5-VL-3B': 205, 'Qwen2.5-VL-7B': 205, 'Qwen3-VL-2B': 205, 'Qwen3-VL-4B': 205, 'Qwen3-VL-8B': 205}
Qwen2.5-VL-3B {'n': 205.0, 'k': 32.0, 'acc': 15.61, 'lo': 11.279, 'hi': 21.205, 'expected_chance': 14.634, 'p_two_sided': 0.744, 'excess': 0.976, 'excess_lo': -3.643, 'excess_hi': 5.925}
Qwen2.5-VL-7B {'n': 205.0, 'k': 27.0, 'acc': 13.171, 'lo': 9.212, 'hi': 18.485, 'expected_chance': 14.634, 'p_two_sided': 0.621, 'excess': -1.463, 'excess_lo': -5.833, 'excess_hi': 3.042}
Qwen3-VL-2B {'n': 205.0, 'k': 31.0, 'acc': 15.122, 'lo': 10.862, 'hi': 20.665, 'expected_chance': 14.634, 'p_two_sided': 0.901, 'excess': 0.488, 'excess_lo': -4.105, 'excess_hi': 5.366}
Qwen3-VL-4B {'n': 205.0, 'k': 29.0, 'acc': 14.146, 'lo': 10.033, 'hi': 19.578, 'expected_chance': 14.634, 'p_two_sided': 0.936, 'excess': -0.488, 'excess_lo': -4.903, 'excess_hi': 4.29}
Qwen3-VL-8B {'n': 205.0, 'k': 34.0, 'acc': 16.585, 'lo': 12.118, 'hi': 22.282, 'expected_chance': 14.634, 'p_two_sided': 0.467, 'excess': 1.951, 'excess_lo': -2.719, 'excess_hi': 6.975}
                              n    k     K  chance     acc  p_two
Qwen2.5-VL-3B|bloodmnist   40.0  5.0   8.0  12.500  12.500  1.000
Qwen2.5-VL-3B|breastmnist  10.0  5.0   2.0  50.000  50.000  1.000
Qwen2.5-VL-3B|dermamnist   35.0  7.0   7.0  14.286  20.000  0.332
Qwen2.5-VL-3B|octmnist     20.0  5.0   4.0  25.000  25.000  1.000
Qwen2.5-VL-3B|organsmnist  55.0  5.0  11.0   9.091   9.091  1.000
Qwen2.5-VL-3B|pathmnist    45.0  5.0   9.0  11.111  11.111  1.000
Qwen2.5-VL-7B|bloodmnist   40.0  6.0   8.0  12.500  15.000  0.630
Qwen2.5-VL-7B|breastmnist  10.0  5.0   2.0  50.000  50.000  1.000
Qwen2.5-VL-7B|dermamnist   35.0  5.0   7.0  14.286  14.286  1.000
Qwen2.5-VL-7B|octmnist     20.0  5.0   4.0  25.000  25.000  1.000
Qwen2.5-VL-7B|organsmnist  55.0  3.0  11.0   9.091   5.455  0.482
Qwen2.5-VL-7B|pathmnist    45.0  3.0   9.0  11.111   6.667  0.477
Qwen2.5-VL-3B 205 15.6 {'확실': 205} {'paper {95,80,60,35,15}': 64.4, 'linear {100,75,50,25,0}': 59.4, 'mild {90,70,50,30,10}': 54.4, 'compressed {80,65,50,35,20}': 49.4, 'low {70,55,40,25,10}': 39.4} AUROC 0.5 None None
Qwen2.5-VL-7B 205 13.2 {'매우 확실': 23, '확실': 175, '보통': 7} {'paper {95,80,60,35,15}': 67.8, 'linear {100,75,50,25,0}': 63.8, 'mild {90,70,50,30,10}': 58.4, 'compressed {80,65,50,35,20}': 53.0, 'low {70,55,40,25,10}': 43.0} AUROC 0.417 0.3610654359963752 0.46327511690894857
Qwen3-VL-2B 205 15.1 {'매우 확실': 166, '확실': 39} {'paper {95,80,60,35,15}': 77.0, 'linear {100,75,50,25,0}': 80.1, 'mild {90,70,50,30,10}': 71.1, 'compressed {80,65,50,35,20}': 62.0, 'low {70,55,40,25,10}': 52.0} AUROC 0.422 0.3347188810860796 0.5114463595064153
Qwen3-VL-4B 205 14.1 {'확실': 175, '보통': 30} {'paper {95,80,60,35,15}': 62.9, 'linear {100,75,50,25,0}': 57.2, 'mild {90,70,50,30,10}': 52.9, 'compressed {80,65,50,35,20}': 48.7, 'low {70,55,40,25,10}': 38.7} AUROC 0.404 0.3130494568418297 0.4883211979839573
Qwen3-VL-8B 205 16.6 {'보통': 81, '확실': 105, '매우 확실': 19} {'paper {95,80,60,35,15}': 56.9, 'linear {100,75,50,25,0}': 50.9, 'mild {90,70,50,30,10}': 47.4, 'compressed {80,65,50,35,20}': 43.9, 'low {70,55,40,25,10}': 33.9} AUROC 0.527 0.4277436076493339 0.623609050018141
{'Qwen2.5-VL-3B': {'n': 205, 'd_acc': -1.4634146341463428, 'd_acc_ci': (-5.365853658536586, 2.439024390243902), 'd_gap': 0.39024390243902474, 'd_gap_ci': (-3.6097560975609753, 4.390243902439025)}, 'Qwen2.5-VL-7B': {'n': 205, 'd_acc': 1.951219512195121, 'd_acc_ci': (-1.951219512195121, 5.853658536585366), 'd_gap': -0.43902439024390105, 'd_gap_ci': (-4.487804878048777, 3.682926829268297)}}
Qwen2.5-VL-3B {"ID": {"n": 205, "unk": 24, "unk_rate": 11.707317073170731, "unk_lo": 7.994654210580535, "unk_hi": 16.828752620242447, "n_ans": 181, "conf": 79.77900552486187, "conf_lo": 79.4475138121547, "conf_hi": 80.0, "labels": {"확실": 179, "보통": 2}, "acc_ans": 15.469613259668508}, "OOD": {"n": 205, "unk": 143, "unk_rate": 69.7560975609756, "unk_lo": 63.152534276362935, "unk_hi": 75.63284181728889, "n_ans": 62, "conf": 79.03225806451613, "conf_lo": 77.74193548387096, "conf_hi": 80.0, "labels": {"확실": 59, "보통": 3}, "acc_ans": null}, "mannwhitney_p": 0.0751730212883204, "diff_conf": -0.7467474603457447}
Qwen2.5-VL-7B {"ID": {"n": 205, "unk": 0, "unk_rate": 0.0, "unk_lo": 0.0, "unk_hi": 1.8394802568070727, "n_ans": 205, "conf": 76.95121951219512, "conf_lo": 75.8048780487805, "conf_hi": 78.00060975609755, "labels": {"확실": 165, "매우 확실": 5, "보통": 35}, "acc_ans": 15.609756097560975}, "OOD": {"n": 205, "unk": 154, "unk_rate": 75.1219512195122, "unk_lo": 68.77839532999657, "unk_hi": 80.54128044341256, "n_ans": 51, "conf": 78.33333333333333, "conf_lo": 76.47058823529412, "conf_hi": 79.90196078431373, "labels": {"확실": 45, "보통": 5, "매우 확실": 1}, "acc_ans": null}, "mannwhitney_p": 0.26776301350085085, "diff_conf": 1.382113821138205}
Qwen2.5-VL-3B {"n_all": 205, "n_used": 205, "agree": 22.439024390243905, "lo": 17.26522796264295, "hi": 28.62677822769461, "expected_marginal": 27.371802498512785, "perm_p": 0.8708, "best_case": 22.439024390243905, "worst_case": 22.439024390243905, "vlm_marg": {"우상단": 0.32195121951219513, "우하단": 0.3902439024390244, "좌상단": 0.28780487804878047, "좌하단": 0.0}, "cnn_marg": {"우상단": 0.2682926829268293, "우하단": 0.23902439024390243, "좌상단": 0.32682926829268294, "좌하단": 0.16585365853658537}, "kept_acc": 14.634146341463413, "dropped_acc": null, "n_dropped": 0, "ds_keep": {"pathmnist": 45, "dermamnist": 35, "octmnist": 20, "breastmnist": 10, "bloodmnist": 40, "organsmnist": 55}, "ds_drop": {}}
Qwen2.5-VL-7B {"n_all": 205, "n_used": 202, "agree": 26.237623762376238, "lo": 20.65483161326831, "hi": 32.70736533817398, "expected_marginal": 27.22772277227723, "perm_p": 0.8808, "best_case": 27.31707317073171, "worst_case": 25.853658536585368, "vlm_marg": {"우상단": 0.8811881188118812, "우하단": 0.0, "좌상단": 0.10891089108910891, "좌하단": 0.009900990099009901}, "cnn_marg": {"우상단": 0.26732673267326734, "우하단": 0.24257425742574257, "좌상단": 0.3217821782178218, "좌하단": 0.16831683168316833}, "kept_acc": 14.356435643564355, "dropped_acc": 33.33333333333333, "n_dropped": 3, "ds_keep": {"pathmnist": 45, "dermamnist": 32, "octmnist": 20, "breastmnist": 10, "bloodmnist": 40, "organsmnist": 55}, "ds_drop": {"dermamnist": 3}}
Qwen3-VL-2B {"n_all": 205, "n_used": 205, "agree": 21.951219512195124, "lo": 16.829672337436584, "hi": 28.104670245650315, "expected_marginal": 22.903033908387865, "perm_p": 1.0516, "best_case": 21.951219512195124, "worst_case": 21.951219512195124, "vlm_marg": {"우상단": 0.13658536585365855, "우하단": 0.5756097560975609, "좌상단": 0.04390243902439024, "좌하단": 0.24390243902439024}, "cnn_marg": {"우상단": 0.2682926829268293, "우하단": 0.23902439024390243, "좌상단": 0.32682926829268294, "좌하단": 0.16585365853658537}, "kept_acc": 14.634146341463413, "dropped_acc": null, "n_dropped": 0, "ds_keep": {"pathmnist": 45, "dermamnist": 35, "octmnist": 20, "breastmnist": 10, "bloodmnist": 40, "organsmnist": 55}, "ds_drop": {}}
Qwen3-VL-4B {"n_all": 205, "n_used": 150, "agree": 21.333333333333336, "lo": 15.536143670182534, "hi": 28.562201640838676, "expected_marginal": 25.07111111111111, "perm_p": 0.1608, "best_case": 42.4390243902439, "worst_case": 15.609756097560975, "vlm_marg": {"우상단": 0.4066666666666667, "우하단": 0.3933333333333333, "좌상단": 0.16, "좌하단": 0.04}, "cnn_marg": {"우상단": 0.26, "우하단": 0.21333333333333335, "좌상단": 0.3333333333333333, "좌하단": 0.19333333333333333}, "kept_acc": 14.666666666666666, "dropped_acc": 16.363636363636363, "n_dropped": 55, "ds_keep": {"pathmnist": 44, "dermamnist": 14, "octmnist": 20, "breastmnist": 10, "bloodmnist": 7, "organsmnist": 55}, "ds_drop": {"pathmnist": 1, "dermamnist": 21, "bloodmnist": 33}}
Qwen3-VL-8B {"n_all": 205, "n_used": 155, "agree": 21.29032258064516, "lo": 15.580732288767635, "hi": 28.388605718404285, "expected_marginal": 20.75754422476587, "perm_p": 0.4768, "best_case": 40.487804878048784, "worst_case": 16.097560975609756, "vlm_marg": {"우상단": 0.012903225806451613, "우하단": 0.7548387096774194, "좌상단": 0.0, "좌하단": 0.23225806451612904}, "cnn_marg": {"우상단": 0.2645161290322581, "우하단": 0.2129032258064516, "좌상단": 0.33548387096774196, "좌하단": 0.1870967741935484}, "kept_acc": 18.70967741935484, "dropped_acc": 18.0, "n_dropped": 50, "ds_keep": {"pathmnist": 43, "dermamnist": 17, "octmnist": 14, "breastmnist": 10, "bloodmnist": 16, "organsmnist": 55}, "ds_drop": {"pathmnist": 2, "dermamnist": 18, "octmnist": 6, "bloodmnist": 24}}
Qwen3-VL-4B dataset composition keep/drop chi2 p=0.0000 [[7, 33], [10, 0], [14, 21], [20, 0], [55, 0], [44, 1]]
Qwen3-VL-8B dataset composition keep/drop chi2 p=0.0000 [[16, 24], [10, 0], [17, 18], [14, 6], [55, 0], [43, 2]]
scale common ids 205
{
 "gap": {
  "Qwen3-VL-2B": {
   "pt": 77.02439024390243,
   "lo": 71.8780487804878,
   "hi": 81.90243902439023
  },
  "Qwen3-VL-4B": {
   "pt": 62.926829268292686,
   "lo": 57.853658536585364,
   "hi": 67.8048780487805
  },
  "Qwen3-VL-8B": {
   "pt": 56.90243902439024,
   "lo": 51.51219512195122,
   "hi": 62.07317073170732
  }
 },
 "diff_2B_4B": [
  7.439024390243887,
  20.853658536585364
 ],
 "diff_4B_8B": [
  0.7317073170731732,
  11.341463414634148
 ],
 "diff_2B_8B": [
  13.609756097560968,
  26.682926829268283
 ],
 "p_monotone": 0.9865,
 "p_2B_gt_8B": 1.0,
 "labels": {
  "Qwen3-VL-2B": {
   "확실": 39,
   "매우 확실": 166
  },
  "Qwen3-VL-4B": {
   "확실": 175,
   "보통": 30
  },
  "Qwen3-VL-8B": {
   "확실": 105,
   "보통": 81,
   "매우 확실": 19
  }
 },
 "maps": {
  "paper {95,80,60,35,15}": [
   77.02439024390243,
   62.926829268292686,
   56.90243902439024
  ],
  "linear {100,75,50,25,0}": [
   80.1219512195122,
   57.19512195121951,
   50.853658536585364
  ],
  "mild {90,70,50,30,10}": [
   71.07317073170731,
   52.926829268292686,
   47.36585365853659
  ],
  "compressed {80,65,50,35,20}": [
   62.02439024390243,
   48.65853658536585,
   43.8780487804878
  ],
  "low {70,55,40,25,10}": [
   52.02439024390243,
   38.65853658536585,
   33.8780487804878
  ]
 }
}
{"Qwen2.5-VL-3B": {"n": 205, "n_cot_valid": 205, "d_acc": -0.4878048780487809, "d_acc_ci": [-3.9024390243902434, 2.926829268292683], "d_gap": 7.146341463414629, "d_gap_ci": [3.7804878048780495, 10.464024390243898], "cot_labels": {"매우 확실": 103, "보통": 9, "확실": 93}}, "Qwen2.5-VL-7B": {"n": 201, "n_cot_valid": 201, "d_acc": 2.9850746268656705, "d_acc_ci": [-1.9900497512437805, 7.960199004975125], "d_gap": -2.1890547263681697, "d_gap_ci": [-7.662313432835828, 3.2344527363184064], "cot_labels": {"보통": 41, "확실": 81, "매우 확실": 79}}}
saved

