Skip to content

Retrain heuristic weights (2026-08-14) - #13

Open
github-actions[bot] wants to merge 1 commit into
mainfrom
retrain
Open

Retrain heuristic weights (2026-08-14)#13
github-actions[bot] wants to merge 1 commit into
mainfrom
retrain

Conversation

@github-actions

Copy link
Copy Markdown

Automated re-fit. Review the weight diff below, then update the golden tests
in HeuristicArenaDataSourceTests with the printed values before merging.

Golden gate (run in the retrain job — a GITHUB_TOKEN PR does not trigger build):
FAIL — goldens must be updated from the trainer output before merging

training dir: D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training
live fetch; snapshotting payloads to D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training\.snapshot
fetching HSReplay arena card stats...
HSReplay pooled (card,class) rows: 2208
rows: 2208 (single source)
features: 31 fitted, 19 dropped
  dropped: kw_charge(n=11), kw_colossal(n=11), kw_combo(n=21), kw_echo(n=5), kw_forge(n=3), kw_freeze(n=23), kw_magnetic(n=13), kw_outcast(n=18), kw_poisonous(n=37), kw_reborn(n=39), kw_spellpower(n=18), kw_stealth(n=13), kw_windfury(n=8), tx_armor(n=28), tx_destroy_minion(n=28), tx_discover(redundant), tx_restore_amt(n=18), tx_silence(n=17), tx_transform(n=34)
penalty BEFORE normalizing weights: alpha=10 vs mean(diag)=77471.3  -> shrinkage 0.000129, df_eff=30.0/31
CV: 5-fold grouped by card (1093 cards, 2208 rows), 5 repeats, 11 alphas

     alpha   withinClassRho       se
      0.03           0.1733   0.0016
       0.1           0.1733   0.0016
       0.3           0.1735   0.0016
         1           0.1740   0.0016
         3           0.1755   0.0014
        10           0.1793   0.0011
        30           0.1844   0.0009
       100           0.1835   0.0012
       300           0.1748   0.0013
      1000           0.1554   0.0025
      3000           0.1289   0.0023
CV best alpha=30 (rho=0.1844); 1-SE rule picks alpha=30 (rho=0.1844, threshold 0.1835)
penalty AFTER: alpha=30 vs mean(diag)=2396.5  -> shrinkage 0.012364, df_eff=28.9/31

holdout evaluation (pooled Spearman on held-out rows, alpha=30):
  random cards (baseline)            n= 109  rho= 0.2271  (se 0.0346)
  lowest-games decile                n= 109  rho= 0.2463  (up to 219 games)
  -> covariate-shift gap vs baseline: +0.0192
  target spread / model error:
      thin  sd(y)=6.582  MAE=5.190
      rest  sd(y)=4.986  MAE=3.210
  calibration slope (truth regressed on prediction):
      thin decile  1.2622   random holdout  0.9544  (se 0.1101)
      -> measured ModelOnlyShrink = 1.0000 (thin/random; set the runtime constant from THIS, not from a correlation ratio)
  leave-out set CORE                 n= 210  rho= 0.1539
  leave-out set TIME_TRAVEL          n= 171  rho= 0.2879
  leave-out set BATTLE_OF_THE_BANDS  n= 158  rho= 0.3495
  leave-out set WHIZBANGS_WORKSHOP   n= 151  rho= 0.2435

fitting ridge (alpha=30) on 2208 rows, 31 features...
display anchor (pool median raw): -0.97, robust sigma: 1.2466 -> 15 pts per robust SD
wrote D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training\arena_weights.generated.json

feature                  committed   generated     diff
intercept                   +1.09      +1.05    -0.04
anchor_median_raw           -0.92      -0.97    -0.05  <-- differs
attack                      -0.15      -0.12    +0.03  <-- differs
cost                        +0.15      +0.08    -0.07  <-- differs
has_tribe                   +0.12      +0.39    +0.27  <-- differs
health                      -0.18      -0.12    +0.06  <-- differs
is_hero                     +0.81      +3.83    +3.02  <-- differs
is_loc                      -0.88      -0.77    +0.11  <-- differs
is_minion                   +2.61      +2.42    -0.19  <-- differs
is_neutral                  -1.94      -2.08    -0.14  <-- differs
is_spell                    -2.75      -2.53    +0.22  <-- differs
is_weapon                   -0.94      -1.22    -0.28  <-- differs
kw_battlecry                +0.70      +0.60    -0.10  <-- differs
kw_deathrattle              -0.41      -0.74    -0.33  <-- differs
kw_discover                 +0.86      +0.62    -0.24  <-- differs
kw_divine_shield            +0.10      +0.22    +0.12  <-- differs
kw_lifesteal                +0.84      +0.89    +0.05  <-- differs
kw_reborn                   +1.83      +0.00    -1.83  <-- differs
kw_rush                     +1.05      +1.16    +0.11  <-- differs
kw_secret                   -0.19      -0.07    +0.12  <-- differs
kw_taunt                    -0.10      +0.00    +0.10  <-- differs
kw_tradeable                +0.71      +0.78    +0.07  <-- differs
stat_per_mana               -2.40      -2.31    +0.09  <-- differs
statline                    +0.34      +0.36    +0.02  <-- differs
tx_aoe                      +0.51      +0.43    -0.08  <-- differs
tx_damage_amt               +0.08      +0.08    +0.00
tx_dmg_per_mana             +0.00      -0.07    -0.07  <-- differs
tx_draw                     -0.31      -0.26    +0.05  <-- differs
tx_gain_card                -0.87      -0.49    +0.38  <-- differs
tx_mana_cheat               +1.05      +0.75    -0.30  <-- differs
tx_persistent               +0.15      +0.23    +0.08  <-- differs
tx_random                   +0.32      +0.13    -0.19  <-- differs
tx_summon                   +0.95      +1.38    +0.43  <-- differs
weapon_value                -0.20      -0.28    -0.08  <-- differs

max abs weight diff: 3.02
weights differing by >= 0.01: 32
GATE: generated weights are close but not identical (see rows above).

golden scores for HeuristicArenaDataSourceTests.Goldens
(paste the VALUES over the literals on adopt; keep the trailing comments):
  { HSCard.PlatedBeetle, 22.57 },
  { HSCard.ElvenArcher, 52.11 },
  { HSCard.DefenderOfArgus, 39.15 },
  { HSCard.FieryWarAxe, 50.96 },
  { HSCard.Fireball, 52.48 },
  { HSCard.ColdlightOracle, 43.50 },
  { HSCard.DireFrenzy, 47.71 },
  { HSCard.NorthshireCleric, 46.99 },
  { HSCard.DarkIronDwarf, 29.11 },
  { HSCard.Deathwing, 17.63 },

bootstrapping 300 card-resampled refits...

pick instability under resampling, by how clearly the model named a winner:
  winner leads by >  0.0 pts:  27.58% of those picks flip
  winner leads by >  1.0 pts:  25.69% of those picks flip
  winner leads by >  3.0 pts:  22.23% of those picks flip
  winner leads by >  5.0 pts:  19.18% of those picks flip
  winner leads by > 10.0 pts:  13.14% of those picks flip

feature                    weight       se    |w|/se   signCons
attack                      -0.12     0.11       1.1       0.90  <-- within noise
cost                        +0.08     0.12       0.7       0.74  <-- within noise
has_tribe                   +0.39     0.25       1.5       0.93  <-- within noise
health                      -0.12     0.13       0.9       0.81  <-- within noise
is_hero                     +3.83     4.45       0.9       0.68  <-- within noise
is_loc                      -0.77     1.46       0.5       0.66  <-- within noise
is_minion                   +2.42     1.18       2.0       1.00
is_neutral                  -2.08     0.32       6.6       1.00
is_spell                    -2.53     1.04       2.4       1.00
is_weapon                   -1.22     1.54       0.8       0.71  <-- within noise
kw_battlecry                +0.60     0.29       2.0       0.99
kw_deathrattle              -0.74     0.31       2.4       0.99
kw_discover                 +0.62     0.33       1.9       0.96  <-- within noise
kw_divine_shield            +0.22     0.53       0.4       0.69  <-- within noise
kw_lifesteal                +0.89     0.45       2.0       0.97  <-- within noise
kw_rush                     +1.16     0.41       2.8       1.00
kw_secret                   -0.07     0.55       0.1       0.60  <-- within noise
kw_taunt                    -0.04     0.39       0.1       0.54  <-- within noise
kw_tradeable                +0.78     0.66       1.2       0.84  <-- within noise
stat_per_mana               -2.31     1.64       1.4       0.85  <-- within noise
statline                    +0.36     0.14       2.6       0.99
tx_aoe                      +0.43     0.38       1.1       0.89  <-- within noise
tx_damage_amt               +0.08     0.15       0.5       0.68  <-- within noise
tx_dmg_per_mana             -0.07     0.63       0.1       0.54  <-- within noise
tx_draw                     -0.26     0.41       0.6       0.72  <-- within noise
tx_gain_card                -0.49     0.34       1.4       0.94  <-- within noise
tx_mana_cheat               +0.75     0.42       1.8       0.97  <-- within noise
tx_persistent               +0.23     0.36       0.7       0.70  <-- within noise
tx_random                   +0.13     0.30       0.4       0.73  <-- within noise
tx_summon                   +1.38     0.33       4.2       1.00
weapon_value                -0.28     0.26       1.1       0.90  <-- within noise
release-gate noise floor (p95 of the bootstrap null): 27.58% of picks flip on resampling alone
wrote D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training\metrics.json
pick flips vs committed: 16.94% of same-class triples (resampling noise floor 27.58%); per-card score volatility p95 17.1 pts
MATERIAL: this refit changes more recommendations than noise would - worth a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants