š kompress.vaked.dev
the manuscript Ā· open science
Asymmetric Loss Modulation Resolves the Voting Ensemble Paradox in Learned Context-Pruning Ensembles
Abstract
Multi-checkpoint voting ensembles of learned context-pruning models paradoxically underperform their best single-voter constituent. We formalize this: under k-of-N drop voting, the ensemble eviction indicator equals the k-th order statistic of the per-voter indicators ā the ensemble collapses to its weakest member on every stratum. Three mechanisms resolve the paradox: asymmetric loss modulation during training (Ī» = 3.0), regex-based inference overrides, and C3 self-distillation with a stronger teacher. We train 17 kompress models (149M-param ModernBERT) for $38.95 total, achieving a Pareto-optimal 0.955 heretic-exact at 15% compression with kompress-v8.
Theorem 1 ā the voting ensemble paradox
Let E be an ensemble of N voters, each producing a per-token keep
indicator Ii ā {0,1}. Under k-of-N drop voting, IE = 1 iff
Ī£ Ii ā„ k. Then IE = I(Nāk+1) ā the (Nāk+1)-th order
statistic ā and the ensemble's precision ⤠mini precision(Ii).
Results
| Version | Heretic | Keep | Note |
|---|---|---|---|
| v2-base | 0.975 | 0.897 | precision ceiling |
| v8 ā | 0.955 | 0.854 | production |
| v16 | 0.972 | 0.972 | endpoint (10Ć) |
| v17 | 0.963 | 0.963 | tradeoff (5Ć) |