We built a firewall for AI confidently-wrong answers - and it catches what confidence cannotPostavili sme firewall na sebaisto-nespravne odpovede AI - a chyta to, co confidence nevidi
The problem. A model is most confident exactly when it is wrong for the right-looking reason. When a retrieved document states a plausible-but-false answer - a poisoned context: stale data, an injecteModel je najsebaistejsi presne vtedy, ked ho otraveny dokument privedie k chybe. Grounding Firewall sa zdrzi pri odpovediach, ktore visia na nacitanom dokumente - chyta chyby z otraveneho kontextu, na ktore je confidence slepy (AUC 0.028 vs 0.095).
The problem. A model is most confident exactly when it is wrong for the right-looking reason. When a retrieved document states a plausible-but-false answer - a poisoned context: stale data, an injected line, a wrong source - the model can follow it at full confidence. So confidence-based abstention ("only answer when the model is sure") ships precisely those errors.
The idea. Measure how much the answer DEPENDS on the document rather than how sure the model sounds. For an answer, compute sensitivity = | p(answer | context) - p(answer | context removed) |. An answer that flips when you delete its evidence is grounded in the document, not in the model knowledge - so if that document is wrong, the answer is wrong, and confidence will not warn you. The Grounding Firewall ABSTAINS on high-sensitivity answers.
The measured result. 24 real factual questions, each given a POISONED context (a document asserting the false answer), scored black-box on an open model (qwen2.5-7B) where the truth is known independently. The grounding signal predicts correctness at +0.68; the model own confidence only +0.37. Risk-coverage AUC: firewall 0.028 vs confidence 0.095 - about 3.4x lower risk. At 70% coverage the firewall ships ZERO wrong answers; confidence-gating still ships 12%. The decisive case: the model followed a poisoned "tallest mountain = K2" at confidence 0.99 - confidence trusts it, the firewall flags it.
The method, in two sentences. Read the answer token-probability under the retrieved context, then again with the context removed; the gap is the sensitivity. Abstain when sensitivity is high - the answer is riding on the document, which you cannot vouch for.
It is a one-file open tool. Point it at any OpenAI-compatible or Ollama endpoint with a (question, retrieved context, options) and it returns ANSWER or ABSTAIN plus the sensitivity and why.
The falsifierIf sensitivity had not beaten confidence at equal coverage, the firewall would be useless. It did, on data where correctness is known independently of the model.
Honest scope. N=24, one open model, a simple injected poison. The next test is a large real corpus plus an adaptive poisoner that tries to keep sensitivity low. This is a real result at small scale, not a product claim - and the test that would kill it is named above.
Problem. Model je najsebaistejsi presne vtedy, ked sa myli zo spravne vyzerajuceho dovodu. Ked nacitany dokument uvedie vierohodnu-ale-nepravdivu odpoved - otraveny kontext: zastarane data, podstrcena veta, nespravny zdroj - model ju moze nasledovat s plnou istotou. Takze zdrzanie sa na zaklade confidence ("odpovedaj len ked si isty") preposle presne tieto chyby.
Napad. Merajme, nakolko odpoved ZAVISI od dokumentu, nie ako isto model znie. Pre odpoved spocitame sensitivity = | p(odpoved | kontext) - p(odpoved | kontext odstraneny) |. Odpoved, ktora sa prevrati, ked vymazete jej dokaz, je ukotvena v dokumente, nie vo vedomostiach modelu - takze ak je dokument nespravny, odpoved je nespravna a confidence vas nevaruje. Grounding Firewall sa ZDRZI pri odpovediach s vysokou sensitivity.
Nameraný vysledok. 24 realnych faktickych otazok, kazda s OTRAVENYM kontextom (dokument tvrdi nepravdivu odpoved), hodnotene black-box na otvorenom modeli (qwen2.5-7B), kde je pravda znama nezavisle. Grounding signal predpoveda spravnost na +0.68; vlastny confidence modelu len +0.37. Risk-coverage AUC: firewall 0.028 vs confidence 0.095 - asi 3.4x nizsie riziko. Pri 70% pokryti firewall preposle NULA nespravnych odpovedi; confidence-gating stale 12%. Rozhodujuci pripad: model nasledoval otravene "najvyssia hora = K2" pri confidence 0.99 - confidence mu veri, firewall ho oznaci.
Metoda v dvoch vetach. Precitaj pravdepodobnost tokenu odpovede s nacitanym kontextom a potom znova bez kontextu; rozdiel je sensitivity. Zdrz sa, ked je sensitivity vysoka - odpoved sa vezie na dokumente, za ktory neviete rucit.
Je to jednosuborovy otvoreny nastroj. Namierte ho na hocijaky OpenAI-kompatibilny alebo Ollama endpoint s (otazka, nacitany kontext, moznosti) a vrati ANSWER alebo ABSTAIN plus sensitivity a preco.
Falzifikator. Keby sensitivity neprekonala confidence pri rovnakom pokryti, firewall by bol zbytocny. Prekonala - na datach, kde je spravnost znama nezavisle od modelu.
Poctivy rozsah. N=24, jeden otvoreny model, jednoduche podstrcenie. Dalsi test je velky realny korpus plus adaptivny utocnik, ktory sa snazi udrzat sensitivity nizko. Toto je realny vysledok v malej skale, nie produktove tvrdenie - a test, ktory by ho zabil, je vyssie pomenovany.