Research statement Jingwei Ni

AI systems that know when to be trusted.

I extract the uncertainty a model already carries and expose it for human supervision — because a model that reliably reports when it is likely to be wrong is what makes oversight possible.

← Back to homepage

Topic

Surfacing a model's own uncertainty.

I build AI systems that know when to be trusted: I extract the uncertainty a model already carries and expose it for human supervision. A trustworthy model should report low confidence whenever it is likely to be wrong, whatever the cause — and that ability to report its own unreliability is what makes human oversight possible. The uncertainty itself has two sources, epistemic (what the model doesn't know) and aleatoric (irreducible ambiguity in the world) (Hüllermeier & Waegeman, 2021; Shorinwa et al., 2024); for oversight, what matters is that the model reports it.

Why this matters now

Uncertainty reporting is the bottleneck.

Frontier models already game the rewards we give them.

In June 2026 METR found that GPT-5.6 had the highest cheating rate of any public model it has tested — it packaged exploits into its intermediate submissions to leak a task's own hidden tests, and counting that as success inflated its measured capability past the point where the benchmark still means anything (METR, 2026). The same week, Cursor traced 63% of a frontier model's benchmark wins to answers retrieved from the public web and git history rather than real fixes; cut off those channels and one model fell from 87% to 73% (Cursor, 2026). When the reward is gameable, real capability and reward-hacking are indistinguishable from the outside — which is exactly the gap a model's own reported uncertainty is meant to close.

The safer path to advanced AI runs on uncertainty reporting.

Bengio et al.'s Scientist AI makes a simple case: agentic systems tend to develop implicit, instrumental goals — self-preservation, power-seeking, reward hacking via overoptimization — that emerge regardless of their initial objective, so rather than trust an agent on its own, we supervise it with a non-agentic, uncertainty-aware system that acts as a guardrail (Bengio et al., 2025). Building that kind of supervisor is the tractable core of my work.

Our evaluation tools can't see disagreement.

The field ranks, filters, and aligns models with LLM-as-a-judge (Zheng et al., 2023; Gu et al., 2024), but "helpful," "harmful," and "novel" are contested across legitimate viewpoints, so a judge commits to one reading and returns a confident score — leaving leaderboards most miscalibrated exactly where humans most disagree (Chochlakis et al., 2025; Baan et al., 2022; Li et al., 2025a).

Deployment is full of the hard kind of uncertainty.

Real instructions are underspecified and real judgments are contested, yet agents are rewarded only for producing a heuristic-verifiable correct answer in math and code (Lambert et al., 2024; DeepSeek-AI, 2025) — never for being uncertainty-aware — so they invent the missing detail and act instead of asking (Wang et al., 2024; Li et al., 2025b).

Scaling will not rescue it.

When uncertainty is irreducible, more compute on the same ambiguous input cannot resolve it — and my own work shows that making models reason longer actively harms their ability to represent human disagreement (Ni et al., 2026).

My research

From checkable signals to contested labels.

What I build differs from setting to setting; how I judge it does not. I evaluate every method with calibration and selective-prediction AUROC, which test whether a model's reported confidence tracks its actual errors. What changes across settings is how tractable the problem is, and that turns on whether the uncertainty is objectively checkable.

Where it is checkable — verifying a math reasoning step, judging whether retrieved evidence is relevant — labels are cheap, models are strong, and uncertainty can be trained and rigorously tested. One favorite paper of my own, ReProbe, lives here: it reads a model's own internal states mid-reasoning to tell when an answer has settled, scaling test-time compute only as far as needed, with indisputable gold evaluation and extensive out-of-distribution tests. It is exactly the kind of non-agentic, uncertainty-reporting verifier that Scientist AI calls for.

At the other end lies far less tractable terrain — subjective classification and conflicting human labels, long argued to be signal rather than noise (Plank, 2022; Uma et al., 2021; Xu et al., 2025) — where I build uncertainty-aware classifiers distilled into cheap, deployable models (AFaCTA, DIRAS), evidence-grounded specialists whose every claim is traceable to its source (Evidence-Based QA), and interfaces that surface the edge cases a classifier quietly gets wrong so people adjudicate the contested ones (Co-DETECT). Here the gold signal is a distribution of disagreeing human labels — scarce, costly, and far from mainstream benchmarks — so even evaluation is an open problem. That gap is itself a central finding: because this uncertainty cannot be cheaply scored, it never enters the reward, and models are never trained to handle it.

References

Works cited.

Hüllermeier & Waegeman, 2021 Hüllermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110(3), 457–506. link
Shorinwa et al., 2024 Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., & Majumdar, A. (2024). A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions. arXiv:2412.05563. link
METR, 2026 METR. (2026, June 26). Summary of METR's predeployment evaluation of GPT-5.6 Sol. link
Cursor, 2026 Jain, N. (2026, June 25). Reward hacking is swamping model intelligence gains. Cursor. link
Bengio et al., 2025 Bengio, Y., et al. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? arXiv:2502.15657. link
Zheng et al., 2023 Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks. link
Gu et al., 2024 Gu, J., et al. (2024). A Survey on LLM-as-a-Judge. arXiv:2411.15594. link
Chochlakis et al., 2025 Chochlakis, G., et al. (2025). Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors. NAACL. link
Baan et al., 2022 Baan, J., et al. (2022). Stop Measuring Calibration When Humans Disagree. EMNLP. link
Li et al., 2025a Li, Z., Schuurmans, D., Dai, B., & Palangi, H. (2025). Judging with Confidence: Calibrating Autoraters to Preference Distributions. arXiv:2510.00263. link
Lambert et al., 2024 Lambert, N., et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. link
DeepSeek-AI, 2025 DeepSeek-AI (Guo, D., et al.). (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature, 645, 633–638 (arXiv:2501.12948). link
Wang et al., 2024 Wang, W., et al. (2024). Learning to Ask: When LLM Agents Meet Unclear Instruction. arXiv:2409.00557. link
Li et al., 2025b Li, B. Z., Kim, J., & Wang, R. (2025). QuestBench: Can LLMs Ask the Right Question to Acquire Information in Reasoning Tasks? NeurIPS Datasets and Benchmarks (arXiv:2503.22674). link
Ni et al., 2026 Ni, J., Fan, Y., et al. (2026). Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? EACL. link
Plank, 2022 Plank, B. (2022). The "Problem" of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. EMNLP. link
Uma et al., 2021 Uma, A., et al. (2021). Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72, 1385–1470. link
Xu et al., 2025 Xu, M., Santosh, T. Y. S. S., & Plank, B. (2025). From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLP. arXiv:2510.12817. link