ai-evaluation

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.

[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification

[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

build and benchmark deep research

Ranking LLMs on agentic tasks

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

开源劳动者AI研究与评测项目：建设劳动知识、公开评测基准与共同治理机制｜Open worker-centered AI research and evaluation project.

Agentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.

A reproducible local LLM benchmarking platform with versioned datasets, deterministic scoring, and auditable reports.

A benchmark for AI-driven CAD generation and editing

Vivaria is METR's tool for running evaluations and conducting agent elicitation research.

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

Statistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.

Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.

one click to open multi AI sites ｜ 一键打开多个 AI 站点，查看 AI 结果

Universal AI Charter: plain-language governance principles and a public certification registry for AI models, built on disclosure instead of a development slowdown.

[Public preview] Hard human-written reasoning puzzles: 749 answered items in 15 topics (707 scored core), with protocol and scorer.

[Public preview] The first public quant trading benchmark scored against what professional traders actually made on the same asset