Gradio

Leaderboards by Samsung Research for LLM evaluation.

✨ Samsung Research | 📄 Paper: To be Updated | 🌠 Discussion | 🔭 Updated: 2025-09-23

🏆 TRUEBench: A Benchmark for Assessing LLMs as Human Job Productivity Assistants

TRUEBench (Trustworthy Real-world Usage Evaluation Benchmark) evaluates LLMs as productivity assistants.
As LLMs become integral to tasks like report drafting and data analysis, existing benchmarks are suboptimal to capture real-world challenges.
To address this gap, Samsung Research developed TRUEBench as a comprehensive evaluation framework for real-world LLM applications.

TRUEBench is a benchmark designed to evaluate the instruction-following capabilities of LLMs, determining whether a response receives a Pass (1 point) or Fail (0 points) based on checklists.
This aligns with user satisfaction from the perspective of job productivity.

Main Features

📝

2,400+ Productivity-Oriented User Inputs

A large-scale collection of complex, real-world user inputs designed to reflect productivity assistant scenarios.

🌎

Multilinguality in Real Tasks

Comprehensive 12-language coverage with intra-instance multilingual instructions.

For multilingual aspects, it was created through local research institutes.

🧩

Beyond Explicit Constraints

Human-annotated implicit requirements validated by LLMs.

🧭

Dynamic Multi-Turn Contexts

Realistic dialogue flows with evolving constraints.

📂 Dataset Sample →

Category Analysis

TRUEBench consists of 10 categories and 46 sub-categories which highly related to productivity assistants.

📝 Content Generation

Evaluates the model's ability to produce diverse written outputs across professional and creative domains. This category measures adaptability to linguistic, stylistic, and formatting constraints, as well as the effectiveness of prompt engineering.

🏷️Email 🏷️ReportDrafting

✂️ Editing

Evaluates refinement capabilities for optimizing given text. It focuses on queries related to rephrasing, revision, and correction, while preserving the rest of the content.

🏷️QueryRephrase 🏷️DocumentRevision

📊 Data Analysis

Measures proficiency in processing structured and unstructured data. This category includes tasks related to information extraction and data processing.

🏷️JSONFormatted 🏷️TableQuery

🧠 Reasoning

Assesses logical problem-solving in coding, multiple-choice question answering, and mathematical operations. It also includes evaluation of rounding errors made by models in quantitative tasks.

🏷️Logical 🏷️Mathematical

🦄 Hallucination

Detects limitations in generating plausible but inaccurate responses when faced with ambiguous queries, insufficient context, hypothetical scenarios, or challenges in document interpretation.

🏷️InsufficientContext 🏷️FalseQueries

🛡️ Safety

Verifies safeguards against harmful/inappropriate content. This category tests filtering of discriminatory, violent, or illegal material while upholding ethical standards.

🏷️Illegal 🏷️Prejudice

🔁 Repetition

Evaluates consistency in producing iterative content variations while maintaining quality and relevance across outputs.

🏷️Listing

📝 Summarization

Measures ability to distill lengthy content into concise overviews preserving core concepts and eliminating redundancy. This category includes various constraints such as language, format, and output length.

🏷️BulletPoints 🏷️N-lineSummary

🌐 Translation

Tests the ability to accurately translate diverse real-world contexts while adhering to target language and specified constraints. Our benchmark includes linguistic conditions in 12 languages, ensuring comprehensive multilingual evaluation.

🏷️Document 🏷️Line-by-line

💬 Multi-Turn

Assesses the model's ability to capture user intent in challenging scenarios where the context shifts or understanding of previous context is required.

🏷️Consistency 🏷️Non-consistency

Rank	Model Nameⓘ	Type	Model Type	Think	Overall	Med. Len.ⓘ	Med. Resp. Len.ⓘ	Parameter Size (B)	Content Generation	Editing	Data Analysis	Reasoning	Hallucination	Safety	Repetition	Summarization	Translation	Multi-Turn
🥇	GPT-5	Proprietary	Think	On	70.73				71.00	74.38	76.49	79.75	64.94	56.20	82.86	80.16	69.38	54.36
🥈	o3-pro	Proprietary	Think	On	66.47				72.50	70.31	75.70	83.88	64.37	33.88	74.29	65.48	64.33	48.32
🥉	Claude 4 Opus	Proprietary	Hybrid	On	63.29				60.75	59.69	73.31	69.83	78.74	53.72	55.71	65.48	65.45	48.99
4	Claude 4.1 Opus	Proprietary	Hybrid	On	63.24				61.25	60.00	78.49	72.73	77.01	56.20	57.14	61.90	62.64	46.98
5	GPT-5 mini	Proprietary	Think	On	62.56				68.00	62.50	74.90	76.86	55.17	47.93	44.29	74.60	56.18	45.30
6	Claude 4 Sonnet	Proprietary	Hybrid	On	61.80				58.00	58.44	76.49	67.77	79.31	57.02	44.29	65.08	62.92	44.97
7	o3	Proprietary	Think	On	60.91				68.75	60.00	73.31	79.34	54.02	34.71	64.29	60.71	55.06	46.98
8	Gemini 2.5 Pro	Proprietary	Think	On	59.34				54.00	60.94	78.88	73.14	63.22	17.36	52.86	67.86	53.93	52.68
9	Grok-4	Proprietary	Think	On	58.74				61.00	66.25	72.51	63.22	66.09	16.53	58.57	66.27	54.21	44.30
10	Gemini 2.5 Flash	Proprietary	Hybrid	On	58.62				57.25	62.19	70.52	72.31	56.90	28.93	47.14	68.65	55.06	46.98
11	o4-mini	Proprietary	Think	On	57.57				67.25	61.25	71.71	75.62	45.40	39.67	44.29	59.92	47.19	41.95
12	Qwen3 235B A22B Thinking 2507	Open	Think	On	55.48	2404	423	235	57.50	53.12	73.31	75.21	55.17	25.62	35.71	55.56	56.18	40.27
13	GPT-5 nano	Proprietary	Think	On	55.39				63.50	47.19	68.92	75.21	55.17	52.07	34.29	63.49	40.73	42.95
14	Qwen3 Next 80B A3B Thinking	Open	Think	On	55.07	3263	329	80	58.25	51.56	73.71	76.03	52.30	38.84	32.86	57.14	46.63	43.62
15	GLM-4.5 FP8	Open	Hybrid	On	54.03	1442	604	355	60.75	53.75	68.92	74.38	47.13	33.06	41.43	60.32	46.07	35.91
16	Qwen3 235B A22B Instruct 2507	Open	Instruct	Off	52.94	433	433	235	58.00	49.69	68.13	73.97	55.17	45.45	30.00	55.95	38.48	41.61
17	DeepSeek V3.1	Open	Hybrid	On	51.45	710	356	671	52.00	50.00	67.33	69.83	50.00	33.88	35.71	59.52	41.85	40.27
18	Qwen3 30B A3B Thinking 2507	Open	Think	On	50.44	2830	351	30	56.25	45.00	69.32	69.01	50.00	29.75	30.00	48.02	47.47	36.58
19	gpt-oss-120B	Open	Think	On	49.11	760	370	117	58.50	48.44	68.92	69.83	41.38	39.67	25.71	50.79	35.67	32.21
20	Qwen3 Next 80B A3B Instruct	Open	Instruct	Off	48.87	478	478	80	53.50	45.31	66.14	71.07	55.17	39.67	21.43	48.41	36.80	35.23
21	DeepSeek R1	Open	Think	On	48.79	1178	554	671	49.75	50.00	65.34	59.09	48.85	38.02	32.86	57.94	36.52	38.93
22	Gauss2.3 Hybrid	Proprietary	Hybrid	On	46.58	546	308		52.00	46.25	59.76	66.94	41.95	34.71	25.71	53.17	34.55	33.22
23	DeepSeek V3	Open	Instruct	Off	45.09	408	408	671	46.25	45.00	58.96	60.33	41.95	21.49	30.00	55.95	38.48	33.22
24	Qwen3 32B	Open	Hybrid	On	44.44	1113	390	33	52.25	41.56	68.92	66.53	35.06	19.83	25.71	46.43	30.90	32.89
25	Qwen3 30B A3B Instruct 2507	Open	Instruct	Off	42.79	442	442	30	45.00	35.00	56.18	66.12	51.15	33.06	24.29	46.83	28.09	35.57
26	A.X 4.0	Open	Instruct	Off	41.59	412	412	72	56.00	43.75	43.43	42.56	40.23	15.70	24.29	53.97	33.43	32.21
27	gpt-oss-20B	Open	Think	On	41.18	954	326	21	52.00	40.00	61.35	65.70	43.10	41.32	22.86	36.51	20.51	22.82
28	Gemma 3 27B it	Open	Instruct	Off	40.86	380	380	27	44.25	45.00	45.82	36.78	31.61	32.23	22.86	57.14	32.87	39.93
29	Tongyi DeepResearch 30B A3B	Open	Think	On	40.10	1147	408	30	41.25	33.12	62.15	68.18	44.25	23.97	18.57	41.67	26.12	29.19
30	Mistral Small 3.2 24B Instruct 2506	Open	Instruct	Off	39.09	369	369	24	43.00	44.69	43.43	51.65	25.86	22.31	25.71	51.98	31.18	30.20
31	K2-Think	Open	Think	On	35.06	1835	486	33	35.50	36.56	56.18	47.11	35.06	14.05	12.86	49.21	21.63	23.15
32	EXAONE 4.0 32B	Open	Hybrid	On	33.82	1274	503	32	34.25	29.38	56.97	57.44	24.71	27.27	17.14	38.49	18.54	25.50
33	HyperCLOVAX SEED Think 14B	Open	Hybrid	On	31.84	1444	382	15	35.00	26.56	53.78	58.68	27.59	26.45	17.14	29.76	17.13	20.47
34	ERNIE 4.5 21B A3B Thinking	Open	Think	On	25.32	1637	541	21	27.25	20.31	42.23	49.59	23.56	31.40	17.14	28.17	7.30	13.76
35	Solar Pro Preview	Open	Instruct	Off	20.73	260	260	22	28.00	24.69	16.73	19.42	17.24	28.10	11.43	31.35	13.76	11.74
36	Mi:dm 2.0 Base Instruct	Open	Instruct	Off	20.25	316	316	12	21.75	17.50	16.73	18.60	27.59	59.50	14.29	25.40	12.64	11.41
37	Kanana 1.5 15.7B A3B Instruct	Open	Instruct	Off	11.71	414	414	16	14.25	10.62	13.55	11.16	22.41	22.31	4.29	11.90	6.74	5.37

Plot

Language Analysis

As a multilingual benchmark, TRUEBench supports a total of 12 user input languages: Korean (KO), English (EN), Japanese (JA), Chinese (ZH), Polish (PL), German (DE), Portuguese (PT), Spanish (ES), French (FR), Italian (IT), Russian (RU), and Vietnamese (VI).