Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Mar 24, 2024·
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
·
0 min read
Zhouhong Gu
Xiaoxuan Zhu
Haoning Ye
Lin Zhang
Jianchen Wang
Yixin Zhu
Sihang Jiang
Zhuozhi Xiong
Zihan Li
Weijie Wu
Qianyu He
Rui Xu
Wenhao Huang
Jingping Liu
Zili Wang
Shusen Wang
Weiguo Zheng
Hongwei Feng
Yanghua Xiao
Abstract
New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by \myred{Xiezhi-Specialty with 14,041 questions and Xiezhi-Interdiscipline with 10,746 questions}. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management.
Type
Publication
Proceedings of the AAAI Conference on Artificial Intelligence
