Evaluating large language models on the core cognitive capabilities expected of technology managers: a hierarchical benchmark

Abstract

Technology managers are instrumental in facilitating technology transfer. While large language models (LLMs) have demonstrated potential in this domain, evaluating them on the core cognitive capabilities expected of technology managers remains challenging due to the lack of domain-specific benchmarks. To address this challenge, we construct TMBench, a domain-specific benchmark for evaluating the cognitive capabilities expected of technology managers. It comprises a collection of 3474 questions spanning diverse scenarios, covering three tasks and five question types. We integrate typical metrics with LLM-as-a-Judge using both fixed and entropy-based weights to calculate the corresponding dimensional and composite scores. Leveraging this mechanism, we evaluate nine representative models under varying prompt configurations and temperature settings to investigate their capabilities, characterize their error patterns, and benchmark them against human experts. Our results show heterogeneous capabilities, with no single model ranking first across all three capability levels. For higher-order capabilities, LLMs show more favorable point estimates than the human baseline, but the differences are not statistically significant. Chain-of-thought-based prompting is associated with prompt-sensitive capability profiles, with more favorable open-weight performance in the two higher-order tasks. Error analysis identifies knowledge base, retrieval, and application deficits as the most prevalent error category. Building on these findings, this study provides a reference for model selection, prompt engineering, and targeted optimization when applying LLMs in technology transfer.

Keywords

Technology transfer / Technology manager / Benchmark / Large language model (LLM)

References

[1] Alibaba Group , 2025. Qwen3-Next-80B: instruction-tuned models for RAG workflows.https://qwen.ai/blog/qwen3-next-ultimate-efficiency/[Accessed on Jan. 20, 2026].

[2] Altermatt FR , Neyem A , Sumonte NI , et al., 2025. Evaluating the performance of large language models on the CONACEM anesthesiology certification exam:a comparison with human participants. Appl Sci, 15 (11): 6245.

[3] Anthropic , 2026. Introducing Claude Haiku 4.5: high-efficiency iterations of the Claude family.https://www.anthropic.com/news/claude-haiku-4-5[Accessed on Jan. 20, 2026].

[4] Association of University Technology Managers , 2024. Technology Transfer Practice Manual.https://autm.net/surveys-and-tools/techtransferpractices-manual [Accessed on Jan. 20, 2026].

[5] Badenes-Olmedo C , Eyzaguirre-Barreda P , Chu-Artzt N , et al., 2026. ESQAD:a curriculum-aligned dataset for question answer generation in Spanish. Knowl-Based Syst, 335: 115255.

[6] Bastani H , Karger E , Chen YH , et al., 2025. ForecastBench: a dynamic benchmark of AI forecasting capabilities. Proc 13^th Int Conf on Learning Representations.

[7] BenYoash N , Brief M , Ovadia O , et al., 2025. SECQUE: a benchmark for evaluating real-world financial analysis capabilities. Proc 4^th Workshop on Generation, Evaluation and Metrics, p.212- 230.

[8] Betancourt J , Coral A , Fraga A , et al., 2025. Intelligent virtual assistant for calculating technology readiness levels using large language models (LLM). IEEE Access, 13: 137709- 137723.

[9] Binegde GN , Zhang HP , 2026. Exploring cultural commonsense in multilingual large language models:a survey. Inform Syst, 138: 102649.

[10] Busch K , Leopold H , 2024. Towards a benchmark for large language models for business process management tasks.

[11] Center for AI Safety , Scale AI , HLE Contributors Consortium , 2026. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649 (8099): 1139- 1146.

[12] Cetindamar D , Phaal R , Probert DR , 2016. Technology management as a profession and the challenges ahead. J Eng Technol Manag, 41: 1- 13.

[13] Chadli K , Botterweck G , Saber T , 2026. PEARL:performance and energy aware routing for LLMs. Fut Gener Comput Syst, 176: 108218.

[14] Chang CC , Chen CY , Lee HS , et al., 2024. Benchmarking cognitive domains for LLMs:insights from Taiwanese Hakka culture. Proc 27^th Conf of the Oriental COCOSDA Int Committee, p.1- 6.

[15] Chang YP , Wang X , Wang JD , et al., 2024. A survey on evaluation of large language models. ACM Trans Intell Syst Technol, 15 (3): 39.

[16] China Association of Science and Technology Evaluation and Management of Scientific and Technical Achievement (CASTEM) , National Center for Science and Technology Evaluation (NCSTE) , 2025. Advanced Tutorial Technical Manager. Science and Technology of China Press, Beijing, China (in Chinese).

[17] Chiu YY , Jiang LW , Lin BY , et al., 2025. CulturalBench:a robust, diverse and challenging benchmark for measuring LMs' cultural knowledge through human-AI red-teaming. Proc 63^rd Annual Meeting of the Association for Computational Linguistics, p.25663- 25701.

[18] Chowa SS , Alvi R , Rahman SS , et al., 2026. From language to action:a review of large language models as autonomous agents and tool users. Artif Intell Rev, 59 (2): 71.

[19] COGR , 2021. The evolution of technology transfer.https://www.cogr.edu/sites/default/files/COGR%20Tech%20Transfer10621.pdf[Accessed on Jan. 20, 2026].

[20] Dai H , Teehan R , Ren MY , 2025. Are LLMs prescient? A continuous evaluation using daily news as the oracle.

[21] Decoupes R , Interdonato R , Roche M , et al., 2025. Evaluation of geographical distortions in language models. Mach Learn, 114 (12): 1- 32.

[22] DeepSeek AI , 2025. DeepSeek-V3.2: high-compute optimization and reasoning performance.https://api-docs.deepseek.com/news/news251201/[Accessed on Jan. 20, 2026].

[23] Deroy A , Ghosh K , Ghosh S , 2026. Investigating legal question generation using large language models. Artif Intell Law, 34: 775- 813.

[24] DeVore S , 2025. Exploring the ability of LLMs to classify written proficiency levels. Comput Speech Lang, 90: 101745.

[25] Fan ZY , He JL , Li XQ , et al., 2025. CAMIEval:enhancing NLG evaluation through multidimensional comparative instruction-following analysis. Proc Conf of the Nations of the Americas Chapter of the Association for Computational Linguistics:Human Language Technologies, p.8708- 8733.

[26] Gao MQ , Hu XY , Yin XJ , et al., 2025. LLM-based NLG evaluation:current status and challenges. Comput Ling, 51 (2): 661- 687.

[27] Genç O , 2026. Is conversational AI ready for engineering licensure? A ChatGPT performance benchmark and prompting strategy evaluation. J Asian Archit Build Eng, p.1- 11.

[28] Georgiev P , Lei VL , Burnell R , et al., 2024. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context.https://arXiv.org/abs/2403.05530.

[29] Gignac GE , Ilić D , 2025. Psychometrically derived 60-question benchmarks: substantial efficiencies and the possibility of human-AI comparisons. Intelligence, 110: 101922.

[30] Google DeepMind , 2025. Gemini 3 Flash: frontier intelligence built for speed.https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/[Accessed on Jan. 20, 2026].

[31] Gorschek T , Garre P , Larsson S , et al., 2006. A model for technology transfer in practice. IEEE Softw, 23 (6): 88- 95.

[32] Guo SK , Shariatmadari AH , Xiong GZ , et al., 2025. IDEABench:benchmarking large language models for research idea generation. Proc 31^st ACM SIGKDD Conf on Knowledge Discovery and Data Mining V.2, p.5888- 5899.

[33] Guo X , Xia HT , Liu ZW , et al., 2025. FinEval:a Chinese financial domain knowledge evaluation benchmark for large language models. Proc Conf of the Nations of the Americas Chapter of the Association for Computational Linguistics:Human Language Technologies, p.6258- 6292.

[34] Halawi D , Zhang F , Chen YH , et al., 2024. Approaching human-level forecasting with language models. Proc 38^th Int Conf on Neural Information Processing Systems, p.50426- 50468.

[35] Harrison W , 2006. Technology transfer and the tech broker. IEEE Softw, 23 (5): 5- 7.

[36] Huang L , Yu WJ , Ma WT , et al., 2025. A survey on hallucination in large language models:principles, taxonomy, challenges, and open questions. ACM Trans Inform Syst, 43 (2): 42.

[37] Hunan University , National Center for Science and Technology Evaluation (NCSTE) , 2025. Elementary Tutorial Technical Manager. Science and Technology of China Press, Beijing, China (in Chinese).

[38] Jiang LK , Zhang CQ , Scherz PA , et al., 2025. Can large language models generate high-quality patent claims? Findings of the Association for Computational Linguistics, p.1272- 1287.

[39] Jung J , Lu M , Benker SC , et al., 2025. How model size, temperature, and prompt style affect LLM-human assessment score alignment. Proc Artificial Intelligence in Measurement and Education Conf, p.265- 273.

[40] Kampelopoulos D , Tsanousa A , Vrochidis S , et al., 2025. A review of LLMs and their applications in the architecture, engineering and construction industry. Artif Intell Rev, 58 (8): 250.

[41] Khera B , Alamian R , Scherz PA , et al., 2025. Can large language models understand as well as apply patent regulations to pass a hands-on patent attorney test?

[42] Kim J , Podlasek A , Shidara K , et al., 2025. Limitations of large language models in clinical problem-solving arising from inflexible reasoning. Sci Rep, 15 (1): 39426.

[43] Kim JS , Kim BK , 2022. Examining different technology transfer capabilities and their counterpart works from two different positions. Technol Soc, 68: 101856.

[44] Krathwohl DR , 2002. A revision of Bloom's taxonomy: an overview. Theory Pract, 41 (4): 212- 218.

[45] Lane JP , 1999. Understanding technology transfer. Assist Technol, 11 (1): 5- 19.

[46] Li HT , Chen Y , Ai QY , et al., 2024. LexEval:a comprehensive Chinese legal benchmark for evaluating large language models. Proc 38^th Int Conf on Neural Information Processing Systems, p.25061- 25094.

[47] Li JW , Gao YZ , Yang Y , et al., 2025. Fundamental capabilities and applications of large language models:a survey. ACM Comput Surv, 58 (2): 38.

[48] Li KY , Jiang MH , Fu DY , et al., 2025. DatasetResearch: benchmarking agent systems for demand-driven dataset discovery.

[49] Li SW , Jiang J , Yang XD , 2025. Preliminary assessment of large language models' performance in answering questions on developmental dysplasia of the hip. J Child Orthop, 19 (3): 207- 212.

[50] Li YJ , Sun Y , 2026. A fine-grained evaluation framework for language models:combining pointwise grading and pairwise comparison. Inform Process Manag, 63 (1): 104270.

[51] Liang C , Huang ZQ , Wang HF , et al., 2026. AECBench:a hierarchical benchmark for knowledge evaluation of large language models in the AEC field. Adv Eng Inform, 71: 104314.

[52] Lin CB , Liang HY , Su TY , et al., 2022. Multi-criteria decision-making approach for analyzing competency model of technology managers. Math Probl Eng, 2022 (1): 1909851.

[53] Liu AX , Feng B , Xue B , et al., 2025. DeepSeek-v3 technical report.

[54] Liu QY , Zou T , Xu CH , et al., 2025. Shanghai Service Manual about Technology Commercialization. Shanghai Jiao Tong University Press, Shanghai, China (in Chinese).

[55] Longpre S , Perisetla K , Chen A , et al., 2021. Entity-based knowledge conflicts in question answering. Proc Conf on Empirical Methods in Natural Language Processing, p.7052- 7063.

[56] MacMillan IC , McGrath RG , 2004. Nine new roles for technology managers. Res-Technol Manag, 47 (3): 16- 26.

[57] Manifold , 2026. Manifold Markets: The World's Largest Prediction Market Platform.https://manifold.markets/[Accessed on Jan. 20, 2026].

[58] Matrinex Academy of International Innovation & Strategy (MAIIS) , National Center for Science and Technology Evaluation (NCSTE) , 2025. Intermediate Tutorial Technical Manager.Science and Technology of China Press, Beijing, China (in Chinese).

[59] Meta AI , 2025. Llama 4 Maverick: sparse mixture-of-experts (MoE) architectures.https://ai.meta.com/blog/llama-4-multimodal-intelligence/[Accessed on Jan. 20, 2026].

[60] Miao ZY , Sun QY , Wang JY , et al., 2025. Easy Dataset:a unified and extensible framework for synthesizing LLM fine-tuning data from unstructured documents. Proc Conf on Empirical Methods in Natural Language Processing:System Demonstrations, p.960- 968.

[61] Mistral AI , 2025. Mistral Small 3.1: latency-optimized 24B parameters model.https://mistral.ai/news/mistral-small-3-1/[Accessed on Jan. 20, 2026].

[62] Moonshot AI , 2025. Kimi-K2-0905: scaling 1T-parameter MoE architectures.https://www.moonshot.cn/news/kimi-k2-announcement/[Accessed on Jan. 20, 2026].

[63] Morrissey MT , Almonacid S , 2005. Rethinking technology transfer. J Food Eng, 67 (1-2): 135- 145.

[64] NASA , 2020. Technology readiness assessment best practices guide.NASA Special Publication, NASA/SP-20205003605.https://ntrs.nasa.gov/citations/20205003605 [Accessed on Jan. 20, 2026].

[65] NASA , 2026. TechPort 101.https://techport.nasa.gov/help/TechPort101 [Accessed on Jan. 20, 2026].

[66] Naveed H , Khan AU , Qiu S , et al., 2025. A comprehensive overview of large language models. ACM Trans Intell Syst Technol, 16 (5): 106.

[67] Ni YQ , Huang JS , Song LL , 2024. Guidelines for Enhancing the Capacity of National Technology Transfer Professionals. University of Science and Technology of China Press, Hefei, China (in Chinese).

[68] Nie Y , Yan BW , Guo TY , et al., 2025. CFinBench:a comprehensive Chinese financial benchmark for large language models. Proc Conf of the Nations of the Americas Chapter of the Association for Computational Linguistics:Human Language Technologies, p.876- 891.

[69] OpenAI , 2026. GPT-5-mini technical report.https://openai.com/index/introducing-gpt-5-for-developers/[Accessed on Jan. 20, 2026].

[70] Park J , Mendes E , Stanovsky G , et al., 2026. Look before you leap: estimating LLM benchmark scores from descriptions.https://arxiv.org/abs/2509.20645v2

[71] Pei DS , Wu YW , He JH , et al., 2025. Methodology and benchmark for automated driving theory test of large language models. IEEE Trans Intell Transp Syst, 26 (10): 17013- 17026.

[72] Peters U , Chin-Yee B , 2025. Generalization bias in large language model summarization of scientific research. R Soc Open Sci, 12 (4): 241776.

[73] Pham DK , Vo BQ , 2024. Towards reliable medical question answering: techniques and challenges in mitigating hallucinations in language models.

[74] Pombal J , Yoon D , Fernandes P , et al., 2025. M-Prometheus: a suite of open multilingual LLM judges.

[75] Rahman ABMA , Anwar S , Usman M , et al., 2025. DefAn:definitive answer dataset for LLM hallucination evaluation. Information, 16 (11): 937.

[76] Reese ML , Smirnova A , 2024. Comparing ChatGPT and humans on world knowledge and common-sense reasoning tasks:a case study of the Japanese Winograd schema challenge. Extended Abstracts of the CHI Conf on Human Factors in Computing Systems, p.1- 9.

[77] Renze M , 2024. The effect of sampling temperature on problem solving in large language models. Findings of the Association for Computational Linguistics, p.7346- 7356.

[78] Rifkin KI , Fineman M , Ruhnke CH , 1999. The human side:developing technical managers-first you need a competency model. ResTechnol Manag, 42 (2): 53- 57.

[79] Rogers EM , Takegami S , Yin J , 2001. Lessons learned about technology transfer. Technovation, 21 (4): 253- 261.

[80] Saxena A , Aggarwal D , Badathala N , et al., 2025. A framework to synthetically generate fine-grained hallucinated data. Lang Resour Eval, 59 (4): 4223- 4252.

[81] Schaufeld J , 2015. The commercialization model: the value of a process. In: Commercializing Innovation: Turning Technology Break-throughs into Products.Apress, Berkeley, CA, p.17- 30.

[82] Schoenegger P , Tuminauskaite I , Park PS , 2024. Wisdom of the silicon crowd:LLM ensemble prediction capabilities rival human crowd accuracy. Sci Adv, 10 (45): eadp1528.

[83] Servantez S , Barrow J , Hammond K , et al., 2024. Chain of logic:rule-based reasoning with large language models. Findings of the Association for Computational Linguistics, p.2721- 2733.

[84] Sheft J , 2008. Technology transfer and idea commercialization. Nat Biotechnol, 26 (6): 711- 712.

[85] Song PY , Han PR , Goodman N , 2026. Large language model reasoning failures.

[86] Srivastava A , Rastogi A , Rao A , et al., 2023. Beyond the imitation game: quantifying and extrapolating the capabilities of language models.

[87] Sun LT , Han Y , Zhao ZH , et al., 2024. SciEval:a multi-level large language model evaluation benchmark for scientific research. Proc 38^th AAAI Conf on Artificial Intelligence, p.19053- 19061.

[88] Tak R , Pele DT , 2025. LLM-driven stock prediction:capturing market trends with LLaMA. Proc Int Conf on Busin Excell, 19: 529- 540.

[89] Tao YC , Xu HT , 2026. TheoremForge: scaling up formal data synthesis with low-budget agentic workflow.

[90] Ting YS , Nguyen TD , Ghosal T , et al., 2025. AstroMLab 1: who wins astronomy jeopardy!? Astron Comput, 51: 100893.

[91] Tofeeq K , Naseer A , Wali A , 2025. Large language models in healthcare:a systematic evaluation on medical Q/A datasets. Health Inform Sci Syst, 14 (1): 2.

[92] Toroghi A , Guo W , Sanner S , 2025. CoLoTa:a dataset for entity-based commonsense reasoning over long-tail knowledge. Proc 48^th Int ACM SIGIR Conf on Research and Development in Information Retrieval, p.3444- 3454.

[93] Vu T , Iyyer M , Wang XZ , et al., 2024. FreshLLMs:refreshing large language models with search engine augmentation. Findings of the Association for Computational Linguistics, p.13697- 13720.

[94] Wang MS , Li ZQ , Zhao XG , et al., 2026. Eliminate-then-select:a humancentric reasoning framework for educational question answering with LLMs. Inform Process Manag, 63 (2): 104422.

[95] Wang QY , Chen GH , Wang HB , et al., 2025. IPBench: benchmarking the knowledge of large language models in intellectual property.

[96] Wang RQ , Guo JY , Gao CY , et al., 2025. Can LLMs replace human evaluators? An empirical study of LLM-as-a-Judge in software engineering. Proc ACM Softw Eng, 2 (ISSTA): 1955- 1977.

[97] Wang XZ , Wei J , Schuurmans D , et al., 2023. Self-consistency improves chain of thought reasoning in language models.

[98] Wei KL , Paskov P , Dev S , et al., 2025. Recommendations and reporting checklist for rigorous & transparent human baselines in model evaluations.

[99] xAI , 2025. Grok 4.1 Fast and Agent Tools API.https://x.ai/news/grok-4-1-fast [Accessed on Jan. 20, 2026].

[100] Xu K , Lian ZY , 2026. WaferSAGE: large language model-powered wafer defect analysis via synthetic data generation and rubric-guided reinforcement learning.

[101] Xu LQ , Li QW , Peng TS , et al., 2025. Can large language models be good language teachers? Proc Conf on Empirical Methods in Natural Language Processing, p.23957- 23971.

[102] Yang QC , Mahns S , Li SD , et al., 2025. LLM-as-a-Prophet: understanding predictive intelligence with Prophet Arena.

[103] Yao SY , Yu D , Zhao J , et al., 2023. Tree of thoughts: deliberate problem solving with large language models. Proc 37^th Int Conf on Neural Information Processing Systems, p.11809- 11822.

[104] Yu JF , Wang XZ , Tu SQ , et al., 2024. KoLA: carefully benchmarking world knowledge of large language models.

[105] Zhang S , Huang QC , Zhang Q , et al., 2025. ElecBench:a large language model benchmark in electric power domain. Eng Appl Artif Intell, 162: 112310.

[106] Zhao XK , Huang H , Yang T , et al., 2025. Urban planning in the age of large language models:assessing OpenAI o1's performance and capabilities across 556 tasks. Comput Environ Urban Syst, 121: 102332.

[107] Zhao ZH , Huang ZX , Li JR , et al., 2025. SUPERChem: a multimodal reasoning benchmark in chemistry.

[108] Zheng LM , Chiang WL , Sheng Y , et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Proc 37^th Int Conf on Neural Information Processing Systems, p.46595- 46623.

[109] Zografos G , Moussiades L , 2025. Beyond the benchmark:a customizable platform for real-time, preference-driven LLM evaluation. Electronics, 14 (13): 2577.

Rights & permissions

The Authors. Published by Zhejiang University Press Co., Ltd.