1 Introduction
Big Data-based education evaluation is a more scientific, professional and objective approach to evaluating education, which integrates the concept of Big Data and related technologies into all elements and stages of education evaluation. This approach features of diverse subjects under education, inclusive evaluation contents, comprehensive and rapid data collection, scientific and efficient value analysis, and intelligent and precise evaluation feedback.
In 2020, the Central Committee of the Communist Party of China (CPC) and China’s State Council (2020) released the Overall Plan for Deepening the Reform of Education Evaluation in the New Era (hereinafter referred to as the Plan), which points out that information and communications technology (ICT) should be fully utilized to improve the scientificity, professionalism and objectivity of education evaluation. The document emphasizes the need to innovate evaluation tools and use modern information technologies such as artificial intelligence (AI) and Big Data.
In March 2022, to implement the Plan, Minzu University of China (MUC) made efforts to establish a quality contribution-oriented education evaluation system, achieve significant progress in crucial areas including school, teacher, student, and employment evaluations, and build a multi-dimensional effectiveness evaluation system that reflects connotation and characteristic development (
Li & Wu, 2022).
Against the above background, this study intends to build a Big Data-based evaluation model of education, elucidate its key components and associated technologies, and put forward application suggestions tailored to the specific context of MUC. The present study aims to contribute to some extent to the ongoing reform of higher education evaluation, serve the high-quality development of higher education, and promote the digital transformation and intelligent upgrade of higher education evaluation.
2 Literature Review
Previous studies have made explorations into the significance, functions, problems, and paths of applying Big Data to education evaluation, which are summarized as follows: First, Big Data is of great value and role in implementing the Plan. The evaluation and decisions made based on Big Data is more convincing and credible, contributing to enhancing people’s satisfaction with the education evaluation reform (
Song et al., 2021). Big Data can empower the education evaluation with various forms such as result evaluation, process evaluation, value-added evaluation, and comprehensive evaluation (
Liu et al., 2021). Second, Big Data will change the elements and processes of education evaluation. The subject of evaluation will change from single to multiple. The content of evaluation will extend to the entirety students’ learning processes (
Tian et al., 2022), and all the elements of moral grounding, intellectual and physical ability, aesthetic sensibility, and work skills. The targets of evaluation service will cover all groups or individuals who care about and participate in education. For evaluation data collection, leveraging technologies such as Internet of Things (IoT), video recording, image recognition and platform collection allow for completely automated, end-to-end, comprehensive sampling, and inclusive data collection without any loss (
Qiu et al., 2021). In the analysis of evaluation data, utilizing data fusion, data mining and other technologies enables multi-dimensional and global data analysis and accurate evaluation (
Zhu & Ma, 2019). In terms of feedback on evaluation results, through technologies such as personalized customization and intelligent recommendation, the evaluation results can be promptly and precisely delivered to those evaluated in the form of interactive charts, which may effectively improve the self-awareness of the evaluation object, help them to adjust and optimize its goals and strategies in time, and thus effectively promotes teaching, learning, and management (
Zhu & Yan, 2018). Third, some problems exist in the implementation of Big Data-based education evaluation, such as uneven development of regional informatization, data islands, inconsistent standards, and the inproficiency of teachers’ data literacy. Evaluation data are often stored in multiple platforms and institutions, with different data structure standards and uneven data quality. Chinese teachers do not have a strong awareness of data utilization, do not have enough keen insight into data, still use outdated concepts to guide data processing activities, and even doubt the role and value of Big Data (
Song et al., 2021). Fourth, since 2001, the U.S. Department of Education has implemented data-driven practices for education evaluation, instructional improvement, and education decision-making, and established a nationwide education data system, significantly enhancing the users’ data literacy in education, and providing support and guarantee for teachers’ data application from a variety of perspectives, including policy, technology, and research (
Zheng & Liu, 2015).
In summary, previous studies can inform research on the direction of Big Data-based education evaluation. However, many problems still need resolution: First, the heterogeneity of data within the education system poses challenges to integration. As pervious studies have explained, high-quality education evaluation relies on a comprehensive and effective foundation (
Song et al., 2021;
Zhu & Ma, 2019). However, due to the uneven development of education informatization in China, information islands are widespread. According to incomplete statistics, there are more than 200,000 information platforms in China’s education system, in which data about educational elements and educational processes are scattered. The integration of heterogeneous data from these platforms is almost impossible. Second, although the framework for Big Data-based education evaluation has been established, the key technologies for evaluating each link of data processing have not been verified yet. Previous studies have assumed the expected characteristics of each link of evaluation data processing, but lack of detailed demonstrations regarding technical implementation. Determining the calculation model and dataset for data processing requires repeated verification and optimization, and engineering solutions are necessary for integration, enabling a process-oriented and automated approach. Third, the effect of Big Data-based education evaluation has not been tested in practice. Previous studies focused on the practical application of Big Data-based education evaluation, leaving its performance in terms of scientific rigor, professionalism, and objectivity largely untested.
3 Model Construction
The Big Data-based evaluation model of education includes the construction of an element model and a process model.
3.1 Element Model
As shown in Fig.1, an element model of Big Data-based educational evaluation is built around “who will evaluate,” “what to evaluate,” “when to evaluate,” “how to evaluate,” and “why to evaluate.”
Fig.1 Element model of Big Data-based education evaluation. |
Full size|PPT slide
The essential characteristics of Big Data-based education evaluation includes multiple evaluation subjects, comprehensive evaluation contents, process-oriented evaluation methods, diverse evaluation methods, and rich evaluation functions. Evaluation subjects encompass of teachers, students, parents, and experts. The evaluation content integrates knowledge, skills, and emotions, covering the entirety of students’ learning in all grades and all terms of moral grounding, intellectual and physical ability, aesthetic sensibility, and work skills. The evaluation method focuses on the process and accompanying evaluation, relies on the acquisition and analysis system of Big Data to achieve the whole, full, and automatic lossless acquisition of multi-source, multi-dimensional, heterogeneous and real-time changing data, and enables the optimization of multi-dimensional and global data processing and analysis (
Zhu & Ma, 2019). The evaluation function aims at not just screening and selection, but promoting students’ development.
3.2 Process Model
This study constructs a process model framework of Big Data-based education evaluation.
As shown in Fig.2, the main features of the process model of Big Data-based education evaluation including: full-time, global, and lossless data acquisition; multi-dimensional, intelligent, and fusion data analysis; and intuitive, personalized, and accurate result feedback. To ensure data acquisition, various multi-modal information perception technologies play a pivotal role, including IoT, speech recognition, optical character recognition (OCR), and document analysis. In addition, it is necessary to ensure that all data acquisition points and data acquisition equipment can work continuously and stably. Regarding data analysis, the key lies in the fusion of multimodal data (see Fig.3), which involves the fusion of video data from the classroom, structured log data of the teaching platform, physiological data of teachers and students, and self-reported data of teachers and students. It is considered that experience API (xAPI) data specification can be used for multimodal data fusion. In the result feedback, information generation is the key, which include abstract generation, chart generation, report writing, and personalized push.
Fig.2 Process model framework. |
Full size|PPT slide
Fig.3 Multimodal education data. |
Full size|PPT slide
3.3 Evaluation System Design
Based on the above two models, a Big Data-based education evaluation system can be further designed. The Big Data-based education evaluation system is divided into 3 parts, including the infrastructure layer, technology layer, and application layer (see Fig.4). It is planned to use agile methods to develop the system rapidly and iteratively in the application process.
Fig.4 Framework of Big Data-based education evaluation system. |
Full size|PPT slide
The infrastructure layer is mainly filled with computing resources and storage resources. The former comprises elastic cloud servers and distributed computing frameworks, while the latter includes relational databases for structured data and object storage for unstructured data. In the technical layer, common technologies of education evaluation are mainly used, such as the technical middle platform, including data acquisition, data governance, cognitive computing, multimodal analysis, intelligent output, and visualization components. In the application layer, there are mainly functional modules related to the evaluation business, including evaluation project management, evaluation tool management, and evaluation index design, evaluation data acquisition, evaluation data analysis, evaluation result feedback and other modules corresponding to evaluation preparation, implementation and feedback, etc.
4 Typical Application
MUC makes full use of new-generation information technologies such as Big Data, IoT, and AI, and facilitates the deep integration of information technology into school operations, merging virtual space and physical space to build a smart campus (
Li & Wu, 2022). The smart teaching system captures every learning step of every student’s learning journey from enrollment to graduation, generating personalized profiles based on their learning process and outcomes. School administrators can learn about the teaching, student management, and logistics support of the whole school via the school data portal. To date, the MUC has accumulated a dataset and provided initial data application support. Efforts are underway to design additional application scenarios and continually develop new micro-applications based on the aforementioned Big Data-based education evaluation framework.
4.1 Application Design
According to the characteristics of digital transformation of education, the construction and application of digital education will encompass the entire spectrum of teaching, management, research, and service; the whole process of lesson preparation, teaching, and evaluation; and all elements such as teachers, students, and courses. Against the background of digital transformation of education, education evaluation is now capable of comprehensive evaluations, full process evaluation, and full factor evaluation. Some typical diagnostic evaluation projects are shown in Fig.5.
Fig.5 Design of Big Data-based education evaluation and diagnosis project. |
Full size|PPT slide
Next, the teaching evaluation of teachers and students in the whole process evaluation, the early warning of teachers’ appointment term assessment, and the early warning of students’ abnormal behavior in the all-factor evaluation will be further introduced.
4.2 Evaluation of Teaching and Learning
Teachers and students are the main evaluation objects of the Big Data-based education evaluation system. In the evaluation of teachers, teaching evaluation and work evaluation are the main evaluation contents. Furthermore, teacher-oriented teaching diagnostic evaluation is the core of the Big Data-based education evaluation. This section will mainly discuss the application of classroom teaching diagnosis and online teaching diagnosis.
MUC has deployed 112 smart classrooms in its new campus. Each smart classroom is outfitted with automatic recording and broadcasting features, enabling simultaneous recording of teachers’ lectures and students’ learning. Based on these video data, students’ behavior statistics, classroom attendance, teachers’ behavior statistics, teachers’ teaching type analysis, RT-CH analysis, S-T teaching analysis and teacher–student activity analysis can be carried out, which can provide data support for teaching analysis and assist teachers to improve their teaching process.
With the help of the “Chaoxing Xuexitong” application, the interaction between teachers and students extends beyond the classroom. Behavior logs during this process for teachers and students are recorded for further analysis on online teaching and learning behaviors, temporal-spatial behavioral characteristics, interactive characteristics of teachers and students, and activity paths of teachers and students.
By integrating classroom records, online platform data, and final course scores, comprehensively evaluation can be made on teachers’ teaching input, students’ learning input, and the achievement of course goals, and predict students’ course scores and academic progress, so as to achieve early warning, early intervention, and early improvement.
4.3 Early Warning of Teachers’ Appointment Term Assessment
MUC applies the 3-year appointment term system to assess teachers. The assessment comprises four major items and ten minor items, including discipline and team building, teaching and student guidance, research and social service. However, often by the term’s end, through the assessment it discovers discrepancies in meeting certain standards, which are difficult to rectify within the stipulated period. Therefore, it is necessary to integrate the appointment term assessment into the regular monitoring, comprehensively collecting the data of four major items and ten minor items. This entails primarily consolidating existing data from the educational administration platform, the postgraduate management platform, and the scientific research management platform, and supplementing it with data related to social services. The monitoring module calculates the progress and predicts the passing rate for the appointment term assessment at the end of the first, second year, and the middle of the third year respectively. A warning will be sent to the teacher, their affiliated college, and the human resourses department if the predicted passing rate is low. These measures aim to enhance the teachers’ work accomplishments during their employment tenure, and decrease failure rate of the appointment term assessment.
5 Practical Path
To adhere to the principles of application-oriented and service-oriented, it is necessary to explore the application scenarios of Big Data-based education evaluation, continuously accumulate data, expand data sharing, and maximize the utilization of data elements. Specifically, universities should fully apply all available data types in the assessment and evaluation of various works, emphasize evidence-based evaluation, and implement evaluations and applications one by one in the form of a micro-service approach. Secondly, it is essential to develop and improve the management methods for school affairs data, and establish a fundamental principle of comprehensive sharing within educational institutions, with the exception of non-sharing. While maintaining the security of school affairs data, all departments should provide all kinds of information data sharing services within the scope of their duties, so that the data can meet the needs of various works and support the decision-making process. Thirdly, the establishment of Big Data centers, comprising public data platforms including data standard management systems, data exchange systems, and public databases is essential to accumulate school Big Data and maintain continuous data governance while providing high-quality data-sharing and exchange services.
To adhere to the principle of leading by example, it is necessary to carry out teacher training and improve teachers’ data literacy constantly. Teachers are the designers of teaching activities and the main producers of teaching data. Teachers’ data literacy determines the depth and breadth of application of Big Data-based education evaluation and the degree of mining of educational data elements. Practically, universities should organize training regularly, take the case of Big Data-based education evaluation as the traction, and adopt the output-oriented training concept to develop teachers’ awareness, knowledge, skills, and ethics of data in the on-the-spot training, and improve their ability of using education Big Data to evaluation. Besides, universities should establish dedicated education reform project to guide teachers in conducting research and applying practice of Big Data-based education evaluation.
To adhere to the principle of safe operation, prioritizing data security is imperative because it is the base of data-based teaching reform and the demonstration of the value of data. Universities should strictly abide by the requirements of data security-related laws and regulations, implement data security protection measures, and improve their ability to prevent hacker intrusion, data leakage, data abuse, and data destruction. Moreover, data security protection need to strengthen from three technical aspects: data integrity, data confidentiality, and data backup and recovery.
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
PDF(6340 KB)
Fig.1 Element model of Big Data-based education evaluation.
AI Summary
