Introduction
Background
With the development of the Internet of Things (IoTs), cloud computing, wearable devices, and social media, big data has become ubiquitous. At present, the global data volume is growing exponentially. Advancements in technologies such as cluster computing and cloud computing have made storage, analysis, sharing, and distribution of big data easier and cheaper (
Wani and Ashtankar, 2017). Many companies have realized the huge business value contained in big data and combined their respective advantages to use it. For example, Amazon analyzes customers’ previous orders, shopping cart information, browsing history, store collections, and other information to predict whether to order so that goods can be shipped to nearby areas of potential customers in advance to reduce shipping time. ZARA timely feeds back to its designers and production departments a large amount of customer feedback collected from its online store and experience information from offline brick-and-mortar customers to adjust product style and output. The use of big data technology to provide support for management decisions has become a hot topic in current management research.
As one of the most important functions of management, evaluation is a process of measuring the attributes of things and giving reliable conclusions by some data and methods based on certain goals and standards. In a big data environment, the data has the unique features of “massive, high dimensional, heterogeneous, unstructured, incomplete, noisy, and erroneous”. The traditional evaluation method is suitable for limited static sample data, and it is difficult to adapt to the big data with high-frequency dynamic changes. However, evaluation has a positive value. First, evaluation can directly support management decisions, such as organizational performance evaluation, employee performance evaluation ratio, asset value evaluation, academic evaluation, qualification evaluation, credit rating, etc. Second, evaluation can provide information support for other methods and indirectly assist in complex management decisions, such as risk assessment for insurance pricing decisions, reliability assessment for equipment management, customer perceived value assessment for marketing decisions, and health assessment for diagnosis and treatment decisions. Therefore, evaluation methods in a big data environment need to be studied urgently.
Furthermore, evaluation is widely used in the improvements of various systems, for example, the performance evaluation of the supply chain system, the strategic analysis of the inventory system, the management of the risk system, the sales forecast of the marketing system, the ranking recommendation of the online platform, assessments of disease diagnosis and treatment in medical systems, the road prediction of the transportation system, and safety assessment of fire protection systems. Combining big data analysis technology can have a more scientific understanding of system evaluation, thereby improving the system and creating better benefits. There are still some difficulties and challenges in the application of system evaluation and decision making in the big data environment. How to take advantage of the development opportunities brought by the big data environment, research the system evaluation work in-depth, and explore the system evaluation and improvement methods in the big data environment has important theoretical and practical significance.
At present, comprehensive analysis and research of evaluation methods and systematic improvement in a big data environment are lacking. This article reviews the systematic evaluation methods and improvement researches in the big data environment from four aspects. First, we explored and reviewed the evaluation methods based on big data analytics techniques. Second, focused on the different data characteristics, we reviewed the evaluation methods based on the characteristics of big data. Third, we focus on the improvement methods of different systems in the big data environment. Fourth, the application areas of assessment activities in the big data environment are given. Finally, we summarize the new application areas of evaluation methods in the big data environment and the future research directions of evaluation methods. Through the research in this article, we hope to break through the limitations of traditional evaluation, have a clear understanding of the current state of system evaluation and improvement research in the big data environment, and provide a reference for future researchers.
Literature review search strategies
A search within the timeframe ranging from 2011 to 2019 was considered to represent the period covering the emergence of “big data”. We first focus on searching via the following databases: Informs, Elsevier, Taylor and Francis, Wiley, and IEEE Xplore. We use major keywords such as “big data, data-driven, data analytic techniques, evaluation methods, assess, assessment, systematic improvement, data mining, statics, optimization, simulation, deep learning, association rules, missing data, data loss, data noise, visualization, systematic assessment” to search related literature, supplemented by secondary keywords like “clustering, classification, operations, supply chain management, inventory management, risk assessment, risk analysis, marketing, forecast, system”. Based on our experience in the field and related research in the literature, we also get some additional papers. We focus on a list of journals that are considered to be the leading journals in the operation and management fields: Management Science, Journal of Operations Management, Operations Research, Production and Operations Management, Decision Sciences, Manufacturing and Service Operations Management, Marketing Science, Information Systems Research. The search results show that the evaluation methods are widely used in business value, retail forecasting, risk analysis, inventory management, transportation route planning, consumer preferences, product categories, and disease forecast and assessment.
Our search started on November 13, 2019, and ended on December 25, 2019. Regarding the literature retrieval process, we take the literature review in Section 2.1 as an example. Specifically, we first focused on searching via Informs advanced search using major keywords such as “big data, data mining”, supplemented by secondary keywords like “clustering, classification, association rule”. For example, when we input primary keyword “data mining” and the timeframe ranging from 2011 to 2019, the initial search resulted in 32 articles. These references, including the abstracts of all articles, have been downloaded into Endnote, a reference management software package, for further analysis. A co-author then conducted the screening of the abstract of each article to assess its relevance with our research goals. We obtained the related literature (Lutu and Engelbrecht, 2013;
Bai et al., 2015;
Das et al., 2016;
Liu et al., 2016b). Afterward, secondary keywords “clustering, classification, association rule” were input and several joint papers were held (
Jagabathula et al., 2018;
Kopcso and Pachamanova, 2018;
Roy et al., 2019). Subsequently, supplement with Google Scholar searches, we found the relevant reference (
Hastie et al., 2005;
Choi et al., 2018;
Sato et al., 2019). At the end of this process, 11 articles were deemed relevant for evaluation methods based on data mining. The literature search process in other parts is similar, the only difference is that the literature search platform and the concerned journal are different. For example, Section 3 used Google Scholar more. In Sections 4 and 5, more articles are retrieved in Taylor and Francis, and Wiley.
Evaluation methods based on big data analytics techniques
Big data processing techniques are different when it comes to data from different sources. For example, data can come from mobile devices, the web, social media, and cloud platforms, and their formats can be text, graphic, images, and videos. Therefore, there are terms of text analytics, web analytics, social analytics, and multimedia analytics (
Hu et al., 2014). Big data techniques involve a number of disciplines, including statistics, data mining, machine learning, neural networks, optimization methods, and visualization approaches (
Chen and Zhang, 2014). There are many specific techniques in these disciplines, and they overlap with each other to some extent. Traditionally, data mining and deep learning techniques are widely used in forecasting, revenue management, marketing, and risk analysis. Statistical methods are powerful tools for business intelligence analytics (
Sivarajah et al., 2017). Optimization and simulation are widely accepted tools to improve system performance, applying to inventory management and supply chain analytics (
Wang et al., 2016). We focus on the application of big data analytic techniques in business intelligence, supply chain management, and operations management. Similar to the literature of
Choi et al. (2018), we overview the evaluation methods in the big data environment based on four analytic techniques, including data mining, statistics, optimization and simulation, and deep learning.
Evaluation methods based on data mining
Data mining is to extract information from a data set and transform it into an understandable structure for further use. Commonly used data mining methods are classification, clustering, association rule analysis, estimation, and prediction (
Hastie et al., 2005). Data mining plays an important role in business intelligence and big data analytics (
Choi et al., 2018).
Focused on the problem of product assortment,
Bai et al. (2015) present a method for assortment planning and optimization for multiple stores that can identify the optimal product classification for each store and allow analyses of classification efficiency assessments in all existing stores. Their methodology offers solutions on product assortment for complements versus substitutes and conducts sales efficiency evaluation and assortment optimization. Considering the potential values from online social platforms, such as Google Trends, Twitter assessments, IMDb (Internet Movie Database) reviews, Wikipedia views, and Huffington Post news,
Liu et al. (2016b) conduct massive analyses on of nearly 2 billion Tweets and 400 billion Wikipedia pages and conclude that extracting and sorting from information of online platforms can reflect consumer intent in a timely manner, which has critical implications for forecasting purchases.
Kopcso and Pachamanova (2018) design an example where predictive analytics is used to determine the input to a customer service specification model. They then illustrate how calculations of business value for business stakeholders. They evaluate the level of an organization’s maturity by using a predictive and prescriptive analytics model.
Jagabathula et al. (2018) develop a method based on the embedding technique that takes the customer’s observations and probability classes generating the observations as inputs and outputs the embedded results for each customer. They show that this method outperforms empirical Bayesian, standard latent class, and demographic-based techniques.
In addition, the evaluation methods based on data mining include quantitative attributes based on text mining (
Das et al., 2016), positive-versus-negative (pVn) classification (
Lutu and Engelbrecht, 2013), a method for evaluating different platforms (
Roy et al., 2019), and knowledge discovery tool (
Sato et al., 2019). The relevant evaluation methods in this section are reviewed in Table 1.
Evaluation methods based on statistics
Statistics, as the most fundamental technique for data analysis, exist in almost all subject areas of research. The statistical methods commonly used in management evaluation include regression analysis, maximum likelihood estimation, Bayesian estimation, and Markov stochastic process. With the maturity of big data analysis technology, evaluation work based on statistical methods is also increasing.
To improve the accuracy of sales predictions in the tire industry,
Sagaert et al. (2018) propose a forecasting method, which can automate the identification of key leading indicators, driving to generate accurate forecasts. In their case study, the accuracy of their proposed method improves by 16.1%. Furthermore, this method also can handle external indicators of short-term and long-term dynamics. In analysis management,
Jiang et al. (2019) propose a logistic regression model using data generated in past simulation experiments to estimate portfolio risk and classifying portfolio risk levels in real-time. They show that the simulation analytics idea is viable and promising in the field of financial risk management.
Chehrazi and Weber (2015) construct a dynamic collectability score (DCS) that can be used to estimate the probability of delinquent credit-card accounts. The DCS framework is applied to a large set of account-level repayment data. Compared to standard bank-internal scoring methods, the DCS framework has significant improvements in classification and prediction performance. A new stochastic variational Bayesian (SVB) approach is used to estimate movie ratings and semantic tags by a large data set. The approach is very useful in actual recommendation contexts (
Ansari et al., 2018). Other evaluation methods are used in transport risk management (
Shang et al., 2017), ranking and selection (
Salemi et al., 2019), audit quality (
DeFond et al., 2017), project portfolio optimization (
Yang et al., 2015), and stock-keeping unit (SKU)-clustering problem (
Park et al., 2017). The relevant evaluation methods are shown in Table 2.
Evaluation methods based on optimization and simulation
The evaluation of the algorithm is mainly considered from the time complexity and space complexity. The optimization algorithms commonly used in management evaluation mainly include gradient descent method, simulated annealing method, Newton method, and quasi-Newton method (
Simon, 2013).
Considering the problem of travel time and routing plan,
Bertsimas et al. (2019a) leverage a simple approach to solve the travel time estimation and route planning problem in a real-world situation. Given travel times for any number of origin-destination pairs, the method can estimate the travel time as well as provide a sensible path associated with this travel time. Their algorithm is robust against a high input uncertainty and can successfully exploit noisy data to provide results characterized by their accuracy. Based on flow procedures,
Hochbaum (2018) proposes a combinatorial method to solve efficiently the classification problem as a network flow problem on a graph, which has higher accuracy and shorter running time in pattern recognition, image segmentation, and general data mining. Focused on the design of mechanisms for a sequencing problem,
Hoeksma and Uetz (2016) combine an exponential size linear programming with a convex decomposition algorithm to find the optimal linear programming solutions. Increasing the integration of local generators is a challenge in the planning, design, and operation of the distribution system.
Naghdi et al. (2018) present a quasi-Newton trust-region algorithm to evaluate the planning, design, and operation of the distribution system. Two networks were used for testing, and the obtained results revealed the accuracy and validity of the proposed method.
Huang et al. (2019) develop a novel two-stage data-analytic method that can serve as a template for modeling customer-firm interactions. The application of a new method can improve decision making in real-time. Their paper is one of the first studies to examine the evolution of player participation based on motivational factors using observational data.
Other evaluation methods based on optimization algorithms and simulation include stochastic annealing (
Ball et al., 2018), reserving relief supplies for earthquake (
Yang et al., 2016b), the routing optimization algorithm (
Bertsimas et al., 2019b), and evaluation of recommender systems (Adomavicius and Zhang, 2016). The relevant evaluation methods are reviewed in Table 3.
Evaluation methods based on deep learning
Deep learning is an algorithm based on representational learning of data in machine learning. Typical deep learning models include the convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM) network and more. Deep learning also brings many development opportunities for evaluation work.
To explore the nature of intertemporal cross-product patterns,
Xia et al. (2019) propose a conditional restricted Boltzmann machines (CRBM)-based model in an enormous consumer purchase data set. By using the proposed model, retailers can potentially capture and predict each consumer’s complex shopping patterns with greater accuracy for personalized marketing. Focused on the quality assessment of Wikipedia,
Wang and Li (2019) select state-of-the-art deep-learning models to conduct quality evaluation from classification performance and training performance and validate the effectiveness of the proposed model.
Considering finance risk,
Borovkova and Tsiamas (2019) propose a long short-term memory (LSTM) neural network for intraday stock predictions by many technical analysis indicators. They evaluate the predictive power of their model on several US large-cap stocks and find the proposed model has better performance than the benchmark models or equally weighted ensembles. Accurate prediction of forex rates is an essential element of an effective response to hedging or speculation strategies in the forex market.
Galeshchuk and Mukherjee (2017) explore the ability of deep convolution neural networks to forecast the direction of forex rates change, finding that trained deep networks can achieve satisfactory prediction accuracy. In addition, a hybrid architecture based on deep learning (
Amorin et al., 2019), the impact of personality similarity on subsequent purchase (
Adamopoulos et al., 2018), and a deep-learning approach to identify customers’ needs (
Timoshenko and Hauser, 2019) are studied. The relevant evaluation methods are shown in Table 4.
Evaluation methods based on big data characteristics
Big data are huge and complex which are difficult to process using traditional data-warehousing tools (
Kalbandi and Anuradha, 2015). Organizations often collect data from internal and external sources. Internal sources usually provide data related to their internal operations and business processes and external sources are provided by suppliers, retailers, customers, and market information (
Geczy, 2014). Data from different sources are often interconnected. In this paper, we focus on the characteristics of big data include the association features between data, data quality (data loss and data noise), and visualization features.
Evaluation methods based on data association features
Association rule is a way to discover the hidden relationships between variables in large databases (
Agrawal et al., 1993). Traditionally, input data sets can come from mobile devices, the web, social media, and cloud platforms, and their formats can be text, graphics, images, and videos. The premise of systematic evaluation and improvement in a big data environment is to understand the features of input data sets. If the features of the input data set are obvious, they can be modeled and analyzed by traditional statistical and econometric methods (
Chen and Zhang, 2014). However, it is not obvious which features should be input. There are many methods for data feature selection (
Bennasar et al., 2015;
Abedinia et al., 2017;
Ambusaidi et al., 2016). Association rules are widely used to discover hidden patterns from large databases and find interesting knowledge and information. This makes sense for nominal features and constructing full-fledged models.
Cang and Yu (2012) developed a fitness function based on association rules, which has been shown to be effective for input feature selection. It systematically improves the generalization ability of the evolution model. Based on the data association features, the management field has also carried out a large number of evaluation activities.
Focused on customer churn problem in retail sales,
Aung et al. (2019) apply the FP (Frequent Pattern)-Growth method to the customer churn data set. They develop a customer churn prediction model to help the retail company to make decisions on estimating the loss of clients or the promotion activities.
Zhang et al. (2019) propose an improved method based on association rule to evaluate energy efficiency, and they show that the proposed approach is effective in outlier identification and data transformation. Considering a large amount of associated data in the audit business,
Parkinson et al. (2016) develop a novel method of modeling file system permissions to evaluate auditing efficiency. Their method can correctly identify irregularities with an average accuracy rate of 91%, minimizing the reliance on expert knowledge.
Association rule-based evaluation is also used in many other fields, including risk management (
Bhatia, 2019), the electric vehicle data anomaly detection clinical (
Wang and Wu, 2019), and bioinformatics (
Boudellioua et al., 2016). Besides, some researches focus on developing new mining methods for association rule (
Feng et al., 2016;
Czibula et al., 2019). The relevant evaluation methods are shown in Table 5.
Evaluation methods considering data loss
Due to technical, human, and user privacy reasons, there is often a large number of missing data. The lack of data will hinder the subsequent analysis of big data and then affect decision-making.
Soley-Bori (2013) proposes basic concepts and methods for dealing with missing data. After explaining the missing data mechanisms and missing patterns, the author reviews some main conventional methods in data analytics, including imputation methods, multiple imputation, maximum likelihood, Bayesian methods, and listwise deletion. Advantages and limitations are listed so that the reader can identify the main trade-offs when using each method. Likewise,
Graham et al. (2012) review the methods for handling missing data from the view of psychology.
Little and Rubin (2019) review some methods for handling missing data, including imputation, multiple imputations, and maximum likelihood.
Clinical data sets often suffer from high missingness, which seriously impacts the diagnosis and prediction of disease. Imputing missing values provides an opportunity to resolve the issue. Based on imputation methods,
Wu et al. (2019b) propose a machine learning method that can improve the quality of breast cancer datasets. The results reveal that the proposed method gains strong robustness and discriminant power even the data set experiences a high missing rate (>50%). It always happens the loss of relevant information when aggregating the high-frequency traffic collision data into the lower frequency.
Li et al. (2019) propose a vector auto-regression (VAR) approach to evaluate traffic collision and they show that the proposed VAR demonstrates better performance than other missing value imputation techniques. In evaluating the impacts of products and processes,
Moreau et al. (2012) develop a statistical approach to carry out a life cycle assessment. The authors show how missing data of material and energy flows to evaluate the hydropower plants.
Jia and Wu (2019) use Monte Carlo simulation to assess the five methods for dealing with data loss and show robust full information maximum likelihood (RFIML) and MI-LV (multiple imputation-latent variable) combined with cat-DWLS (diagonally weighted least squares) seemed the best methods. In addition, evaluation activities based on missing data are also used in many other fields, including clinical endpoint bioequivalence (
Lou et al., 2019), estimating men’s fertility (Dudel and Klüsener, 2018), and power system (
Yang et al., 2020). The relevant evaluation methods are shown in Table 6.
Evaluation methods considering data noise
There are often various interference factors in the data collection process, which makes noise in the original data we obtain (
Ilow and Hatzinakos, 1998). Noise can affect the data analysis to varying degrees. Many big data analysis techniques use algorithm iteration to obtain the optimal solution. If the data set contains a lot of noisy data, it will greatly affect the convergence speed of the data and the accuracy of data analysis.
Recent evaluation work based on noisy data has also made some progress. Principal component analysis (PCA) is one of the powerful dimension reduction techniques, but PCs are still contaminated with noise in the data. Rezghi and Obulkasim (2014) proposes a noise-free PCA (NFPCA) method by introducing regularization to mitigate the effect of noise. And the authors show that NFPCA produces highly informative than the ordinary PCA method and it has a lower computational cost. Leveraging the encoder-decoder framework for neural machine translation,
Zoph et al. (2016) propose a transfer learning to assess the performance of machine translation. The results show the transfer learning mode can improve the syntax based on the machine translation by an average of 1.3 BLEU (Bilingual Evaluation Understudy).
Van Vliet and Salmelin (2020) present a framework that decomposes the weight matrix of a fitted linear model into three subcomponents and develop a post-hoc modification of linear models. They show that the decoding accuracy of two example linear models can be boosted by incorporating the information. Furthermore, the relevant evaluation methods including a self-organizing incremental neural network approach (
Wiwatcharakoses and Berrar, 2019), a new modeling and two-dimensional mapping approach (
Ball et al., 2018), a Bayesian approach to online robust parameter design (
Huang et al., 2017), and an evaluation of typical flow cytometry (Cao and Grima, 2019). The relevant evaluation methods are reviewed in Table 7.
Evaluation methods based on visualization (convert big data into small data)
The term “big data” is related to machines, while “small data” is related to people. Small data are data that is easy to access and the process in capacity and format contains useful information and is understandable by humans (
Kitchin and Lauriault, 2015). A common method to transform big data into small data is to visualize big data, such as histograms, violin plots, heat-maps, and scatter plots (
Gatto et al., 2015).
With the improvement of data availability, using big data to help managers make scientific decisions has become a trend.
France and Ghose (2016) introduce a statistical likelihood to evaluate visualizing submarkets in product categories. A series of experiments show their method is better at identifying market structure than other methods described. Likewise,
Ringel and Skiera (2016) also focus on the visualized competitive structure in large markets. They integrate large-scale data into new modeling and two-dimensional mapping methods, enabling users to visualize asymmetric competition in large markets and identify different submarkets. An empirical application of the LED-TV market with 1124 products and 56 brands resulted in valid and useful insights and showed that their method outperforms traditional models. Aiming to establish a new approach identifying cultivars of Chrysanthemi Flos (CF),
Nie et al. (2019a) develop a multimodal quantitative method combining principal component analysis (PCA) and similarity evaluation system (SES) to identify four cultivars of CF. The results show that the comprehensive method is effective. Furthermore, other evaluation methods based on data visualization can be seen in the literature
Nie et al. (2019b)and
Rajwan et al. (2013). The relevant evaluation methods are shown in Table 8.
Systematic improvement methods in the big data environment
With the rapid development of big data technology, massive data information can be tracked, collected, and utilized. The analytics techniques can provide new knowledge on their own without human intervention, helping decision-makers understand and predict consumer behavior (
Dhar, 2013). Studying the improvement methods of different application systems is of great significance to promote system evaluation and optimization. In this section, we focus on systematic improvement methods under big data processing techniques and information characteristics and review some hot operations management problems.
Systematic improvement methods based on big data processing
With the widespread applications of smart service systems in the fields of home, transportation, energy, and healthcare sectors,
Lim and Maglio (2018) combine metrics and machine learning algorithms to preprocess and analyze text data from smart service systems. Based on an analysis of 5378 scientific articles and 1234 news articles, they establish a common evaluation ground for understanding modern service systems. Considering current ranking algorithms in social media platforms ignore consumers’ multidimensional preferences for products,
Ghose et al. (2012) propose a new method to improve the hotel system ranking. They use a real data set from a website and show the qualitative comments are the first step in text mining.
Liu et al. (2016b) conduct an extensive analysis of nearly 2 billion Tweets and 400 billion Wikipedia pages, concluding that extracting and sorting from the information on online platforms can provide a timely representation of consumer intentions, which has important implications for forecasting purchases.
Statistics is a mature area whose purpose is to provide a scientific framework for collecting, analyze, and conclusion (
Choi et al., 2018).
Distelhorst et al. (2017) analyze an intervention by Nike to promote the adoption of lean manufacturing in its apparel supply chain across 11 developing countries. They find that lean manufacturing and high involvement work practices can improve social performance by estimating from a panel of more than 300 factories.
Ramasubbu and Kemerer (2016) analyze the impact of technical debt on system reliability by utilizing a large-scale longitudinal data set. Their empirical results illustrate how firms could evaluate business risk exposure due to technical debt accumulation, and they also assess the estimated net effects.
Bai et al. (2012) provide an effective and viable means for managing the risk associated with data quality in accounting information systems. Compared to previous approaches to data quality risks, their methodology is more cost effective and easier to implement.
As a standard analytical approach, optimization and simulation can solve the optimal (or near-optimal) solutions in quantitative decision-making problems.
Naghdi et al. (2018) present a quasi-Newton trust-region algorithm to evaluate the planning, design, and operation of the distribution system. Two networks were used for testing, and the obtained results revealed the accuracy and validity of the proposed method.
Ansari et al. (2018) develop a new stochastic variational Bayesian (SVB) approach for scalable estimation and used it to estimate movie ratings and semantic tags by a large data set. Their approach is very useful in actual recommendation contexts. Focused on the transportation system,
Buijs et al. (2016) propose an evaluate method to structure and improve Fritom’s existing collaborative transport planning process.
Deep learning models mainly include neural networks.
Sun and Vasarhelyi (2018) develop a deep neural network (DNN) that can assess the risk of credit card defaults based on the personality characteristics and spending behaviors. Compared with other machine-learning algorithms, DNN has a higher F score and better overall predictive performance. Flight delays are another common problem in the transportation field, which seriously affects the travel experience.
Chung et al. (2017) use a large data set from a major Hong Kong airline to analyze flight delays at 112 airports around the world. They leverage a cascading neural network to improve the flight schedule prediction, and then applied it to the crew optimization problem. The new method can improve the accuracy of flight delay prediction, which greatly improves the crew matching performance.
Systematic improvement based on data processing technology usually involves a large amount of data for secondary processing, which gives some management inspiration. The improvement of systems based on big data processing technology requires more in-depth discussions to dig out more valuable information. The relevant systematic improvement researches are shown in Table 9.
Systematic improvement method based on information characteristics
As an abstract and intangible special resource, information has a corresponding use-value, and it can meet people’s needs in some aspects. Reasonable extraction of information can effectively improve decision-making (
Adnan and Akbar, 2019). In a big data environment, information sharing, disclosure, and security issues are particularly important. In this part, we focus on the systematic improvement research under the characteristics of information sharing, information disclosure, and information privacy.
Li and Gu (2019) propose an integrated approach for the support system to allow users to query data simultaneously from both relational Standard Query Language (SQL) systems and NoSQL (not only SQL) systems. The proposed approach can effectively reduce development complexity and improve development efficiency. Kishore et al. (2020) link call detail records (CDR) with influenza-like illness (ILI) registry and evaluate the role that international travelers played in the introduction of epidemics (A/H1N1). Their methods carry out a similar assessment of domestic airports and the system efficiency is improved significantly. To explore how the individuals’ valuations change in the presence of multiple privacy factors,
Buckman et al. (2019) use an incentive-compatible mechanism to capture individuals’ willingness to accept disclosure. The results show participants’ privacy valuations are largely unaffected by requiring the disclosure of personal identifying information, the information context, and the intended secondary use of the disclosed information.
In addition, to examine trends in academic research on personal information privacy,
Choi et al. (2017a) extract 2356 documents published between 1972 and 2015 (by August) to carry out topic modeling using Scopus database. They show the topics of algorithms, online social networks, and Facebook privacy have become promising. Moreover, the top journals put more attention on both e-business and healthcare. The system operations based on information characteristics are shown in Table 10.
Supply chain management and operations
With the improvement of enterprise resource planning (ERP) software, capturing and storing data at different levels of operations become easier. Companies also hope more efficient processes by analyzing this data. Big data analytics techniques are widely used in supply chain management and operations to make smarter decisions (
Wamba et al., 2015). Key areas of operational management in a big data environment include supply chain and logistics management, inventory management, retail forecasting, and risk assessment.
Big data has a significant impact on supply chain management. Researchers have studied capacity sharing contracts where the demand is uncertain (
Yang et al., 2017a).
Wu et al. (2019a) examine the relationship between data analytics capabilities and innovation. The results suggest data analytics capabilities are most strongly associated with innovation. Firms might receive the most benefits from using data analytics if they historically focused on specific types of innovation.
Newman et al. (2014) propose a parameter estimation routine for multinomial logit discrete choice models. Their method is computationally efficient and can easily incorporate price and other product attributes. They simulate the hotel industry data and demonstrate their method has superior computational performance over alternative estimation methods that are capable of estimating price effects. Focused on the sustainability of supply chain systems,
Badiezadeh et al. (2018) propose a “double frontier network data envelopment analysis (DEA)” to assess the sustainability of supply chain systems in a big data environment. Their method can rank the sustainability scores of supply chains. Considering strategic customers,
Yang et al. (2019c) investigate the impact of selling effort on the pricing decisions of the supply chain.
Inventory management is an important topic in operations management (OM). Considering the inventory control problem,
Huang and van Mieghem (2014) develop a decision support model, which can reduce 3% inventory holding cost and 5% back-ordering cost. Likewise,
Bertsimas et al. (2016) explore the inventory control issue by conditional stochastic optimization method in a big data environment. They analyze 4-year inventory data from the retail network, obtaining the optimal inventory management scheme for a retail company with multiple inventory and retail locations.
Retail forecasting has been another key area of research for the OM, especially in multi-channel retail (
Mehra et al., 2018). In the research of traditional retail, the forecast more depends on historical data, expert advice, and market information. However, in a big data environment, increasing available information sources can potentially enhance prediction performance. In recommendation systems, some firms reduce consumer search workloads by using big data technologies (
Dutta et al., 2017).
Cui et al. (2018) propose a comprehensive system that predicts the total daily sales of online men’s exclusive retailer though feature extractions and machine learning, finding the system can improve the forecast accuracy by utilizing social media inputs. They analyze data posted by Facebook (recording the company’s interactions with customers) to supplement operational data to generate sales forecasts. Furthermore, the service optimization of after-sales operations also influenced by big data analytics.
Boone et al. (2018) add customer search data to time series models to improve out-of-sample forecast errors, finding that the models added with data from Google Trends can improve sales forecast accuracy for online specialty food and cookware retailer, especially in multiple products.
Yang et al. (2014) use a DEA method to forecasting the production abilities of recycling systems and show the reuse level of water in China is still low.
Risk assessment, for both business operations and non-profit organization, would benefit from the advancements of big data techniques (
Choi et al., 2017b). To assess the risk of having rail failures,
Jamshidi et al. (2017) develop a novel method to explore rail surface defects. They collect big data on track surface defects through intelligent image processing, which includes measurable lengths of these defects. Finally, they conduct a practical case study on Dutch railways and find that the railway fault assessment system they proposed performs well. Considering the risk assessment of procuring infrastructure mega-projects,
Chan et al. (2018) develop a fuzzy evaluation model and demonstrate the practicality of the risk evaluation model by analyzing Hong Kong–Zhuhai–Macao Bridge project. Supply disruption affects the efficiency of the supply chain.
Yang et al. (2019b) find the licensing strategies can effectively reduce the negative effects caused by the risk of interruption, improving the performance of the supply chain. The evaluation of environmental impact indicators for sustainable maritime transportation systems is studied (
Lizzette et al., 2019). The relevant supply chain management and operations are shown in Table 11.
Application research of system evaluation in the big data environment
In this section, we discuss the applications of system evaluation in the big data environment in the medical industry, finance, business, information systems, transportation, and other areas. Figure 1 summarizes the key areas and future researches.
Medical industry
Since big data can deal with massive data volume and variety at high velocity, it has the potential to create significant value in healthcare by improving outcomes while lowering costs. Data analytics plays an important role in improving the quality of care, increasing the efficiency of operational processes, predicting and planning responses to disease epidemics, and optimizing healthcare spending at all levels (Nambisan et al., 2015). Hence, we explore the applications of system evaluation on the medical industry from medical image informatics, healthcare management, and privacy concerns.
The medical image informatics can help the doctor better diagnose the disease. Insights derived from the analysis of patient data can help healthcare professionals better identify disease symptoms and predict the cause and incidence of disease, ultimately improving the overall quality of care (
Genta and Sonnenberg, 2014).
Nambisan et al. (2015) focus on social media communications to identify individuals suffering from depression. In the field of healthcare management,
Hydari et al. (2018) analyze a new patient data set from the Pennsylvania Patient Safety Authority (PSA), finding that advanced electronic medical records (EMRs) can decline 17.5% patient safety events caused by reductions in medication errors, falls, and complication errors.
Senot et al. (2016) examine the impact of combined quality and experiential quality on hospitals’ readmissions and cost performance. They analyze six years of data from 3474 US acute care hospitals and show that combined quality and experiential quality decrease the likelihood of readmission for a patient.
A major problem facing information systems today is the protection of consumer privacy. Building Healthcare Information Exchanges (HIE) frameworks containing security principles and privacy is necessary in the big data environment. Considering the privacy issues in the smart healthcare framework,
Adjerid et al. (2016) explore the impact of the forms of privacy regulations and policies on HIE efforts. They find that privacy regulation alone negatively affects HIE efforts while privacy regulations with incentives have positive impacts on HIE efforts. Some researchers investigate privacy-preserving data mining (PPDM) operation. PPDM is a data mining method that cannot leak any data containing the client’s sensitive information (
Xu et al., 2014).
Finance
The finance research in a big data environment mainly focused on risk analysis. The major financial institutions have millions of credit card accounts. Because of the large scale data, these loan pools and returned securities are computationally difficult to analyze.
Allodi and Massacci (2017) develop a quantitative scheme to assess cybersecurity risk. Their scheme can give quantitative probability estimates to help deal with untargeted cyber-attacks against the organization. They use a real data set from a financial institution to verify the new scheme, finding that the new risk assessment scheme is effective. Focusing on a broad class of dynamic, discrete-time models of loan-level risk,
Sirignano and Giesecke (2018) develop a numerical method for the analysis of large pools of loans as well as asset-backed securities backed by such pools. Applying their approach to a data set of over 25 million mortgages, the accuracy and speed of the approximation have improved for a variety of pools with different risk profiles. Furthermore, big data analytics is also employed in various other topics such as the relationship between ratings of new issuances and the number of rating analysts (
Jiang et al., 2018) and predictor of US real economic activity (
Faccini et al., 2018).
Business
Akter and Wamba (2016) make a systematic review of e-commerce in a big data environment. They show that data analytics can transform data into insights by the dynamics processes and technology, providing more value for robust decision-making and business problem solutions for e-commerce companies. Some growth areas of e-commerce research include advertising strategies and recommendation systems for online firms (
Ghoshal et al., 2015). Web and mobile advertising have also been other interesting areas of research.
Mookerjee et al. (2017) set up a model to predict the clicks of visitors for online advertisements. They propose an approach to managing online advertisements. The results show that both click-rate and firm’s revenue are increased.
Yang et al. (2019a) develop a winner-take-all model to improve the total values of corporations. Their approach assesses how a firm’s competitiveness and development potential of corporations help managers effectively avoid overcapacity.
Information systems (IS)
Information systems (IS) has become an interdisciplinary research area that combines computing techniques with big data from business practice (
Agarwal and Dhar, 2014). Most research in the IS areas focuses on improving the efficiency of business operations. Focused on IS-Marketing research,
Ruths and Pfeffer (2014) analyze the data from online social media to explore consumer behaviors and forecast events. In this direction,
Qiu and Kumar (2017) design a randomized field experiment to examine the performance of prediction markets, finding that the increasing size of audiences and the online endorsement often produce more precise predictions. The results suggest that the prediction is more accurate when targeting people with intermediate abilities. Considering the pitfalls of social media,
Kumar et al. (2018) summarize several collective behaviors of users on social media and describe them uniformly. They then propose a new hierarchical supervised-learning approach that can assess the likelihood of finding anomalies in online reviews. The results suggest that it is difficult to detect dishonest online reviews owing to complex interactions.
Transportation
With the development of intelligent mobility, the transportation industry has also entered the era of big data. The analysis of big data can significantly improve traffic conditions.
Liu et al. (2016a) use big data that contains truck driver behavior information before the accident and geographic location information to study the direct and indirect relationships between truck traffic safety and related factors. They propose a methodology to improve trucks at railway crossings. Considering the pedestrian crash,
Xie et al. (2017) explore the possibility of accidents in the logistics network by big data analytics. They build a new model based on a grid-based unit structure framework that can simultaneously utilize large data sets of transit turnstiles, taxis, and even social media. They believe that big data analysis can more accurately estimate the relevant risk factors, which can help identify hot spots in traffic accidents so that positive measures can be taken. Focused on traffic flow predictions,
Lv et al. (2015) derive a novel method considering both temporal and spatial correlations and then develop a stacked auto-encoder that is trained by a greedy layer-by-layer method to learn traffic flow characteristics.
Shang et al. (2017) use a Bayesian statistics method to explore the cargo logistics risk (CLR). The authors strive to flexibly estimate the conditional density function of the CLR by a large amount of air cargo data set. Their findings help logistics companies distinguish whether the source of CLR is recurrent or disrupt.
Other areas
Focused on the assessment of weather risk,
Biffis and Chavez (2017) develop a data-driven risk transfer scheme and demonstrate how to use weather data of rainfall and temperature to create a risk profile. They conduct a real case study for Mozambique’s maize production. The results show that their proposed framework can save 30% cost (from insurance). Considering the management of the e-government system,
Joseph and Johnson (2013) demonstrate how big data improves operational efficiency and process effectiveness in the US government. Furthermore, the systemic evaluation in a big data environment is also employed in other fields such as the environment system, the recycling economy evaluation index system, and logistic system. Interested readers can refer to
Bi et al. (2014),
Yang et al. (2016a), and
Zhou et al. (2016) for more discussions.
Way ahead: Potential applications and challenges
Potential applications of evaluation methods
The Internet of Things has created a world of interconnected sensing devices that can collect and store information from their respective real environments (
Hashem et al., 2016). The combination of the Internet of Things and the application of big data analysis may bring breakthrough changes to various industries and academic research. For example, smart healthcare is a promising area for future development. The Internet of Things and cloud-based big data analytics can more accurately detect and treat diseases while requiring lower healthcare costs (
Varshney and Chang, 2016). The Internet of Things and block-chain ensure the authenticity and traceability of all processes, reducing the transaction risk of various entities. The risk assessment combined with block-chain technology is also worth studying. In addition, the IoTs and related big data applications can play key roles in advancing environmental sustainability (
Bibri, 2018). Making a reasonable assessment of the sustainable economy and proposing improvement methods is also of great significance today in response to environmental degradation.
Logistics services are becoming more intelligent and can be tracked in real-time (
Yang et al., 2017b). The development of the Internet of Things and big data has also promoted the development of smart logistics. According to the China Smart Logistics Development Report released in 2018, the industry value of smart logistics will exceed one trillion by 2025. Focusing on the evaluation of smart logistics will be a promising topic based on the Internet of Things and big data analysis technology. Furthermore, with the rapid development of social platforms and short video Apps, evaluating consumer’s online behavior is also a trend (
Zheng et al., 2019).
The development of new evaluation methods
The big data put forward new requirements on behavioral cognition of evaluation, evaluation modes, and evaluation functions. First, behavioral cognition of evaluation. The evaluator shows a series of new behavioral characteristics in the context of big data. For example, complete evaluation accuracy is impossible to achieve in the context of big data. Therefore, the evaluator’s requirements for evaluation accuracy may need to be reduced. Specifically, the accuracy of evaluation depends on the extracting costs of information. After all, the acquisition of big data requires huge economic costs. Word-of-mouth service on the Internet is a representative type of public comment. Evaluation in the context of public comment may cause some behavioral deviations (such as group polarization). These behavioral deviations also need to be understood. Second, changes in evaluation modes. In the context of big data, online evaluation models will emerge. The online evaluation has the characteristics of high-speed computing and instant response. In addition, the background of big data also emphasizes the dependence on distributed human-computer interaction. The current evaluation model is not profound for the introduction of distributed human-computer interaction. Third, a new expansion of the evaluation function. Collecting and processing big data through a large number of mobile, distributed terminals and processors is to meet the needs of decision-makers to mine the value of data, which provides opportunities for evaluating the emergence of new features. For example, public safety evaluation based on mobile social network data, energy efficiency evaluation based on smart meter data, residential credit evaluation based on smart payment data, and building security evaluation based on IoTs data are all new evaluations in the context of big data.
Furthermore, the age of big data brings new opportunities to the evaluation methods, which are mainly manifested as follows: (1) Evaluation of data quality. Although big data contains sufficient information, it also means big garbage (such as repeated, redundant, disturbing, and distorted information), which hinders the reliability of the evaluation results. Therefore, to evaluate data quality has become a top priority. (2) New data analysis techniques to reduce computational difficulty. Evaluation methods based on statistical models are difficult to implement, while evaluation methods based on optimized models often perform poorly on large data scales. How to reduce the computational difficulty on the basis of ensuring the completeness and rationality of data information is the basic performance required by the new evaluation method and its implementation technology. (3) Robust evaluation methods. Most of the current evaluation methods deal with single or several discrete static data sets, and cannot perform continuous dynamic evaluation throughout the process. In the big data environment, the status of the evaluation object is updated in real-time. But decision-makers want some robust assessment results. Therefore, it is necessary and valuable to develop a robust evaluation method that can adapt to dynamically changing data. (4) Evaluation methods for unstructured data. In the big data environment, many data are unstructured or semi-structured, and most of the current evaluation methods can only deal with structured data. Due to the huge amount of data, the diversity of information dimensions, the personalization of expressions, and the complex relationships between data, it is a very difficult task to transform unstructured data into structured data. To develop novel evaluation methods for unstructured data will greatly enrich the evaluation theory. (5) Sampling and population contain different amounts of information. For a large amount of data in the traditional sense, it is assumed that the sampling and the population have the same amount of information, and the performance of the overall data can be indirectly measured by random sampling. In a real big data environment, the total amount of information that cannot be contained in the sampling. Hence, the big data environment poses severe challenges to the traditional random sampling method. (6) Evaluation methods for new features. The use of big data are the conscious behavior of decision-makers. Collecting and processing big data through a large number of mobile and distributed terminals and processors is to meet the needs of decision-makers to mine the value of data. This provides the possibility of the emergence of new evaluation features. This is the evaluation of new features such as data based on mobile social networks used for public safety assessment, smart meter data for energy efficiency assessment, smart payment data for resident’s credit assessment, etc.
Conclusions
This article reviews the systematic evaluation and improvement researches in the big data environment. We first focus on the development of evaluation methods under different big data analytics techniques and data characteristics. The systematic improvement researches under different data analysis techniques and different information characteristics are also reviewed. It also summarizes the applied research of system evaluation from six aspects: Medical industry, finance, business, information systems, transportation, and other areas. Then it discusses the future development of evaluation methods in the context of big data. The research on the evaluation methods and systematic improvement work in the big data environment is a great innovation of traditional evaluation theories and methods, which will help promote the scientific process of evaluation. The study of evaluation theory and methods in the context of big data is a new field of big data research, and its research results can be used as an important module of management decision-making. Systematic evaluation and improvement also have broad research and application prospects.
PDF
(308KB)
