In the quest for interpretable models, two versions of a neural network rule extraction algorithm were proposed and compared. The two algorithms are called the Piece-Wise Linear Artificial Neural Network (PWL-ANN) and enhanced Piece-Wise Linear Artificial Neural Network (enhanced PWL-ANN) algorithms. The PWL-ANN algorithm is a decomposition artificial neural network (ANN) rule extraction algorithm, and the enhanced PWL-ANN algorithm improves upon the PWL-ANN algorithm and extracts multiple linear regression equations from a trained ANN model by approximating the hidden sigmoid activation functions using N-piece linear equations. In doing so, the algorithm provides interpretable models from the originally trained opaque ANN models. A detailed application case study illustrates how the generated enhanced-PWL-ANN models can provide understandable IF-THEN rules about a problem domain. Comparison of the results generated by the two versions of the PWL-ANN algorithm showed that in comparison to the PWL-ANN models, the enhanced-PWL-ANN models support improved fidelities to the originally trained ANN models. The results also showed that more concise rule sets could be generated using the enhanced-PWL-ANN algorithm. If a more simplified set of rules is desired, the enhanced-PWL-ANN algorithm can be combined with the decision tree approach. Potential application of the algorithms to domains related to petroleum engineering can help enhance understanding of the problems.
Acknowledgements
The first author is grateful for the scholarships and generous support from the Faculty of Graduate Studies and Research, University of Regina and from the Canada Research Chair Program. The authors also wish to acknowledge the contributions of Dr. Raphael Idem and Dr. Paitoon Tontiwachwuthikul for their insights and for allowing us to use the datasets on the carbon dioxide capture process system of the Clean Energy Technology Research Institute in Saskatchewan, Canada.
| [1] |
A. Sheikhtaheri, F. Sadoughi, Z. Hashemi Dehaghi, Developing and using expert systems and neural networks in medicine: a review on benefits and challenges, J. Med. Syst. 38 (9) (2014), https://doi.org/10.1007/s10916-014-0110-5.
|
| [2] |
M. Tkáč, R. Verner, Artificial neural networks in business: two decades of research, Appl. Soft Comput. 38 (2016) 788-804, https://doi.org/10.1016/j.asoc.2015.09.040.
|
| [3] |
V. Chan, C. Chan, Towards developing the piece-wise linear neural network algorithm for rule extraction, Int. J. Cogn. Inf. Nat. Intell. 11 (2) (2017).
|
| [4] |
M. Lichman, UCI Machine Learning Repository, University of California, School of Information and Computer Science, Irvine, CA, 2013 http://archive.ics.uci.edu/ml.
|
| [5] |
G.G. Towell, J.W. Shavlik, Extracting refined rules from knowledge-based neural networks, Mach. Learn. 13 (1) (1993) 71-101.
|
| [6] |
R. Setiono, H. Liu, Understanding neural netowrks via rule extraction, Proceeding of 14th International Joint Conference on Artificial Intelligence, 1995, pp. 480-485.
|
| [7] |
R. Setiono, H. Liu, Neurolinear: from neural networks to oblique decision rules, Neurocomputing 17 (1) (1997) 1-24.
|
| [8] |
R. Krishnan, G. Sivakumar, P. Bhattacharya, A search technique for rule extraction from trained neural networks, Pattern Recognit. Lett. 20 (3) (1999) 273-280.
|
| [9] |
D. Kim, J. Lee,Handling continuous-valued attributes in decision tree with neural network modelling, Machine Learning: ECML 2000, 11th European Conference on Machine Learning, 2000, pp. 211-219 Barcelona, Catalonia, Spain, May 31 -June 2, 2000.
|
| [10] |
R. Setiono, W.K. Leow, J.M. Zurada, Extraction of rules from artificial neural networks for nonlinear regression, IEEE Trans. Neural Netw./a Publ. IEEE Neural Netw. Counc. 13 (3) (2002) 564-577.
|
| [11] |
R. Setiono, J.Y.L. Thong, An approach to generate rules from neural networks for regression problems, Eur. J. Oper. Res. 155 (1) (2004) 239-250.
|
| [12] |
K. Odajima, Y. Hayashi, G. Tianxia, R. Setiono, Greedy rule generation from discrete data and its use in neural network rule extraction, Neural Netw. 21 (7) (2008) 1020-1028.
|
| [13] |
R. Nayak, Generating rules with predicates, terms and variables from the pruned neural networks, Neural Netw. 22 (4) (2009) 405-414, https://doi.org/10.1016/j.neunet.2009.02.001.
|
| [14] |
J. Wang, B. Qin, W. Zhang, W. Shi,Regression rules extraction from artificial neural network based on least squares, Proceedings -2011 7th International Conference on Natural Computation, ICNC 2011, 2011, pp. 203-207.
|
| [15] |
N. Tsopze, E. Mephu-Nguifo, G. Tindo,Towards a generalization of decompositional approach of rule extraction from multilayer artificial neural network, Proc. Int. Jt. Conf. Neural Netw. (2011) 1562-1569, https://doi.org/10.1109/IJCNN.2011.6033410.
|
| [16] |
Y. Hayashi, R. Setiono, A. Azcarraga, Neural network training and rule extraction with augmented discretized input, Neurocomputing 207 (2016) 610-622.
|
| [17] |
S.B. Thrun, Extracting Provably Correct Rules from Artificial Neural Networks, University of Bonn, 1993, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10. 1.1.2.2110&rep=rep1&type=pdf.
|
| [18] |
M.W. Craven, J.W. Shavlik,Extracting tree-structured representations of trained neural networks, Adv. Neural Inf. Process. Syst. 8 (1996) 24-30.
|
| [19] |
E.W. Saad, D.C. Wunsch, Neural network explanation using inversion, Neural Netw. 20 (1) (2007) 78-93.
|
| [20] |
E.J. de Fortuny, D. Martens, Active learning based rule extraction for regression, 2012 IEEE 12th International Conference on Data Mining Workshops, 2012, pp. 926-933.
|
| [21] |
M.G. Augasta, T. Kathirvalavakumar, Rule extraction from neural networks -a comparative study, Proceedings of the International Conference on Pattern Recognition, Informatics and Medical Engineering, 2012.
|
| [22] |
K.K. Sethi, D.K. Mishra, B. Mishra, KDRuleEx: a novel approach for enhancing user comprehensibility using rule extraction, Proceedings -3rd International Conference on Intelligent Systems Modelling and Simulation, ISMS 2012, 2012, pp. 55-60, https://doi.org/10.1109/ISMS.2012.116.
|
| [23] |
M. Hall, H. National, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I.H. Witten, The WEKA data mining Software : an update, SIGKDD Explor. 11 (1) (2009) 10-18.
|
| [24] |
V. Chan, C. Chan, Learning from a carbon dioxide capture system dataset: application of the piecewise neural network algorithm, Petroleum 3 (2017) (2017) 56-67.
|
| [25] |
T.M. Therneau, E.J. Atkinson, An Introduction to Recursive Partitioning Using the RPART Routines vol. 61, (1997), p. 452 Mayo Foundation: Technical report.
|
PDF
