Incorporating prior knowledge into learning by dividing training data

Abstract

In most large-scale real-world pattern classification problems, there is always some explicit information besides given training data, namely prior knowledge, with which the training data are organized. In this paper, we proposed a framework for incorporating this kind of prior knowledge into the training of min-max modular (M³) classifier to improve learning performance. In order to evaluate the proposed method, we perform experiments on a large-scale Japanese patent classification problem and consider two kinds of prior knowledge included in patent documents: patent’s publishing date and the hierarchical structure of patent classification system. In the experiments, traditional support vector machine (SVM) and M³-SVM without prior knowledge are adopted as baseline classifiers. Experimental results demonstrate that the proposed method is superior to the baseline classifiers in terms of training cost and generalization accuracy. Moreover,M³-SVM with prior knowledge is found to be much more robust than traditional support vector machine to noisy dated patent samples, which is crucial for incremental learning.

Keywords

prior knowledge / patent classification / support vector machine / min-max modular network / task decomposition

References

[1] Liu B, Li X L, Lee W S, Yu P S. Text classification by labeling words. AAAI, 2004

[2] Wu X Y, Srihari R. Incorporating prior knowledge with weighted margin support vector machines. In: Proceedings of International Conference on Knowledge Discovery and Data Mining, 2004, 326-333

[3] Schapire R E, Rochery M, Rabim M, Gupta N. Boosting with prior knowledge for call classification. IEEE Transactions on Speech and Audio Processing, 2005, 13, 174-181

[4] Zhu J B, Chen W L. Improving text categorization using domain knowledge In: Proceedings of International Conference on Applications of Natural Language to Information Systems, 2005, 103-113

[5] Dayanik A, Lewis D D, Madigan D, Menkov V, Genkin A. Constructing informative prior distributions from domain knowledge in text classification. In: Proceedings of ACM’S Special Interest Group on Information Retrieval, 2006

[6] Lu B L, Ito M. Task decomposition based on class relations: a modular neural network architecture for pattern classification. Biological and Artificial Computation: From Neuroscience to Technology. Springer, LNCS, 1997, 1240: 330-339

[7] Lu B L, Ito M. Task decomposition and module combination based on class relations: A modular neural network for pattern classification. IEEE Transactions on Neural Networks, 1999, 10: 1244-1256

[8] Anand R, Mehrotra K G, Mohan C K, Ranka S. An improved algorithm for neural network classification of imbalanced training sets. IEEE Transaction on Neural Netwook, 1993, 4: 962-969

[9] Lu B L,Wang K A, Utiyama M, Isahara H. A part-versus-part method for massively parallel training of support vector machines. In: Proceedings of International Joint Conference on Neural Networks, 2004, 735-740

[10] Krier M, Zaccá F. Automatic categorization applications at the European patent office. World Patent Information. Elsevier, 2002, 24(3): 187-196

[11] Larkey L. Some issues in the automatic classification of US patents. Learning for Text Categorization. Technical Report WS-98-05, 1998, 87-90

[12] Larkey L. A patent search and classification system. In: Proceedings of the fourth ACM conference on Digital libraries, 1999, 179-187

[13] Mase H, Tsuji H, Kinukawa H, Ishihara M. Automatic patents categorization and its evaluation. Transactions of Information Processing Society of Japan(IPSJ), 1998

[14] Fall C J, Benzineb K. Literature survey: Issues to be considered in the automatic classification of patents. World Intellectual Property Organization, 2002, 29

[15] Fall C J, Torcsvári A, Benzineb K, Karetka G. Automated categorization in the international patent classification. In: Proceedings of ACM’S Special Interest Group on Information Retrieval. New York: ACM Press, 2003, 37: 10-25

[16] Fujii A, Iwayama M, Kando N. Test collections for patent retrieval and patent classification in the 5th NTCIR workshop. In: Proceedings of the 5th international conference on language resources and evaluation, 2004, 1643-1646

[17] Fujii A, Iwayama M, Kando N. Introduction to the special issue on patent processing. Information Processing and Management, 2007, 1149-1153

[18] Wen Y M, Lu B L, Zhao H. Equal clustering makes min-max modular support vector machine more efficient. In: Proceedings of International Conference on Neural Information Processing, 2005, 77-82

[19] Lian H C, Lu B L, Takikawa E, Hosoi S. Gender recognition using a min-max modular support vector machine. In: Proceedings of International Conference on Natural Computation, 2005, 438-441

[20] Yang Y M, Pedersen J O. A comparattive study on feature selection in text categorization. In: Proceedings of International Conference on Machine Learning, 1997, 187-196

[21] Sebastiani F. Machine learning in automated text categorization. ACM Computing Surveys, 2002, 34: 1-47

[22] Zhao H, Lu B L. A modular k-nearest neighbor classification method for massively parallel text categorization. In: Proceedings of First International Symposium on Computational and Information Science. Springer, LNCS, 2004, 3314: 867-872

[23] Wu K, Lu B L, Uchiyama M, Isahara H. An empirical comparison of min-max-modular k-NN with different voting methods to largescale text categorization. Soft Computing - A Fusion of Foundations, Methodologies and Applications, 2008, 12(7): 647-655

[24] Joachims T. Making large-scale support vector machine learning practical. Advances in Kernel Methods: Support Vector Learning. Cambridge: MIT Press, 1998

[25] Lewis D D, Yang Y, Rose T, Li F. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 2004, 5: 361-397

[26] Liu W, Xue G R, Yu Y, Zeng H J. Importance-based web page classification using cost-sensitive SVM. In: Proceedings of International Conference on Web-Age Information Management, 2005, 127-137

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag Berlin Heidelberg