Resolution of overlapping ambiguity strings based on maximum entropy model

Abstract

The resolution of overlapping ambiguity strings (OAS) is studied based on the maximum entropy model. There are two model outputs, where either the first two characters form a word or the last two characters form a word. The features of the model include one word in context of OAS, the current OAS and word probability relation of two kinds of segmentation results. OAS in training text is found by the combination of the FMM and BMM segmentation method. After feature tagging they are used to train the maximum entropy model. The People Daily corpus of January 1998 is used in training and testing. Experimental results show a closed test precision of 98.64 % and an open test precision of 95.01 %. The open test precision is 3.76 % better compared with that of the precision of common word probability method.

Keywords

Chinese information processing, Chinese automatic word segmentation, overlapping ambiguity strings, maximum entropy model