Sparse decomposition is a new theory in signal processing, with the advantage in that the base (dictionary) used in this theory is over-complete, and can reflect the nature of a signal. Thus, the sparse decomposition of signal can obtain sparse representation, which is very important in data compression. The algorithm of compression based on sparse decomposition is investigated. By training on and learning electrocardiogram (ECG) data in the MIT-BIH Arrhythmia Database, we constructed an over-complete dictionary of ECGs. Since the atoms in this dictionary are in accord with the character of ECGs, it is possible that an extensive ECG datum is reconstructed by a few nonzero coefficients and atoms. The proposed compression algorithm can adjust compression ratio according to practical request, and the distortion is low (when the compression ratio is 20∶1, the standard error is 5.11%). The experiments prove the feasibility of the proposed compression algorithm.