EMelodyGen: Emotion-Conditioned Melody Generation in ABC Notation with Musical Feature Templates

Abstract

The EMelodyGen system mainly focuses on melody generation in ABC notation controlled by emotional conditions. To overcome the scarcity of emotional labeled sheet music, we utilize statistical correlations derived from small-scale symbolic music datasets with emotion labels and music psychology conclusions to guide subsequent feature extraction, emotional control and automatic annotation. We then automatically annotate a large, well-structured sheet music collection with rough emotional labels, convert the annotated dataset into ABC notation format, and apply data augmentation to address label imbalance, resulting in the creation of a dataset named Rough4Q. We demonstrate that our system backbone pre-trained on Rough4Q can achieve up to 99% music21 parsing rate. Our emotional control parameters, categorized into directly modifiable, embedding, dual-stage, and guidance features, can be selected and assembled to design customized emotional control templates that can lead to a 91% alignment in emotional expression in blind listening tests. Ablation studies further validate the impact of these control conditions on emotional accuracy.

Keywords

music generation / melody / ABC notation / LLM

Author Contributions

M.Z.: contributed to conceptualization, methodology, software development, data curation, and original draft preparation, writing, reviewing, editing and validation. X.L.: conceptualization, reviewing and supervision. F.Y.: conceptualization, reviewing and supervision. W.L.: conceptualization, reviewing, validation, editing and supervision. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the National Social Science Fund of China (21ZD19), and the Special Program of the National Natural Science Foundation of China (T2341003).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The code of this work is available at https://github.com/monetjoe/EMelodyGen.

Conflicts of Interest

The authors declare no conflict of interest.

References

[1] Ji S.; Yang X.; Luo J. A survey on deep learning for symbolic music generation: Representations, algorithms, evaluations, and challenges. Acm Comput. Surv. 2023, 56, 1-39.

[2] Xin Y. MusicEmo: transformer-based intelligent approach towards music emotion generation and recognition. J. Ambient. Intell. Humaniz. Comput. 2024, 15, 3107-3117.

[3] Bao C.; Sun Q. Generating Music With Emotions. IEEE Trans. Multimed. 2023, 25, 3602-3614.

[4] Ji S.; Yang X. Emomusictv: Emotion-conditioned symbolic music generation with hierarchical transformer vae. IEEE Trans. Multimed. 2023, 26, 1076-1088.

[5] Zhu H.; Wang S.; Wang Z. Emotional music generation using interactive genetic algorithm. In Proceedings of the 2008 International Conference on Computer Science and Software Engineering, Wuhan, China, 12-14 December 2008.

[6] Zheng L.; Li C. Real-Time Emotion-Based Piano Music Generation using Generative Adversarial Network (GAN). IEEE Access 2024, 12, 87489-87500.

[7] Huang J.; Chen K.; Yang Y.H. Emotion-driven Piano Music Generation via Two-stage Disentanglement and Functional Representation. In Proceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR, San Francisco, CA, USA, 10-14 November 2024.

[8] Zhang J.; Fazekas G.; Saitis C. Fast diffusion gan model for symbolic music generation controlled by emotions. arXiv 2023, arXiv:2310.14040.

[9] Sulun S.; Davies M.E.; Viana P. Symbolic music generation conditioned on continuous-valued emotions. IEEE Access 2022, 10, 44617-44626.

[10] Ferreira L.N.; Mou L.; Whitehead J.; et al. Controlling Perceived Emotion in Symbolic Music Generation with Monte Carlo Tree Search. In Proceedings of the Eighteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE, Pomona, CA, USA, 24-28 October 2022.

[11] Grekow J.; Dimitrova-Grekow T. Monophonic music generation with a given emotion using conditional variational autoencoder. IEEE Access 2021, 9, 129088-129101.

[12] Oliwa T.M. Genetic algorithms and the abc music notation language for rock music composition. In Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation; Association for Computing Machinery: New York, NY, USA, 12-16 July 2008.

[13] Wu S.; Li X.; Yu F.; et al. TunesFormer: Forming Irish Tunes with Control Codes by Bar Patching. In Proceedings of the 2nd Workshop on Human-Centric Music Information Retrieval 2023 Co-Located with the 24th International Society for Music Information Retrieval Conference (ISMIR), Milan, Italy, 10 November 2023.

[14] Casini, L.; Jonason N.; Sturm B.L.T. Investigating the Viability of Masked Language Modeling for Symbolic Music Generation in abc-notation. In Artificial Intelligence in Music, Sound, Art and Design; Johnson, C., Rebelo, S.M., Santos, I., Eds.; Springer Nature: Cham, Switzerland, 2024; pp. 84-96.

[15] Wu S.; Wang Y.; Li X.; et al. MelodyT5:A Unified Score-to-Score Transformer for Symbolic Music Processing. In Proceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR, San Francisco, CA, USA, 10-14 November 2024.

[16] Hung H.T.; Ching J.; Doh S.; et al. EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR, Online, 7-12 November 2021.

[17] Ferreira L.; Whitehead J. Learning to Generate Music With Sentiment. In Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR, Delft, The Netherlands, 4-8 November 2019.

[18] Cuthbert M.S.; Ariza C. Music21:A Toolkit for Computer-Aided Musicology and Symbolic Music Data. In Proceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR, Utrecht, The Netherlands, 9-13 August 2010.

[19] Scherer K.R.; Coutinho E. How music creates emotion:A multifactorial process approach. In The Emotional Power of Music, Multidisciplinary Perspectives on Musical Arousal, Expression, and Social Control; Oxford University Press: Oxford, UK, 2013; pp. 121-145.

[20] Thompson W.F.; Quinto L. Music and emotion: Psychological considerations. Aesthetic Mind Philos. Psychol. 2011, 357-375.

[21] Granot R.Y.; Eitan Z. Musical tension and the interaction of dynamic auditory parameters. Music. Percept. 2011, 28, 219-246.

[22] Juslin P.N.; Sloboda J.A. Music and Emotion. In The Psychology of Music; Academic Press: Cambridge, MA, USA, 2001.

[23] Russell J.A. Measures of Affect: A Review. In Emotion: Theory, Research, and Experience; Plutchik, R., Kellerman, H., Eds.; Academic Press: Cambridge, MA, USA, 1989; Volume 4, pp. 83-111.

[24] Russell J. A Circumplex Model of Affect. J. Personal. Soc. Psychol. 1980, 39, 1161.

[25] De S.; Dutta P. Computational Intelligence for Human Action Recognition; CRC Press: Boca Raton, FL, USA, 2020.

[26] Pearson K. II. Mathematical contributions to the theory of evolution. II. Skew variation in homogeneous material. Proc. R. Soc. Lond. 1985, 57, 257-260.

[27] Mu¨ller J.M. Computational Approaches to Symbolic Music Analysis: Weighting by Note Duration. J. Comput. Musicol. 2021, 29, 45-60.

[28] Parzen E. On estimation of a probability density function and mode. Ann. Math. Stat. 1962, 33, 1065-1076.

[29] McFee B.; Raffel C.; Liang D.; et al. librosa: Audio and Music Signal Analysis in Python. In Proceedings of the 14th Python in Science Conference 2015 (SciPy 2015), Austin, TX, USA,6-12 July 2015.

[30] Liu Z.; Li Z. Music Data Sharing Platform for Computational Musicology Research (CCMUSIC DATASET); Zenodo: Beijing, China, 2021. https://doi.org/10.5281/zenodo.5676893,

[31] Wu S.; Li X.; Sun M. Chord-conditioned melody harmonization with controllable harmonicity. In Proceedings of the ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 4-10 June 2023.

[32] The Nottingham Music Database. 2011. Available online: accessed on 16 September 2025).

[33] Simonetta F. Enhanced Wikifonia Leadsheet Dataset; Zenodo: Beijing, China, 2018. https://doi.org/10.5281/zenodo.1476555.

[34] Essen Folk Song Database. 2013. Available online: accessed on 16 September 2025).

[35] Vaswani A.; Shazeer N.; Parmar N.; et al. Attention is All you Need. In Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4-9 December 2017.

[36] Kang C.; Lu P.; Yu B.; et al. EmoGen: Eliminating Subjective Bias in Emotional Music Generation. arXiv 2023, arXiv:2307.01229.

[37] Simpson E.H. The interpretation of interaction in contingency tables. J. R. Stat. Soc. Ser. 1951, 13, 238-241.