Singing Voice–Melody Co-Generation and Automatic Arrangement for Intelligent Music Composition
Keywords:
intelligent music composition, singing voice synthesis, melody generation, automatic arrangement, generative modeling, music information retrieval, socio-technical systemsAbstract
The integration of singing voice synthesis with melodic co-generation and automatic arrangement represents a substantial shift in intelligent music composition. This paper presents a system-level examination of coupled vocal and musical generation, focusing on architectural trade-offs, multimodal representation, training infrastructure, evaluation, fairness, and governance. Rather than treating voice synthesis and melody generation as separate downstream tasks, the paper argues for a coupled modeling paradigm in which symbolic musical structure, linguistic content, and acoustic vocal detail are aligned through hierarchical representations. The study examines how speech synthesis architectures, audio language models, symbolic music generators, and arrangement systems can inform this paradigm. Attention is given to the structural tensions between end-to-end integration and modular engineering, between high-fidelity acoustic modeling and interpretable symbolic control, and between expressive diversity and deterministic consistency. The paper further discusses data requirements for singing-specific corpora, the role of automatic arrangement as constrained generation, and the robustness risks that arise when generative models are deployed in creative workflows. Infrastructural concerns include compute demands, dataset governance, licensing, voice identity protection, and the environmental footprint of large-scale training. Evaluation frameworks are considered in terms of objective metrics and human musical judgment, with particular attention to the inadequacy of speech-oriented or instrument-oriented metrics for sung expression. The discussion extends to policy implications, including consent, provenance, cultural representation, and the need for transparent model documentation. The paper concludes that sustainable progress in singing voice and melody co-generation requires not only advances in generative architectures but also institutional support for open benchmarks, ethical data collection, and interdisciplinary collaboration among engineers, musicians, and legal scholars.
References
1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
2. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., ... & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.
3. Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., ... & Kavukcuoglu, K. (2018). Efficient neural audio synthesis. In Proceedings of the 35th International Conference on Machine Learning (pp. 2410-2419). PMLR.
4. Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33, 17022-17033.
5. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2019). FastSpeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems, 32.
6. Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., ... & Wu, Y. (2018). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4779-4783). IEEE.
7. Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., ... & Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. In Interspeech 2017 (pp. 4006-4010). ISCA.
8. Dhariwal, P., Jun, H., Payne, C., Kim, J., Radford, A., & Sutskever, I. (2020). Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341.
9. Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., ... & Frank, C. (2023). MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.
10. Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., ... & Adi, Y. (2024). Simple and controllable music generation. Advances in Neural Information Processing Systems, 36.
11. Dong, H.-W., Hsiao, W.-Y., Yang, L.-C., & Yang, Y.-H. (2018). MuseGAN: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 34-41.
12. Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C., Shazeer, N., ... & Eck, D. (2019). Music Transformer: Generating music with long-term structure. In International Conference on Learning Representations.
13. Briot, J.-P., Hadjeres, G., & Pachet, F.-D. (2020). Deep learning techniques for music generation: A survey. Neural Computing and Applications, 32(20), 16171-16200.
14. Herremans, D., Chuan, C.-H., & Chew, E. (2017). A functional taxonomy of music generation systems. ACM Computing Surveys, 50(5), 1-30.
15. Ji, S., Yang, X., & Luo, J. (2023). A survey on deep learning for symbolic music generation: Representations, algorithms, evaluations, and challenges. ACM Computing Surveys, 56(1), 1-39.
16. Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., ... & Eck, D. (2019). Enabling factorized piano music modeling and generation with the MAESTRO dataset. In International Conference on Learning Representations.
17. Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., ... & Ellis, D. P. W. (2014). mir_eval: A transparent implementation of common music evaluation metrics. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 367-372).
18. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.
19. Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., ... & Zeghidour, N. (2023). AudioLM: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 2523-2533.
20. Bittner, R. M., Salamon, J., Tierney, M., Mauch, M., Cannam, C., & Bello, J. P. (2014). MedleyDB: A multitrack dataset for annotation-intensive MIR research. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 155-160).
21. Simon, I., Morris, D., & Basu, S. (2008). MySong: Automatic accompaniment generation for vocal melodies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 725-734). ACM.
22. Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., & Simonyan, K. (2017). Neural audio synthesis of musical notes with WaveNet autoencoders. In Proceedings of the 34th International Conference on Machine Learning (pp. 1068-1077). PMLR.
23. Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Cao, Y., & Wang, Y. (2019). Hierarchical generative modeling for controllable speech synthesis. In International Conference on Learning Representations.
24. Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017). Convolutional recurrent neural networks for music classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2392-2396). IEEE.
25. Oore, S., Simon, I., Dieleman, S., Eck, D., & Simonyan, K. (2020). This time with feeling: Learning expressive musical performance. Neural Computing and Applications, 32(4), 955-967.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 International Journal of Artificial Intelligence Engineering and Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
