Language Independent Speech-to-Singing-Voice Conversion

抄録

This research addresses the challenge of converting spoken voice into singing voice in a language-independent manner. Traditional speech-to-singing systems often rely on language-specific phoneme alignment or require parallel singing datasets, which limits their applicability across languages and speakers. To overcome these constraints, the authors propose a novel framework that utilizes voiced/unvoiced (V/UV) classification and music state modeling to align speech with musical scores without relying on linguistic content. The approach begins by extracting a V/UV state sequence from input speech using a convolutional layer built on top of a pretrained HuBERT model. Simultaneously, a music state sequence is generated from a monophonic musical score using a decay function that models note intensity over time. These two sequences are then aligned using Dynamic Time Warping (DTW), allowing the system to synchronize speech features with musical timing and pitch. After alignment, the World vocoder is employed to analyze and synthesize the singing voice. The spectral and aperiodic components of speech are aligned to the music sequence, while pitch is replaced with musical pitch to produce the final singing output. Experimental results demonstrate the effectiveness of the proposed V/UV classification using the ATR speech database. The system could generate singing voices from spoken input without requiring phoneme-level annotations or parallel singing data, but still, there is room for quality improvement.