ReadPaper Blog
Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization
<Society logo(s) and publica- tion title will appear here.> Received XX Month, XXXX; revised XX Month, XXXX; accepted XX Month, XXXX; Date of publication XX Month, XXXX; date of current version XX Month, XXXX. Digital Object Identifier 10.1109/XXXX.2022.1234567 Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization RYOTA KOMATSU1, KOTA KAWAKITA1, TAKUMA OKAMOTO2 (Member, IEEE), AND TAKAHIRO SHINOZAKI1 (Member, IEEE) 1Institute of Science T okyo, Meguro, T okyo 152-8550, Japan 2National Institute of...
Source: Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization
Why syllables?
Why syllables?. spoken language, and have emerged as a foundation for spoken dialogue modeling [4]–[7]. To transfer linguistic knowledge from text LMs to speech LMs, SpiRit-LM introduces word-level speech-text interleav- ing, where textually pretrained LMs are continually trained on sequences that alternate between phonetic and text tokens at word boundaries [8]. However, a fundamental challenge lies in the mismatch of token granularity between speech and text. Learned phonetic tokens typically occur at a high frame rate (12.5–50 Hz), whereas text is encoded using coarser subword tokens. This lower linguistic information density in speech tokens reduces computational efficiency and exacerbates the granularity mismatch, which can hinder speech-text alignment. To mit
The problem
The problem. Ms [23]. In this work, we focus on learning phonetic tokens for linguistic content modeling. To combine the complementary strengths of both token types, phonetic tokens can be integrated into neural audio codecs via semantic distillation [6], [22]. B. Learning coarse phonetic tokens for speech LMs An orthogonal line of research explores learning coarser subword- or syllable-level phonetic tokens. Some training- free approaches apply subword tokenization or deduplication to phonetic tokens [24], [25]. Although simple, these meth- ods still operate at relatively high token frame rates of around 2 VOLUME , <Society logo(s) and publication title will appear here.> 20 Hz. Another direction inserts speech adapters into the LM front-end to aggregate frame-l
Core idea
Core idea. nd WER, as it operates on continuous features with an unbounded bitrate. In contrast, our model achieves the high- est UTMOS score. It can resynthesize clean speech from utterances with background noise such as rain, indicating its robustness to noise. Furthermore, our proposed method matches the TWIST synthesizer in CER and WER at a 2.3× lower bitrate, demonstrating its high coding efficiency. De- spite its higher bitrate, a chunk size of 1 impairs intelligibility due to inaccurate syllable segmentation in Table 3. H. Impact of the merge threshold on downstream tasks Table 6 shows the impact of the merge thresholdτ SylReg on downstream tasks. This threshold controls the token frame rate. The defaultτ SylReg = 0.7yields an oversegmented frame rate of
What improved
What improved. rove with in- creasing depth. Recall peaks at the 8th Transformer layer and decreases at the 9th layer. Although the 9th layer yields the highest precision, this precision can be improved in the subsequent self-segmentation distillation stage. We therefore use the 8th layer for syllabic tokenization. E. Learning dynamics of SylReg Figure 4a plots the learning curve of SylReg on the Lib- riSpeech development set. Recall peaks at 10k steps, and the overall F1 begins to decline after 12k steps. As depicted in Figure 4b, the boundaries between adjacent syllables become blurred, and the similarity matrix shifts toward a more uni- form distribution. Consequently, 10k training steps represent a reasonable choice to prevent representation collapse. F . Speec
Why it matters
Why it matters. ls are largely marginalized. In contrast, acoustic tokens are produced by neural audio codecs trained to faithfully reconstruct input waveforms [21]. Although acoustic tokens can encode general audio, includ- ing environmental sounds and music, they are weakly aligned with textual content [22]. This misalignment hinders lexical, syntactic, and semantic understanding in speech LMs [23]. In this work, we focus on learning phonetic tokens for linguistic content modeling. To combine the complementary strengths of both token types, phonetic tokens can be integrated into neural audio codecs via semantic distillation [6], [22]. B. Learning coarse phonetic tokens for speech LMs An orthogonal line of research explores learning coarser subword- or syllable-lev
