ZipL-Dialog: Memory-Efficient Long-Form Spoken Dialog Synthesis via Latent Flow Matching
Jihwan Kim, Nam Soo Kim
Abstract: Zero-shot dialog TTS benefits from flow-matching, but minute-scale generation on dense mel-spectrograms causes severe memory bottlenecks, often forcing unnatural chunked synthesis. We propose ZipL-Dialog, which shifts conditional flow-matching into a 4x time-compressed (25 Hz) latent space. To preserve acoustic fidelity under compression, we employ a deterministic mel autoencoder with auxiliary mel-domain supervision and optimize the ZipFormer's hierarchical downsampling schedule. Evaluations demonstrate that ZipL-Dialog reduces maximum peak GPU memory by 11.22x and accelerates inference by 2.23x over the baseline. This enables seamless, single-pass multi-minute dialog synthesis on commodity GPUs while maintaining perceptual naturalness.
1. CoVoMix2 Dialog Generation
Comparison of our proposed ZipL-Dialog against the baseline models on the CoVoMix2 test set. Notice the significant reduction in peak memory and inference time while maintaining natural dialog flow. We additionally include CoVoMix2 baseline samples (sourced from the official CoVoMix2 demo)
Dialog Example 1
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialog to Synthesize:
[S1] Sun-set hotel. May I help you?
[S2] Yes, I have booked a room for 24th. It’s a double room.
[S1] Hold on, please. Let me check it for you. Yes, you’re right. You will keep it for 3 days.
[S2] Well, now I want to change the date from 24th to 28th.
[S1] OK, that shall be arranged.
Model
Audio
Peak Mem. (MB)
Inference (s)
RTF
Ground Truth
-
-
-
VibeVoice 1.5B AR Baseline
5390.93
41.52
2.71
ZipVoice-Dialog NAR Baseline (Single-pass)
1513.96
2.52
0.15
ZipL-Dialog (Ours)Latent NAR (Single-pass)
925.13
2.33
0.15
Dialog Example 2
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialog to Synthesize:
[S1] Why’d you pull me over?
[S2] Are you aware that you drove through a red light?
[S1] I ran a red light?
[S2] Yes, you did.
[S1] I apologize, but I didn’t realize that I did.
[S2] Weren’t you taught that yellow means slow down, not speed up?
[S1] I did learn that.
[S2] So, then why did you speed up?
[S1] I don’t know what to tell you.
[S2] I’m going to have to write you a ticket.
[S1] I understand.
[S2] Here you go. Don’t do that again.
Model
Audio
Peak Mem. (MB)
Inference (s)
RTF
Ground Truth
-
-
-
VibeVoice 1.5B
5391.08
88.73
2.56
ZipVoice-Dialog
2169.50
2.71
0.11
ZipL-Dialog (Ours)
962.89
1.37
0.06
Dialog Example 3
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialog to Synthesize:
[S1] Hey man, you wanna buy some weed?
[S2] Some what?
[S1] Weed! You know? Pot, Ganja, Mary Jane some chronic!
[S2] Oh, umm, no thanks.
[S1] I also have blow if you prefer to do a few lines.
[S2] No, I am ok, really.
[S1] Come on man! I even got dope and acid! Try some!
[S2] Do you really have all of these drugs? Where do you get them from?
[S1] I got my connections! Just tell me what you want and I’ll even give you one ounce for free.
[S2] Sounds good! Let’s see, I want.
[S1] Yeah?
[S2] I want you to put your hands behind your head! You are under arrest!
Model
Audio
Peak Mem. (MB)
Inference (s)
RTF
Ground Truth
-
-
-
VibeVoice 1.5B
5266.47
98.91
2.57
ZipVoice-Dialog
2002.97
2.71
0.09
ZipL-Dialog (Ours)
937.73
1.67
0.06
Dialogue Example 4
* Note: The prompts and transcript below are from the official CoVoMix2 test set. However, the ground truth audio for this specific long-form sample is not publicly available in their official release.
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialogue to Synthesize:
[S1] Excuse me. I have an appointment with Mr. Li at nine. May I come in?
[S2] Yes, come in please. I am Mr. Li. You must be My Liu, right?
[S1] Yes, I am My Liu. Thanks.
[S2] I'd like to start this interview with some questions. Why do you think you are qualified for this position?
[S1] According to your advertisement, you want an experienced software engineer. I think my background meets the requirement of this position.
[S2] Then tell me something about your background.
[S1] My major was computer science when I was at college, and I am quite familiar with Visual C + + and Java language.
[S2] Well, what do you think about the development in computers?
[S1] The developments in software are going ahead very quickly and more and more problems are resolved by software. In some regions, the hardware is completely replaced by software. So I think the software industry has a great future.
[S2] Have you ever designed any programs concerning network?
[S1] Yes, I have designed some programs for the network with Visual C + + and I have passed the test for programmers - MUSE.
[S2] Have you got anything to ask me?
[S1] Yes, can you tell me what's my responsibility in this position?
[S2] Yes, of course. You would be responsible for the development of software products.
[S1] I see. This is my advantage.
[S2] Good. Have you got any other questions?
[S1] No.
[S2] Ok, I will contact you in a week. See you.
[S1] Thank you, bye-bye.
Model
Audio
Peak Mem. (MB)
Inference (s)
RTF
Ground Truth
N/A (Not released)
-
-
-
VibeVoice 1.5B
5658.54
160.71
1.48
ZipVoice-Dialog
15291.17
8.05
0.09
ZipL-Dialog (Ours)
1783.71
1.44
0.02
2. OpenDialog Test Set Dialog Generation
Comparison on the highly expressive OpenDialog test set. We additionally include DiA and Mooncast baseline samples (sourced from the official ZipVoice-Dialog demo).
Dialog Example 1
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialog to Synthesize:
[S1] I'm a big bookworm.
[S2] Yeah, I have a few books-
[S1] Yeah.
[S2] ... that I've been trying to pick up, but, um, haven't had the time yet. But, um, I think I'll get into it at some point this summer.
[S1] Uh, what are the titles?
[S2] Oh, um, okay, well, one of them, it's, um, it's the, um, biography of, uh, Alexander Hamilton. I bought the book-
[S1] Oh.
[S2] ... when, when Hamilton first came out.
Model
Audio
Peak Mem. (MB)
Inference (s)
RTF
DiA
-
-
-
Mooncast
-
-
-
VibeVoice 1.5B
5349.98
107.03
2.60
ZipVoice-Dialog
1736.97
2.42
0.10
ZipL-Dialog (Ours)
962.62
1.44
0.05
Dialog Example 2
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialog to Synthesize:
[S1] Yeah, well, what were, what were you gonna say? 'Cause I was just gonna change subject.
[S2] Oh, well, I, I was just going to ask you, um, have you ever encountered any signs that you missed or any hints, like I did?
[S1] Oh, not that I know of. Well, okay, for my life, oh my God, a sob story type, I'm just kidding. But, um, it's, it's always, um, uh, at least I, I have, I haven't known anyone to be interested, or like, if they did, they were, like, too subtle about it, you know? 'Cause, uh, I'm still on the, like, um, Yeah, like, uh, I mean, there's one person that I think maybe he could've been interested, but honestly, like, we're just, like, really close friends, so I don't know if, I, I don't even know if he was interested, you know? Um, it's just, um, it's just like occasionally, like, the, the vibes are like kinda off, but that's all.
Model
Audio
Peak Mem. (MB)
Inference (s)
RTF
DiA
-
-
-
Mooncast
-
-
-
VibeVoice 1.5B
5356.59
129.60
2.60
ZipVoice-Dialog
5932.97
5.49
0.09
ZipL-Dialog (Ours)
1165.04
1.76
0.03
Dialog Example 3
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialog to Synthesize:
[S1] I know, it could be worse, but-
[S2] They didn't count the pages for you, right?
[S1] Yeah, but, meh. What have you spent money on lately?
[S2] I actually haven't been spending money on games that much.
[S1] Oh, yeah, that's good.
Model
Audio
Peak Mem. (MB)
Inference (s)
RTF
DiA
-
-
-
Mooncast
-
-
-
VibeVoice 1.5B
5253.75
44.90
2.55
ZipVoice-Dialog
748.45
1.42
0.15
ZipL-Dialog (Ours)
883.09
1.48
0.15
Dialog Example 4
Voice Prompt [Speaker 1]
Voice Prompt [Speaker 2]
Target Dialog to Synthesize:
[S1] Well, what was it, uh, like elementary to middle school, as long as I didn't fail, I was good.
[S2] That was, yeah, that's, that was my mindset, uh, throughout high school. If you, if you pass it, uh, and you did your, and you did your best at the last, uh, at the final stretch, you have no shame. There's nothing to be shameful about.
[S1] Yeah, but, uh, in high school, I actually started worrying about, like, GPAs and all that.
[S2] Oh, yeah.
[S1] Yeah, I'm not gonna lie, history, history class was, was a vibe, like.