Transplantation of Conversational Speaking Style with Interjections in Sequence-to-Sequence Speech Synthesis

Raul Fernandez1, David Haws1 Guy Lorberbom1 Slava Shechtman1 and Alexander Sorin1
1Authors are listed alphabetically and contributed equally

Submitted to Interspeech 2022

Inventory and definitions of Dialog Acts used to label events within dialogues

Orthographic forms and functions of interjections

Audio Samples


Speaker F1
Systems with Transpanted Style References Script
Base Adv Aug Target Speaker Target Style
<waiting> <uh>, one sec, let me take a look. </waiting> <agreement> Yes, definitely! All rooms include breakfast, which is served between 7 and 10 in the morning. </agreement>
<surprise> <aha>, I think I figured it out! I think you're trying to use the wrong email address to log in. </surprise>
Let me see. <waiting> <uh>, it looks like the item may be out of stock now, but let me put you on hold while I take a look at this, OK? </waiting>
<positive_feedback> <oh> no problem! I'll show you! </positive_feedback> <instruction> Start by logging in, and then go to where it says "Personal Banking" at the top of the page. </instruction>
<empathy> I apologize John. But unfortunately your order is already at the warehouse waiting to go out, and, <uh>, I can't make any changes. Sorry. </empathy>
<greeting> Hi Ralph! I'm doing great, thanks! I'll be happy to help you solve your issue. What can I do for you today? </greeting>
<waiting> I'm so sorry about the wait. Please bear with me while my system comes back. <hmm>, I'm afraid it's <emphasis> really </emphasis> slow today. </waiting>
<instruction> I'm mailing you a link so that I can access your desktop remotely. Click on it, and check the box that says <emphasis> accept. <emphasis> Then I will take over. <instruction>
<surprise> <oh> , I think I know what happened! You're using the wrong credentials. </surprise> Let me send your username to the email address we have on file.
<empathy> You're just outside our free delivery area. I'm sorry about the inconvenience. </empathy> <positive_feedback> But I'll be happy to expedite your order and get it to you A.S.A.P. </positive_feedback>
Let me look into it. <surprise> <aha>, I figured it out! Your first payment bounced, and we weren't able to withdraw from your account until 2 days after the due date. That would explain it! </surprise>
<agreement> <uh-huh>, I agree! </agreement> <uncertainty> I'm not sure why there's an overdraft fee on your statement. I have to say I find this a bit strange. </uncertainty>
<empathy> <huh> looks like we've made a mistake. My apologies. </empathy> I'm gonna go ahead and reimburse you for the price of the extra item.
<positive_feedback>Well, I have some excellent news! The loan was approved at a fixed interest rate of only 3 percent! </positive_feedback>
<uncertainty> That's confusing. I'm not sure what happened. Did you report one of them stolen or damaged by any chance? </uncertainty>
Let me confirm your search. <agreement> You want a reservation for <emphasis> 4 </emphasis> people, at an <emphasis> Italian </emphasis> restaurant, around <emphasis> 8 </emphasis> PM. </agreement> Is that correct?
<uncertainty> I don't see any work being done in your area, so, <um>, I can't explain your connectivity issues. </uncertainty> Let me troubleshoot this!
<empathy> Yes, I'm afraid so. </empathy> <waiting> Sorry, <um>, do you mind if I place you on hold for another minute? Thanks for being so patient with me here. </waiting>
<farewell> Thanks for giving us a call Steven. Enjoy the rest of your afternoon! Bye for now! </farewell>
<positive_feedback> Glad to help! </positive_feedback> If you wish to stay on the line, there's a brief survey at the end to help us improve our service. <thanks> Thanks in advance if you choose to participate. </thanks>


Speaker F2
Systems with Transpanted Style References Script
Base Adv Aug Target Speaker Target Style
<waiting> <uh>, one sec, let me take a look. </waiting> <agreement> Yes, definitely! All rooms include breakfast, which is served between 7 and 10 in the morning. </agreement>
<surprise> <aha>, I think I figured it out! I think you're trying to use the wrong email address to log in. </surprise>
Let me see. <waiting> <uh>, it looks like the item may be out of stock now, but let me put you on hold while I take a look at this, OK? </waiting>
<positive_feedback> <oh> no problem! I'll show you! </positive_feedback> <instruction> Start by logging in, and then go to where it says "Personal Banking" at the top of the page. </instruction>
<empathy> I apologize John. But unfortunately your order is already at the warehouse waiting to go out, and, <uh>, I can't make any changes. Sorry. </empathy>
<greeting> Hi Ralph! I'm doing great, thanks! I'll be happy to help you solve your issue. What can I do for you today? </greeting>
<waiting> I'm so sorry about the wait. Please bear with me while my system comes back. <hmm>, I'm afraid it's <emphasis> really </emphasis> slow today. </waiting>
<instruction> I'm mailing you a link so that I can access your desktop remotely. Click on it, and check the box that says <emphasis> accept. <emphasis> Then I will take over. <instruction>
<surprise> <oh> , I think I know what happened! You're using the wrong credentials. </surprise> Let me send your username to the email address we have on file.
<empathy> You're just outside our free delivery area. I'm sorry about the inconvenience. </empathy> <positive_feedback> But I'll be happy to expedite your order and get it to you A.S.A.P. </positive_feedback>
Let me look into it. <surprise> <aha>, I figured it out! Your first payment bounced, and we weren't able to withdraw from your account until 2 days after the due date. That would explain it! </surprise>
<agreement> <uh-huh>, I agree! </agreement> <uncertainty> I'm not sure why there's an overdraft fee on your statement. I have to say I find this a bit strange. </uncertainty>
<empathy> <huh> looks like we've made a mistake. My apologies. </empathy> I'm gonna go ahead and reimburse you for the price of the extra item.
<positive_feedback>Well, I have some excellent news! The loan was approved at a fixed interest rate of only 3 percent! </positive_feedback>
<uncertainty> That's confusing. I'm not sure what happened. Did you report one of them stolen or damaged by any chance? </uncertainty>
Let me confirm your search. <agreement> You want a reservation for <emphasis> 4 </emphasis> people, at an <emphasis> Italian </emphasis> restaurant, around <emphasis> 8 </emphasis> PM. </agreement> Is that correct?
<uncertainty> I don't see any work being done in your area, so, <um>, I can't explain your connectivity issues. </uncertainty> Let me troubleshoot this!
<empathy> Yes, I'm afraid so. </empathy> <waiting> Sorry, <um>, do you mind if I place you on hold for another minute? Thanks for being so patient with me here. </waiting>
<farewell> Thanks for giving us a call Steven. Enjoy the rest of your afternoon! Bye for now! </farewell>
<positive_feedback> Glad to help! </positive_feedback> If you wish to stay on the line, there's a brief survey at the end to help us improve our service. <thanks> Thanks in advance if you choose to participate. </thanks>