Web and mobile applications combined with smart devices like
Amazon Echo and Google Home can provide very interesting interactions today.
Recently, Google Home announced
an amazing feature that you can listen the specific steps of your favorite meal
recipe while cooking. You can view the video on YouTube. For more information, see Google Home Help.
For a few months, I have been writing about various AWS
services in my blog. I have created a digital card store web application
for managing digital game cards and added different functionalities by using various
AWS services. We can add a card with a card image, put the card on sale and buy
a card in this simple application.
After Google Home announcement, I was thinking whether it
was possible to notify the owners of the cards with a natural speech when their
cards are sold. With Amazon Polly, it sure is.
Amazon Polly, is a service that turns text into speech. With Polly, we can create
applications that talk in a natural human like voice. Polly supports 47
different voices across 24 languages and Turkish is one of them :). You can
try Filiz, a Turkish female voice, here.
In this post, I will use Amazon Polly to generate a
speech based notification that notifies the users when their cards are bought
by other users. I will use Server Sent Events for providing real time notification and play an audio
file generated by Amazon Polly. The picture below shows the notification
flow.
For this post, I will use English voice. You can listen a
sample speech here,
generated for the text below.
Your card,
Gandalf, has been bought by John Doe, for 45$
I will use the code I have developed in this post as a
starting point, which can be found in my GitHub repository.
Also you can find a comparison of Server Sent Events with
Long Polling, WebSockets and Comet here.
The steps are below.
1. Enable Server Sent Events
2. Send notification when cards are bought
3. Enable Polly
4. Generate audio with Amazon Polly
5. Change the dashboard to play audio notifications
Let's start.
1.
Enable Server Sent Events
I will use Spring's SseEmitter class to implement Server Sent Events. Add sseEmitters field to the UserController class like the below to hold a map from user name to the
notification channel for that user.
@Controller
public class UserController {
public static final String USER_KEY_FOR_SESSION = "USER";
private static Map<String, SseEmitter> sseEmitters = new HashMap<String,
SseEmitter>();
Add the support methods like the below for creating a new
notification channel for a user, getting the channel for a user, removing the
channel for a user and sending notification to the user.
private synchronized SseEmitter newEmitterForUser(String username) {
SseEmitter emitter = new SseEmitter();
Runnable remover = new Runnable() {
@Override
public void run() {
removeEmitter(username);
}
};
emitter.onCompletion(remover);
emitter.onTimeout(remover);
sseEmitters.put(username, emitter);
return emitter;
}
private synchronized SseEmitter getEmitterForUser(String username) {
return sseEmitters.get(username);
}
synchronized void removeEmitter(String username) {
sseEmitters.remove(username);
}
void notifyUser(String username, String eventName, Object data) {
SseEmitter emitter = getEmitterForUser(username);
if (emitter != null)
try {
emitter.send(SseEmitter.event().name(eventName).data(data));
} catch (Exception e) {
e.printStackTrace();
}
}
Add feed method like the below
for establishing the notification channel for the user when a user is logged
in. This method will be called from dashboard.jsp and it will create a channel for sending notifications
from the server to the browser.
@RequestMapping("/feed")
public ResponseBodyEmitter feed(HttpSession
session) {
SseEmitter
emitter = null;
User
user = userfromSession(session);
if (user != null) {
emitter
= newEmitterForUser(user.getUsername());
}
return emitter;
}
2. Send
notification when cards are bought
Add CardSoldEvent
class to the com.cardstore.entity package with the fields below. This class will be used to
hold data about a card sold event.
public class CardSoldEvent {
private String name;
private double price;
private String oldOwner;
private double oldOwnerBalance;
private String newOwner;
public CardSoldEvent(String name, double price, String oldOwner, double oldOwnerBalance, String newOwner) {
super();
this.name = name;
this.price = price;
this.oldOwner = oldOwner;
this.oldOwnerBalance = oldOwnerBalance;
this.newOwner = newOwner;
}
Add userController
field to CardController
class like the below.
@RestController
public class CardController {
@Autowired
CardRepository cardRepository;
@Autowired
UserRepository userRepository;
@Autowired
UserController userController;
And add the lines below to the buyCard method to send a
notification to the seller of the card. The notification is sent as an event
with the name cardSold.
CardSoldEvent event = new CardSoldEvent(cardToBuy.getName(), cardToBuy.getPrice(), seller.getName(), seller.getBalance(), currentUser.getName());
userController.notifyUser(seller.getUsername(), "cardSold", event);
3.
Enable Polly
Add Maven dependency for Amazon Polly SDK.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-polly</artifactId>
<version>1.11.62</version>
</dependency>
Add PollyHelper
class with the code below. Amazon Polly is not available
in every region for now. region parameter is used for setting the region that will be
used. synthesize method
creates an audio stream for the text in the output format requested.
public class PollyHelper {
private final AmazonPollyClient polly;
private final String voiceId = "Joanna";
public PollyHelper(Region region) {
// create an Amazon Polly
client in a specific region
polly = new AmazonPollyClient();
polly.setRegion(region);
}
public InputStream synthesize(String text, OutputFormat format) throws IOException {
SynthesizeSpeechRequest synthReq = new SynthesizeSpeechRequest().withText(text).withVoiceId(voiceId).withOutputFormat(format);
SynthesizeSpeechResult synthRes = polly.synthesizeSpeech(synthReq);
return synthRes.getAudioStream();
}
}
4. Generate audio with Amazon Polly
Create AudioController
class as below to return a MP3 file after generating it
by Amazon Polly. I have used eu-west-1 region as it is the only region in
Europe that Polly is available as of today.
@Controller
public class AudioController {
@RequestMapping(path="/audio", produces="audio/mpeg3")
public @ResponseBody byte[] textToSpeech(@RequestParam("msg") String msg) throws IOException {
PollyHelper helper = new PollyHelper(Region.getRegion(Regions.EU_WEST_1));
InputStream is = helper.synthesize(msg, OutputFormat.Mp3);
return StreamUtils.copyToByteArray(is);
}
}
5. Change
the dashboard to play audio notifications
Add the functions below to the dashboard.jsp file. initNotifications method creates an EventSource object that make a
request to /feed url. This request creates a notification channel from
server to the browser. When a notification sent from the server, EventSource object calls the message handler.
In this code, we add an event listener for the 'cardSold' event. When a 'cardSold' event is sent from the server, the text for the notification is shown in
the notification area and the audio is played with the playAudio method.
Audio will be requested from /audio
url, which corresponds to AudioController.textToSpeech method that is created in the
previous step.
function messageForData(data) {
return "Your card, " + data.name + ", has been bought by " + data.newOwner + ", for " + data.price + "$";
}
function playAudio(data) {
var audio = new Audio('audio?msg=' + messageForData(data));
audio.play();
}
function animateSpeaker() {
$('#speaker').fadeTo("slow", 0.15).delay(400).fadeTo("slow", 1).delay(400).fadeTo("slow", 0.15).delay(400).fadeTo("slow", 1).delay(400).fadeTo("slow", 0.15).delay(400).fadeTo("slow", 1);
}
function setNotification(data) {
var msg = messageForData(data);
var spn$ = $('<span/>').html(msg + ' <img id="speaker"
src="images/speaker.png"/>');
$('#notifications').empty().append(spn$);
animateSpeaker();
}
function processCardSoldEvent(data) {
setBalance(data.oldOwnerBalance);
setNotification(data);
playAudio(data);
}
function initNotifications() {
if (typeof (EventSource) !== "undefined") {
var source = new EventSource("/feed");
source.addEventListener('cardSold', function(event) {
var data = JSON.parse(event.data);
processCardSoldEvent(data);
});
}
}
After creating the functions we can use them as below. initNotifications method is called to initialize the notification
functionality when the dashboard page is loaded.
<body onload="initNotifications()">
<div id="notif-container">
<div id="notif-title">
<img src="images/notification.png"/>
<span>Notifications</span>
</div>
<div id="notifications"></div>
</div>
After completing the application, you can use it as shown in the video below.
Other
Considerations
In this post, I have used a fixed language and voice. In
real applications, we can enable users to select their preferred language and
the voice they like.
Also I have developed a generic endpoint to generate
audio for any text sent by the browser for simplicity. In production, the audio
should be prepared and possibly cached in the server to prevent uncontrolled
Polly usage.
For simplicity, I have used Server Sent Events for
real-time notification. SSE is not supported by IE, please check browser support
here.
I want to remind one more thing about SSE. SseEmitter
objects are valid only in the JVM they are created. If we use multiple EC2
instances, SseEmitter may not be in the JVM that processes the buyCard request.
In this post, I have used one instance for simplicity. In production, each EC2
instance that establish the SseEmitter connection would subscribe to a topic
specific to a user.
Summary
In this post, I have developed a notification
functionality with Server Sent Events and Amazon Polly to notify users with a
natural speech.
The code can be found here.
To read more about AWS services, stay tuned.
