The use case
Build a conversational voice assistant for the Citizens Advice Scotland (CAS) phone service, a charity that offers free legal advice, practical help and information on consumer and political rights.
The assistant had to determine which type of service the user was in need of and collect their postcode. The postcode was then used to determine which local 'bureau' the user would be connected to, with some postcodes having multiple options to choose from.
User research and testing
I first wanted to understand why users would call this service in the first place. This was challenging as we were not able to contact end-users due to data privacy concerns. To overcome this, I talked with client stakeholders and created documentation on what services they offer.
From here, I created flow charts of user journeys and early prototypes of the bot architecture. As the assistant was using intent-based NLU, all utterances had to be carefully written to meet client expectations. I wrote carefully crafted utterances to ensure a warm and comforting tone of voice that also stayed suitably formal due to the nature of the use case.
This architecture and content was shared with stakeholders for validation and feedback. I pushed for several rounds of feedback but was careful not to over-optimise on feedback that was not from end-users.
A key consideration for this use case was the performance of our automated speech recognition (ASR) systems against Scottish dialects and place names. We deployed a pilot voice assistant that collected postcodes and was shared with speakers of Scottish dialects. I then manually assessed the quality of our ASR and highlighted problematic phonemes and clusters. This data was collated and presented to Machine Learning Engineers to improve our models.
Images of internal PolyAI work blurred due to IP constraints.
Voice
The voice of the assistant was carefully selected. I collated several voice actor samples that were friendly, natural and were speakers of Scottish dialects. After working with the client to choose a voice, I coached the actor to deliver a natural, conversational tone.
As the voice of the assistant is the primary interface the user interacts with, I used pre-recorded voice actor audio. At the time of development, this presented a far more natural-sounding assistant compared to contemporary synthetic voices and typically increased user engagement.
As non-synthesised voices require concatenating audio files together during runtime, I implemented new techniques to capture different intonations of values that would be collected from a user and repeated back. For example, postcode alphanumerics were recorded with different intonations that mapped depending on the position of the digit or letter in the string. When played in sequence, these values therefore sounded more natural and moved the attention away from the system they were interacting with and instead to the task at hand.
Design and Deployment
In my discovery, I found CAS had no 'back up' support team to transfer the user to should things fail. If a caller was not able to be understood, this would have to be solved by the assistant. Recognising that many vulnerable users in difficult situations would be interacting with the system, I went beyond the initial brief and designed extensive behaviour that would direct users to alternate solutions. I also designed a version of the bot that could be operated with keypad input, rather than voice, to account for users with accessibility needs.
The main flow of the agent was, after feedback, implemented by myself and a Software Engineer. After the initial build, this was presented to the client team for testing.
In order to get further validation, I introduced mock user persona testing. Mock user personas were written by me and given to PolyAI staff with no context on the project. They were told to adopt the user persona and goals and provide quantitative and qualitative feedback.
The rounds of feedback from client and internal testers were used to further improve the bot while I was waiting for other technical work to finish before go-live.
After iterative rounds of improvements, the project went live to a small volume of end users as I insisted on end-user validation before scaling to 100% of calls.
After go-live, I manually reviewed user interactions to find failures and areas of improvements. This involved filtering calls by metrics to target interactions that indicated quantifiable failures (such as calls that could not be routed or were over a certain duration). I also assessed random samples of calls to provide qualitative assessments of user experience.
Exports from these reviews were grouped, collated and shared with the team to prioritise. These improvements were then iteratively rolled out until the bot reached acceptable success levels.
For example, I identified a number of calls failing at the postcode collection stage. I dug into agent logs to ensure our technology was working as expected but found the drop in performance was caused due to users providing English postcodes. I worked with engineers to validate users were not being incorrectly routed to our system but were instead calling the wrong number. To mitigate this issue, I designed and implemented new behaviour that would recognise postcodes from outside of Scotland, ask a user if they were calling from a different country, and then explain what service they had called. This meant that only this user group would hear this information, as it was not relevant to the vast majority of users and would make their experiences worse. This resulted in increased success rates and reduced callbacks from outside of Scotland.
Outcomes
Before deployment, callers requiring local advice could be waiting days for a local advisor to call them back. After deployment, callers can reach an expert adviser in their local area in under 30 seconds. Call success for the project has consistently been at over 90% since go-live.
The project won a 2023 ScotlandIS Tech for Good Award and was runner-up for DigiLeaders AI Innovation of the Year.
George Eckton, Director of Advice Services at CAS said:
“The PolyAI voice assistant has allowed our network to solve the long-standing issue of routing all calls to our collective entry portal to the most local Citizens Advice Bureaux for the client. For such a major project, the process was swift – we hatched the plan in October 2021 and deployed it in early February 2022. This has been transformative in terms of ensuring our clients get high quality, impartial, free and confidential advice that the CAB network is justifiably proud of from local advisors within their communities”.