At the beginning of 2020, SCIC announced a pilot project - Speech to Text (S2T) - to develop a corporate speech recognition solution that leveraged the latest advancements in artificial intelligence and natural language processing. Since then a lot of progress has been made, thanks to the contribution of SCIC’s interpreters. SCIC uses Microsoft Azure – the current cloud solution in the Commission which provides a standard speech recognition service - to create transcriptions, which are subsequently assessed by volunteer interpreters. Any errors are corrected and the revised transcription is fed back into the model to improve its accuracy, and in this way the machine ‘learns’ and gets better.
Colleagues from the 23 booths participated in the project with their linguistic expertise: verifying and validating automated transcriptions in their respective languages produced by Microsoft Azure. This ensures the multilingual nature of the project and contributes to SCIC establishing itself as a key player in the field of speech activities within the EC.
After the initial pilot phase, the number of interpreters working on the project has been gradually expanded and the project is now at cruising speed. As interpreters verify the automated transcription, they need to see the text with a new set of eyes altogether. For example, they have had to learn how to ignore punctuation, capital letters, and hyphens as the transcription at this stage comes out ‘naked’. That feels counterintuitive to language professionals and the shift has not been easy.
Each week, the volunteers ‘meet’ to discuss issues they come across during their work. Discussions cover a broad range of transcription issues and help improving the transcription guidelines available to all participants. Gems produced by the machine, like bringing up Applebee repeatedly in a Greek text or Brad Pitt in a Portuguese transcription, add a light-hearted touch to the discussions. For the time being the machine performs better in some languages than others, and it has come up with some very funny transcriptions, such as transcribing ‘commissioner Wojciechowski’ ‘commissioner virtual husky’ and ‘charity’ as ‘cherry tea’, or transcribing Brexit as ‘backseat’, or ‘to fill in the cows’ for ‘to fill up vacancies’. As the machine gets better of course the humorous mistakes will be fewer and much further between.
For the interpreters to have something to validate, though, there is a lot of activity behind the scenes. Large audio files need to be downloaded, prepared and uploaded into Microsoft Azure. This is done by the technical teams, as well as some specially trained interpreters. Needless to say, any project involving such a large group of people requires a whole lot of coordination, which has been taken on board by three interpreters, from the Romanian, Dutch and Portuguese booths respectively. They act as the interface to some extent, between the technical team and the volunteer interpreters. They are also in charge of sending the work packages to colleagues, answering their questions and helping with any issue they may encounter. After the validation of the transcription made by ‘the machine’, colleagues send back comments they have concerning the corrections they introduced to the coordinators, who then report this to the technical team.
One of the most exciting discoveries for the team has been to see how the system behaves and how to make it evolve; interpreters have described it as being like teaching a multilingual child basic things like not to mix languages, but also letting this ‘robot’ child its freedom to evolve by itself. The machine learns to cope with the difficult parts, to correct itself, to refine constantly its technique and not to repeat the same mistakes.
It is a future-oriented project and a breakthrough in terms of the prospect it offers, especially for the hearing-impaired. It will lead to a whole range of very useful and concrete applications, not only across the European Commission, for example, in the form of automated subtitles in press conferences, but also, in the longer run, by way of offering tailor-made solutions to European citizens. Making material accessible to persons with disabilities is one of the main driving forces behind the project.