Automatic translator from Portuguese (voice and text) to Portuguese sign language

Introduction

Disability an evolving concept that results from the interaction between people with disabilities and people with behavioral, orientation or spatial barriers, which prevents a person’s full and effective participation in society on an equal basis with others National Institute for Rehabilitation (3). One of these disabilities is people with deaf and hearing impairment. In Portugal, there are about 100,000 to 150,000 people with some level of hearing loss, and of these, around 30,000 people use Portuguese sign language as their mother tongue [(1); National Institute for Rehabilitation, (2)].

To overcome the existing gaps, it is often necessary to have empathy and think about what can be done to improve our services for disabled citizens. Being hearing impaired and having had certain negative experiences, which intensified with the COVID-19 pandemic, as a result of the mandatory use of the face mask, it made sense to create an automatic translator from voice to text and from text to Portuguese sign language (LGP), which will certainly be useful for people with the same limitations.

The difference between hearing impaired and deaf people lies in the depth of hearing loss and how well they can communicate (4). People with hearing impairment can hear certain sounds, but with difficulty; to circumvent this pathology in order to be able to communicate, they use speech in conjunction with medical devices, such as hearing aids or cochlear implants, depending on the degree of hearing loss (1). People with profound hearing loss, who cannot hear even using hearing aids or cochlear implants, use sign language as communication (1).

According to several authors, the greatest difficulties encountered by the deaf and hard of hearing are poor communication with the listeners, since there is a great ignorance on the part of the hearing community about the deaf community’s own characteristics, namely, LGP, and the lack of access to properly trained interpreters (5).

In addition to the problem of poor communication between hearing-impaired people and listeners, in 2019, the COVID-19 pandemic brought even more complications with regard to communication (6). The mandatory use of face mask to prevent the transmission of the SARS CoV 2 virus started to have a negative impact in this context, since many deaf and hearing impaired people, even those using sign language, need to visualize facial expressions, as well as use lip reading (6).

Paper Informatics, programming and machine learning have a key role in improving the communication of hearing-impaired and deaf people in the society. The work presented here is an example of the application of an automatic translator for written text, intended for hearing- impaired people who do not know Portuguese sign language (LGP), and for LGP, intended for people who can only communicate through sign language.

Portuguese sign language is the sign language used by the Portuguese deaf community, which is characterized as a form of communication through hand movements and facial and body expressions that have their own vocabulary and grammar (1).

Portuguese sign language has a very specific sentence structure, quite different from the structure used in the Portuguese language (LP), as it does not use linguistic connectors and the verb is always applied in the infinitive Goncalves et al. (7). This way, LGP respects a subject-object-verb grammatical syntax (SOV), whereas the LP presents a subject-verb-object sentence structure (SVO).

In Portugal, according to the Portuguese Association of the Deaf, there are about 100,000 to 150,000 people with some level of hearing loss, and of these, about 30,000 people use LGP as their mother tongue [(1); National Institute for Rehabilitation, (2)].

In the last decade, there has been a special interest in the development of machine translators due to the evolution of technology, as well as a greater focus on promoting the social inclusion of the deaf community, making communication between deaf and hearing people more effective.

In 2016, a bidirectional translator of Portuguese sign language (called VirtualSign) was developed, a model that facilitates the access of deaf and hearing-impaired people to digital content, especially educational and learning content (8).

At an international level, it is worth mentioning the automatic translator HandTalk, which uses an avatar named “Hugo”. It is considered the largest automatic translation platform for sign language in the world, and it was elected in 2013 by the United Nations (UN) as the best social app in the world (9). However, according to a questionnaire to the deaf community in Portugal (Appendix A and Appendix B), it was possible to verify that the automatic translators that use avatars are not in fact the best method of communication, since the part of facial/body language expression is compromised because it is a crucial element in the phonology of sign languages.

This way, with this work, we intended to create an automatic translator that doesn’t use an avatar in order not to compromise the parameter related to facial and corporal expression inherent to LGP. A database was created which presents several images with LGP gestures. When the system refers to the word “nurse”, for instance, the respective image with the “nurse” gesture appears together with the caption. For that to happen, it was necessary to use different programming languages in order to achieve a code that can receive the word “nurse” as input by voice recognition and write it in the text field, and the code will search for this word in the library, and if it finds it, it will show on the screen the image of the gesture corresponding to the word “nurse” with the respective legend above the image and its degree of confidence.

Materials and methods

To develop the voice-text and text-LGP machine translator, the authors got access to a set of tools that enabled its proper functioning. Figure 1 presents the flowchart of the process, where it is possible to verify all the steps taken in order to develop the translator.

Each block of the flowchart is briefly explained below, which receives as input a word or a set of words (sentence) entered by the user.

Text input

This stage uses a microphone to receive the input from the user and uses the Google Recognition tool to convert audio to text. Figure 2 shows the interface of the translation system.

Sentence recognition

1. If the spoken sentence is not recognized: “Sentence not found” will be displayed on the screen.

2. If it is recognized: The recognized sentence and its confidence level are displayed on the screen, and these two variables are stored in the computer’s RAM. In Figure 3, one can see the sentence “O enfermeiro chegou ao hospital” (it will be spoken in the SOV structure saying “Enfermeiro hospital chegar”).

Split function

This function splits a string into an ordered list of substrings, makes those substrings into an array, and returns the array. In this case, the function splits the variable sentence into several variables (words), as exemplified next with the sentence “hospital nurse arrive”. If only one word is spoken in the microphone, it just returns the word itself.

Input (text): enfermeiro hospital chegar

Output (function SPLIT): “enfermeiro,” “hospital,” “chegar”

Database (BD)

After the division of the sentence into words by the SPLIT function, the stored word is compared with the words in the library list (the database library shows the words and their respective representative images, if they exist in the BD). The database presents 21 images; however, only 16 images are being recognized, because the words, “Ainda nao,” “Nao ha,” “Nos,” “Dificil” and “OS,” are recognized by the microphone, but they cannot match the respective images.

Existence of the word in a file name.jpg

1. If the word doesn’t exist in the BD: The screen shows the symbol of the image not found, as shown in Figure 4.

2. If the word exists in the DB: Displays the corresponding image that takes the value of the word found, as in the example of the word “enfermeira”, shown in Figure 5.

Results and discussion

To evaluate the machine translation system developed, we applied a questionnaire that proposes a set of tasks, which are to speak five sentences in the translator’s microphone and, after these tasks, to answer seven questions in order to evaluate the translator and his/her performance.

The selected sentences are in SOV structure since the translator has a limitation regarding the ability to change the structure from SVO to SVO. Thus, the translator can recognize the voice, switch to text and display on the screen the image corresponding to that text. The evaluation is done based on the translator’s goal, voice-to-text-to-image conversion.

The sentences selected and the output of each sentence are presented as follows:

1. LGP: FILHA TUA ENFERMEIRA (LP: A tua filha ei enfermeira).

2. LGP: EU CHOCOLATE NAO GOSTAR (LP:Eu nao gosto de chocolate).

3. LGP: FILHO MEU DOENTE (LP:O meu filho estai doente).

4. LGP: ELES LISBOA CHEGAR (LP: Eles chegaram a Lisboa).

5. LGP: ENFERMEIRO HOSPITAL CHEGAR (LP: O enfermeiro chegou ao hospital).

The questionnaire was answered by three people from the Associacao de Surdos do Concelho de Sintra, two of whom were born deaf and both use cochlear implants. The third respondent was born hearing, but due to measles infection, he became profoundly deaf, and he doesn’t use any hearing aid, because he did not adapt. All respondents use LGP to communicate in their daily lives (Figures 6 to 10).

The results obtained are presented in Table 1.

Given the answers obtained, the respondents assessed the automatic translator positively, and it is an asset to the deaf community (Table 1).

Constructive criticism was made at the level of images, which could present a more prominent outline in order to highlight the gesture more and could also have slightly larger images for better visualization of the gesture, if the device presenting the images is at a greater distance. It was also suggested to make the inverse translation of the proposed translation system (from LGP to text), so that listeners understand what is being gestured.

Respondents, despite finding the proposed translation system “very good,” mentioned that they have a preference for LGP interpreters or video translation systems, knowing that both have some limitations. According to them, interpreters are the first choice for them; however; it is necessary to pay for the service all the time, and the State doesn’t help financially in this sense, making it very complicated economically. They also mentioned that, as described in the literature, SNS 24 has communication channels with deaf people through video call and webchat, but according to the respondents, they are constantly busy and end up not being able to contact any interpreter.

Respondents shared the same opinion about the avatar machine translators. According to them, these translators are often very confusing and cannot show emotion, as is the case of the VirtualSign avatar, where they say they cannot understand what is being gestured by the avatar, as the hand arrangement is very confusing in relation to the space. The HandTalk (Libras) translator, however, is one that respondents agree is a fairly good and perceptible translator.

Thus, it can be concluded that the translation system developed had a good evaluation by the respondents, both at the translation level and at the level of understanding of the figures.

Limitations of the study

The major limitation of this study was undoubtedly the little (practically non-existent) information available on LGP and its structure. Undoubtedly, more studies and coherent information on this theme are necessary.

Relative to the translation system developed, it presents some limitations, namely:

1. Of the 23 images that are in the database library, only 16 images are working correctly, because the words “Ainda não” and “Não há” in sign language are represented only with a gesture; however, the translator has “read” them as two distinct words and will look for two images “Ainda” and “não”, for example. The words “Nós”, “Difícil” and “ÓS” are recognized by the microphone, but are not matched with their respective images, as they are accented words

2. This translator cannot switch the text recognized by the microphone from the Portuguese sentence structure (SVO) to the sign language structure (SOV).

3. It is not possible to write directly into the text box to translate.

Conclusion and future prospects

Undoubtedly, the last decade has been marked by strong growth in the area of technology, which has had a positive impact on all sectors, namely, the health sector. However, there are still many obstacles to overcome regarding the adaptation of society to people with disabilities, namely, hearing impairment.

Few works have been developed, at a national level, which tried to create a system that could help this community, but two works should be highlighted, namely, “PE2LGP: From Text to Sign Language (and vice-versa)” by the student Ruben Santos from Tecnico de Lisboa and the “Real-Time Bidirectional Translator of Portuguese Sign Language” by Professor Paulo Escudeiro, responsible for the project, from Instituto Superior de Engenheiria do Porto. Both projects have some limitations, namely, the parameter related to facial and body expressions inherent to LGP and the linguistic level, because although LGP is an official language since 1997 in Portugal, there is very little information about it.

Thus, this study aimed to create an automatic voice and text translator for LGP, which could help hearing-impaired people, without resorting to the use of avatars in order not to compromise the parameter related to facial and corporal expressions inherent to LGP. In the first instance, in order to understand what deaf people and people who use LGP as a means of communication think about translators, a questionnaire was made to this community, and 183 answers were obtained, of which 39 were from people who had some degree of deafness. From this questionnaire, it was concluded that most of the respondents agree that the translator is a good way of inclusion for the deaf community, and what translator is more perceptible to the respondents is the translator by images.

In this way, an automatic translator was developed from several programming languages, and it was necessary to create a database library, where the 23 images with LGP gestures are stored. When opening the translator interface, a button allusive to a microphone appears, and when clicking on it, we can say the word or a set of words we want to translate. Having said this, the translator can receive a given word, as input, by voice recognition and write it in the text field, and the code will look for this word in the library, and if it finds it, it will show on the screen the image of the gesture corresponding to that word with the respective subtitle above the image and its confidence level.

Despite the great progress that the health and technology sectors have made in society in recent decades, there are still many gaps when it comes to adapting society to people with disabilities, of whatever kind. Disabled people need adapted services so that they can have access to them without feeling frustrated, put aside, or even thinking that the problem lies within themselves as individuals, when in fact the problem lies in the lack of means and funds on the part of institutions that fail to adapt services for disabled people.

The translation system was evaluated through a questionnaire made by three deaf people from the Associacao de Surdos do Concelho de Sintra, who use LGP as a means of communication in their daily life. They evaluated that the translator has a good performance and will be useful for the deaf community. The first part of the questionnaire consisted of five sentences that the respondents had to say in the microphone. After this stage, they answered the questionnaire with seven questions related to the good functioning and perception of the translator. The answers were quite similar, as they classified the translator as “very good” and as being a useful translation system for the deaf community.

In future studies, it would be essential to overcome the limitations presented above, as it would also be interesting to use this translator as a means of learning LGP, presenting didactic games for children and dictionaries with the words and the respective gestures to be performed. In a more advanced stage, it would also be interesting to make the translation module from LGP to text, making the translator bidirectional.

References