Language and Humor Blog

Imagine you’re a signing Deaf person and you find out about a spoken lecture. You need a human interpreter, but there’s no time. What do you do?

In the UK, soon you’ll be able to use IBM UK’s Say It Sign It (SiSi) translation system.

SiSi converts spoken English to British Sign Language (BSL) via speech recognition, translation, and a computer-animated avatar character.

Demonstration video on Youtube (14 seconds)

Link to video

This is a wonderful invention for simple communication and last-minute occasions. I hope, however, that people don’t try to replace human interpreters with speech-recognition avatars. The nuances of the face, which can carry important grammatical information in sign languages, can’t be perfectly rendered in animation in all their complexity and subtlety.

I also wonder how good the speech-recognition and translation programs are. I suspect the device is really translating into a pidgin sign language that includes elements of BSL and English. The way information is presented in natural sign languages and spoken languages can be quite different. Sign languages use a lot of simultaneity/overlapping of information in space and what are called size and shape specifiers, or classifiers (such as for flat objects or round objects or vehicles or people). Translation is not easy.

Still, I hope IBM can make Say It Sign It better and better.

If you use American Sign Language (ASL) or another sign language to communicate, how do you participate in the cellphone/mobile phone revolution? You don’t very much. You participate in the texting (and E-mail) revolution using written English or another language, but cellphone video quality isn’t good enough to properly capture the flow of sign languages, including their grammar.

But now University of Washington researchers are working to improve video compression with a project called MobileASL.

What makes this a language-related project is how the researchers are trying to compress the video. They’re exploiting how sign languages are perceived.

With sign languages the face is more important than the hands. When people sign, you look at their faces (with your acute “foveal” vision, as opposed to peripheral vision). The signer’s face carries not only the emotions of the signer but grammatical markers as well. With ASL, markers for yes-no questions, wh-questions (who/what/where/when/why/how), negatives, sentence topics, and conditionals are made with the face. The face also conveys commentary on whether something described is ordinary, of poor quality, and so forth.

Thus, instead of compressing the video evenly, the researchers are compressing the area around the signer’s face a little less and the area around the torso a little more. They’re also using somewhat fewer but higher quality frames of video (PDF research paper).

I hope they have success. I would think companies involved with mobile video phones would be interested in this targeted compression. Even if you’re watching people speaking English on video, you want to see their facial expressions and you just want to see their faces.

Two professors, sign language researcher Ronnie Wilbur and animator Nicoletta Adamo-Villani, at Purdue University (Indiana, U.S.) are developing computer animation and virtual reality (VR) using motion capture of human signing to teach math/maths to signing Deaf children via American Sign Language.

From Animation World Magazine:

Two Purdue University professors are studying how to use computer animation and virtual reality to teach deaf children in grades K through 4 about mathematics principles, and to communicate math terminology using American Sign Language. The reason for the project is that, for deaf children, reading instruction is delayed; therefore most kids in this age group cannot use traditional textbooks. They also aren’t able to take full advantage of secondary learning opportunities that non-disabled kids have, such as television and dinner table conversation, and their parents may not be fluent enough in sign language to teach them about math.

Project sites (with demos):

• Computer animation: Mathsigner
• VR: Signed Math in an Immersive Learning Environment (SMILE)

The VR of counting money in a toy store and so forth should be useful and fun. However, most deaf children have hearing parents and siblings, who often don’t sign. This can be isolating if the child can’t (or can’t yet) adequately communicate via oral-aural language. I just hope teachers and parents are careful not to take too much time away from interacting with other humans in a natural visual language in order to have limited interaction between the child and animated characters.