In cooperation with engineers at The George Washington University and cognitive scientists at the University of Rochester, the Georgia Tech team is developing a system that acts as a one-way American Sign Language-to-English phrase book. The system bridges a gulf in communication that exists for people deaf from birth, who often learned a sign language as their first language and are not always fluent in spoken languages. And because the average deaf adult in the United States reads at a fourth-grade level, written communication is, generally speaking, not an effective workaround when trying to talk to the hearing.

The system, called TeleSign, consists of a miniature video camera and wrist-mounted accelerometers to pick up a person's signs and software-based machine learning techniquesto interpret them. The system is intended as a tourist-style phrase book. "It limits the vocabulary to a few phrases—currently about 20—that are sufficient for a variety of situations and are most likely to elicit responses like a nod or a point in a particular direction," says Thad Starner, director of the contextual computing group at Georgia Tech.

TeleSign was inspired by VoxTec International's Phrasalator system now used by U.S. troops around the world. The user speaks an English phrase into the Phraselator and the device attempts to match it to a phrase in the target language. The Annapolis, Md., company's device then speaks the phrase.

TeleSign does the same thing, but instead of using a microphone to detect the user's speech, it uses a camera mounted on the brim of a hat and accelerometers worn on wristbands to detect the user's signing. The lens is aimed at the area in front of the chest where most sign language hand gestures are made. The user activates the system by clicking a button on a wristband that sends signals to a computer processor via the Bluetooth wireless protocol. After signing a phrase, he or she presses the button again—telling the system to stop capturing the hand movements and to begin searching its database for the closest English-language matches.

This matching is done with the help of algorithms known as hidden Markov models—the same type of algorithms used in speech recognition. They take the data from the accelerometers and the camera and find the most likely match for the signed gesture based on models of what the signs for particular phrases should look like. To ensure that the system has correctly matched the signed phrase and the proposed English phrase, the two or three most likely matches are displayed on a portable gadget like a PDA. (In the prototype it is a head-up display mounted on eyeglasses.) The user may select a phrase from the list, or chose to re-sign.

In an attempt to simplify the system, Starner's team worked with Jose Hernandez-Rebollar at The George Washington University, in Washington, D.C., on another interface design featuring Hernandez-Rebollar's AcceleGlove. Accelerometers placed inside the AcceleGlove provide orientation and acceleration information with respect to each other. Meanwhile, other sensors fixed to the elbow and shoulder determine the hand's absolute position with respect to the body. The glove interface works because many signs are recognized by differences in their beginning hand shapes, intervening movements, and ending hand shapes.

Once it recognizes the beginning hand position, it can safely eliminate all the phrases in the database whose beginnings don't match it. As the intervening movement progresses, more phrases are eliminated until, when the end shape is formed, theoretically, only a single match remains. Though the version using the glove interface has a larger, 141-sign vocabulary, it still needs less processing power than the head-mounted camera prototype does.

To keep the translator from incorrectly interpreting the move from a relaxed state with the hands down at the sides to the signing position and vice versa, it begins processing the data from movement of the glove only after the signer makes the ASL gesture meaning "start sentence." The "end sentence" gesture deactivates the system. Valerie Henderson, a Ph.D. candidate working in the Georgia Tech lab, says, "We're still at an early stage in the research, [but] we suspect that the AcceleGlove could remove the need for the hat-based camera."

Dozens of research teams around the globe are working on the technology that will allow translation between sign language and spoken languages. But even this phrase book-style mode of communication is broad enough to include several research paths. The British government, for example, has been supporting work on a phrase book that translates in the other direction—spoken English to British Sign Language. The Text and Sign Support Assistant, or Tessa, developed at the at the University of East Anglia, Norwich, England, in partnership with the UK's Post Office, combines speech recognition technology and virtual human animation to enable postal workers to communicate with deaf customers. When the postal worker speaks into a microphone, the phrase is recognized by a computer speech recognition system and matched to its equivalent in British Sign Language. An avatar, an onscreen virtual human they devised, signs to the deaf person.

By Willie D. Jones