Journal
Scientific and technical journal of information technologies, mechanics and optics
UDK20.19.29
Issue:5 (93)
Download PDF0 Kbyte
BILINGUAL MULTIMODAL SYSTEM FOR TEXT-TO-AUDIOVISUAL SPEECH AND SIGN LANGUAGE SYNTHESIS
Annotation
We present a conceptual model, architecture and software of a multimodal system for audio-visual speech and sign language synthesis by the input text. The main components of the developed multimodal synthesis system (signing avatar) are: automatic text processor for input text analysis; simulation 3D model of human's head; computer text-to-speech synthesizer; a system for audio-visual speech synthesis; simulation 3D model of human’s hands and upper body; multimodal user interface integrating all the components for generation of audio, visual and signed speech. The proposed system performs automatic translation of input textual information into speech (audio information) and gestures (video information), information fusion and its output in the form of multimedia information. A user can input any grammatically correct text in Russian or Czech languages to the system; it is analyzed by the text processor to detect sentences, words and characters. Then this textual information is converted into symbols of the sign language notation. We apply international «Hamburg Notation System» - HamNoSys, which describes the main differential features of each manual sign: hand shape, hand orientation, place and type of movement. On their basis the 3D signing avatar displays the elements of the sign language. The virtual 3D model of human’s head and upper body has been created using VRML virtual reality modeling language, and it is controlled by the software based on OpenGL graphical library. The developed multimodal synthesis system is a universal one since it is oriented for both regular users and disabled people (in particular, for the hard-of-hearing and visually impaired), and it serves for multimedia output (by audio and visual modalities) of input textual information.
Keywords
Постоянный URL
Articles in current issue
- ACCOUNTING OF MANY-PARTICLE INTERACTIONS IN MOLECULAR J-AGGREGATES AND NONLINEAR OPTICAL EFFECTS IN THESE SYSTEMS
- SERS OF BACTERIORHODOPSIN WITH OUT-DIFFUSED SILVER NANOISLANDS
- SOLID BODY ABLATION UNDER EXPOSURE TO ULTRA SHORT LASER PULSES: STUDY BY MOLECULAR DYNAMICS METHODS
- FUNDAMENTAL MATRIX OF LINEAR CONTINUOUS SYSTEM IN THE PROBLEM OF ESTIMATING ITS TRANSPORT DELAY
- THERMAL AND ELECTRIC FIELDS AT SPARK PLASMA SINTERING OF THERMOELECTRIC MATERIALS
- INFLUENCE OF QUARTZ CERAMICS SINGLE-STAGE PROCESSING BY GEL-FORMING WATER SOLUTIONS ON ITS STRENGTH
- INVESTIGATION OF SORPTION CHARACTERISTICS OF POLYMERIC MINERAL-FILLED COMPOSITES FOR MEDICINE
- CRYSTALLIZATION KINETICS OF POLYMERIC NANOCOMPOSITES BASED ON POLYAMIDE 12 MODIFIED BY Cr2O3 NANOPARTICLES
- INORGANIC PHOSPHORS IN GLASS BASED ON LEAD SILICATE GLASSES
- IMPROVEMENT OF RECOGNITION QUALITY IN DEEP LEARNING NETWORKS BY SIMULATED ANNEALING METHOD
- EXPERIMENTAL STUDY OF FIRMWARE FOR INPUT AND EXTRACTION OF USER’S VOICE SIGNAL IN VOICE AUTHENTICATION SYSTEMS
- SYSTEMS FOR SUPPORT OF COLLABORATIVE STUDIES IN THE COLLA ENVIRONMENT BASED ON THE EVENT BUSH METHOD
- POLICE OFFICE MODEL IMPROVEMENT FOR SECURITY OF SWARM ROBOTIC SYSTEMS
- RELATIONAL THEORY APPLICATION FOR OPTIMAL DESIGN OF INTEGRATED CIRCUITS
- INTERVALS OPTIMIZATION OF SYSTEMS INFORMATION SECURITY INSPECTION
- ALGORITHM FOR SEMANTIC TEXT ANALYSIS BY MEANS OF BASIC SEMANTIC TEMPLATES WITH DELETION
- ARCHITECTURE OF WEB BASED COMPUTER-AIDED MANUFACTURING SYSTEM
- SIMULATION OF PULSED BREAKDOWN IN HELIUM BY ADAPTIVE METHODS
- ON ALGORITHMS CREATION FOR STRAPDOWN STABILIZED GYROCOMPASS OPERATION BASED ON ELECTRICALLY SUSPENDED GYROSCOPE
- MODELING AND EXPERIMENTAL STUDY OF A FIBER OPTIC HYDROPHONE SENSING ELEMENT
- NEW APPROACHES TO EFFICIENCY OF MASSIVE ONLINE COURSE
- INFORMATION INFRASTRUCTURE OF THE EDUCATIONAL ENVIRONMENT WITH VIRTUAL MACHINE TECHNOLOGY
- TWO-LAYER PHASE COMPENSATING INTERFERENCE SYSTEMS
- A PROTOTYPE OF BARENTSNET PROFESSIONAL SOCIAL NETWORK FOR INFORMATION SUPPORT OF DEVELOPMENT MANAGEMENT FOR BARENTS EURO-ARCTIC REGION