Journal
SCIENTIFIC AND TECHNICAL JOURNAL OF INFORMATION TECHNOLOGIES, MECHANICS AND OPTICS
UDK004. 93
Issue:5 (99)
APPLICATION OF PARTIAL LEAST SQUARES REGRESSION FOR AUDIO-VISUAL SPEECH PROCESSING AND MODELING
Annotation
Subject of Research. The paper deals with the problem of lip region image reconstruction from speech signal by means of Partial Least Squares regression. Such problems arise in connection with development of audio-visual speech processing methods. Audio-visual speech consists of acoustic and visual components (called modalities). Applications of audio-visual speech processing methods include joint modeling of voice and lips’ movement dynamics, synchronization of audio and video streams, emotion recognition, liveness detection. Method. Partial Least Squares regression was applied to solve the posed problem. This method extracts components of initial data with high covariance. These components are used to build regression model. Advantage of this approach lies in the possibility of achieving two goals: identification of latent interrelations between initial data components (e.g. speech signal and lip region image) and approximation of initial data component as a function of another one. Main Results. Experimental research on reconstruction of lip region images from speech signal was carried out on VidTIMIT audio-visual speech database. Results of the experiment showed that Partial Least Squares regression is capable of solving reconstruction problem. Practical Significance. Obtained findings give the possibility to assert that Partial Least Squares regression is successfully applicable for solution of vast variety of audio-visual speech processing problems: from synchronization of audio and video streams to liveness detection.
Keywords
Постоянный URL
Articles in current issue
- THE INFLUENCE OF PLASMONIC AND DIELECTRIC INCLUSIONS ON ANTIREFLECTIVE PROPERTIES OF HOMOGENEOUS COATINGS FOR SILICON PHOTOVOLTAIC STRUCTURES
- SIMULATION OF FORWARD AND BACKWARD WAVES EVOLUTION OF FEW-CYCLE PULSES PROPAGATING IN AN OPTICAL WAVEGUIDE WITH DISPERSION AND CUBIC NONLINEARITY OF ELECTRONIC AND ELECTRONIC-VIBRATION NATURE
- RESEARCH OF LINEAR AND NONLINEAR PROCESSES AT FEMTOSECOND LASER RADIATION PROPAGATION IN THE MEDIUM SIMULATING THE HUMAN EYE VITREOUS
- OUT-OF-FOCUS REGION SEGMENTATION OF 2D SURFACE IMAGES WITH THE USE OF TEXTURE FEATURES
- IMPACT STUDY OF ANISOTROPIC OPTICAL FIBERS WINDING WITH DIFFERENT TENSION VALUE ON THE H-PARAMETER INVARIANCE DEGREE
- APPLICATION OF SPATIAL LIGHT MODULATORS FOR GENERATION OF LASER BEAMS WITH A SPIRAL PHASE DISTRIBUTION
- REFINED MODEL OF THE OPTICAL SYSTEM FOR SPACE MINI-VEHICLES WITH LASER PROPULSION
- RESEARCH OF NIGHT LIGHT EFFECTS ON COLORIMETRIC CHARACTERISTICS OF IMAGE PERCEIVED BY THE PILOT IN AN AIRCRAFT COCKPIT
- ELECTRON MICROSCOPIC INVESTIGATION OF YTTRIUM ALUMINUM GARNET POWDERS Y3AL5O12, SYNTHESIZED BY SOL–GEL METHOD
- CENTRAL WAVELENGTH ADJUSTMENT OF LIGHT EMITTING SOURCE IN INTERFEROMETRIC SENSORS BASED ON FIBER-OPTIC BRAGG GRATINGS
- MODELING OF DYNAMIC SYSTEMS WITH MODULATION BY MEANS OF KRONECKER VECTOR-MATRIX REPRESENTATION
- PERMITTIVITY DISPERSION FEATURES OF A NEMATIC LIQUID CRYSTAL WITH QUANTUM DOTS
- INTERACTION OF SILVER MOLECULAR CLUSTERS, INTRODUCED BY LOW-TEMPERATURE ION EXCHANGE METHOD, WITH NANOPARTICLES OF CdS IN FLUORINE PHOSPHATE GLASSES
- EFFICIENCY EVALUATION OF ENTERPRISE INFORMATION SYSTEMS WITH NON-UNIFORM LOAD
- SEMSIN SEMANTIC AND SYNTACTIC PARSER
- RISK MANAGEMENT AUTOMATION OF SOFTWARE PROJECTS BASED ОN FUZZY INFERENCE
- ACCURACY RESEARCH OF THE DIAMETRICAL SIZES FORMING AT GEAR SHAPING BY STEPPED CUTTER
- MASS FLOW METER FOR LIQUIDS
- SOFTWARE TOOLS FOR COMPUTING EXPERIMENT AIMED AT MULTIVARIATE ANALYSIS IMPLEMENTATION
- INVESTIGATION OF STURM-LIOUVILLE PROBLEM SOLVABILITY IN THE PROCESS OF ASYMPTOTIC SERIES CREATION
- ELEMENT DESIGN FOR AN INKJET SYSTEM OF HYDROSTATIC GAS BEARING CONTROL
- INTERFERENCE HYSTERESIS OF COUNTERPROPAGATING SHOCK WAVES AT A CHANGE IN MACH NUMBER
- ASYMMETRICAL INTERFERENCE OF COUNTER OBLIQUE SHOCK WAVES
- THE APPROACHING TRAIN DETECTION ALGORITHM
- ON THE PAPER BY L.S. KONEV ET AL. “SIMULATING EVOLUTION OF FORWARD AND BACKWARD WAVES OF FEW-CYCLE PULSES PROPAGATING IN AN OPTICAL WAVEGUIDE WITH DISPERSION AND CUBIC NONLINEARITY OF ELECTRONIC AND ELECTRONIC-VIBRATION NATURE”