Search Results (14)

This paper explores advancements in real-time talking-head generation, focusing on overcoming challenges in Audio Feature Extraction (AFE), which often introduces latency and limits responsiveness in real-time applications. To address these issues, we propose and implement a fully integrated system that replaces conventional AFE models with OpenAI’s Whisper, leveraging its encoder to optimize processing and improve overall system efficiency. Our evaluation of two open-source real-time models across three different datasets shows that Whisper not only accelerates processing but also improves specific aspects of rendering quality, resulting in more realistic and responsive talking-head interactions. Although interviewer training systems are considered a potential application, the primary contribution of this work is the improvement of the technical foundations necessary for creating responsive AI avatars. These advancements enable more immersive interactions and expand the scope of AI-driven applications, including educational tools and simulated training environments. Full article

Audio-driven cross-modal talking head generation has experienced significant advancement in the last several years, and it aims to generate a talking head video that corresponds to a given audio sequence. Out of these approaches, the NeRF-based method can generate videos featuring a specific person with more natural motion compared to the one-shot methods. However, previous approaches failed to distinguish the importance of different regions, resulting in the loss of information-rich region features. To alleviate the problem and improve video quality, we propose MLDF-NeRF, an end-to-end method for talking head generation, which can achieve better vector representation through multi-level feature dynamic fusion. Specifically, we designed two modules in MLDF-NeRF to enhance the cross-modal mapping ability between audio and different facial regions. We initially developed a multi-level tri-plane hash representation that uses three sets of tri-plane hash networks with varying resolutions of limitation to capture the dynamic information of the face more accurately. Then, we introduce the idea of multi-head attention and design an efficient audio-visual fusion module that explicitly fuses audio features with image features from different planes, thereby improving the mapping between audio features and spatial information. Meanwhile, the design helps to minimize interference from facial areas unrelated to audio, thereby improving the overall quality of the representation. The quantitative and qualitative results indicate that our proposed method can effectively generate talk heads with natural actions and realistic details. Compared with previous methods, it performs better in terms of image quality, lip sync, and other aspects. Full article

Imagine a future where language is no longer a barrier to real-time conversations, enabling instant and lifelike communication across the globe. As cultural boundaries blur, the demand for seamless multilingual communication has become a critical technological challenge. This paper addresses the lack of robust solutions for real-time face-to-face translation, particularly for low-resource languages, by introducing a comprehensive framework that not only translates language but also replicates voice nuances and synchronized facial expressions. Our research tackles the primary challenge of achieving accurate lip synchronization across culturally diverse languages, filling a significant gap in the literature by evaluating the generalizability of lip sync models beyond English. Specifically, we develop a novel evaluation framework combining quantitative lip sync error metrics and qualitative assessments by human observers. This framework is applied to assess two state-of-the-art lip sync models with different architectures for Turkish, Persian, and Arabic languages, using a newly collected dataset. Based on these findings, we propose and implement a modular system that integrates language-agnostic lip sync models with neural networks to deliver a fully functional face-to-face translation experience. Inference Time Analysis shows this system achieves highly realistic, face-translated talking heads in real time, with a throughput as low as 0.381 s. This transformative framework is primed for deployment in immersive environments such as VR/AR, Metaverse ecosystems, and advanced video conferencing platforms. It offers substantial benefits to developers and businesses aiming to build next-generation multilingual communication systems for diverse applications. While this work focuses on three languages, its modular design allows scalability to additional languages. However, further testing in broader linguistic and cultural contexts is required to confirm its universal applicability, paving the way for a more interconnected and inclusive world where language ceases to hinder human connection. Full article

The article discusses Pope Francis’s visit to Bahrain in 2022, which, although religiously and politically important, has not yet been exposed to science. This is the second visit of the head of the Catholic Church to the countries of the Persian Gulf, which is part of the refreshing interreligious dialogue between Christians and Muslims. This article analyzes changes in the Catholic Church’s understanding of dialogue with Islam, as illustrated by Pope Francis’s visit to Bahrain. Using comparative analysis, the article highlights the evolving nature of interreligious dialogue and its role in strengthening Muslim–Christian connections. The papal visit to Bahrain caused much controversy among human rights activists. Bahrain is seen, on one hand, as a tolerant and religiously inclusive country, but on the other, as a nation that frequently violates human rights, particularly in cases involving differences among its members. Pope Francis, as a diplomat, was not afraid to talk about some social issues in Bahrain (death penalty, discrimination, labor law) from the beginning of his visit. Pope Francis’s attitude and the benefits of interreligious dialogue that he has generated may not only confirm the presence of Christians in Bahrain but, above all, must change their destiny in the countries of the region that have so far treated this religion as an enemy. This article is an analysis of sources and their systematic review. The authors have focused on the interpretation of Pope Francis’s statements and their reception in the world. Interreligious and intercultural dialogue, as well as interpersonal fraternities, are extremely important in the international policy of the Holy See. Full article

Audio-driven facial animation is a rapidly evolving field that aims to generate realistic facial expressions and lip movements synchronized with a given audio input. This survey provides a comprehensive review of deep learning techniques applied to audio-driven facial animation, with a focus on both audio-driven facial image animation and audio-driven facial mesh animation. These approaches employ deep learning to map audio inputs directly onto 3D facial meshes or 2D images, enabling the creation of highly realistic and synchronized animations. This survey also explores evaluation metrics, available datasets, and the challenges that remain, such as disentangling lip synchronization and emotions, generalization across speakers, and dataset limitations. Lastly, we discuss future directions, including multi-modal integration, personalized models, and facial attribute modification in animations, all of which are critical for the continued development and application of this technology. Full article

Background: According to prevailing views in neuroscience, near-death experiences (NDE) occurring after severe head trauma, critical illness, or coma are often life-transforming experiences in which no awareness or sensory experience of any kind is possible. Although there are general patterns, each case is quite different from the other and requires accurate recording and reporting to potentially explain the phenomenon. Aim: This narrative study aimed to explore a pregnant woman’s NDE due to complications from MERS-CoV. Methods: This was a qualitative narrative study with the administration of two unstructured interviews. After the second interview, the participant completed the Greyson NDE scale, presented through descriptive statistics. Qualitative data were analyzed using Labov’s model of narrative analysis through abstract, orientation, complicating action, evaluation, resolution, and coda. Results: The Greyson scale resulted in a total score of 12, confirming that the patient had experienced an NDE. Labov’s model of narrative analysis revealed that the patient’s experience was not limited to the NDE but had implications for her recovery and life. The patient experienced all three types of NDEs: out-of-body, transcendental, including the transition of consciousness to another dimension, and a combined experience. She also suffered from prolonged hallucinations, neuropathy, and post-intensive care syndrome (PICS). At the same time, the patient experienced what is known as NDE aftereffects, which are caused by a change in beliefs and values; she began to lead a more altruistic life and became interested in the meaning of life. Conclusions: NDE survivors should be encouraged to talk more and share their stories with others if they wish. This study not only investigates the NDE but also considerably adds to the existing literature by integrating a unique cultural view from a country outside of the US and other Western nations, and it highlights the significant role of healthcare providers in NDEs and the importance of communication with comatose patients. It underscores the need for compassion when dealing with patients with NDEs. Full article

In today’s digital era, the realms of virtual reality (VR), augmented reality (AR), and mixed reality (MR) collectively referred to as extended reality (XR) are reshaping human–computer interactions. XR technologies are poised to overcome geographical barriers, offering innovative solutions for enhancing emotional and social engagement in telecommunications and remote collaboration. This paper delves into the integration of (AI)-powered 3D talking heads within XR-based telecommunication systems. These avatars replicate human expressions, gestures, and speech, effectively minimizing physical constraints in remote communication. The contributions of this research encompass an extensive examination of audio-driven 3D head generation methods and the establishment of comprehensive evaluation criteria for 3D talking head algorithms within Shared Virtual Environments (SVEs). As XR technology evolves, AI-driven 3D talking heads promise to revolutionize remote collaboration and communication. Full article

This work presents BlinkLinMulT, a transformer-based framework for eye blink detection. While most existing approaches rely on frame-wise eye state classification, recent advancements in transformer-based sequence models have not been explored in the blink detection literature. Our approach effectively combines low- and high-level feature sequences with linear complexity cross-modal attention mechanisms and addresses challenges such as lighting changes and a wide range of head poses. Our work is the first to leverage the transformer architecture for blink presence detection and eye state recognition while successfully implementing an efficient fusion of input features. In our experiments, we utilized several publicly available benchmark datasets (CEW, ZJU, MRL Eye, RT-BENE, EyeBlink8, Researcher’s Night, and TalkingFace) to extensively show the state-of-the-art performance and generalization capability of our trained model. We hope the proposed method can serve as a new baseline for further research. Full article

Angiotensin II-converting enzyme inhibitors (ACEIs) and selective angiotensin II receptor antagonists (ARAIIs) are widely used antihypertensive agents. Their use has generated controversy due to their possible influence on the health status of chronic patients infected with COVID-19. The objective of this work is to analyze the influence of COVID-19 on chronic hypertensive patients treated with ACEI and ARAII inhibitors. A systematic review and meta-analysis in the databases Pubmed, Pro-Quest and Scopus were carried out. The systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The search equation descriptors were obtained from the Medical Subject Headings (MeSH) thesaurus. The search equation was: “Older AND hypertension AND (COVID-19 OR coronavirus) AND primary care” and its equivalent in Spanish. Nineteen articles were obtained, with n = 10,806,159 subjects. Several studies describe the COVID-19 association with ACEI or ARAII treatment in hypertension patients as a protective factor, some as a risk factor, and others without a risk association. In the case of ACEI vs. ARAII, the risk described for the former has an odds ratio (OR) of 0.55, and for ARAII, an OR of 0.59. Some authors talk about mortality associated with COVID-19 and ACEI with a half ratio (HR) of 0.97, and also associated ARAIIs with an HR of 0.98. It is recommended to maintain the use of the renin–angiotensin–aldosterone axis in the context of the COVID-19 disease. Full article

Virtual human is widely employed in various industries, including personal assistance, intelligent customer service, and online education, thanks to the rapid development of artificial intelligence. An anthropomorphic digital human can quickly contact people and enhance user experience in human–computer interaction. Hence, we design the human–computer interaction system framework, which includes speech recognition, text-to-speech, dialogue systems, and virtual human generation. Next, we classify the model of talking-head video generation by the virtual human deep generation framework. Meanwhile, we systematically review the past five years’ worth of technological advancements and trends in talking-head video generation, highlight the critical works and summarize the dataset. Full article

PLEA is an interactive, biomimetic robotic head with non-verbal communication capabilities. PLEA reasoning is based on a multimodal approach combining video and audio inputs to determine the current emotional state of a person. PLEA expresses emotions using facial expressions generated in real-time, which are projected onto a 3D face surface. In this paper, a more sophisticated computation mechanism is developed and evaluated. The model for audio-visual person separation can locate a talking person in a crowded place by combining input from the ResNet network with input from a hand-crafted algorithm. The first input is used to find human faces in the room, and the second input is used to determine the direction of the sound and to focus attention on a single person. After an information fusion procedure is performed, the face of the person speaking is matched with the corresponding sound direction. As a result of this procedure, the robot could start an interaction with the person based on non-verbal signals. The model was tested and evaluated under laboratory conditions by interaction with users. The results suggest that the methodology can be used efficiently to focus a robot’s attention on a localized person. Full article

Due to the population aging in Japan, more elderly people are retaining their driver’s licenses and the increase in the number of car accidents by elderly drivers is a social problem. To address this problem, an objective data-based method to evaluate whether elderly drivers can continue driving is needed. In this paper, we propose a car driving evaluation system based on gaze as calculated by eye and head angles. We used an eye tracking device (TalkEye Lite) made by the Takei Scientific Instruments Cooperation. For our image processing technique, we propose a gaze fixation condition using deep learning (YOLOv2-tiny). By using an eye tracking device and the proposed gaze fixation condition, we built a system where drivers could be evaluated during actual car operation. We describe our system in this paper. In order to evaluate our proposed method, we conducted experiments from November 2017 to November 2018 where elderly people were evaluated by our system while driving an actual car. The subjects were 22 general drivers (two were 80–89 years old, four were 70–79 years old, six were 60–69 years old, three were 50–59 years old, five were 40–49 years old and two were 30–39 years old). We compared the subjects’ gaze information with the subjective evaluation by a professional driving instructor. As a result, we confirm that the subjects’ gaze information is related to the subjective evaluation by the instructor. Full article

Car operation requires advanced brain function. Currently, evaluation of the motor vehicle driving ability of people with higher brain dysfunction is medically unknown and there are few evaluation criteria. The increase in accidents by elderly drivers is a social problem in Japan, and a method to evaluate whether elderly people can drive a car is needed. Under these circumstances, a system to evaluate brain dysfunction and driving ability of elderly people is needed. Gaze estimation research is a rapidly developing field. In this paper, we propose the gaze calculation method by eye and head angles. We used the eye tracking device (TalkEyeLite) made by Takei Scientific Instruments Cooperation. For our image processing technique, we estimated the head angle using the template matching method. By using the eye tracking device and the head angle estimate, we built a system that can be used during actual on-road car operation. In order to evaluate our proposed method, we tested the system on Japanese drivers during on-road driving evaluations at a driving school. The subjects were one instructor of the car driving school and eight general drivers (three 40–50 years old and five people over 60 years old). We compared the gaze range of the eight general subjects and the instructor. As a result, we confirmed that one male in his 40s and one elderly driver had narrower gaze ranges. Full article

In next-generation cars, safety equipment related to assisted driving systems commonly known as ADAS (advanced driver-assistance systems) are of particular interest for the major car-makers. When we talk about the “ADAS system”, we mean the devices and sensors having the precise objective of improving and making car driving safer, and among which it is worth mentioning rain sensors, the twilight sensor, adaptive cruise control, automatic emergency braking, parking sensors, automatic signal recognition, and so on. All these devices and sensors are installed on the new homologated cars to minimize the risk of an accident and make life on board of the car easier. Some sensors evaluate the movement and the opening of the eyes, the position of the head and its angle, or some physiological signals of the driver obtainable from the palm of the hands placed in the steering. In the present contribution, the authors will present an innovative recognition and monitoring system of the driver’s attention level through the study of the photoplethysmographic (PPG) signal detectable from the palm of the driver’s hands through special devices housed in the steering of the car. Through a particular and innovative post-processing algorithm of the PPG signal through a hyper-filtering framework, then processed by a machine learning framework, the entire pipeline proposed will be able to recognize and monitor the attention level of the driver with high accuracy and acceptable timing. Full article