Brain-To-Speech technology represents a fusion of interdisciplinary applications encompassing fields of artificial intelligence, brain-computer interfaces, and speech synthesis. It refers to a brain signal-mediated communication method that converts brain activities of silent or imagined speech into audible speech. Brain-To-Speech technology directly connects neural activity to the means of human linguistic communication which may greatly enhance the naturalness of communication using brain signals.

With the current discoveries on neural features of imagined speech and the development of the speech synthesis technologies, direct translation of brain signals into speech has shown significant promise. This talk introduces the current Brain-To-Speech technology with the possibility of speech synthesis from non-invasive brain signals, which may ultimately facilitate silent communication via brain signals.

In this talk, I will discuss the evolution of signal processing algorithms over the years and the connections between modern approaches and classical approaches. Modern algorithms have better performance, at the expense of additional complexity, and often hard to connect to classical techniques. But, in fact, these modern algorithms are connected to the classical algorithms and understanding these connections can help us gain perspective and insight into the newer algorithms. I will discuss insights in two specific cases, one in sparse signal recovery and the other in deep learning.

The first problem we will discuss is the source localization problem in array processing where the Minimum Power Distortionless Response (MPDR) algorithm, also known as the Minimum Variance Distortionless Response (MVDR) algorithm, is a widely used approach. Recently, sparse signal recovery (SSR) algorithms have been developed to solve the linear inverse problem associated with the source localization problem. The sparse Bayesian learning (SBL) algorithm for sparse signal recovery is one such algorithm and the EM-SBL algorithm will be used as the basis for this discussion. The iterative approach used in the EM algorithm will be provided an MPDR beamformer interpretation making the algorithm more transparent. Additionally, it will enable understanding the new attributes that emerge from the SBL algorithm. Similar insights can be also obtained for the other SSR algorithms.

A second problem is how to estimate one random vector given observations of another random vector. Linear estimation techniques are widely used in many signal processing applications and extended to nonlinear estimation with linear estimation on data augmented with handcrafted nonlinear features. With the advent of deep neural networks (DNNs), nonlinear estimation techniques have become attractive. We will delve into ResNEst, a variant of ResNet, to understand the nonlinear feature learning process, and the linear estimator being constructed based on these features. This insight will allow for replacement of linear estimators with DNNs with the assurance of better estimation performance.

The advent of deep neural networks has brought significant advancements in the development and deployment of novel AI technologies. Recent large-scale neural network architectures have demonstrated remarkable performance in object classification, scene understanding, language processing, and multimodal generative AI.

How can we understand how the representations of input signals are transformed within deep neural networks? I will explain how statistical insights can be gained by analyzing the high-dimensional geometrical structure of these representations as they are reformatted by neural network hierarchies of basic perceptron units.

Humans are social animals. Perhaps this is why we so enjoy watching movies, TV shows and YouTube videos, all of which show people in action. A central problem for artificial intelligence therefore is to develop techniques for analyzing and understanding human behavior from images and video.

I will present some recent results from our research group towards this grand challenge. We have developed highly accurate techniques for reconstructing 3D meshes of human bodies from single images using transformer neural networks. Given video input, we link these reconstructions over time by 3D tracking, thus producing "Humans in 4D" (3D in space + 1D in time). As a fun application, we can use this capability to transfer the 3D motion of one person to another e.g. to generate a video of you performing Michael Jackson's moonwalk or Michelle Kwan's skating routine.

The ability to do 4D reconstruction of hands is a source of imitation learning for robotics and we show examples of reconstructing human-object interactions. In addition to 4D reconstruction, we are also now able to recognize actions by attaching semantic labels such as "standing", "running", or "jumping". However, long range video understanding, such as the ability to follow characters' activities and understand movie plots over periods of minutes and hours, is still quite a challenge, and even the largest vision-language models struggle on such tasks. There has been substantial progress, but much remains to be done.