Peter Ochieng — Projects to Supervise
Projects to Supervise
Cross-modal alignment model for audio to sentence.
Supervisor: Peter Ochieng
Establishing vector representations of input features has been crucial to the success of machine learning especially natural language processing. The vector representations( embeddings) are always exploited in downstream tasks such as machine translation where the intuition is that words appearing in similar context should generate similar word embeddings hence should be aligned. The idea of similar words generating similar representation has been extended to cross-lingual alignment. It has been exploited to perform machine translation between two different languages without any annotated dataset linking these languages. This is due to the discovery that similar words from different languages share similar structures in a continuous word embedding space. This eliminates the need for a large parallel training corpus to train machine translation systems. work in [1] extends this concept to perform cross-modal alignment between text and audio where audio segments are aligned to words with similar embeddings in a joint continuous embedding space. This is helpful to generate audio transcription without any transcribed dataset to train the transcription model. For low resource languages this is crucial. Despite the remarkable results were reported in [1] and while the word based generation from audio may appear natural for speech, it is still not clear how to chunk speech audio in a lengths that correctly generates words. Also the word based generation of textfrom audio may be slow for downstream applications. This project proposes an audio to sentence generation as opposed to existing audio to word generation. Can we develop a model where a whole sentence is generated at once based on a given chunk of speech. Basically can cross-modal alignment be developed where an audio chunk and a sentence share a continuous embedding space. Will this speed up transcription of a recorded speech ? will the long dependencies within the sentence level audio chunks affect the quality of transcription. How can we effectively identify sentences boundaries with a given audio ?. In this project, we will utilize the dataset and model proposed in [1] and extend it to handle sentence level text generation. reference [1]Chung, Yu-An, et al. “Unsupervised cross-modal alignment of speech and text embedding spaces.” Advances in neural information processing systems 31 (2018).
Investigating the Effect of Sequence Length on the quality of clean speech generated
Supervisor: Peter Ochieng
To effectively perform speech separation, the speech separation tools need to model long sequence dependencies that exist within the audio signal input. The initial speech separation models relied on deep neural network to estimate clean speech from a noisy one. However, feedforward DNN models are ill poised for speech data since they are unable to effectively model long dependencies across time that are present in the speech data. Due to this researchers progressively introduced recurrent neural network (RNN) which has a feedback structure such that the representations at given time step t is a function of the data at time t, the hidden state and memory at time t − 1. One such RNN that has been exploited in speech separation is long-short-term memory (LSTM). Through LSTM structures one can learn sequential prediction networks which are able to exploit use of long-term contextual information. Due to their inherently sequential nature, RNN models are unable to support parallelization of computation. This limits their use when working with large datasets with long sequences. Convectional convolution neural (CNN) has been used to design speech separation models, however they are limited in their ability to model long range dependencies due to limited receptive field. Due to the shortcomings of the CNN and RNN, dilated temporal convolution network (TCN) has been exploited to encode long-range dependencies using hierarchical convolutional layers. Speech separation models have also explored the use of transformers. Transformers are attention based that have been successful in the recent past in modeling sequences and allows uncovering of dependencies that exist within an input without regard to the distance between any two values of the input. One common thing in all these progression is that researchers have focused on designing models that are large to effectively model audio input without looking at the impact of input data size. In audio separation, a typical model’s input is a sequence of frames, each of which is 25 seconds long. It is possible to reduce the length of the input hence reducing the burden of modelling long sequence dependencies ? Will this reduce the training cost of the models without impacting on the quality of their output? Basically by focusing on the data side of the training spectrum can we reduce the training cost without impacting on the quality of the separations generated. The project will focus on utilizing already existing systems such as [1] and evaluating its performance on variable frame size. Reference [1] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June:21–25, 2021. ISSN 15206149. doi:10.1109/ICASSP39728.2021.9413901.