Multimedia Signal Processing
Audio, voice, image, and video representation, compression algorithms, and perceptual modeling.
Interactive Jupyter Notebooks & Google Colab Labs
Hands-on interactive notebooks covering core signal processing and data compression techniques. Run them live in your browser via Google Colab or view them rendered on Jupyter nbviewer:
- Introdução ( nbviewer ) ( colab )
- imagem em cores (nbviewer) ( colab )
- Normalização de histograma (nbviewer) ( colab )
- move-to-front (nbviewer) ( colab )
- RLE (nbviewer) ( colab )
- Quantização escalar (nbviewer) ( pdf ) ( colab )
- Dithering, Floyd-Steinberg (nbviewer) ( colab )
- Reamostragem (nbviewer) ( colab )
- Filtros polifásicos (nbviewer) ( colab )
- Espectrograma e Tom Shepard (nbviewer) ( colab )
- Modelo AR (nbviewer) ( colab )
- DPCM (nbviewer) ( colab )
- DCT (nbviewer) ( colab )
- k-médias (nbviewer) ( colab )
- Vogais LPC (nbviewer) ( colab )
- Síntese LPC (nbviewer) ( colab )
- JPEG ( nbviewer ) ( colab )
- PCA ( nbviewer ) ( colab )
- LPC Matlab, Mathworks
- cjpeg - compress an image file to a JPEG file
- Wavelets ( nbviewer ) ( colab )
Course Schedule & Reading Plan
- Introduction. Multimedia signal. Audio, Voice, Image, Video, Text. ( Salomon and Motta, Cap. 1 ) (notebook image introduction 1 , 2 , 3 )
- Spectrogram ( notebook )
- Digital Signal Processing. Resampling. Multirate Signal Processing. Oversampling. ( Oppenheim )
- Scalar Quantization and Vector Quantization. ( notebook ) ( Gray and Neuhoff, 1998 ) ( Gray, 1984 ) ( Cap. 2, 3 You, 2010 )
- Basic compression. RLE. Dictionary methods. Statistical methods. ( Salomon and Motta, Cap. 2, 5, 6 ) ( RLE, fileformat ) ( rle notebook ) ( move-to-front notebook ) ( slides )
- Orthogonal transform. Discrete Cosine Transform (DCT). Wavelets. ( Cupertino, 2002 )
- Image compression. JPEG Standard. ( Wikipedia , ITU-T T.81 ), ( Kerr, 2012)
- Linear Prediction. Differential Pulse Code Modulation (DPCM). Linear Predictive Coding (LPC). ( Yehia, 1993 ) (slides dpcm , lpc )
- Audio and voice compression. ( Gray, preprint ) ( wikipedia ) ( Cap. 1, 6, 7, 10 You, 2010 )
- Speech (vocal apparatus video 1 , 2 , 3 )
- Vision ( khanacademy ), Hearing ( TED-ed , khanacademy ), Perception ( Kandel ).
- Video compression. MPEG Standard.
- Principal Component Analysis ( Shlens 2014 ) (slides pca )
- The Unreasonable Effectiveness of JPEG: A Signal Processing Approach ( youtube )
- How PNG Works: Compromising Speed for Quality ( youtube )
Introduction & Video Overviews
- Compression: Crash Course Computer Science #21
- JPEG DCT, Discrete Cosine Transform (JPEG Pt2)- Computerphile
- Digital Audio Compression - Computerphile
- Entropy in Compression - Computerphile
- JPEG - Wikipedia
- How computers learn to recognize objects instantly | Joseph Redmon
- How we teach computers to understand pictures | Fei Fei Li
- How a driverless car sees the road | Chris Urmson
Recommended Bibliography
- Salomon, D., Data Compression: The Complete Reference , Springer. ( google books , archive )
- Salomon, D., Motta, G., Handbook of Data Compression, Springer. ( google books )
- Gibson, J. D., Multimedia Communications: Directions and Innovations, Academic Press. ( google books )
- Oppenheim, A.V., Schafer, R., Digital Signal Processing, Prentice-Hall. ( google books )
- Gonzalez, R.C., Woods, R.E., Digital Image Processing , Pearson. ( google books )
- Sayood, K., Introduction to Data Compression . Morgan Kaufmann. ( google books )
- Bocharova, I., Compression for Multimedia . Cambridge University Press. ( google books )
- Huang, X., Acero, A., Hon, H., Spoken Language Processing . Prentice Hall. ( google books )
Suggested Readings, Standards & Resources
- Gallager, R.G., Principles of Digital Communications, Chapter 3. ( MIT )
- Gray, R., Neuhoff, D. L., Quantization , IEEE Transactions on Information Theory, 44(6), October 1998. ( IEEE )
- Gray, R. Vector quantization . IEEE ASSP Magazine, 1 (2), 4-29,1984. ( IEEE )
- Graham Hudson et al, JPEG at 25: Still Going Strong, 24, 2, 2017 ( IEEE ).
- Vaidyanathan, P.P., The Theory of Linear Prediction , Morgan & Claypool Publishers, 2008. ( google books , archive )
- Yehia, H. C., Análise de funções de erro em sistemas de codificação LPC , ITA, 1993. ( ITA ) - capítulos 1 e 2
- John Makhoul, Linear Prediction: A Tutorial Review, Volume: 63, Issue: 4 , April 1975 ( IEEE )
- Atal, B. S., The History of Linear Prediction , IEEE Signal Processing Magazine 23 (2), 2006. ( IEEE )
- Lima, P. C., Wavelets: uma introdução , Matemática Universitária, 33, 2002. ( matemática universitária )
- Gomes, J., Velho, L., From Fourier Analysis to Wavelets , IMPA Springer, 2015. ( google books )
- Gomes, J., Velho, L., Goldenstein, S., Wavelets: Teoria, Software e Aplicações , 21 Colóquio Brasileiro de Matemática, 1997.
- Schroeder, M., Atal , B., Code-excited linear prediction(CELP): High-quality speech at very low bit rates, ICASSP, 1985. ( IEEE )
- Rabiner, L. R., Schafer, R. W., Digital Processing of Speech Signals , Pearson, 1978.
- Gray, R. M., Linear Predictive Coding and the Internet Protocol: A survey of LPC and a History of of Real time Digital Speech on Packet Networks , Journal Foundations and Trends in Signal Processing, 3 (3), 2010. ( preprint )
- Musmann, H. G., Genesis of the MP3 audio coding standard , IEEE Transactions on Consumer Electronics, 52 (3), 2006. ( IEEE )
- Karwowski, D., et al, 20 Years of Progress in Video Compression - from MPEG-1 to MPEG-H HEVC , International Conference on Image Processing and Communications, 2017.
- You, Y., Audio Coding: Theory and Applications , Springer, 2010. ( google books )
- Kerr, D. A., Chrominance Subsampling in Digital Images, 2012. ( online )
- Encyclopedia of Graphics File Formats: https://www.fileformat.info/mirror/egff/index.htm
- Dave Litwiller, CCD vs. CMOS: facts and fiction, Photonics spectra 35(1):154-158 · January 2001. ( online )
- Richard W. Harold, An introduction to apperance analysis, SS Number 84, 2001 ( link )
- Karlheinz Brandenburg, MP3 and AAC explained, 1999 ( link )
- J. Zeng, O. C. Au, W. Dai, Y. kong, L. Jia, W. Zhu, A Tutorial on Image/Video Coding Standards, APSIPA, 2013 ( link )
- Russ, J. C., The Image Processing Handbook , CRC Press, 2006 ( google books )
- Lepton image compression ( link ), Guetzli ( link ), WaveNet ( link )
- TED talks: How computers learn to recognize objects instantly ( link ); How we teach computers to understand pictures ( link )
- Computer-based synthesis of speech and song on IBM 704 ( link ) ( yt-link )
- The JPEG XL Image Coding System: History, Features, Coding Tools, Design Rationale, and Future ( link )
- rules (portuguese)
- homework (portuguese)
- slides (portuguese)
- video lecture (portuguese)
- old exams (portguese)
- introduction to GNU Octave (portuguese)
- slides: métodos de dicionários , quantização escalar , quantização vetorial , ...