Subject: Re: audio-visual correlation From: at <TODDFS4.PSY.MAN.AC.UK> Date: Fri, 13 Feb 1998 10:39:52 GMTDear List I have also been reading with great interest the recent contributions on A-V interactions. My own research has been concerned with the relationship between auditory rhythm and visual motion and over the last ten years I have accumulated a body of quotes on this matter, from which selected items are listed below. "I am speaking of things moved in the way that the voice is moved in speaking and singing, and the body in making a gesture and dancing". Aristoxenus of Tarentum c. 320 BC "The material of music is sound and bodily movement". Aristdes Quintilianus c. 300 AD "Rhythm in speech, rhythm in sound and rhythm in motion , were in the beginning, parts of the same thing". Herbert Spencer, 1870. "...hence it becomes possible for motion in music to imitate the peculiar characteristics of motive forces in space, that is, to form an image of the various impulases and forces which lie at the root of motion. And on this, as I believe, essentially depends the power of music to picture emotion". Helmholtz, 1885. "The dynamic form of a phrase is a form of a movement.. In reciting verse, or in singing, a phrase becomes a single act of expiration; indeed, just this movement of breathing is probably the origin of musical phrasing." R.H. Stetson, 1905 "The origin of music lies in inner motion...a pure sensation of motion...which may be counted among the vestibular sensations...Musical motion can be likened to an invisible, imaginary dance which is free from all phusical constraints." Alexander Truslitt, 1938. Paul Fraisse (1982) also gives an account of the etymolology of the word rhythm. "Rhythm comes from the Greek words 'rithmos' [sorry can't get Greek characters] and 'reo' (to flow).... 'Rhythmos' appears as one of the key words in Ionian philosophy, generally meaning 'form', but an improvised, momentary, and modifiable form. 'Rhythmos' literally signifies a 'particular way of flowing'..." >From a computational view this essential unity of rhythm and motion can be realised if one considers the spatio-temporal power spectrum of visual and auditory images. E.g. Heeger, D. (1987) A model for the extraction of image flow. J. Opt. Soc. Am. 4(8), 1454-1470. Todd, N.P.McAngus (1998) A model of auditory image flow. Proc. 16th ICA. In terms on neurophysiology, as has been mentioned by Robert Zatorre, there exist various multi-modal areas, including in the cortex. V5, the visual motion area of the cortex, lives in the temporal lobe. It makes sense for the brain to evolve a common strategy for coding motion across all senses and indeed recent physiology (e.g. Kowalski et al, 1996. J. Neurophys. 76(5), 3504-3523.) is providing strong evidence for this. It should not suprise us that audio-visual interactions occur if there is high cross-correlation in the temporal power spectrum. On this basis I have developed some software for real-time visualisation of rhythm. I gave a little demo. at the ICMPC96 at Montreal. Goes down a treat with Manchester ravers! cheers Neil Todd Dept. of Psychol. Univ. of Manc. Manc. UK