The well-known perceptual magnet effect [e.g., P. K. Kuhl, Percept. Psychophys. 50, 93--107 (1991)] is characterized by a warping of perceptual space near phonemic category centers. Whereas previous explanations of this effect have been formulated within the theoretical framework of cognitive psychology, the model proposed here builds on research from both psychology and neuroscience to account for the effect. This model embodies two principal hypotheses supported by considerable experimental and theoretical research from the neuroscience literature: (1) Sensory experience guides development of an auditory neural map, and (2) a population vector analysis can predict psychological phenomena from map cell activities. These hypotheses are realized in a simple, self-organizing neural network model. The magnet effect arises in the model from language-specific nonuniformities in the distribution of map cell firing preferences. These nonuniformities result from exposure to language-specific distributions of attended sounds experienced by the network during training, in keeping with neurophysiological results from auditory cortex [G. H. Recanzone, C. E. Schreiner, and M. M. Merzenich, J. Neurosci. 13, 87--103 (1993)] and other sensory areas. Numerical simulations verify that the model captures the known general characteristics of the magnet effect and provides very accurate fits to specific psychophysical data.