Listener's segmentation of continuous speech is a subject which has been investigated cross-linguistically, proposing that listening strategies are language specific, which is based upon rhythm in language [Cutler, Cognition 50, 79--81 (1994)]. Listeners may, however, form conscious representations of potential within-word segmentations. In earlier work, it was proposed that listeners' representations of within-word structure could be much richer knowledge of word-internal structure, investigating monolingual speakers of Japanese and English [Otake et al., Proc. of Eurospeech 95 3, 1703--1796 (1995)]. Monolingual speakers of Japanese and English and bilingual speakers of Japanese and English were presented with spoken words in two languages, and asked to mark on a written transcript of each word the second natural division point in the word which is based upon a methodology used in the earlier work. The results showed clear and strong patterns of consensus, indicating that monolingual speakers of both Japanese and English exploit morae and syllables, respectively, as a unit of listeners' representations of within-word structure. Bilingual speakers of Japanese and English, however, exploit syllables, suggesting that they may be able to suppress a rhythm-based unit as listeners' representations of within-word structure.