Recent studies have shown that neural activity in auditory cortex encodes the envelope of speech, and that this encoding is more robust for attended speech than for unattended speech during multi-speaker (cocktail party) situations. To determine if this effect could form the basis for a novel brain-computer interface (BCI), we investigated the accuracy with which a subject's locus of attention during a cocktail party task could be ascertained from envelope responses present within single trials of EEG. We found that the attended speaker can be determined reliably from short periods of EEG, with accuracy improving as a function of trial length. Furthermore, we compared the performance of this envelope-based attention classifier to others based on changes in steady-state responses (elicited via 40 and 41 Hz amplitude modulations of the speech) and hemispheric lateralization of alpha power. We found that the neural responses to the speech envelopes were far more robust indicators of attention than the others. These results suggest that envelope-related signals recorded in EEG data can be used to form robust auditory BCI's that do not require artificial manipulation (e.g., amplitude modulation) of stimuli in order to function.