Classifying remotely sensed images from urban environments is challenging. Urban land cover classes are spectrally heterogeneous and materials from different classes have similar spectral properties. Image segmentation has become a common preprocessing step that helped to overcome such problems. However, little attention has been paid to impacts of segmentation on the data’s spectral information content. Here, urban hyperspectral data is spectrally classified using support vector machines (SVM). By training a SVM on pixel information and applying it to the image before segmentation and after segmentation at different levels, the classification framework is maintained and the influence of the spectral generalization during image segmentation hence directly investigated. In addition, a straightforward multi-level approach was performed, which combines information from different levels into one final map. A stratified accuracy assessment by urban structure types is applied. The classification of the unsegmented data achieves an overall accuracy of 88.7%. Accuracy of the segment-based classification is lower and decreases with increasing segment size. Highest accuracies for the different urban structure types are achieved at varying segmentation levels. The accuracy of the multi-level approach is similar to that of unsegmented data but comprises the positive effects of more homogeneous segment-based classifications at different levels in one map.