Atherosclerotic plaque at the carotid bifurcation is the underlying cause of the majority of ischemic strokes. Noninvasive imaging and quantification of the compositional changes preceding gross anatomic changes within the arterial wall is essential for diagnosis of disease. Current imaging modalities such as duplex ultrasound, computed tomography, positron emission tomography are limited by the lack of compositional contrast and the detection of flow-limiting lesions. Although high-resolution magnetic resonance imaging has been developed to characterize atherosclerotic plaque composition, its accessibility for wide clinical use is limited. Here, we demonstrate a fiber-based multispectral photoacoustic tomography system for excitation of lipids and external acoustic detection of the generated ultrasound. Using sequential ultrasound imaging of ex vivo preparations we achieved ∼2 cm imaging depth and chemical selectivity for assessment of human arterial plaques. A multivariate curve resolution alternating least squares analysis method was applied to resolve the major chemical components, including intravascular lipid, intramuscular fat, and blood. These results show the promise of detecting carotid plaque in vivo through esophageal fiber-optic excitation of lipids and external acoustic detection of the generated ultrasound. This imaging system has great potential for serving as a point-of-care device for early diagnosis of carotid artery disease in the clinic.