Brain imaging genetics is an important research topic in brain science, which combines genetic variations and brain structures or functions to uncover the genetic basis of brain disorders. Imaging data collected by different technologies, measuring the same brain distinctly, might carry complementary but different information. Unfortunately, we do not know the extent to which phenotypic variance is shared among multiple imaging modalities, which might trace back to the complex genetic mechanism. In this study, we propose a novel dirty multi-task SCCA to analyze imaging genetics problems with multiple modalities of brain imaging quantitative traits (QTs) involved. The proposed method can not only identify the shared SNPs and QTs across multiple modalities, but also identify the modality-specific SNPs and QTs, showing a flexible capability of discovering the complex multi-SNP-multi-QT associations. Compared with the multi-view SCCA and multi-task SCCA, our method shows better canonical correlation coefficients and canonical weights on both synthetic and real neuroimaging genetic data. This demonstrates that the proposed dirty multi-task SCCA could be a meaningful and powerful alternative method in multi-modal brain imaging genetics.