Early change in memory performance is a key symptom of many brain diseases, but its underlying mechanism remains largely unknown. While structural MRI has been playing an essential role in revealing potentially relevant brain regions, increasing availability of diffusion MRI data (e.g., Human Connectome Project (HCP)) provides excellent opportunities for exploration of their complex coordination. Given the complementary information held in these two imaging modalities, we hypothesize that studying them as a whole, rather than individually, and exploring their association will provide us valuable insights of the memory mechanism. However, many existing association methods, such as sparse canonical correlation analysis (SCCA), only manage to handle two-way association and thus cannot guarantee the selection of biomarkers and associations to be memory relevant. To overcome this limitation, we propose a new outcome-relevant SCCA model (OSCCA) together with a new algorithm to enable the three-way associations among brain connectivity, anatomic structure and episodic memory performance. In comparison with traditional SCCA, we demonstrate the effectiveness of our model with both synthetic and real data from the HCP cohort.