Fundamental physics with blazar spectra: a critical appraisal