After Akhilesh jibe, CM Yogi cuts Karnataka trip short