Unless your role demands you to be in office, I think saying that you want to avoid travel should be enough. Can your doctor write you a note to mention that? I just told my manager verbally and they were okay with it, but it may not be the case for everyone.
Does your office have a strict in-person work policy? Usually you can say you want to avoid the travel and if there is any sickness then that should be a good enough reason. At our office we usually work from home and occasionally go for meetings but I avoid those too quoting the above reasons. No doctor's note was asked