If you have doubts, you should always keep your options open. Medicine is not for everyone. Choosing to pursue a career in medicine for prestige or financial rewards will likely be disappointing, as most physicians find that medicine is a vocation that requires a commitment to service, lifelong learning and the dedication to practice competently and compassionately.
The decision whether or not to stay on a pre-medical track can usually be deferred through the first two years of college without loss of credit when changing majors. This will give you time to gather as much information as possible about what medicine has to offer.