Dad and I have Medical and Financial POA. We are ONLY caregivers, they do not help. I am main caregiver during the day. Monitoring vitals, dispensing medication, cook, clean, transfer to all doctors appts. Am present with all exams and consultations. I am the main contact for all 9 physicians (granny gets confused.) Mom and dad care for them at night as they live with them. We have asked other siblings (aunts) to help, even for a weekend. They said they work and have their own lives and cannot and "will not" help (?!) BUT...they want to come and scoop them up at will and take them out of town on a whim. They are pretty much homebound. Grandad is in 6th year Alzheimer/dementia, congestive heart failure, many other issues. Grandmother has heart, anxiety, and blood pressure issues to name a few. She is unable to make smart decisions and sends money to an uncle who is on drugs and lives on and off the streets. To grandmother, WE are the bad guys because we won't let them go. Dad and I have Medical and Financial POA. Long story, short....can the aunts disregard our Power and take them??
As to impulsive trips, that wouldn't fit with your schedule; you would require advance notice.
I don't really understand what's going on if they want to take your grandparents on impulsive trips but perhaps that's all they're capable of doing - they may not be able to handle the caregiving aspects.
I think that it would be confusing for both to suddenly be taken out of town, especially for someone with dementia and another with anxiety issues.
You'd have to review your DPOA and see if it grants authority to restrict movement, including with relatives.
If you want medical backup, which I think might be appropriate, ask their internists or PCPs about the visits, and whether it might cause confusion and be detrimental to their health. Check the medical POA to see if it covers those kinds of conditions. Depending on whether it was drafted by a good EP attorney or is just a downloadable Internet form, it may have broad authority to restrict activities that aren't healthful, such as out of town visits.
Alternately, suggest that the family come and visit at your house; if there's not enough room, you can always plan a picnic locally at a park, or someplace that's not so unique that it causes confusion for your GPs.
But be sure to be back before dark if there's any sundowning involved. I would think that just seeing relatives who aren't around much would cause some confusion for your grandfather.