My mother has mild vascular dementia, and normally does well. Her cognitive level was tested and is the same as one year ago. However we had two bad weeks of what we thought was agitation and hallucinations. The neurologist said he believes she has REM sleep disorder, with a non determined cause at this time. He mentioned stress, dementia and sleep apnea as possible causes. His decision was to wait and see. No meds, no treatment, no tests at this time. I don't think he understands what we have been thru these past two weeks. Things are better, but wait and see? I'm a little terrified to face those problems again.
(I agree with MACinCT: if you won't do the treatment, don't do the test. It is just that RBD has a different treatment than sleep apnea.)
My husband had RBD for a least a decade before the other dementia symptoms showed up. It was very easily controlled by a medication. If he took it he slept peacefully. If he didn't he was punching me in his sleep.
When the dementia was fully evident he needed an additional pill for agitation to help him sleep through the night.
I think many doctors have no idea how hard it is to care for a loved one who doesn't sleep. Those who do understand this do not "wait and see."
If you'd like to see an article about the relationship between RBD and Lewy Body Dementia, Google
REM sleep problems predict Parkinson's, Lewy body dementia
Of course, your mother may not have RBD. Without an overnight sleep study it would be hard to be sure. If this comes up again, I think I'd insist on a sleep study.