My parents are still living on their own but are being harassed by their son for money and signing on loans with him. I am the only other child and I don't or can't get them to give me full authority over these things. I have POA but because they are competent by definition, they are losing much of the financial freedom they once enjoyed
I would discuss what they are doing with their doctor. Is this a sign of cognitive decline? Maybe; a lack of ability to appreciate consequences can be a symptom.
I would talk with an attorney about what your duties as POA entail. Are you responsible for them having funds to care for themselves down the road?
I would talk to my parents and tell them that I can no longer hold their poa, since there will be a long look back at funds gifted, and that I might be held responsible.
You say you brother is harassing them. Is he threatening them, intimidating them? Call their lawyer and ask for his advice about this, as well.
http://www.dads.state.tx.us/news_info/publications/brochures/pub395-guardianship.pdf
I hope it can be of some help to you, even if you don't live in Texas, some of the information might be the same.
Perhaps they have an agreement with your brother that these loans are an advance on his inheritance?