We provided a key safe for emergency use only. This was given to a carer to gain entry an hour earlier than she was due. She appeared in my ninety two year old mother in laws bedroom unannounced frightening the wits out of her. We complained and made it clear this was not to happen again, however it did. We were promised continuity of care yet she rarely had the same carer twice. Her morning carer was scheduled for 8am but often had not arrived by 11am to wash, dress and give her breakfast. Her evening visit was scheduled for 6 pm to get her ready for bed, but often arrived at 4pm.
You may need to work with a different agency. Or you may just need a better way to track what is going on. For example, can you call each day at 8:30, and if a carer is not there by then, call the agency immediately? Could you arrange for the carers to call you first thing when they arrive? (Many service people do this.)
I can well understand your frustration. Just be sure you really know what is happening before you act.