Jessica's right. The only way to take a diary off your switch list is to manually log into that diary and delete yourself off their list.
And if you don't have the login information, then it's not possible. Can you contact the creator of the diary via their personal diary?
And, as far as I know, Steve is the only one who may have the capacity to remove diaries off switch lists. So if you wish to raise a ticket, address it to Steve. =)