Territorial Instincts

Is it wrong for me to feel annoyed when a co-worker stops by my desk to chat and proceeds to peel an entire orange using my wastebasket? Personal space violation!