Just going to put my 2 cents in,

I was on the server when this happened, and according to Tadgee he entered your base through a different door and he assumed it was not building because there was no sign.

He needs to check the whole building for a building sign in the future, but I can see how he could assume there was no sign.