Why does having a certain attribute make it okay to speak negatively about it?

For example a Jewish person insulting or poking fun at Jewish traditions is usually acceptable whereas someone who has different religious beliefs saying the same things would like be pegged as insulting and anti-semitic. Personally, I think everyone should have the right should have the right to speak their mind. I'm just curious as to why an insult from someone outside of a certain group is judged more harshly than one from inside of it.