There are some jokes almost anyone can get, and then there are ones that can only be understood by people with certain knowledge. Jokes about computer programming that can only be understood by technicians, jokes about a Film that only make sense if you listened to the director's commentary, jokes about a country's political system or culture that only citizens of that place could ever laugh at. If you are in on the joke, these can be hilarious; if you aren't, then they get irritating fast.
From a sociological point of view, in-jokes are used by groups to identify those who are Not Us—anyone who's not a member of the group is not likely to get the joke.
If these are frequently placed in a show or book, then it can become a case of Viewers Are Geniuses, or if they are language related, Bilingual Bonus. Fandoms are often teeming with them, so seeing a certain one everywhere may be a result of Fan Myopia. In some cases, may even lead to Memetic Mutation.
An Outside Joke relies on an opposite effect.