Reality shows are marked primarily by a lack of scripted lines for at least some of the participants.
The Real World was the first break-out reality show, and forms the basis of one of the key aspects of many reality programs—groups of people living together in a single home. In The Real World that was the basis of the whole show. Future reality series expanded on the premise by introducing competitive elements -- gradually removing people from the group, leading to a prize for the last man standing. Most successfully, the Survivor series put the groups in exotic locations and forced them to survive both the physical and social environment.
A second sub-genre of reality shows is the talent competition. Somewhere between Star Search and Survivor, these longer running shows, like American Idol and Last Comic Standing, pit contestants against each other in a long running talent competition. Competitors are eliminated each week, with the last man standing (and most likely several of the runners-up) awarded a contract to help them on their road to stardom (or, you know, obscurity, whichever comes first). Note that the key difference between these and a regular Game Show is their structure; regular game shows tend to be very rigid and self-contained while a reality game show tends to be more open and subjective.
An argument exists that these shows are presented as reality shows because that's what sells right now, but in the viewing experience little distinguishes them from a Talent Show.
A third sub-genre is the "day in the life" show, which follows an individual, family or group throughout their daily lives and records the many dramas that can occur. Usually this person is someone who achieved a small measure of fame, or infamy, in the past and sees it as an attempt to return to their Glory Days by inviting the viewing public into their lives. In other cases, a group of people is assembled based on looks, outrageousness or potential for drama and set loose in front of the camera.
There is also the "recovery" show, which features people with serious substance abuse issues or mental illness and follows them through the recovery process, even if this process is not ultimately successful. There has been some debate about whether these types of shows are actually helpful to people with similar problems and their loved ones or whether it's the modern day equivalent of the rabble being entertained by paying a penny to gawk at the inmates of Bedlam House.
Relationship shows are perhaps the most numerous of Reality programs. Put together a man or woman and a pool of potential love interests. Let the target pick and choose amongst the pool, throw in a few Shocking Twists, and hope sparks fly. Who Wants to Marry a Millionaire? led to Joe Millionaire led to Average Joe. Each show ends with the final choice. Then maybe the chosen one's shocking choice. And possibly the retrospective show. In one lone case out of all the relationship shows thus far -- The Bachelorette—even the Wedding Show.
Even more recently there have been subverted Reality Shows, where only one participant is "genuine", and all the others are actors; the interest in the show comes from the implicit railroading toward the "win" that the sole "contestant" undergoes without his knowledge.
See Reality TV for examples. These types of programs, due to their cheap production values and broad appeal, are often at the root of Network Decay. One important thing to remember about shows of this nature is that their benefit to a network is almost completely short-term. Their cheap production value and frequent high ratings send investors flocking to them, which make them great shows for filling in bare spots on a TV schedule; however, since most of them involve some type of contest, their appeal dies the second they announce the winner. Most are never released on DVD (those that are are usually better classified as "documentaries" instead of "reality TV"), and syndication is very hit or miss. Fox Reality shows quite a few, and America's Next Top Model has had some success on MTV, VH-1, and Oxygen, but that's usually for daytime filler (similar to the endless Real World marathons MTV used to do), and American Idol Rewind works mostly because they have commentary and "where are they now" style interviews with the contestants, similar to DVD episode commentary.