This is when the broadcast networks have their evening network programming on their affiliates. It's when most of the possible demographics are off work, out of school, and awake—and thus, it's the time with the largest possible general audience. It usually is when the most people are actually watching as well, especially early in the week. (Later in the week, people often go out, and since the networks are all tied in to movie studios these days, they don't want to discourage the impulse completely.)
In America, Prime Time is 8-11 p.m. Eastern, 7-10 p.m. Central weekdays, and 7-11 p.m. Eastern, 6-10 p.m. Central on Sundays. (It's been a while since the American broadcast networks have taken Saturday seriously.) FOX, The CW (and its predecessors), and My Network TV don't use the last hour.
The first hour or two of prime time used to be called the Family Hour during The Seventies and The Eighties, and was supposed to air family-friendly programming. Innuendo, Bloodless Carnage, and Technical Pacifists could be aired, but not outright sex or bloody violence. This has changed for many reasons, including changes in the Federal Communications Commission and the advent of the v-chip. Few people use the v-chp directly, but the parental guideline icons that tell you which "block" range contains a program work as a Ratings System, and they are aired publicly every half hour. If you know to avoid, or can program your TV to avoid, all TV-14 or TV-MA programming, there's no need to force it all after 10 p.m., is there?
Well into The Nineties, the first hour of prime time was always two half-hour Sit Coms, followed by two hour-long drama/action shows in the 9-10 and 10-11 ET slots, with the only variations being two hours of sitcoms (8-10) on Fridays and Saturdays (when more kids were watching) and maybe a newsmagazine or two later in the week- see ANSI Standard Broadcast TV Schedule. There's a lot more variation now.
In Germany, prime time starts at 20:15 (8:15 pm). The main reason is that the Tagesschau, the oldest newscast on German TV, has been aired from 20:00 until 20:15 since The Fifties. Other stations use the same timing, and an experiment by a few private stations in the 1990s to start their prime time programs on the full hour was not popular.