Cartridge

You mean those plastic things NES games were stored in? Yes and no. The plastic cases were just a covering, although that is why they are called cartridges (ink, bullets, and 8 track music in such cases are also called cartridges). The actual game is stored on a chip board, most of which is stored safely in the case, save for the connectors to the system.

Expansion

Cartridges have some advantages over Optical Discs. First is durability. The cartridges only have the outer connectors exposed to wear, and the systems themselves have no moving parts to wear out or overheat. Second, the games run faster; load-times are very short since data doesn't have to be physically sought on a disc or hard drive. But most importantly, the game itself only takes up a small physical part of the cartridge. The rest of the space can be used for extra chips, which can be used to increase the power of a system. Let's say the system doesn't have enough memory to run the game on the chip, but can accept extra memory from the cart. So an extra RAM chip is included. Early games with save points (like the NES Zelda games) used this trick, and included a watch battery inside the case to keep the RAM from erasing itself (Flash Memory, which doesn't need a backup battery, was several years off at this point).

The NES, SNES, and Sega Genesis all used this kind of modular system to get more out of their games than the main specs allowed; the SNES was particularly known for these "expansion systems", as some of the cartridges had the Super FX chip (a dedicated GPU that added 3D capability), an add-on DSP for physics calculations, or (in the case of Super Mario RPG and a few others) an entire second CPU clocked much faster than the main CPU.

Disadvantages

The disadvantage was the cost. With enough chips, games could cost around $100. That's about how much Chrono Trigger and Phantasy Star IV cost when they came out. Even at the bare minimum, a cartridge generally cost around $14 to manufacture as opposed to about a few cents for discs, which was getting lower and lower. That's why many developers left the N64 for the original PlayStation in the fifth generation. Carts were just too expensive to sell games at a decent profit margin. Nintendo switched to discs with its sixth-generation console.

Handhelds

But what about handhelds? The Game Boy's simple monochrome-screen design and emphasis on battery life meant that expensive coprocessors weren't of much value; aside from a few "memory mapper" chips (which allowed bigger games than usual via bank switching), add-on chips were barely used in Game Boy carts. This meant that the games could sell for half as much as home console games, and still have a decent profit margin. While expensive add on chips were rarely used, some games were able to use the cartridge to add light or gyro sensors, expanding gameplay possibilities. Nintendo kept this through all their handhelds, even the DS, 3DS, and the Nintendo Switch. The technology was also quite different: the Nintendo DS, for example, used flash ROM later in its run, the same kind you'd find in a USB drive, allowing on cartridge saves without a battery. The Switch would use read only memory and keep saves on device.

Present Day

So nowadays carts seem to be back on track. As the prices of Flash Memory continue to fall and the data they hold rises while optical discs are slowing down a bit, there's a real chance that solid state distribution media could become economical again, and rather soon. In addition to the physical advantages mentioned above, they make Digital Piracy significantly more complicated since you need both a compatible blank cartridge to write on, as well as the hardware to actually do the writing. Rumors that eighth generation game consoles would store games on flash cards never panned out, apart from the handhelds. In the ninth generation, entry level consoles would simply forgo any removable media in favor of being digital exclusive, with the high end consoles retaining an optical drive. (per gigabyte optical is still cheaper, and the fact it can be a movie player is a selling point.)