The History of Middle-earth

JRR Tolkien was subject to Attention Deficit Creator Disorder and perfectionism, with the result that The Hobbit and The Lord of the Rings are rare among his works in that he actually finished them - though even then continually making notes for revised second editions. The vast majority of his works were in a disorganised array of disparate parts written across more than 50 years when he died. His son Christopher Tolkien put together the published version of The Silmarillion using some of these parts, but later decided to present more of the source materials alone — with commentary — to demonstrate how the conception of Middle-earth had evolved across the years.

The result was The History of Middle-earth, 12 volumes covering the evolution of Tolkien's legendarium from 1916 to 1973. The content ranges from earlier forms of the Silmarillion legends and early drafts of The Lord of the Rings, to narrative texts (some more complete, some less), to essays about the history of the world, its culture, languages, and more. Along the way, we are also introduced to previously unknown and interesting offshoots of the legendarium, such as Eriol the Mariner of The Book of Lost Tales, Alboin and Audoin of The Lost Road, the cast of The Notion Club Papers and many more.

Volumes I and II of The History of Middle-earth are also known as The Book of Lost Tales. These were initially published on their own, and only after the conception of The History of Middle-earth they were re-published as the first two instalments of that series.

Not for the faint-hearted by any means, but a must for anyone who wants to really understand Middle-earth inside and out.

The History of Middle-earth contains examples of:
"Be he foe or friend, be he foul or clean
 * All Myths Are True: In particular with The Book of Lost Tales and still to a lesser extent later. As Tolkien's friend CS Lewis would go on to do with Narnia, the works reconcile the idea of pagan gods with Christian theology, and include stories based on those from Norse Mythology.
 * Animal Jingoism: The original Tale of Tinúviel is a mythological origin for Cats vs. Dogs - in that version, Huan fights great cats rather than wolves (except for Karkaras/Carcharoth).
 * Badass Creed: The Oath of Feanor, as expressed in The Lays of Beleriand:

Brood of Morgoth or bright Vala,

Elda or Maia or Aftercomer,

Man yet unborn upon Middle-earth,

Neither law, nor love, nor league of swords,

Dread nor danger, not Doom itself

Shall defend him from Fëanáro, and Fëanáro’s kin,

Whoso hideth or hoardeth, or in hand taketh,

Finding keepeth or afar casteth

A Silmaril. This swear we all…

Death we will deal him ere Day’s ending,

Woe unto world’s end! Our word hear thou,

Eru Allfather! To the everlasting

Darkness doom us if our deed faileth…

On the holy mountain hear in witness

and our vow remember,

Manwë and Varda!"

"Indeed afterward Melko heard all and he cursed Tevildo and his folk and banished them, nor have they since that day had lord or master or any friend, and their voices wail and screech for their hearts are very lonely and bitter and full of loss, yet there is only darkness within and no kindliness."
 * Canis Major: Huan and Karkaras Knife-fang (the later Carcharoth).
 * Cats Are Mean: Actually mythologically justified in the Lost Tales:

"Eriol: Who was Ilúvatar? Was he of the Gods?
 * Composite Character:
 * Sauron grew from a combination of three characters from the Lost Tales - he has the name of the wizard Tu (Tu - Thu - Thaur - Thauron - Sauron), the position of Fankil/Fangli (The Dragon to Melko), and the narrative role of Tevildo Prince of Cats in The Tale of Tinuviel.
 * Inverted with Finwë. Originally, when Fëanor was not his son, he did not die in Valinor, and led the Noldoli (Noldor) at the Battle of Unnumbered Tears. When this was changed so he died early on, Tolkien split his roles between different descendants - hence why so many characters in The Silmarillion have Fin- names, such as Fingolfin and Fingon.
 * Also inverted with Herendil, Elendil's (only) son in The Lost Road. The Lord of the Rings gave him two sons, Isildur and Anarion; Herendil seems most like Isildur, with a fiery and somewhat rebellious spirit and a fascination with Sur (Sauron).
 * Demoted to Extra: Those familiar with The Silmarillion and The Lord of the Rings will recognise throwaway names in those works who originally had much greater roles. For example, Rúmil is mentioned as a creator of alphabets in those books, but is an important character in The Book of Lost Tales.
 * Elves vs. Dwarves: The Lost Tales present the origins of the conflict, as in The Silmarillion, but the Dwarves are presented as another evil race comparable to Orcs (the influence of Norse Mythology being obvious). It wouldn't be until The Hobbit that they got to be sympathetic characters - and indeed given the Wood-elves of that book were essentially a recycled version of Tinwelint's folk from the Lost Tales, it can even be considered a Perspective Flip.
 * Evil Albino: Lungorthin, from The Lays of Beleriand, is a Balrog that was encased in white flames, as opposed to red like the other Balrogs.
 * Expy and Self-Insert Fic: The Notion Club Papers is Tolkien parodying his own meetings with the Inklings — 'Notion Club' being a pun on that name — and a way for him to criticise C.S. Lewis' ideas in his Space Trilogy by having characters discuss it. It later evolved into a story about characters having ancestral memories of Númenor.
 * The Fair Folk: Subverted, the Lost Tales say that the Celtic notion of evil elves is due to conflicts and misunderstanding, and that the more positive English view is correct. To take one example, Tolkien plays with the idea that eating fairy food traps you in their land forever - the Elves of Tol Eressea drink something called limpë, but warn the human traveller Eriol against drinking it, for his heart will forever be filled with their own sadness and he will be compelled to fight for their causes against his own kindred.
 * God: Ilúvatar. Interestingly treated in The Lost Tales, which uses 'Gods' in a pagan sense to describe the Valar:

Rúmil: Nay, that he was not, for he made them. Ilúvatar is the Lord for Always who dwells beyond the world; who made it and is not in it or of it, but loves it."

"They say that the One will himself enter into Arda, and heal Men and all the Marring from the beginning to the end."
 * Heel Face Turn: Figuratively speaking, some characters in early drafts of The Lord of the Rings started out evil but were then reworked into different, good characters. Farmer Maggot was originally an antagonist who was even responsible for setting the Black Riders onto Frodo's trail, while Treebeard was originally an evil giant who had the same role as Saruman in the final book, imprisoning Gandalf.
 * Hidden Depths: Revealed for some characters who may have seemed one-dimensional in the published Silmarillion, such as the more antagonistic of the Sons of Fëanor.
 * Hoist By Her Own Petard: Just barely Subverted with Ungoliant in Morgoth's Ring, who, prior to her meeting with Melkor, was starving to death in her lair, due to her own webs of darkness blocking off any light from entering her lair for her to devour.
 * Humans Through Alien Eyes: Done in The Notion Club Papers, when a character travels through space in his dreams by astral projection, observes various alien worlds, and eventually comes across a planet where he sees what looks like a great heaving anthill come into existence, despoiling a landscape. It's a shock to him when he realises that what he is actually looking at is the history of his own home city of Oxford, sped up to a great pace.
 * Immortality Begins At Twenty: Averted with the Númenóreans in The Lost Road, who age in proportion to their long lifespan, for example Herendil being considered an adolescent when he's in his forties. It's unclear whether this idea survived.
 * Interspecies Romance: Besides the ones already present in The Silmarillion and The Lord of the Rings, Athrabeth Finrod ah Andreth adds a failed one between the Elf Aegnor and the mortal woman Andreth, noteworthy because all the others Tolkien wrote about were between a mortal man and an Elf woman. Also, Beren and Luthien were originally written as two Elves, albeit from different kindreds, and the idea of their Interspecies Romance was added later.
 * Jesus: Implied in a prophecy of Men mentioned in the Athrabeth Finrod Ah Andreth. Tolkien, after writing it, thought it was a bit too explicit and seemed like 'a parody of Christianity', so he dropped it.

"Frodo: Who are you, Master?
 * MacGuffin: The Silmarils in The Book of Lost Tales. Tolkien rewrote it because he realised that there was nothing to justify their importance in driving the later story, resulting in the idea that they preserve the light of the Two Trees.
 * Literary Agent Hypothesis:
 * In the oldest draft, the history of the Elves supposedly came to us through Eriol, a sea-farer from 5th century Jutland; in the second draft Eriol was replaced by Ælfwine, an 10th century Anglo-Saxon mariner who stumbled upon Tol Eressëa.
 * Also, CS Lewis wrote a review of the Lay of Leithian as though it was an actual piece of mediaeval literature that had been rediscovered.
 * Magic Realism: The Notion Club Papers.
 * Name's the Same: Tolkien changed names of characters and locations at least once with every draft and recycled old ones, meaning names that would later be applied to well-known characters in The Lord of the Rings show up many years before the book was conceived. For example, one Gimli first appears in the Tale of Tinuviel (written in 1916) as a "Gnome" and fellow prisoner of Beren in Tevildo's kitchens. Another example is Legolas, who first appeared as the elf leading the flying people of Gondolin over the plains of Tumladen and over the pass of the Cirith Thoronarth.
 * Nice Job Breaking It, Hero: In The Book of Lost Tales: Part 1, A Gnome named Daurin rushes up to attack Ungoliant as she drains the Two Trees of their sap. Although he manages to wound one of her legs with his sword, its blade becomes stained with her Black Blood, turning it into a poisonous weapon. This doesn't work so well in the end, since he ended up being disarmed by Ungoliant, and killed by Melkor, who then used the tainted weapon to kill the remaining Tree.
 * Offscreen Moment of Awesome: Legions of Balrogs riding robotic dragon-shaped troop carriers to invade Gondolin in the earliest manuscripts of Book of Lost Tales, a scene never elaborated on. Also, the whole concept of Elvenhome (the island-ferry used to transport the Elves to Valinor) becoming England, the Elves being displaced by Eriol/Aelfwine's Anglo-Saxon cousins, and Avallone, the capital of the exiled Noldor, being Warwick.
 * Omniglot: Omar the Vala, who knows all tongues. Didn't survive to the final Silmarillion.
 * Our Dwarves Are All the Same: Dwarves were originally just another 'evil' race similar to Orcs, derived closely from Norse Mythology. It wasn't until The Hobbit that they became a primarily 'good' and sympathetic race.
 * Our Elves Are Better: The evolution of the Elves is shown from the Lost Tales (in which they are still closer to a Victorian conception and referred to as Fairies and, in the case of the Noldor, Gnomes) to the final high and mythic result from The Silmarillion.
 * Our Orcs Are Different: Orcs and goblins were pulled out of nowhere in the Lost Tales because Melko needed some Mooks. Much later on, Tolkien fleshed them out and there are essays written in which he considers the moral implications of whether Orcs are soulless automatons or have free will and choice and could potentially be redeemed. The published Silmarillion has Orcs as ruined Elves, but he considered changing it to ruined Men (even though that doesn't fit the timeline). He also considered changing their origin to 'uplifted' animals and (at least for some individuals) constructs with no 'real' life or will, or very minor fallen Maiar. The origin-of-Orcs question is possibly the messiest in the whole legendarium.
 * Prophecies Are Always Right: The original form of the Doom of Mandos is "Great is the Fall of Gondolin", uttered years before the founding of the titular city (though no-one seems to remember the words when the city gets named...)
 * Really Big Cat: Tevildo Prince of Cats from the Lost Tales.
 * Reincarnation: Both Ælfwine the Anglo-Saxon and Oxford don Alvin Lowdham are supposedly reincarnations of Elendil. The earlier versions of the myths are also more explicit about Elvish reincarnation, saying that Elves are reborn into their own children.
 * Satan: Morgoth (Melko, Melkor). The Athrabeth Finrod ah Andreth explicitly identifies the two.
 * Shoo Out the Clowns: in The Book of Lost Tales, Sauron is a giant cat (who tests Beren's skills as a trapper by having him catch three mice, because his castle is positively overrun with them), Morgoth has a wife and kids, an Elvish defector invents tanks, the sun and moon are ships that sail in the upper atmosphere, and, in one version of the story, Morgoth is banished from the world when he climbs up a gigantic pine-tree to escape the Valar and the Valar cut it down after him. As early as the 1930s, though, The Silmarillion is closer to its modern form.
 * Shoot the Messenger: In The Book of Lost Tales, Part 1, Melkor attempts to persuade the Valar to allow him and his companions to leave Valinor, but the messenger he sends to announce his demands is denounced as a "rogue" and a "traitor", and promptly executed. This actually made the Dark Lord upset, considering he had just Averted this trope earlier, when an Eagle was sent by the Valar to deliver a message to him, and would have expected his own messengers to be treated in kind by the Gods.
 * Shout Out: The original Tale of Turambar includes one to Sigurd from Norse Mythology, in which the teller of the story mentions that men think that eating the heart of a dragon allows you to understand all tongues, but this is a false belief because the blood of a dragon is poisonous.
 * Steampunk: Númenor under Sauron's tutelage in The Lost Road has steam-powered ironclad ships, some form of aircraft and what sound from the description like some kind of guided missiles. In an example of Schizo Tech, however, they also still fight with swords and armour. Also, in The Book of Lost Tales Morgoth attacks Gondolin with mechanic dragons.
 * Time Abyss: The earlier drafts of The Lord of the Rings offer a few more details about Tom Bombadil:

Tom: Eh, what? I am an Aborigine, that's what I am, the Aborigine of this land. I have spoken a mort of languages and called myself by many names. Mark my words, my merry friends: Tom was here before the River or the Trees. Tom remembers the first acorn and the first rain-drop. He made paths before the Big People, and saw the Little People arriving. He was here before the kings and the graves and the Barrow-wights. When the Elves passed westward Tom was here already - before the seas were bent. He saw the Sun rise in the West and the Moon following, before the new order of days was made. He knew the dark under the stars when it was fearless--before the Dark Lord came from Outside."


 * Also in an earlier draft, Treebeard mentions that Tom Bombadil has the longest name in the world. He also says that Ents give people or things names that are longer the older they have existed.


 * The Watson: Ælfwine. He serves as the Audience Surrogate to whom the elves of the Lonely Isle relate the history of the Elder Days.
 * What Could Have Been: The point of the History.
 * What Happened to The Mouse?: The group of rogue Maiar that accompany Melkor and Ungoliant in an early account of the Darkening of Valinor. After assisting Melkor in the theft of the Silmarils and accompanying him when he meets with Ungoliant, they all flee after the destruction of the Two Trees, and never appear again. Although it is mentioned that the Valar found many of these rogues in the Northern regions of Valinor, and slew them, they are mostly forgotten by the time the next part of the story sets in.
 * Several characters, both major and minor, also suffer from this throughout the various drafts of the stories.
 * Write What You Know: The Fall of Gondolin from The Book of Lost Tales is clearly inspired by Tolkien's then-recent experiences in World War One, down to Morgoth's mechanical dragons evoking the early tanks used in the war.
 * Write Who You Know: The Notion Club Papers is an Affectionate Parody of Tolkien's own experiences with the Inklings. Also, Author Avatars Alvin Arundel Lowdham from that story and Alboin Erroll from The Lost Road are both linguists.