The rule is, quite simply:
If it's required by your plot, make one fantastic assumption in your story and only one - and do it in or before the first chapter (or first page or two, for shorter works). Do not add more as the story goes on. And once you have your one assumption, all further fantastic elements must derive from it, not any new assumptions. And just to be fair, it must be either obvious to the reader, or something that can be deduced from evidence present in your story.