The Singularity/Headscratchers

Note to posters: The Singularity Institute has a nice FAQ about the singularity, with several citations and links to scholarly articles. Please give it a look before asking a question.

There are a few unsubstantiated assumptions with the standard "AI will usher in the Technological Singularity" theory.

1: Artificial Intelligence is possible.
- We don't know that it is.
2: AI can upgrade itself.
- For all we know, any modifications will require a reboot to take effect, and that destroys the machine.
3: The machine wants to improve itself.
- As far as we know, reproduction and improvement are a strictly ~~meatbag~~ biological imperative.
4: The machines will be smarter than us.
- Maybe our brains are as smart as you can get for 3 lbs. Or maybe coherent consciousness has an upper limit on intelligence. Or maybe biological systems are intrinsically faster than silicon.

1. I don't know why you would think that it isn't; intelligence can clearly be implemented in organic matter. Carbon and oxygen and hydrogen aren't magic, and machine intelligence continues to increase little by little. It seems clear to me that general intelligence will be implemented on silicon at some point, barring some kind of horrible catastrophe that wipes us all out.
2. Wait, what?
3. Improvement, like survival, is a nearly universal value in that it enables one to better pursue one's goals, and would only be discarded when it was in direct conflict with those goals. It's hard to imagine that, whatever such a machine wants (it's programmed) to do, it won't see improving itself as a logical step in the way of getting it done faster and better and with less risk of failing. See Basic AI Drives
4. If there is an upper limit on intelligence, I would be very surprised if it just happened to be at the human level, or so slightly above it that there was no room for significant improvements. You are gonna need a lot of evidence before I even begin considering that one. Also, who says that an artificial intelligence needs to be run on 3 lbs of hardware?

1. We are all artificially intelligent
2. We are all capable of upgrading ourselves (in many countries, it is illegal NOT to spend over a decade doing nothing but this)
3. Most of us want to improve ourselves
4. Some of us are smarter than the rest of us

Even if intelligence is impossible without a shell made of carbon, hydrogen, and oxygen, such a shell can be artificially made (there something like seven billion living examples to support this statement), and probably created without a traditional body to control (why not?). Such a shell could be much larger than the human brain, and presumably more powerful (perhaps the human brain in its current configuration and size is the most intelligent thing possible, but this is doubtful). At least one such artificial construct would want to improve itself or to improve another intelligence (more such intelligences could be created until one was like this). It would be capable of improving itself or another like it, and so on until at least one is more intelligent than humans. The singularity is inevitable. Only xenocide can stop it.

Specifically in answer to no. 2: take a look at Tool Command Language or PicoLisp; two programming languages that actually let you completely rewrite the application code while it's running (technically possible with any language but these two really showcase it by being almost totally homoiconic). Having to reboot for changes to take effect is a bizarre design decision popularised by certain current-gen operating systems, not universal either to all systems or all programs.
- As an addendum: take a look at the Crawler-Tractor, implemented in newLISP, for a piece of code that doesn't just modify another part of the running program - the same function actually mutates itself, to create an endless cycle while using neither recursion nor a loop. This sort of thing is why Lisp dialects got a reputation for being good for AI work back in the 60s-80s.

Heck, even if the AI did have to reboot to improve itself, it could just save its present mind state, reboot, and go on doing whatever it was doing, except better now. They would have much more reliable non-volatile memory than biological brains do.

Since technology must be based on itself or what came before, it can't increase any faster than exponentially. So why is it named after a mathematical phenomenon that doesn't occur on an exponential curve!? (Oh, and with respect to the four questions - it's one of the basic tenets of science that unless there is a reason for things to be different, they're probably the same. (1) Since humans are organic computers and are intelligent, silicon-based intelligence is not different. (2) See the production line: robots building more advanced computers and robots. (3) There are already path-perfecting programs in existence. These actively try to make the best solution given certain output parameters. If those parameters involve intelligence, then the program will develop intelligence (if given the right starting tools). (4) There is no reason why there would be a limit.

Who the Hell actually wants this sort of thing to happen? The death of baseline humanity isn't a remotely pleasant thought yet scienteists talk about it with a disturbing glee.
- It depends on your definition of idenity. In terms of Mind Uploading, is an exact copy of something essentially the original? Many scientists seem to think so, though I'd have to say it's not the way I see it.
- Who says the death of anyone is a requisite for this?
- Who says that we won't be human anymore? There is nothing to suggest that altering the materials of our bodies or location of our consciousnesses will somehow dehumanise us. And what's so special about a damp mass of organic matter that gets sick?
  - 'What's so special about a damp mass of organic matter that gets sick?' Yeah, that doesn't scare me away at all. What's so special a rusted out tin can?
- Those in favor should keep in mind that Brain Uploading isn't like moving your mind from one shell to another. It's creating a digitcal copy of the original. From your perspective, you're gonna stay in your meat body (unless you get destroyed in the process, which isn't any better).
  - Brain Uploading isn't the only potential option, though.
    - Correct. Personally, I would only Brain Upload if I were about to die and no other technology could preserve my life. I would prefer a slow transition to full artificial intelligence, meaning some type of specialized Nanomachines that behaves like human brain cells and neurons that attach themselves to my existing ones but vastly increases their capabilities. Then as my organic ones die one by one the specialized nanomachines have already been doing majority of the work of my by now centuries old organic ones. It would be hard to argue that I’m no longer me, as which brain cell was me when it died?

Should we ever achieve the Singularity, would it be moral, or even laudable, for us who become part of it to make becoming part of it obligatory? If so, is it a morally good thing to coerce people, by force or otherwise, to become part of said Singularity?
- See also Assimilation Plot for how various writers deal with such.