So what's the difference between the two main branches of speculative fiction? It's not encompassed by the superficial details (ray guns, spaceships and robots versus wizards, elves, and dragons). There are conventions and expectations for each type of story, and not everyone who is attracted to one of them will be attracted to the other.
One convention is that in science fiction the science has to be consistent with real science even when it's way out there. A science fiction writer can put all sorts of things in a story that can't be proven or don't exist in real science at this time, including some very far-fetched speculation indeed, but he can't put anything in that can be proven wrong. That goes for everything from faster than light travel to alternative dimensions to psychic powers to practical probability physics to . . . well, you get the drift. None of that has to be real, but all of it has to be consistent with science to the extent that we can't say with certainty that it's NOT real. Fantasy, even if it's set in the real world (there's a sub-genre called "contemporary fantasy" that is; my own Star Mages trilogy falls into this category) can posit fantastic elements that have no foundation in science whatsoever and as long as the whole thing is logically consistent and lends itself to a fantasy-appropriate theme (see below), that's fine.
While this distinction is important, it's not the real, crucial, central difference between the two genres. That difference, in my opinion, lies in the set of themes appropriate to science fiction versus those appropriate to fantasy. These themes exist alongside of and in addition to the usual themes common to all literature (self-discovery, romantic love, all the personal struggles of life). A good science fiction or fantasy story will always deal with those universal themes in the course of character development, because that's true of all fiction regardless of genre. But in addition to that, a good science fiction story will always have a theme (or at least a subtext) that is political in nature, while a good fantasy story will always have a theme (or at least a subtext) that is religious in nature.
What do I mean by this? Well, science fiction, being set in the real world of a speculative future, must deal with the impact of changing technology on society and how future societies respond to those situations. For example, suppose that a science fiction story deals with the development of a perfect, unlimited, non-polluting energy source that, unfortunately, causes the deaths of a certain number of young children per year in an apparently random fashion. Will the society abandon this technology or make use of it? And suppose that a secret research project exists to discover just how the casualties are selected, so that they can be steered towards less "desirable" children (chosen by whatever -- race, class, intelligence, gender, predilection for social protest, religious belief)? There are moral issues here, of course, and the story may go into those in the course of character and plot development, but the main theme or subtext is always how society adapts to these changes, whether it does so in a positive or negative fashion.
Fantasy is quite different. A fantasy deals with such religious themes as a person's relationship with the gods or God or the cosmos, moral dilemmas, the expansion of consciousness, and the development of personal higher powers. For example, and somewhat in parallel to the hypothetical above, a fantasy story might deal with an item from an ancient tomb discovered by an occult-interested explorer, through which he contacts an ancient god who offers him staggering magical powers, but he must sacrifice a young child each year in order to gain these powers. The explorer must make this choice himself. Does he use the item? Destroy it? Leave it buried in the tomb? If he uses the power, how does he choose the children to be sacrificed? And, most important of all, what does this whole business do to his soul, to who he is as a person? There may be political issues involved in all this, but they are not major themes of the story.
It's been said (quite truly) that the Star Wars movies and novels are fantasy rather than science-fiction, despite their science-fiction trappings. The science in them is very bad, but the main reason they are fantasy is because they present a religious theme or themes (good versus evil, light versus darkness, personal choices, fall and redemption) and not primarily a political theme. Yes, the Galactic Republic is overthrown by the Empire and then restored at the end, but the real story is about what happens with the soul of Anakin Skywalker/Darth Vader, his fall to the Dark Side, and his redemption with the help of his son Luke.
Similarly, a fantasy story that deals primarily with the adjustment of a society to massive changes caused by an invasion of demons or a change in natural law is being written more like a science fiction story than a fantasy.
There are some examples that blend the two. For example, S.M. Stirling's wonderful "Change" series that begins with Dies the Fire has a lot in it about how human society changes and adjusts to the loss of all significant technology and the emergence of magic into the world, and yet it also has a lot about the personal journey, development, and moral evolution of the main characters. That's how it is with fiction and indeed with all art: there are no ironclad, absolute rules. But this one is going to be true most of the time.