This is the famous quip by Einstein often quoted in the literature. By itself, taken out of context it is difficult to see its meaning. Actually this line was a part of a letter written by Einstein to Max Born to express his dissatisfaction with Quantum Theory. Put in proper context it reads as follows:

Quantum mechanics is very impressive. But an inner voice tells me that it is not yet the real thing. The theory produces a good deal but it hardly brings us closer to the secret of the Old One. I am at all events convinced that He does not play dice.

The last three words (put in parentheses in the quote) were not there but implied in the context. Old One was the nickname Einstein used for God. He disliked quantum theory for several reasons, one was its probabilistic nature. In any experiment with several possible outcomes the theory can give only the probability of any particular outcome; it cannot predict the outcome. When you throw a dice there is no way of telling what number it will come up with. All one can have is the probability of a certain number showing. Einstein on the other hand firmly believed that any correct and complete theory should be able to predict the outcome with certainty. The reference to the universe comes up because of the question Why the universe is the way it is? Was it just a probability showing up? If subatomic world is the basis for matter (and hence, for the universe), quantum theory says yes. That is what irritated Einstein. According to him in any physical situation given the same conditions the outcome should always be the same. Not so in quantum theory. No matter how complete our knowledge of a subatomic system, it is not possible to predict with certainty what the results of a measurement will be. Uncertainty is an essential feature of the quantum world.

Another feature of quantum theory that Einstein was vehemently opposed to was action at a distance, which he called spooky. Formally it is called non-locality in physics and implies that communication between two objects (or persons) can occur without any medium or means of communication. The notion of locality had been so firmly entrenched in physics that it was unthinkable to consider the opposite. Relativity theory stated that nothing could travel faster than light and there cannot be any instantaneous communication. Non-locality meant that information was transmitted from one object to the other instantaneously in some mysterious manner.

Einstein, therefore, maintained till the very end of his life that quantum theory was incomplete. For several years he had an ongoing debate with Neils Bohr each trying to prove the other wrong. Finally he devised the famous EPR thought experiment to prove his point, especially about non-locality. Subsequent investigations based on the thought experiment (theoretical as well as experimental) have proved him wrong. Luckily he never knew. It all happened long after his death.