When Aristotle first looked at the heavens, he philosophized that the wandering planets and all the stars must be perfect spheres hung in the heavens. With the advent of the telescope, Galileo saw that the moon isn't smooth and perfect at all; it is a textured ball of craters and mountains. Over the succeeding 400 years, telescope observations of the Solar System have revealed that the planets posses complex surfaces with a mix of volcanoes, graben, mountains, and impact craters giving variety to their imperfect surfaces.
The majority of features on worlds other than Earth are due to volcanism and cratering, but there are a selection of features that are related instead to motions of and energy conducted through the crusts of planets and moons. On the Moon and Mercury, for instance, long linear features called scarps are seen. These features are formed as rocky worlds cool and shrink. Like a tailored suit on a dieting person, the crust starts our fitting snuggly over the world's molten layers, but as the core cools and shrinks, the crust ends up too big. Without support, it slumps, creating long features where essentially a fold forms in the crust.
Here on the Earth, our still molten core is in constant motion beneath the solid floating crust, which itself is fragmenting and moving. At the edges of these plates the mountains and rifts form. Mountains and hills are typically (but not always) the result of tectonic plates interacting — technically called faulting — along shared boundaries. Up heaves in the crust that form hills and mountains often result from collisions, but plates moving side by side can experience drag folding, where one plate will fold up as it fails to smoothly flow past another.
The largest mountains are consistently formed when plates collide and fold. This can occur either when and ocean plate collides with (and typically subducts beneath) a continental plate (e.g. the Rockies and Andes), or when two continental plates collide (e.g. the Himalayas). In some cases, an uplifted plate may crack or fault-block, such that there are lower areas called graben and raised block mountains. The Sierra Nevada mountains are one example of a fault-block mountain range.
The heights of mountain ranges both reflect the duration of a collision and how recently it occurred. The Himalayan mountains only began to form 70 million years ago while the U.S. Rocky Mountains are at their youngest 100 to 65 million years old, with much of the range being older (formation began 1.8 billion years ago). Young mountains are still uplifting and may have clearly defined exposed crustal folds. Older mountains smaller because of erosion. Both wind and water wear at mountains over time, wearing them down like the sharp edges on a beach rock.
The height of mountains is primarily limited by gravity. If a mountain 10 times the height of Everest were to appear on Earth, the pressure due to gravity at its base would melt the rock. The lithosphere would gradually slump, until the load was supported by buoyancy. As a result, no mountain on Earth is higher than about 10 kilometers (or about 6 miles). The Earth's surface is as smooth as a billiard ball, relative to its size. Mars is smaller than the Earth, so its gravity is weaker, and it has cooled more, so its lithosphere is thicker. These factors contribute to the great height of Martian mountains — Olympus Mons rises 27 kilometers (almost 17 miles) above the average elevation of the planet!
It should be noted that not all hills result from tectonic activity. In some cases they are formed by deposits left by melting glaciers, via the build up of deposits from the wind (aeolian processes), and through the erosion of land cut through by water (eroded fluvial processes). Mountains can also be volcanic in origin. These processes are all discussed in other articles.