We overview studies of the natural variability of past climate, as seen from available proxy information, and its attribution to deterministic or stochastic controls. Furthermore, we characterize this variability over the widest possible range of scales that the available information allows, and we try to connect the deterministic Milankovitch cycles with the Hurst–Kolmogorov (HK) stochastic dynamics. To this aim, we analyse two instrumental series of global temperature and eight proxy series with varying lengths from 2 thousand to 500 million years. In our analysis, we use a simple tool, the climacogram, which is the logarithmic plot of standard deviation versus time scale, and its slope can be used to identify the presence of HK dynamics. By superimposing the climacograms of the different series, we obtain an impressive overview of the variability for time scales spanning almost nine orders of magnitude—from 1 month to 50 million years. An overall climacogram slope of −0.08 supports the presence of HK dynamics with Hurst coefficient of at least 0.92. The orbital forcing (Milankovitch cycles) is also evident in the combined climacogram at time scales between 10 and 100 thousand years. While orbital forcing favours predictability at the scales it acts, the overview of climate variability at all scales suggests a big picture of irregular change and uncertainty of Earth’s climate.