When a rubber band is stretched it causes its polymer chains to become very ordered and it expels thermal energy (heat), thus shortening its life. This is increased further when placed in higher temperature environments which increases the oxidisation rate.
Also, the natural rubber that is used to make rubber bands crystallises over time, giving us what is commonly called ‘dry-rot’ – which is where the bands get dry, crumble and no longer have any elasticity.
When a rubber band is in its relaxed state it is very unordered, and will cool when going from a structured state to a relaxed state. So when a rubber band is put in the refrigerator it makes the polymers even more relaxed due to the way they behave in the cold.
Something to be careful of is immediately stretching the rubber band after being in the fridge, as this will cause it to weaken significantly because of the rapid change of temperature state.