Semiconductor quantum dots (QDs) have received a lot of attention because of the possibility of exciting two or more electrons over the band gap after absorption of a single photon via a process known as Carrier Multiplication (CM). By now CM is well known for QDs, especially for the workhorse material PbSe. Surprisingly the rate of CM and the details of the mechanism by which it occurs are still unknown. The common picture is that CM occurs via impact ionization of excited carriers well above the band edges, and that the efficiency is determined via the competition with hot carrier cooling. It is therefore important to understand the details of hot carrier cooling at energies above the threshold of CM.  For this reason we undertook a detailed study of CM and hot carrier cooling using hyperspectral transient absorption spectroscopy and uncovered a surprisingly rich cooling landscape. First we investigated optical transitions at short wavelengths (350-450 nm). After carefully assigning these transitions to asymmetric transitions of electrons and holes to higher bands at the L point in the Brillouin zone (the so called L­₄₆ and L₅₇ transitions), we use the kinetics of the transient absorption of these transitions to distinguish electron and hole cooling. Surprisingly, we find that hole cooling is much faster than electron cooling, in contrast to the common opinion that electron and hole cooling occur with similar rates. Next, we investigate in detail how the electron and hole cooling rates vary as the photon energy is increased, creating a cooling spectrum of both charge carriers. We find constant and clearly distinguishable cooling rates for the 1P-1S, 1D-1P and the Sigma point to 1D intraband cooling transitions for both electrons and holes. These rates are likely governed by energy transfer to ligand vibrational modes. At higher photon energy the cooling rate changes more continuously with energy loss rates that are in line with optical phonon emission. Finally, CM is investigated in PbSe and PbS QDs over a very wide range of sizes. We demonstrate that the threshold of CM follows the onset of the asymmetric L­₄₆ and L₅₇ transitions mentioned above. This can be understood by considering that via those asymmetric transitions all excess energy is stored in a single charge carrier, which consequently has a much higher carrier multiplication rate. This shows that the details of the band structure are very important for CM.