The temperature-dependent thermal conductivity (κ) variation has been actively investigated as a thermal rectification mechanism. However, the intrinsic κ modulation of solid materials is limited. Here we report a shape memory polymer composite (SMPC) by combining the shape-changing ability of cross-linked poly(ethylene-co-vinyl-acetate) with the electrical/thermal functions of silver flakes. The matrix polymer volume change, in contrast to the constant silver flake volume, significantly modulates the volumetric filler fraction during the programming–recovery process. This results in a large change in electrical conductivity (210%) and κ (88%) between 20 and 100 °C, which is employed as a novel thermal rectification mechanism. Only a quarter of the SMPC is partially programmed to realize asymmetry in thermomechanical response. The thermal rectification efficiency (TR) increases as the heat transfer rate and temperature difference across the specimen (ΔT) increase, reaching 38.3% at 0.15 W and 73.4% at ΔT = 45 °C. The SMPC exhibits the highest TR at a significantly smaller ΔT compared with the data in the literature.