The difference between pads for aluminium rims and carbon rims isn't obvious. The ones for alu are made of rubber whilst the ones for carbon rims are made from organic material such as wood or cork. Rubber can melt and since carbon is such a poor conductor of heat, it does melt and then smear rubber on the braking surface, making for grabby and grubby brakes. If you brake and the pads melt - you don't have to brake much for that to happen - the pads effectively glide on liquid rubber and that of course means zero braking. Carbon pads are made from stuff that doesn't melt - think wood. It heats up but skips the liquid stage and goes directly to gas. This works better.
For a short ride and knowing that you could melt the pads and therefore modify your braking, there is no harm in using carbon wheels with rubber pads, other than the aforementioned metal bits inside most rubber pads used on aluminum. Koosltop pads don't pick up metal so if yours are Koolstop, ride away but watch out or melt.