I never played this game as a kid, and I don't think I'd get very far now..but the soundtrack contains some gems by composer Grant Kirkhope (I also reviewed one of his other OSTs, Goldeneye 007).
The tracks here are filled with all kinds of fun little melodies: different kinds of instruments like kazoo (Banjo Kazooie) and banjo bring a bright and upbeat vibe.
There are plenty of covers online, where people go all-out like this "Spiral Mountain" cover by PPF.
Is it worth playing the game just for the soundtrack? That kind of depends on whether you have the patience to hunt down like a million collectibles every level.