To twiddle one's thumbs

I spent the afternoon twiddling my thumbs, waiting for the phone to ring.

There wasn't much to do, so we just twiddled our thumbs.