So I took out my serpentine belt, sprayed some silicone spray on it. Felt the pulleys, all seem fine. The idler pulley seems to rotate freely, but no noise or sand like feel in rotating.
Then I checked the power steering fluid. It was pretty dark. So I replaced the fluid in the reservoir and did another replace one more time. Was running low, so I didn't redo it.
I'm wondering if this isn't the cause for the slight whine noise, so I might buy more and keep replacing until it runs clear.