Arezzo's historic old town is small enough to explore on foot. If you have a rental car, you can park in one of the municipal lots for under €10 for the entire day, then walk up into the historic centre. WARNING: Arezzo is atop a steep incline, and you will feel as though you are walking uphill pretty much everywhere. Wear comfortable shoes.
ATAM (http://www.atamarezzo.it/) runs the city bus service there is also a "Centro Storico" line that covers the historic old town.
The taxi service is efficient and not too expensive.