The obvious place to visit in Saumur is the château - and it amply repays a visit.
In particular, there are some good views of the river Loire and the town. You will also be able to spend time viewing the museum collections in the château: one all about the world-famous Cadre Noire (a training school for the crack cavalry corps) and a fine collection of objets d'art relating to the history of the castle - including an extensive pottery collection.
If you time it right, you can take a tour of the castle in English. You'll hear about how it began as an Abbey but Folques Nerra, Count of Anjou, wanted the site for a small fortress. It became a significant military centre for another Count of Anjou, Henry II Plantagenet, King of England, who extended it. It was later a key part of the protection of the Huguenots, the French Protestants, and also became a prison where English soldiers where kept during the American War of Independence.