Caps and hats are of no use during riding, as you need to be wearing a helmet, but after such ride a cup or a hat will definitely be useful, e.g. during stable works or while taking care of your steed.
In Equishop you can find caps with peaks, useful in summers to protect your eyes against the sunlight during lunging, or winter hats providing warmth and comfort, usually lined with fleece.