While pepper may not be considered to be toxic or poisonous to cats, it is certainly not a type of food that most of them enjoy due to the fact that it has such a pungent smell and taste. A lot of cat repellent products actually use pepper, so you will find that a vast majority of them do not like the taste of it.
If your cat enjoys the taste of pepper however, you will need to make sure that you give it to them in limited amounts. Pepper is not part of their natural diet and you should therefore not give it to them all the time.