It really depends on the size of the data set and the complexity of the classification problem. Generally, the more data you have, the more accurate your model will be, so it's important to use enough data to capture the full complexity of the problem.
If you are dealing with a more complex problem, such as identifying objects in an image, you may want to use a larger data set to ensure that you capture all the features of the objects.
An example might be if you need to classify between different types of animals – you may want a larger dataset to ensure that all the different features of each animal are captured.
When it comes to numbers, if it's a simple test - we've seen great results with even 50 images per category.
More robust projects can easily go over 1K images per category.