Summary
Testing ever-more users in card sorting has diminishing returns, but you should still use three times more participants than you would in traditional usability tests.

Card Sorting: How Many Users to Test