When Lord Chaitanya says that one should avoid the non-sadhus, he means persons who don't follow basic principles of religious life. For example, every Krishna conscious devotee follows the four rules, but the non-sadhus always indulge in illicit sex, meat-eating, intoxication, and gambling.

If a devotee begins to intensively associate with non-sadhus, he will eventually pick up their habits, despite all his knowledge and training.

As stated in the Hari-bhakti-sudhodaya, "Association is very important. It acts just like a crystal stone, which will reflect anything put before it" (The Nectar of Devotion, p. 106).

Those who are on the path of salvation for going back home back to Godhead, are especially advised by all scriptural instruction to become free from all paraphernalia of material attraction. And that is possible only by the association of the devotees of the Lord.