What to do? I suggest that you invest in one of these.
It's a little lint shaver. The same model used to be sold by Dritz, the sewing supply company. It looks as though Knitpicks may have bought the rights to it, because that's the only source for these that I can find these days. I like the little ones, not the great honking big ones that I've occasionally come across. I have two, in case one breaks, and I keep a supply of AA batteries to keep them running. These little gizmos do a fantastic job of refreshing pilled knits. After the first few cleanups, the garment won't shed very much. Keep that in mind.
And, oh yeah, don't do what I did once, and use a lint shaver on cotton. The fibres aren't meant to be knicked off like wool. I destroyed a baby sweater once that way, and had to re-knit the whole thing, which was intended as a gift. Live and learn, as they say.