But this annoying "You May Know" section would keep suggesting personal contacts to me, heck, I'm not going to go through my extended contact list (which google has, from my phone contacts) and add everyone to my G+ circles. That is madness, and there is no setting to make it go away.
Anyways, I finally had enough today, and found a lot of people were recommending Adblock Plus, which seems to be the easiest way to get rid of this unwanted feature. It's quite simple, once Adblock Plus is installed, go to its options tab -> Add your own filters, and add following filter:
plus.google.com##div[guidedhelpid="friendsuggestions"]
Then you are good! The damned "You may know" box is gone from google plus... for now.