You need at least three letters of recommendation, but four strong letters herald a high quality candidate. One of the letters you submit must be a letter from the chair of neurosurgery at your home institution. The absence of this will create speculation about whether there is a reason that your chair is not supportive of your candidacy for neurosurgery. If you cannot obtain one from your home chair, a chair at a program where you did an away rotation may mitigate this. Your other two letters of recommendation should be from people that know you well and would be willing to go to bat for you. If you have done any research, one of the letters could be from your research advisor. There is no doubt it is of great help if the people writing your letters are well-known, but a lukewarm letter from a chair does not stand up to a strong letter from a lesser known neurosurgeon. Strive to get three letters from neurosurgeons and a fourth from another neurosurgeon or research advisor.
In general, letters from non-neurosurgeons (general surgeons, pediatricians, OB-GYNs, etc.), even if they are very strong, are of little benefit as these specialists are not generally able to understand the unique rigors of neurosurgical residency. It is advisable to do one to two away rotations early in the application process (summer), and get letters from these institutions. A strong chair letter from an away rotation is a considerable addition to your application. You should strive for a letter from every program at which you did a sub-internship. Though they may not all be available at the time the application process opens, you must have at least three letters to be highly considered.