I've been through this same cycle and thought process, the smaller frame is hurting your off road ability. The bmxer in you wants a tight feeling bike but for true MTB rocks and steeps you need a big enough bike to move around on.
As for wheel size, I love 29" on a hardtail. 27.5" is just marketing and is effectively the new 26". Unless you're on your game already, moving from 26" to 27.5" with all else constant won't change much of anything for you.
Find a complete for sure. $1500-$2000 will get you a nice new bike like a Kona Honzo or Explosif. You can probably still find a leftover 2014 of most brands for a discount. A slacker HT angle and lower BB will also help you feel better in the rough stuff.
Good luck!
I got into DJ mtb's in 2006, then singlespeed hardtail, then various borrowed and rented XC and AM bikes (FS and hardtail). Also have a Transition Double 4x/slope with 1X9.
I'm not sure I understand what you're saying by "For true MTB rocks and steeps you need a big enough bike to move around on."
If anything, I find that the many people in the mtb world are STILL stuck in the dirt roadie mentality. When you are stretched out over a long frame with a long stem, 29 wheels, a high seat and ultra low bb, that is the most dirt roadie position you can be in. And is NOT a help for technical riding in the sense of getting over rocks and definitely not on steep descents. The only thing it is great for is climbing, which many dirt roadies are obsessed with.
I think 29"s are sluggish. And I think 27.5 is a great compromise.
I rode two 27.5" demo bikes at Winter Park this past summer. The 27.5" DH bike felt big/awkward/non-nimble. But the 27.5" Specialized Enduro Carbon felt great.
i crank around regular xc trails on a 4x/slope bike pretty regularly (Transition Double with 1X9 gearing), and i keep up with geared-up 29'er dudes pretty well. i realize that full leg extension setup does make for more efficient riding... but i also don't like giving up some of the 4x/bmx style handling for hitting the berms rollers jumps that i have built out here, mixed in to the regular mtb trails. or even just the feeling of being able to bunnyhop a log in the way that i'm used to etc. etc. i may actually buy a true xc/am mtb soon, but i will definitely ride a size a little smaller and set it up a little more bmx/DJ-ish. probably with a 60-70mm stem, bmx platform pedals, etc.
29er tires typically measure ~29.25" tall inflated. "26" tires typically measure ~26.25" tall inflated.
650b tires ("27.5" is absolutely a marketing term and nothing more) typically measure ~27"-27.25" tall when inflated. The wheel size change is minimal and makes very little difference when riding, most of the "improvements" on the new breed of 650b enduro bikes are due to changes in geometry, setup (including wide bars + short stem), and refined suspension technology.
For me, actually riding a MTB in the woods is 50% climbing and 50% descending. For that, I need a bike with enough room to get my weight forward while climbing, and then be stable while descending. Coming from bmx I used to think that a short TT was good so I can throw the bike around but that seriously hurt my MTB riding. I hated every Medium sized frame I owned due to the cramped climbing geometry, and now that I am on Larges my riding has improved as well.
DH racers, who ride 100% downhill at mach chicken speed, are even going as long as possible on their top tubes. The school of thought is progressing and geometry is changing to reflect that.
This is all my 2 cents and should not be taken as anything more than that.