dual sports are a study in compromise.
They can do all things proficiently but few things excellently so it comes down to what you want to do.
I agree that starting small and cheap is best, don't do like me and get a big heavy $15,000 BMW with 90hp, it's nowhere nimble enough to really enjoy offroad and simply heartbreaking when it falls over. And yes, dual sporting will fall over. It's part of the fun.
Forget about high speed and quick acceleration, they have no place in real dual sporting. The ability to go anywhere and do anything is usually executed in places where trying to go fast leads to disaster.
What you want is simple & reliable. Easy to service when it falls over. Comfortable enough to spend days in the saddle - reliable enough to push boundaries without fear of catastrophic failure. That rules out exotic bikes in my opinion. I wouldn't trust most KTM & Aprilia bikes (beautiful as they are) to get me around the block, let alone through the wilderness.
If long distance isn't a real concern I like the CRF230L because it's very light and has a bulletproof motor. Throw on some bags and you're ready to roll nearly anywhere that doesn't require you to exceed 60mph.
Bigger bikes I like, the KLR - XR650 (must buy the larger capacity fuel tank) - the DRZ / KLX. You can pick up used ones for a song.
You're not alone in feeling the allure. Dual sporting prettymuch killed my interest in sportbikes for anything but the track, and even there I lean toward supermoto now. Personally I have no use for a gofast bike anymore, I've found the other world of riding and it's a whole different challenge.