That IS where it starts. My Jeepster Commando is so pumped up that there are few fun places near home anymore, but I am only 2 hours from the Rubicon Trail.
Second to a proper safety harness, differentials should be the first modification for an offroader, but they can be expensive, and normal lockers are dangerous on the highway. A stock 1972 Wagoneer with factory limited slips on both ends can do amazing things. I grew up with offroading and Baja racing, and was surprised to find that most tractors have open diffs with mechanical locks. Open is easier for your soil, so that must be why