I used a Nexus One as my smartphone for almost three years, pretty much since it was released in 2010. That's a pretty good testimonial. The N1 wasn't perfect--it was arguably underpowered even at release, and held back for upgrades by the pokey video chip and small memory--but it was good enough. When Google announced the Nexus Four, it was the first time I really felt like it was worth upgrading, and I've been using one for the last couple of months.
One big change, pardon the pun, is just the size of the thing: although it's thinner, the N4 is almost half an inch wider and taller than my old phone (the screen is a full diagonal inch larger). The N1 had a pleasant density, while between the size and the glass backing, the N4 feels less secure in your hand, and at first it doesn't seem like you're getting much for the extra size. Then I went back to use the N1 for something, and the virtual keyboard keys looked as small as kitten teeth. I'm (tentatively) a fan now. Battery life is also better than the N1, although I had to turn wifi on to stop Locale from keeping the phone awake constantly.
I think it's a shame they ditched the trackball after the Nexus One, too. Every time I need to move the cursor just a little bit, pick a small link on a non-mobile web page, or play a game that uses software buttons, I really miss that trackball. Reviewers made fun of it, but it was regularly useful (and with Cyanogen, it doubled as a second power button).
The more significant shift, honestly, was probably going from Android 2.3 to 4.2. For the most part, it's better where Android was already good: notifications are richer, switching tasks is more convenient, and most of the built-in applications are less awful (the POP e-mail client is still a disaster). Being able to run Chrome is usually very nice. Maps in particularly really benefits from a more powerful GPU. Running old Android apps can be a little clunky, but I mostly notice that in K-9 Mail (which was not a UX home run to begin with). The only software feature that I do really miss is real USB hosting--you can still get to internal storage, but it mounts as a multimedia device instead of a disk, which means that you can't reliably run computer applications from the phone drive.
There is always a lot of hullaballoo online around Android upgrades, since many phones don't get them. But my experience has been that most of it doesn't really matter. Most of my usage falls into a few simple categories, none of which were held back by Android 2.3:
Compared to its competitors, Android was always been designed to be standalone. It doesn't rely on a desktop program like iTunes to synchronize files, and it doesn't really live in a strong ecosystem the way that Windows Phone does--you don't actually need a Google Account to use one. It's the only mainstream mobile platform where installing applications from a third-party is both allowed and relatively easy, and where files and data can transfer easily between applications in a workflow. Between the bigger phone size (or tablets) and support for keyboards/mice, there's the possibility that you could do real work on a Nexus 4, for certain definitions of "real work." I think it would still drive me crazy to use it full-time. But it's gradually becoming a viable platform (and one that leaves ChromeOS in kind of an awkward place).
So sure, the Nexus 4 is a great smartphone. For the asking price ($300) it's a real value. But where things get interesting is that Android phones that aren't quite as high-powered or premium-branded (but still run the same applications and OS, and are still easily as powerful as laptops from only a few years ago) are available for a lot less money. This was always the theory behind Nokia's smartphones: cheap but powerful devices that could be "computers" for the developing world. Unfortunately, Symbian was never going to be hackable by people in those countries, and then Nokia started to fall apart. In the meantime, Android has a real shot at doing what S60 wanted to do, and with a pretty good (and still evolving) open toolkit for its users. I still think that's a goal worth targeting.