I've never played any kind of handheld, but I'm really curious about these devices as devices.

What's a low cost device that is fun to play but also fun to play with?

PSP

If you enjoy tinkering, in the past I built a raspberry pi handheld with a 3d printed case that was fun. However, really nice handhelds that emulate everything are pretty cheap these days, and there are several. This one looks pretty good, but I have no experience with it.

How locked down is it? Is that PSP + an open OS?

I’m a fan of any of theAnbernic devices that have a Linux OS on them. Some of them are stupid cheap, and the Mali gpu will give you a good bang for the buck.

The PSP is over 20 years old, I think not a single line of code of it's os and not a single transistor on it's boards hasn't been turned over by the community. It's as cracked open as it can be and especially nowadays fairly hard to brick. They're also very cheap, but the batteries are mostly trash by now.

It has some minding blowingly amazing games that make you question the current hardware requirements of modern realeses.

Highly recommend checking out:

• Monster Hunter Freedom Unite
• Metal Gear Solid Peace Walker
• GTA: Vice City Stories
• Patapon 1, 2
• Ace Combat X: Skies of Deception

Especially the Monster Hunter series had a blooming modding community on the PSP. There are some native ports of PS1 games on the PSP but some also made some homebrew ports.

OP asked about handhelds, not attracting cats.

Freedom Unite my beloved.

A friend hacked his PSP and let me borrow it for a week to try out "this new Monster Hunter game he found" when I was and it single-handedly made me go out and buy a used PSP to do the same. I put so many hundreds of hours into that game, even soloed the guild double-Tigrex mission (scaled for a party of players).

There's a few emulators for older systems that can run on PSP hardware, too -- it's how I first played the NES Castlevania and Megaman games. Looks like it can do up to about the SNES/Genesis era fairly well?

There's an app you can run that cracks the firmware for you, but it doesn't permanently change anything, so there's no chance it'll brick your PSP. It's super easy to turn the crack on and off, too. With the firmware cracked, you can just plug in an SD card and run any native game, any PS1 (as an eboot), and homebrew software which includes games and emulators for GB, GBA, NES, GG, SMS, Genesis, SNES, TG-16, etc. etc. Just a mindbogglingly huge library.

I don't remember the name of it right now, but do a web search for PSP CFWs (Cracked FirmWare) and you should be able to find something.

I mad at myself, that I didn't think of that.