I notice that when watching videos not from the 10ish vids recommended to me, the video buffers alot, even on a 10 sec short.

Many ISPs run a Google cache server which caches a lot of Google-specific data local to your internet provider. This would mean that there's a much shorter path between your home connection and the cached data, but non-cached data would be retrieved externally. I wouldn't be surprised if popular YouTube videos are part of this. Here's some more info: https://support.google.com/interconnect/answer/9058809?hl=en

Probably, but YouTube also compresses popular videos more aggressively, so you need to download less data to watch a second of video.

You want to read about "content delivery networks" (CDN).

https://en.wikipedia.org/wiki/Content_delivery_network

I know netflix does, I would assume that youtube does the same.