Windows used to secretly use green screens to render videos, which is how you could trick MS Paint into becoming a video player

If you opened up the ol’ Windows Media player back in the 95, 98 or XP days, brace yourself for a mild shock: it was lying to you.

And by lying, well, what I really mean is rendering video somewhere other than inside the actual window that was open on your desktop—sort of a parallel plane of existence to the desktop you were actually looking at—before sneakily porting it over.

If you’ve watched a single Twitch stream or Marvel movie in the past 15 years, you’re probably all too familiar with the concept of a green screen, a bright, single-color surface that makes it easy to blank out a background and fill it in with something else. A Hawaiian beach, a bunch of CGI superheroes punching each other, etc. The more accurate color-agnostic term for the technique is chroma-keying, which is actually how Windows rendered all of its video back in the old days.

“The media player program didn’t render the video pixels to the screen,” longtime Microsoft developer Raymond Chen wrote recently in a blog post. Instead, Windows would render a green screen (or a different color, depending on the version), then “render the video pixels to a graphics surface shared with the graphics card.” The final step was to “tell the graphics card that whenever it sees a green pixel about to be written to the screen, it should substitute a pixel from that shared graphics surface.”

The graphics surfaces were usually referred to as overlays because they would (invisibly) overlay the desktop (that’ll come into play shortly). Chen said that the technique provided a couple advantages: avoiding pixel format conversions if the video format differed from that of the user’s monitor, and keeping performance steady by decoupling that graphics surface from the process thread that governed the UI. So even if the Windows shell were to hang for a second, the video could keep playing smoothly.

A more advanced technique, called flipping, would play up this performance advantage with two shared graphics surfaces, one with the “current” video frame and one with the “next” video frame, which the video card would switch between during very vertical blank (as the screen refreshed).

This old video playback technique isn’t very interesting until we arrive at how you could either intentionally or inadvertently trip up how Windows was using chroma-keying, as Chen explained:

“When you took a screen shot, you got the pixels that Windows gave to the video card as the contents of the desktop. If an overlay is active, then these are not the same pixels that came out of the video card and sent to your monitor. The computer never sees these monitor pixels; they are something generated on the fly by the graphics card and sent directly to the monitor. Your screen shot was a screen shot of the desktop screen, and it contains green pixels where the video would go.

“Now, when you load the image into Paint or any other image viewer, Windows sends those green pixels to the video card, but if the media player is still running, then its overlay is still active, and if you put Paint in the same place that the media player window is, then the green pixels in Paint get changed into the pixels of the active video. The video card doesn’t know that the pixels came from Paint. Its job is to look for green pixels in a certain region of the screen and change them into the pixels from the shared surface.

“If you move the Paint window to another position where it doesn’t overlap the media player, or if the media player isn’t playing a video, you will see the bitmap’s true nature: It’s just a bunch of green pixels.”

Rendering video used to be quite a demanding task for computers, but these days there’s no need for these sorts of techniques that allowed Windows to play a tiny 240p file mpeg at 60 frames per second. I know it’s for the better, but personally I’m a little sad that Microsoft Paint is smart enough now to do background removal and even save layers like Photoshop. Computers are just more fun when you can trick them.