Bungie posted their follow-up to the bombshell that Halo and Halo 2 output HD video signals when played on the Xbox 360. This Q/A was included:
How is the resolution change achieved technically?
For Halo and Halo 2, it happens in the “frame buffer� where Full Scene Anti Aliasing is applied. That image is then scaled for the best mix of image quality and compatibility.
I aim to show that, while this information is (unfortunately) expected, it means that you will theoretically NOT want to use the 720p output signal! Based on the results of a subjective, yet-to-be-run image-comparison experiment, we may be able to move this assertion from theory to fact.
First of all, what’s actually happening inside the Xbox 360 with Halo 2? When Halo 2 is running, the original Xbox creates the current image, or frame, based on the internal state of the game world, and the direction your camera is pointed (aimed out the eyes of the Master Chief). Once each frame of the video signal is created, it is stored in the “frame buffer”; a buffer is a temporary storage location in memory, this one intended for storing finished frames.
Next, the image is written from the frame buffer to the screen. Remember, the original image is 576 x 480 pixels - the standard dimensions of a NTSC TV image. High Definition TVs, however, have far more than 576 x 480 pixels: 720p signals require 1280 x 720 pixels. Therefore, to display this lower quality image, a HD TV upscales the image to fill up the space. Imagine taking a four-by-four pixel image and expanding it to fill an eight-by-eight screen - each original pixel would be blown up to be two pixels square. No additional data has been added; the image has simply been expanded to fit in the screen space. If a HD TV couldn’t do this, the image would be displayed in its original size in the middle of the screen, surrounded by black borders.
A game outputs the highest resolution it can, based on the the speed of its underlying program. For Halo 2, native 720p was originally planned, but was not possible due to it lowering the frame rate to an undesirable level. 480p thus became the maximum video output, regardless of what you did to the video signal once it left the Xbox. Once data hits the frame buffer, it is “finished” and ready for display. Any further image processing from here is necessarily defined as “lossy”, which means that you are losing some of the orignal data.
As a side note, upscaling is considered lossy if the frame is being scaled to a resolution that isn’t bigger by a whole integer factor, such as two times bigger, three times bigger, etc. Thus, the next truly lossless upscaling of a 576 x 480 pixel image should be to 1152 x 960. Do you see how this doesn’t fit in a screen that only goes up to 720p resolution = 1280 x 720? Some of the pixels in this scaling will have to be combined with their neighbors, in order to fill the entire screen. Combining pixels means that you are trying to get one pixel to hold the data that two held before. Imagine one was black, and the other white. Is the new pixel black, white, or grey? Data is thus lost. This upscaling is done by your TV - it is throwing away data to expand the signal, if it can’t natively support the upscaled resolution. Generally, we accept this loss, because the alternative is displaying the video on only a small section of our HD TVs. Upscaling algorithms also do a pretty good job in this conversion, so the loss is typically very small. To summarize upscaling, note that no new data is added, and that data is usually lost. Here’s an example:
Now, with Halo 2 loaded and outputting 720p, the Xbox 360 performs a lossy editing routine on the image before it leaves the machine. Upscaling is done in the frame buffer (after the image is stored there) to get the frame into 720p, so that it won’t have to be upscaled by the TV. Fine and dandy, right? The signal has to be upscaled somewhere, after all. However, before the signal is upscaled, the 360 runs an anti-aliasing pass over it (I discussed anti-aliasing in my previous post). In the same way that combining pixels during upscaling loses data, smoothing out stair-step artifacts due to diagonal lines loses data, as well. Essentially, features that are only a few pixels in size can be effectively smoothed into nothingness by anti-aliasing. Look at one of the comparison pictures. Sure, the curves are rounder, but they’re rounder because the jaggy data around the edges has been altered. It actually looks very slightly bleary - the entire scene appears softened. This is a known side effect of anti-aliasing.
What is especially confusing to me is why they are running the anti-aliasing pass before they upscale. Imagine a 3 x 3 grid, with a diagonal line from the lower left corner to the upper right. According to the anti-aliasing algorithm I’m using, pixels above and below this line will be altered, which means that all of the pixels are changed. In a 6 x 6 grid, the line only affects ~67% of the pixels. Here’s an example, showing the difference between upscaling versus anti-aliasing first:
The point is that anti-aliasing before the upscale means that you are editing a greater percentage of the frame, and the new jaggies that result from the upscale (jaggies are always born when you upscale) will be untouched. If the image had been anti-aliased after the upscale, then the jaggies that result from the upscale would have been cleaned up. Now, one possible reason Bungie might not have done this is because the duration of anti-alising algorithms is based on the number of pixels being edited. Increase the number, and you slow down the frame rate. However, I do not believe that was a problem here. Upscaling is rather fast, and I believe there was certinaly power to spare in the Xbox 360 to upscale first, and then anti-alias.
In addition, remember that there is no new data being created. This was confirmed by Bungie when they said the image was upscaled, above. If you take nothing away from this article other than the following, I will be happy: There is a difference between outputting native 720p, and upscaling a lower quality signal to fill a 720p screen. As stated, however, this is not unexpected in the case of emulation by the Xbox 360. Emulated games that can’t output in HD still won’t be able to, and no emulation profile is going to change the code to enable them to. The emulation profiles “fake” the game into thinking it’s being run on different hardware than it really is, but the game is not being changed at all. No bug fixes or additional game content are possible.
So here is our conumdrum. The Xbox 360’s video signal would have been exactly the same on my giant HDTV whether upscaled by the 360, or my TV. The size of the picture is the same. However, because of the anti-aliasing, the image will be (theoretically) slightly degraded when output in 720p mode. Thus, I claim that you will receive slightly more jaggy, but more detailed images if you use standard 480p mode. I recommend not setting your Xbox 360 to 720p mode for Halo 2.
Is this actually true in practice, though? Certainly, some amount of bleariness accompanies all anti-aliasing, but will the anti-aliasing actually be so severe that we can no longer discern some small objects? To measure this, we need to run an experiment. We need to capture video coming from the Xbox 360 in 720p mode, and video leaving my HDTV in 720p that originally came from an Xbox 360’s 480p widescreen signal. We’ll take still shots, and examine subtle and/or small features to see if they remain discernable. If they are, then Bungie has tuned the AA correctly. If not, then we’ll have another reason to look forward to Halo 3: true 720p video.
Footnote: At least one other game being readied for the Xbox 360 uses upscaled HD trickery. It’s not true 720p, folks. However, this is HD - High Def is anything over 576 x 480, in progressive scan mode. It’s still gonna be the prettiest racing game ever.
