Bungie posted their follow-up to the bombshell that Halo and Halo 2 output HD video signals when played on the Xbox 360. This Q/A was included:

How is the resolution change achieved technically?

For Halo and Halo 2, it happens in the “frame buffer� where Full Scene Anti Aliasing is applied. That image is then scaled for the best mix of image quality and compatibility.

I aim to show that, while this information is (unfortunately) expected, it means that you will theoretically NOT want to use the 720p output signal! Based on the results of a subjective, yet-to-be-run image-comparison experiment, we may be able to move this assertion from theory to fact.

First of all, what’s actually happening inside the Xbox 360 with Halo 2? When Halo 2 is running, the original Xbox creates the current image, or frame, based on the internal state of the game world, and the direction your camera is pointed (aimed out the eyes of the Master Chief). Once each frame of the video signal is created, it is stored in the “frame buffer”; a buffer is a temporary storage location in memory, this one intended for storing finished frames.

Next, the image is written from the frame buffer to the screen. Remember, the original image is 576 x 480 pixels - the standard dimensions of a NTSC TV image. High Definition TVs, however, have far more than 576 x 480 pixels: 720p signals require 1280 x 720 pixels. Therefore, to display this lower quality image, a HD TV upscales the image to fill up the space. Imagine taking a four-by-four pixel image and expanding it to fill an eight-by-eight screen - each original pixel would be blown up to be two pixels square. No additional data has been added; the image has simply been expanded to fit in the screen space. If a HD TV couldn’t do this, the image would be displayed in its original size in the middle of the screen, surrounded by black borders.

A game outputs the highest resolution it can, based on the the speed of its underlying program. For Halo 2, native 720p was originally planned, but was not possible due to it lowering the frame rate to an undesirable level. 480p thus became the maximum video output, regardless of what you did to the video signal once it left the Xbox. Once data hits the frame buffer, it is “finished” and ready for display. Any further image processing from here is necessarily defined as “lossy”, which means that you are losing some of the orignal data.

As a side note, upscaling is considered lossy if the frame is being scaled to a resolution that isn’t bigger by a whole integer factor, such as two times bigger, three times bigger, etc. Thus, the next truly lossless upscaling of a 576 x 480 pixel image should be to 1152 x 960. Do you see how this doesn’t fit in a screen that only goes up to 720p resolution = 1280 x 720? Some of the pixels in this scaling will have to be combined with their neighbors, in order to fill the entire screen. Combining pixels means that you are trying to get one pixel to hold the data that two held before. Imagine one was black, and the other white. Is the new pixel black, white, or grey? Data is thus lost. This upscaling is done by your TV - it is throwing away data to expand the signal, if it can’t natively support the upscaled resolution. Generally, we accept this loss, because the alternative is displaying the video on only a small section of our HD TVs. Upscaling algorithms also do a pretty good job in this conversion, so the loss is typically very small. To summarize upscaling, note that no new data is added, and that data is usually lost. Here’s an example:

Now, with Halo 2 loaded and outputting 720p, the Xbox 360 performs a lossy editing routine on the image before it leaves the machine. Upscaling is done in the frame buffer (after the image is stored there) to get the frame into 720p, so that it won’t have to be upscaled by the TV. Fine and dandy, right? The signal has to be upscaled somewhere, after all. However, before the signal is upscaled, the 360 runs an anti-aliasing pass over it (I discussed anti-aliasing in my previous post). In the same way that combining pixels during upscaling loses data, smoothing out stair-step artifacts due to diagonal lines loses data, as well. Essentially, features that are only a few pixels in size can be effectively smoothed into nothingness by anti-aliasing. Look at one of the comparison pictures. Sure, the curves are rounder, but they’re rounder because the jaggy data around the edges has been altered. It actually looks very slightly bleary - the entire scene appears softened. This is a known side effect of anti-aliasing.

What is especially confusing to me is why they are running the anti-aliasing pass before they upscale. Imagine a 3 x 3 grid, with a diagonal line from the lower left corner to the upper right. According to the anti-aliasing algorithm I’m using, pixels above and below this line will be altered, which means that all of the pixels are changed. In a 6 x 6 grid, the line only affects ~67% of the pixels. Here’s an example, showing the difference between upscaling versus anti-aliasing first:

The point is that anti-aliasing before the upscale means that you are editing a greater percentage of the frame, and the new jaggies that result from the upscale (jaggies are always born when you upscale) will be untouched. If the image had been anti-aliased after the upscale, then the jaggies that result from the upscale would have been cleaned up. Now, one possible reason Bungie might not have done this is because the duration of anti-alising algorithms is based on the number of pixels being edited. Increase the number, and you slow down the frame rate. However, I do not believe that was a problem here. Upscaling is rather fast, and I believe there was certinaly power to spare in the Xbox 360 to upscale first, and then anti-alias.

In addition, remember that there is no new data being created. This was confirmed by Bungie when they said the image was upscaled, above. If you take nothing away from this article other than the following, I will be happy: There is a difference between outputting native 720p, and upscaling a lower quality signal to fill a 720p screen. As stated, however, this is not unexpected in the case of emulation by the Xbox 360. Emulated games that can’t output in HD still won’t be able to, and no emulation profile is going to change the code to enable them to. The emulation profiles “fake” the game into thinking it’s being run on different hardware than it really is, but the game is not being changed at all. No bug fixes or additional game content are possible.

So here is our conumdrum. The Xbox 360’s video signal would have been exactly the same on my giant HDTV whether upscaled by the 360, or my TV. The size of the picture is the same. However, because of the anti-aliasing, the image will be (theoretically) slightly degraded when output in 720p mode. Thus, I claim that you will receive slightly more jaggy, but more detailed images if you use standard 480p mode. I recommend not setting your Xbox 360 to 720p mode for Halo 2.

Is this actually true in practice, though? Certainly, some amount of bleariness accompanies all anti-aliasing, but will the anti-aliasing actually be so severe that we can no longer discern some small objects? To measure this, we need to run an experiment. We need to capture video coming from the Xbox 360 in 720p mode, and video leaving my HDTV in 720p that originally came from an Xbox 360’s 480p widescreen signal. We’ll take still shots, and examine subtle and/or small features to see if they remain discernable. If they are, then Bungie has tuned the AA correctly. If not, then we’ll have another reason to look forward to Halo 3: true 720p video.

Footnote: At least one other game being readied for the Xbox 360 uses upscaled HD trickery. It’s not true 720p, folks. However, this is HD - High Def is anything over 576 x 480, in progressive scan mode. It’s still gonna be the prettiest racing game ever.

This idea came to me last night and I wanted to run it by some peeps to make sure I actually have something here.

Ok so picture this…

A game that will completely change the way people view FPSers and RPGs. Ok so it’s been said before but hear this one out, I think it has merit and only now that we’re moving into the next-gen era will it even be possible.

You have been given orders by a secret operations subcommittee of the US Senate to execute a black ops mission in a foreign country. Your assignment is to eliminate (read assassinate) a dangerous cadre of terrorist funders before they can organize and execute a massive attack on US soil. (Pick whatever interesting, exotic country you want at this point–I haven’t thought that far yet.)

To complete this mission you have access to the best of the best of the US government: CIA, NSA, Navy, Army, Marines, Air Force, Special Forces. You name it, you’ve got it. You must assemble and equip a team depending on what kind of approach you decide to use to take out your targets. Will you go in with a crack team of military special ops? Will you infiltrate the country with the best spooks the CIA has to offer? Will you follow the trail of crumbs as NSA agents and then send in the cavalry? It’s all up to you. Everything from team members, to weapons, gadgets, transportation, even what languages they speak and clothes they wear. It’s a completely dynamic system much like an RPG would have before you begin a game.

Next you must enter the country. But how? Insertion via submarine? Halo drop? Border run? Or maybe with a plethora of passports, identities, and spy tricks you enter the country via legitimate means. Anything goes because you are the commander, you assembled your team, and now the mission’s success or failure is soely your responsibility.

So how do you complete your mission? Well first things first, you have to find these terrorist guys. Inteligence knows who they are but they don’t know exactly where they are or who they might be working with or who they might be disguised as. They could be holed up in a military bunker somewhere. Or maybe kickin’ it in their highly secure penthouse suite. Either way, they definitely won’t be all traveling together so you’ll have to track them down and then decide how to take them out quickly, efficiently and without causing too much of a stir with the local authorities.

So you ask, how does one go about accomplishing this? Through a series of scripted, linear, predictable missions?? No! Think Bethesda and you’ll begin to glimpse the extent of my vision. A massive, completely open map that contains countless NPCs, (both friendly, neutral, and hostile), numerous cities, sprawling wilderness…absolutely endless opportunities and replayability. The country you play in would be comparable to the size of a Bethesda game world and definitely more detailed and packed with stuff.

The main mission story will be laid out, ready to be played (obviously with numerous plot twists and turns like any good sophisticated thriller story) but the way you approach this mission is completely open ended. You don’t even have to follow the mission–you could go rogue and follow your own agenda (just be sure your team mates feel the same way). Plus there will be numerous targets of opportunity and other various side missions.

The game could be played as a cutting edge first person shooter, a mind-numbing thinker/mystery, a heart pounding third-person spy adventure, or countless combinations of all three. The possibilities are staggering. This would be the first 21st century action RPG, the first truly replayable military FPS. Throw in a HL2 physics engine and true high def next-gen graphics and your looking at a game that could finally break an out-dated mold that has held strong for more than a decade.

So….comments? Insults? Sarcasm? Cheap shots? I welcome all.