AsciiMii :: An Adobe Pixel Bender Ascii Filter

Introduction
Samples

Images
Video Stream
Web Cam

Source Code
Code Breakdown

Inputs and Parameters
Determine the Color
Determine the Luminosity
Getting the Proper Ascii Pixel
Multiply Blend

Applying to Flash Video
Conclusion, Copyrights, and References

Introduction

On April 1st, 2010 YouTube pulled what I considered to be the greatest Tech Industry prank of the day: TEXTp. For the fools, YouTube claimed that using this video quality mode saved bandwidth; an obvious ruse. In reality it was simply a pixel shader that was applied to the video stream that turned streamed video frames into the ASCII style that we all know and love. I love ASCII art, so I thought this was the coolest thing I’d seen in a long while. I had hoped that YouTube would realize just how awesome this effect truly is and leave it available year round, but my hopes were in vain. At midnight on April 2nd, the effect was gone. You couldn’t even append the &textp=fool parameter into the querystring. Lucky me I had Firefox save me a page on my disk that still had all the proper flashvars to keep using the effect. But sometime on April 5th, YouTube removed the pixel bender shader they were using for the effect from public access. TEXTp was no more as far as YouTube was concerned.

My nerdyness couldn't allow something so awesome to become mere internet history, so did a little digging and wrote my own TEXTp filter: AsciiMii is a kernel for the Adobe Pixel Bender Framework that converts the given input image into colored ASCII art as a homage to YouTube's no-longer-available TEXTp 2010 April Fool's joke.

Sample Images

Obviously, since AsciiMii is a Pixel Bender kernel, it can be used in the Adobe Pixel Bender Toolkit to easily convert images to ASCII art.

Video Stream

AsciiMii can also be used as a filter for DisplayObjects in Flash. Below is an example of applying the filter to a video stream in Flash. Use the radio buttons below to change the movie and use the AsciiMii button in the player to turn the filter on and off. You can also put a direct link to a valid flash video file to play an external video.

The Flash player examples will not work in Flash Lite (sorry, mobile phone users). If the shader fails to load, you may need to update your Flash Player. See Applying to Flash Video for more info.

Do The Cosmonauty
Tron Legacy - Trailer #2
Nobody Beats the Drum - Gridin'
Muse - Sing For Absolution

The source code for the above video player can be downloaded here. For more information on how to use the filter in Flash, see the Applying to Flash Video section below.

Web Cam

Another fun thing to do is attach the filter to a Video object that uses a web cam as its source:

The source code can be downloaded here.

AsciiMii Source Code

The following is the complete source code for the AsciiMii Pixel Bender kernel:

<languageVersion: 1.0;>

kernel NewFilter<
    namespace: "com.greyboxware.asciimii";
    vendor: "Richard Zurad";
    version: 1;
    description: "Filter to mimic the TEXTp effect from YouTube's 2010 April Fools joke";
> 
{
    input image4 src;
    input image4 text;

    output pixel4 dst;

    parameter int size <
        minValue: 1;
        defaultValue: 8;
        maxValue: 32;
    >;

    parameter int charCount <
        minValue: 1;
        defaultValue: 256;
        maxValue: 512;
    >;

    void evaluatePixel() {
        float sizef = float(size);
        float charCountf = float(charCount);
        
        float2 offset2 = mod(outCoord(), sizef);
        pixel4 mosaicPixel4 = sampleNearest(src, outCoord() - offset2);
        
        float luma = 0.2126 * mosaicPixel4.r + 0.7152 * mosaicPixel4.g + 0.0722 * mosaicPixel4.b;

        float range = (1.0 / (charCountf - 1.0));
        float fontOffset = sizef * floor(luma / range);
        float fontmapsize = (sizef * floor(sqrt(charCountf))); 
        float yRow = floor(fontOffset / fontmapsize);
        offset2.y = offset2.y + (sizef * yRow);
        offset2.x = offset2.x + (fontOffset - (fontmapsize * yRow)); 
        pixel4 charPixel4 = sample(text, offset2);

        dst.rgb = mosaicPixel4.rgb * charPixel4.rgb;
        dst.a = mosaicPixel4.a;
    }
}

The above code can be copy/pasted into a new pbk in the Pixel Bender Toolkit, or you can download the pbk for the above source code here

Code Breakdown

At its core, the effect needs to break the image into a grid of a given size, determine what color represents this section of the image, and determine which ASCII character best represents the brightness of this section, and then draw that character with the given color. Recall that during the execution of the shader, the evaluatePixel function is evaluated for each pixel.

Inputs and Parameters

AsciiMii requires two input images:

        input image4 src;
        input image4 text;

An example 8x8 font map.

src is the source image; the image that is being converted into ASCII. text is the font map image that represents the ASCII characters that are used in the conversion. By default, the font map is an image of 256 8x8 pixel cells of ASCII characters in rows of 16 cells ordered from left to right, top to bottom by order of brightness. There really is no limit to the number of characters that can be defined in the font map (except maybe floating point precision), however, the font map image is required to be square. For example, the default image of 256 characters is defined in rows of 16 characters, making the font map a 16x16 grid of characters.

In doing research for this project, I happened to stumble upon the font map that YouTube used for their TEXTp filter (the source of which has since been removed). I also found out just how much time and effort was put into making their font map. Out of respect for the artists and engineers at YouTube, I will not be releasing YouTube's font map with my shader or source code.

There are also two parameters:

    	parameter int size <
            minValue: 1;
            defaultValue: 8;
            maxValue: 32;
        >;
    
        parameter int charCount <
            minValue: 1;
            defaultValue: 256;
            maxValue: 512;
        >;

size is an integer that tells the filter the size (in pixels) of each cell in the font map. By default, each cell is 8x8 pixels. charCount is an integer telling the filter how many characters are defined in the font map. The parameter is defined as an integer, however due to the requirements of the font map, the integer should be a perfect square.

The first lines of the evaluatePixel function

            float sizef = float(size);
            float charCountf = float(charCount);

converts the given inputs from integers to floats. For the sake of ease and common sense, the parameters are supplied as ints. Inside the Pixel Bender Toolkit, there is no problem with converting between floats and ints. In Flash, however, I discovered that its runtime for Pixel Bender is atrocious at truncation. For example: in the Pixel Bender Toolkit, truncating the result of dividing two floats, such as int(float / anotherFloat) works just fine, however in Flash, that statement will crash Flash 90% of the time, and produce incorrect results the other 10% of the time. For this reason, I convert everything to floats right off the bat and keep them as such throughout the function.

Determine the Color

The color is determined by converting the image into a mosaic:

            float2 offset2 = mod(outCoord(), sizef);
            pixel4 mosaicPixel4 = sampleNearest(src, outCoord() - offset2);

The mosaic code simply uses the modulus function break the image into cells of 8x8 pixels and makes each pixel within that grid a clone of the pixel in the upper left corner. The first line calculates the difference in (x, y) between the coordinates of the pixel being evaluated, and the pixel in the upper left corner of its grid cell. Line 2 stores a copy of the pixel determined to be the mosaic pixel for the current pixel.

Determine the Luminosity

The next step is to determine how bright the mosaic pixel is so that we can determine which ASCII character best represents its brightness. It turns out that a couple of geniuses have already derived a function that determines the brightness of a pixel (with respect to how the human eye interprets color) as a number between 0 and 1, zero being completely dark and one being full bright [ITU-R Recommendation BT.709]:

            float luma = 0.2126 * mosaicPixel4.r + 0.7152 * mosaicPixel4.g + 0.0722 * mosaicPixel4.b;

The determined brightness is calculated and stored in the variable luma. The next section is where things get interesting.

Getting the Proper Ascii Pixel

At this point we have a value between 0 and 1 that tells us how bright the pixel is, and we have an image of 256 8x8 ASCII characters that need to tie to this brightness value. First we need to figure out what is the range of values for each ASCII character:

            float range = (1.0 / (charCountf - 1.0));

Take the total number of characters (minus one because we start at zero), and determine the range of luminosity values that a character can represent. For our default 256 characters, range becomes 0.00392. This means that for the first character in our font map, any luma value that is between 0 and 0.00392 will be mapped to the first character; the second character will be any value between 0.00393 and 0.00785, etc.

Using luma and range, we can figure out just which character our luma value actually falls into:

            float fontOffset = sizef * floor(luma / range);

Take the size of each cell (in pixels) multiplied by the floored answer to luma / range, which gives us the proper character number.

Finally we get the proper pixel of the character in the font map that maps to the current pixel we're evaluating:

            float fontmapsize = (sizef * floor(sqrt(charCountf)));
            float yRow = floor(fontOffset / fontmapsize);
            offset2.y = offset2.y + (sizef * yRow);
            offset2.x = offset2.x + (fontOffset - (fontmapsize * yRow));
            pixel4 charPixel4 = sample(text, offset2);

First we need to determine where in the font map the character is (remember, we have the character number as if indexing from a one-dimensional array, but the font map is actually a two-dimensional array of size 16x16). Firstly, we figure out the size of the font map in pixels (we're expecting it to be square, so one calculation gives us width and height), and then use the calculated font offset to determine which row in the font map the character we want is it. From there, we can offset accordingly to get the pixel of the font map that maps to the pixel we're evaluating in the original image. After determining the font map pixel, retrieve it.

Multiply Blend

Ok, so we have our original pixel in the image as a mosaic which gives us the color of the character, and we have a pixel of the character. Because the font map is white (or shades of gray) text on a black background, doing a multiply of the mosaic pixel and the font map pixel forces the font map pixel to be the appropriate mosaic color (unless the font map character is a shade of gray, in which case it will dim the mosaic color slightly which may be desirable for lower luminosity values.

            dst.rgb = mosaicPixel4.rgb * charPixel4.rgb;
            dst.a = mosaicPixel4.a;

As of Pixel Bender 1.0, there is no alpha blending, so we don't include the alpha value in our multiply calculation.

Now the dst output value is set properly, and the pixel returned has been Asciified.

Applying to Flash Video

I'm actually quite surprised by the lack of articles and tutorials online that demonstrate how to apply Pixel Bender kernel's to video streams in Flash, and even less about how to supply a second image source. So without further adue, here's some ActionScript 3.0 code snippets that will make it happen:

var video:Video;
var netConnection:NetConnection;
var netStream:NetStream;
var urlRequest:URLRequest;
var urlLoader:URLLoader;
var shader:Shader;
var shaderFilter:ShaderFilter;

[Embed(source = "fontmap.png")]
var FontMap:Class;

video = new Video();

netConnection = new NetConnection();
netConnection.connect(null);
	
netStream = new NetStream(netConnection);
video.attachNetStream(netStream);

urlRequest = new URLRequest("asciimii.pbj");
urlLoader = new URLLoader();
urlLoader.dataFormat = URLLoaderDataFormat.BINARY;
urlLoader.addEventListener(Event.COMPLETE, urlLoader_complete);
urlLoader.load(urlRequest);

function urlLoader_complete(event:Event):void {
	try {
		urlLoader.removeEventListener(Event.COMPLETE, urlLoader_complete);
		shader = new Shader(event.target.data);
		shaderFilter = new ShaderFilter(shader);
		shader.data.text.input = (new FontMap() as Bitmap).bitmapData;
		shader.data.charCount.value = [225];
		video.filters = [shaderFilter];
	} catch (e:ArgumentError) {
		trace("it fucked up: " + e);
	} //end trycatch
} //end urlLoader_complete

To stream video to Flash, we need Video object, a NetStream object, and a NetConnection object. To actually attach the shader to the video stream, the shader needs to be loaded into the Flash movie (or embedded), assigned the fontmap and parameters (if needed), and applied to the video object.

The font map needs to be supplied as a second input. The easiest way I found to do this was to use the Flex SDK's Embed tag to embed the font map image as a Class which can be then converted to a BitmapAsset to retrieve the bitmap data:

[Embed(source = "fontmap.png")]
var FontMap:Class;

The shader is loaded into the flash movie via the following code:

urlRequest = new URLRequest("asciimii.pbj");
urlLoader = new URLLoader();
urlLoader.dataFormat = URLLoaderDataFormat.BINARY;
urlLoader.addEventListener(Event.COMPLETE, urlLoader_complete);
urlLoader.load(urlRequest);

A URLRequest object is used to get the pbj file (the Pixel Bender kernel exported to bytecode for Flash), and it is loaded by the URLLoader. The urlLoader_complete function is called when the filter has finished loading

function urlLoader_complete(event:Event):void {
	try {
		urlLoader.removeEventListener(Event.COMPLETE, urlLoader_complete);
		shader = new Shader(event.target.data);
		shaderFilter = new ShaderFilter(shader);
		shader.data.text.input = (new FontMap() as Bitmap).bitmapData;
		shader.data.charCount.value = [225];
		video.filters = [shaderFilter];
	} catch (e:ArgumentError) {
		trace("it fucked up: " + e);
	} //end trycatch
} //end urlLoader_complete

The event listener is removed, and a new Shader object is created and supplied the data from the pbj file. The shader is then assigned to a ShaderFilter object that will be supplied to the video object. The font map image is added as a second input source to the shader, and the charCount parameter is changed from default.

I mentioned earlier that Flash tends to calculate things differently than the Pixel Bender Toolkit. After messing around with 256 character font maps, I found that everything worked fine in the Toolkit, but Flash would not properly floor some of the calculations, causing some of the brighter characters to fall off of the font map (everybody loves floats). As an easy workaround was to set the charCount value to 255, effectively removing the last row and column of characters in the font map. I realize this is not an ideal solution, but for now, I'm living with it.

Finally, the shader is added as a filter to the video object via the video.filters = [shaderFilter]; line.

While testing and debugging inside of the Flash IDE, you're probably going to run into problems with the video.filters = [shaderFilter] line. There seems to be some wonkiness with Flash getting confused about the main source of input for the shader... even though it's defined right there in the line. I'm not entirely sure why, but anywhere from 40%-80% of the time, running the swf through the Flash IDE or the Flash Debugger causes the shader to throw an ArgumentError and complain that the source for the shader is either not found or is an unsupported type. It seems to have a mind of its own as to when it decides to throw the ArgumentError. For example: while building the FLV video player the shader worked fine until I added a comment (yes, a comment. An ordinary run-of-the-mill end-of-line as in the-compiler-never-even-evaluates-me-and-therefore-should-not-matter comment). That comment caused the shader to fail to find the source 100% of the time when ran from inside the Flash IDE. The shader seems to load 100% of the time when run in a browser (or just desktop Flash Player).

Conclusion, Copyrights, and References

And there you have it. There's obviously a few small places left to take this filter. Since Pixel Bender is working its way into the whole Adobe way-of-life, I may consider looking into turning the shader into a Photoshop Filter. I'm also considering a small web app that simply takes an image from the user, Asciifies it, and lets the user save it. All this for a later time, though. I think, for now, this fun little experiment to scratch the surface of shaders is good enough

Project and code by Richard Zurad. Based on the TEXTp YouTube Filter by Billy Biggs, Blake Livingston, Peter Bradshaw, and YouTube. All third party images and videos are used for demonstration purposes and are copyrights of their respective owners. All code is licensed under the Beerware License.

Source Code:

References