The dot clock for that display is around 5MHz, which is do-able by the PIC32.
The memory requirements are a bit high, though, 320x240 = 76,800 pixels. The highest PIC32 has is 32k ram, which means you could allocate at most 4 bits per pixel using only the internal memory.
You could expand it with external memory, or even use the external memory to control the display, writing updates to the memory inbetween lines and frames.
Alternately, you can run the display off a single PIC32 if you drive the display with sprites stored in flash. You wouldn't set individual bits on the display memory, you'd instead have a table of objects to display on the screen with their location, sorted by starting pixel line and then column, so as you drive each line to the display you go down this table, find the next object to display and when you get to it's column start taking data from flash and spitting it out to the display.
You can also incorporate vector drawing, including lines, polygons, circles, etc. The math is a bit tricky to understand at first, but it's quite interesting and fun once you understand how to draw vector objects on a raster display without a buffer.
Your fonts, buttons, graphics, etc would mostly be static objects stored in flash, with some ram set aside for custom objects.
Overall, it would feel very limiting if you're used to computer graphics where you set pixels, but if you plan ahead and carefully design your interface your users will not notice the difference, and you may actually find it's easier to develop for.
If you add a little more logic that allows overlapping objects to be combined in interesting ways (add, subtract, or, xor, and, etc on a per-pixel basis) then you can create very sophisticated graphics with many pre-generated graphic objects and vector objects.
Alternately just make the entire display vector - with each line consuming 7 bytes of memory you can draw over ten thousand lines on the display. Circles require less information (x, y, radius, color) so you can draw many more of them. Careful and creative use of vectors can give you very sophisticated and interesting graphics.
Or you can emulate a memory buffer using the flash. Define areas of the display (windows) where information will be changed frequently, and store those in RAM. Write the rest of the display graphics to flash (which you can re-write 3 times a day if you only need the unit to last for 10 years). With careful partitioning you might find that you use very little RAM and still get everything you need in terms of graphics, rapid updates, and eye candy.
And all that will only cost you $10 for another PIC32 (and countless hours of development time). $5 in quantity.

I'll have to add that to my list of projects that would be fun to do...
-Adam