One real oddity is that in your setupPortrayals method you are creating a new FastObjectGridPortrayal2D, then attaching it. But you didn't get rid of (detatch) the old one. Furthermore you create a new Display and register it but don't get rid of the original display!
This is gonna cause a lot of issues.
You shouldn't have any display code in setupPortrayals at all. That should be only in init. It's done once and that's it.
You shouldn't make a new FastObjectGridPortrayal2D in setupPortrayals and attach it unless you've first detached the original one. I suggest instead just taking the original one and setting it to the new field.