Stefan, that's very cool.  So basically you're dumping the data into Java3D geometries for Display3D.  Some comments:

0. Can you do selection and hit-testing?  I'm assuming not because your GeomPortrayal3D code doesn't have any hooks for it.  Instead of going that route, you might try extending MASON's Shape3DPortrayal3D, which can also take geometry and can handle hit-testing.  It'd be nice to have something that's abstract and consistent enough with the existing Display3D codebase that we could include it as part of the package somewhere.

1. Point3D's a problematic name because there's already a javax.vecmath.Point3d.  Same goes for Geometry3D etc.  Naming is forever, so it's worthwhile to think upon this up front.

2. I had previously given some thought to an alternative but less elegant approach: take any FieldPortrayal2D or collection of FieldPortrayal2Ds and dump it [them] to a BufferedImage, then draw the BufferedImage as a texture in a large rect in a Display3D.  You could handle hit-testing and selection by taking the intersection point and translating it into a location in the virtual FieldPortrayal2D world and pushing through that way.

Sean