Currently, upgrading would be pointless. Your computer would have to work harder to process the bigger pictures down to the same size that's already being sent to your phone. In other words, you can't get higher-res images on your phone via iCam. Besides, if iCam did allow higher-res image, your framerate on 3gs would be really slow. Part of the limitations in place by iCam are due to Apple/AT&T.
If your picture isn't clear, it's probably because your camera isn't focused. If there's a piece around the lens that you can twist, twist it while viewing the video until the image is focused. You might also access the camera settings and limit the camera to sending the iCam native resolution (640 x 480 I think). This would reduce processing by icamsource and might improve your image quality a little bit...since processing sometimes results in artifacts, etc.