Headset drifts back towards calibration point after any movement.

I set everything up with an Arduino due cause I had one on hand, and the tracking seems okay except that it will drift back to where the headset was originally calibrated. For example, if I look left it will follow me left but as soon as I stop rotating it starts drifting back to the calibration point. It happens in all directions and is returning to the calibration position which makes me think that it is a code issue, but I am not sure if it was something I changed or something else. Any ideas?