When I work on the X stack I typically compile and install all the packages into an install prefix in my home directory, typically $HOME/install. That way I can keep my system packages intact and even have multiple versions or branches of the stack installed.
