I was going very slowly using Elmer's white to affix bushes, shrubs, etc. a mountain with quite steep sides. I remembered that Elmer's Carpenter's Glue set much more quickly when I was doing my benchwork. I tried it on the sides of the mountain (cliff face, I guess). It sets faster, it grabs the scenery better and dries just as invisible, as far as I can tell.
If it works, that is what to use. Carpenters glue is not water soluble after it dries; white glue is. I have attached track to the baseboard with carpenters glue so that it will not move when I use the white glue to ballast.