The Inside Tree by Linda Smith, illustrated by David Parkins

From chapters.ca:
Mr. Potter lives all alone in his little house. And outside is a little yard with just enough room for one little tree. In fact, it's such a fine tree that good- hearted Mr. Potter feels bad that it stands there all alone in the yard while he stays warm by the fire inside the house. It doesn't seem right that the tree is never invited in. And so he decides to do just that. But what will happen when the tree grows and grows and grows and his little house stays exactly the same size?