Apt. 3 by Ezra Jack Keats

From chapters.ca:
In his apartment building in the city, Sam hears voices, barking dogs, snoring, and all kinds of other noises. One rainy day he also hears the sad sounds of a harmonica, and wonders who's playing. Sam's search leads him to Apt. 3, where he finds not only the source of the music but a new friend.