We had one full day in Portland, primarily because after we checked out of Chebeague, and took the ferry back to the mainland, we had a 6pm reservation at Fore Street Restaurant. It's so good, it's reason enough to go to Portland. I think it's like the Zuni of the East Coast. If you're going to be in the area, call and make a booking, even if it isn't the day or time you want--and then several days before they often call you up and can give you the timing you need.
Just a 10 minute drive north of Portland is a magical walk on Mackworth Island, where you can see fairy houses, and make your own too.