Each person's work situation is different and every person is different in many ways. When you reach a point in your career where your boss invites you to an event that does not fall under the "work requirements" type of event, it is time to tread lightly and think very carefully before accepting or declining.
Depending on the company you work for and the number of employees, as well as the makeup of the group you work in, you may never be invited out for drinks with the boss. Your boss may or may not invite you and your family out to dinner. However, if you work at a close knit company with a rather informal culture, you may very well find yourself in a position to socialize with the boss outside of the job.
Considerations for declining
If your boss is the type of person that you really just can't stand, then it is probably in your best interest to figure out a diplomatic way to decline the invitation without creating any animosity. This may be easier said than done. Some good excuses that keep you looking pristine while getting you out of the engagement are:
If the invite is for drinks with the boss, let he or she know that your spouse has you signed up to review a certain class that night and you promised to do it.
If the invite involves you and your family, tell your boss that your kids are at a rowdy stage in life and you would rather wait until they calm some.
For a general invite, let your boss know that you have some function or other at church where you are volunteering. If you use this excuse, you'd better consider volunteering for something at church!
In essence, try to make it look like you have no choice in the matter and must reluctantly refuse. This way your boss does not take it personally and will hopefully get the message that you have a busy life outside of work.
Considerations for accepting
One of the great things about working with people you get along with is that sometimes, on rare occasions, you make lasting friendships with one of your coworkers or maybe even with the one of your managers. If you feel that socializing with your boss may lead to a rewarding friendship and that your families would fit together like peas and carrots, then you should definitely consider an invite from your boss.
I recommend planning carefully the first couple of events so that you and your family can get acquainted in a neutral area where either family can leave if they are not having fun or just don't get along.
Once you feel comfortable hanging out with your boss and family, start planning more intimate meetings where you may go to each other's homes for dinner. You may even reach a point where you take short trips or vacations with your boss. There is nothing better than being able to develop a friendship like this that lasts a lifetime.
However, remember that this person is your boss and is only human. You need to avoid confrontations and activities that always make you seem better than your boss. If you are a semi-pro bowler and your boss is a novice, don't keep inviting them to bowl. You also need to remember when it's time for work and when it is time to play. All in all, keep in mind that you work for this person and you want to keep your job.