You might think that departing from the pattern at a non-towered airport is simple, and it usually is. However, there is a standard way to do it that is safer than the alternatives.
See this diagram from the Aeronautical Information Manual.
For our purposes, we are only interested in #6. Notice the only two ways out of the pattern are straight ahead, or a 45 degree turn to the left. This is what the AIM says about it:
From AIM 4-3-3 If departing the traffic pattern, continue straight out, or exit with a 45 degree turn (to the left when in a left−hand traffic pattern; to the right when in a right−hand traffic pattern) beyond the departure end of the runway, after reaching pattern altitude.
It’s pretty clear from this that there is only one real way out of the pattern.
When can you turn?
The AIM recommends a straight out departure, but you will need to turn at some point unless you happen to be going where the runway points.
Continue on your straight out or 45 degree departure until you are outside of the normal traffic pattern (2 miles from the airport).
Make a plan that helps you to maintain clearance from the traffic pattern if you need to pass back over the airport. This means being at least 500 feet above pattern altitude before passing over any part of the pattern, including the inbound 45 degree leg.
Why not depart from downwind?
Have you ever heard the story of the high wing airplane flying just under the low wing airplane, and neither can see the other. This kind of thing can really happen, and downwind is where it would occur. If you are climbing through the pattern while others are maintaining altitude in the pattern you are inviting this type of setup.
Conversely, it is hard to imagine this occurring on a straight out departure leg. Keeping traffic separated based on activity makes the pattern safer by making it easier to anticipate where planes are and what they are doing.