I agree with Matt. A small amount off plumb can lead to bigger problems down the road when everything settles and shifts, especially after the first winter/spring cycle. You may not get a freeze, but I'm betting you get a rainy/wet season where your ground conditions are much different than right now. Better safe than sorry.