Order By Sum of Two Fields

Let's say I have a table with karma_up and karma_down. Everytime someone votes up karma_up gets incremented and everytime someone votes down karma_down gets incremented one as well. How can I pull these a selection of rows and have them ordered by the sum of these new values? ORDER BY (karma_up - karma_down) does not seem to work how I want. I simply want the rows with the highest karma up top.