Support your weight on your legs, (which we should all be doing anyway) and take it slow. No problem. The point with keeping light on the handlebars, is that every little bump won't make you change the throttle. If you're weight is on your legs, your throttle is smoother over the bumps. I've got a grass and gravel driveway, and do this every day.
As was previously said, you can be too loose, and the front end will grab, twist and down you'll go. Maintain control of the front end, with your weight off the bars.
You can take most any vehicle down a decent gravel road. With a little gumption, you can take lots of vehicles down lots of roads they were never meant for. I've got stories of a '69 SuperBeetle I had, and some motorcross trails! Whoohoo!