You will have trouble running with a smaller hole saw diameter without the pilot bit. There's nothing to center it then.
If you want to avoid the chisel, then the tip about a router is probably the way to go. Which, by the way, is a very good reason to buy another machine d^_^b