I run the demo, but "Can not open the device in voice mode"

That error message mean the line device which you try to open do NOT support voice mode. Make sure that device is a modem. You can run "DeviceCaps" demo to check the line device capabilities.
You said you can use that modem place calls, that is no strange, maybe you just let it working on the data mode.
How to solve the problem?
1. make sure you open the right device.
2. make sure your modem has voice capability. (you can use the "DeviceCaps" demo to check it)
3. make sure your voice modem 's Unimodem serial wave driver is install correctly. Try to reinstall/update the drivers.