I have just written a program (over a year ago) which allows speaking/voice (no video) using plain server-client architecture. Just in plain C. Reading/writing to/from /dev/dsp (with some ioctl to set Hz, bits etc), next use sppex or vorbis to encode/decode this sound. I would only need to add encryption (which is really simple) to have it 99% secure.
This is the simplest program imagined - no single line of GUI.
Start server as ./cvoice -d
(becomes daemon), next call it from client - no ringing, voice transfer just starts. I've written this because Skype had no option to configure voice parameters - while I could lower it to 8 bit, mono, 6500 Hz + Speex compressions which gives 400-500 BPS bps and allowed me to talk with wife from Spitsbergen while Skype had not enough bandwidth to make a call.