While it is possible, handling media manually is way inferior to using WebRTC. To name a few things: WebRTC supports adaptive media quality and jitter buffering out of the box. We really shouldn’t be reinventing the wheel here.
Edit: also see WEBXDC Audio Conferences.