System and methods of managing reference frame adaptively for video communication under a network are provided. The system adaptively decides the interval between each I frame, the bitrate of the encoded bit steams, and the resolution of the encoded video to achieve quick connection. The system may set the appropriate reference frame(s) for the current frame, manage the reference frame(s) in the DPB to be long/short term reference frame(s) or mark them as unused for reference; mark the current frame to be a long term reference frame, a short term reference frame, unused for reference or skip encoding the current frame; adaptively change the ratio of long term reference frame(s) with respect to the short term reference frame(s) in the DPB; adaptively change the frequency ratio of marking the current frame as long term reference frame and marking the current frame as short term reference frame.