In this paper we propose a novel end-to-end architecture for H.264 unicast video streaming over the Internet. The proposed video streaming architecture is based primarily on a new transport layer protocol, the stream control transmission protocol (SCTP). We show that the network-friendly specification of H.264, and the novel technical characteristics of SCTP, when coupled together are able to provide a highly adaptive and flexible system for unicast video streaming. More specifically, we develop algorithms that handle at the transport layer the following functions concerning video packet transmission: retransmission policy, packet prioritization, implicit receiver feedback. We combine the above algorithms with an R-D optimization strategy at the encoder, that provides more options when adapting to bandwidth variations. This combined optimization strategy leads to more options concerning the streaming parameters. Finally, with simulation results we prove that our system is capable of maintaining good perceptual quality and TCP-friendliness under various loss conditions.