Description

• Send delta frames in another stream: see how the bandwidth is affected, in theory, if we send all delta frames in one stream (not delta frame per stream), then all the delta frames are in order, and more ever, since they are sent in one stream, those delta frames should take much less bandwidth than sending one delta frame in one stream.
  • Verify: does webcodes vp8 delta frames depends only on key frame? Does delta frame also depends on previous delta frame when decoding?

Task List

• [x] what about audio chunks? Do we keep using one chunk per stream? Or 100 chunks per stream? (To align with video delta frames in one stream). Potential HoL blocking?
  • [x] or the underlying principle of sending objs in one stream is dependency? For audio chunks, we can do grouping, stream per audio chunk is a little bit overhead, and by grouping, we can somehow eliminate this overhead.
• server is suppose to handle objs opaquely. RemoteTrack is handling objects reading at object level, so server cannot do anything about sending delta frame objects to audience in one stream. The only place we might be able to send delta frames in one stream lies on the streamer side after it does encoding and opening stream to send those delta frames.
  • [x] quick verification: how does LocalTrack writing objs? does it maintain the stream structure from streamer? Such as if streamer writes key + 29 delta frames in one stream, LocalTrack doesnot interfer with object related stream handling, right? Audience will get those key+29delta frames in one stream from server (just like server got from streamer) For the cureent LocalTrack, it doensot modify anything, it’s just doing one to one mapping. For the upcoming versions of stream mapping, there are 4 options on it: stream per track, stream later MOQT dft, where each subgroup will be mapped to its own stream.
  • Streamer sends objs to the server via the call back func of subscribeOk() from the webtransport session, expend to see code block.
  • we can add another param to the header of the returned func:
    • Code
    • General idea:
      • for video: we only create new uds for video frames if it’s a key frame, for the remaining delta frames, we reuse the same uds. So 1 key + 29 deltas all in one stream (~ 1 sec of video frames)
        • [x] Since all the delta frames depends on the key frame, it makes more sense just to send them in one stream, which ensures order, and doesnt takes more bandwidth than it actually requires logically. Yes, make sense. Sending key + delta frames in one stream is good.
        • the first frame would always be a key frame(ensured in VideoEncoderWorker), then delta frames will reuse the same uds that is created for that key frame. Hence there wonot be the case where some delta frames are generated and send before a key frame is sent.
        • [x] but will this have negative impact on video (burst) buffering? Not, if we start deserializing before all objs received on the stream.
      • for audio: we create a new uds for each chunk, since audio chunks donot have dependencies on each other
• [ ] seems it doesnot work for the current system on the audience decoder/deserializer, and the audience is indeed getting one obj per sec for video, and another obj per sec for audio (if we send 1 sec of audio chunks in one stream). on the audience side, we cannot start process video/audio objects if the sender is not done sending. Hence there will be another second of delay added to the pipe.
  • [x] does WritableStream<Uint8Array> allow reading from ReadableStream (on receiver side) before writer finishes writing to it? Yes. https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream
  • [ ] or can we start deserializing before the stream close? The streamer needs to set boundaries between objects so the audience can start deserializing once a boundary is discovered. (embedding obj size into the header of a chunk in LOC)
    • on sender side, we can prefix the size of each object when sending them, such as if the object is x bytes, and we can encode number "x" in another 8 bytes, and then we write the first 8 bytes as the obj bytes such that the value should be x+8. then when audience read from its readableStream, if extract the first 8 bytes to know the size of the current obj, if the buffer is bigger than the obj size, we slice it off from the buffer and pass the whole object to the deserializer func.
  • [x] since we are sending audio and video chunk in two streams possibly at the same time, on the audience side, would it mix data from two streams since it’s reading from one readableStream. Receiver gets a new ReadableStream when sender opens a uds or bds. So data wonot mix, unless sender mixes them intentionally.
• [ ] Seeing “writing obj” about 30 per sec, but only 1 “obj sent” per sec. we are creating one uds per sec. Seesm the encoder is not sending objs until a new uds is open? Nope, cause we are closing the objectStream when done sending one obj.
  • Code snippet
  • [ ] the prob is, every time we call this function, we have a new instance of the objectStream, the only thing we can do is to pass the func header to the streamer, and streamer define the function?

• [x] We can ensure single stream bandwidth on streamer side via serialization. Keep a stream open for 1 sec to send all delta frames is a long time, maybe keep stream lives short is better. Stream will not be closed unintentially by the client, so the duration of a stream alive is not sth to worry about.