h264

發表於   |   分類於   |   閱讀次數   |  

了解RTSP封包中的H264取得

更新記錄

item note
20170314 第一版
20170321 H264說明

目錄

RTP

  • RTP[rtp-fixed-header-fields]
  • 採用RTP Fix head，只使用前面12byte
  • The first twelve octets are present in every RTP packet, while the
    list of CSRC identifiers is present only when inserted by a mixer.
  • CSRC count (CC): 4 bits
    The CSRC count contains the number of CSRC identifiers that follow the fixed header.
  • byte2,3為SN: 當RTP Data packet send時會將會自動加1
  • 在12byte之後為RTP Padload部分

RTP Fixed Header Fields

RTP Fixed Header Fields

item bit note
versin(V) 2bits 固定為2
padding(P) 1bits
extension(X) 1bits If the extension bit is set, the fixed header MUST be followed by exactly one header extension
marker (M) 1bits
payload type (PT) 7bit This field identifies the format of the RTP payload
sequence number 16bit 送端會自動count
timestamp 32bit 同一包的timestamp是相同的

marker設定方式

由Encoding sample code with TI DSP可知

  • 在NAL Type 1-23中的marker設定如下：
    marker何時使設定為1
    若為IS_ACCESS_UNIT，若只要NAL type為0-5就會設定marker為1
    define IS_ACCESS_UNIT(x) (((x) > 0x00) && ((x) < 0x06))

  • 在FU-A中的marker設定如下
    只有最後一包才會設定marker=1，其它則為0

Payload Type Definitions

Payload Type Definitions
若Padload為H264使用PT:96-127

RTP Packet

  • RTP Packet
    RTP Packet裡面的padload存放NALU資料
    每個NALU資料為由{NAL Header,Payload}組成
    此NALU裡面的Payload可以組成H.264的VCL Frame
    圖片來源：Understand Video Signaling with SDP Debugs[RTP Packet]

H264說明

  • H264 codec有下例兩層

    • VCL: Video Coding Layer
    • NAL: Network Abstraction Layer
      VCL Layer將vidoe image分割為許多的slices，這此slice在分成許多的NALU傳送出去

  • NAL Unit
    一張會動畫面被H.264分成需多NALU(NAL Unit)
    每個NAL Unit如下圖, 圖片來源： Exploring H.264. Part 2: H.264 Bitstream format

    [NAL Structure]
  • RBSP
    RBSP組成如下，是由SOBD(String Of Data Bits)組成
    RBSP至少4byte(因此會將SOBD資料補至4byte對齊)
    圖片來源：H.264 stream structure[RBSP]

資料來源：H.264 stream structure

type note
SODB: String Of Data Bits 原始数据比特流, 熵编码输出结果, 以bit为单位。
RBSP: Raw Byte Sequence Payload 原始字节序列负载, 在SODB最后加上trailing bits补齐为一个字节
EBSP: Extent Byte Sequence Payload 扩展字节序列负载, RBSP里面加入防伪起始码字节(0x03)
NALU: Network Abstraction Layer Units. 网络抽象层单元, 在EBSP前面加上一个字节的NALU type.

RTP Payload Format for H.264 Video

RTP Payload Format for H.264 Video

  • Common Structure of the RTP Payload Format

  • The payload format defines three different basic payload structures
    the first byte of the payload. (第1個byte的NAL unit header可得知type)
    This byte is always structured as a NAL unit header.
    The NAL unit type field indicates which structure is present

  • Single NAL Unit Packet:
    Contains only a single NAL unit in the payload
    unit type: 1-23

  • Aggregation packet:
    Packet type used to aggregate multiple NAL units into a single RTP payload.
    unit type: 24-27

  • Fragmentation unit:
    Used to fragment a single NAL unit over multiple RTP packets.
    unit type: 28-29

[NAL unit types]

Single NAL Unit Packet

  • Single NAL Unit Packet
    nal unit type為1-23的padload格式如下 (即在rtp padload第12byte開始)[Single NAL Unit Packet]

Fragmentation Units (FUs)

  • RTP payload format for FU-A

    [RTP payload format for FU-A]

  • RTP payload format for FU-B

    [RTP payload format for FU-B]

  • FU Indicator
    在FU Indicator裡的Type為28/29 (表示FU-A/FU-B)
    需取得這里的F/NRI 來產生NAL unit he

    [FU Indicator]

  • FU Header
    S:表示片斷開始
    E:表示片斷結束
    Type:表示NAL unit padload type (需要得這裡的TYPE來產生NAL Unit Head)

    [FU Header]

注意地方

重點整理 – H264 Over RTP 之 寶典

  • 需要檢查FU-A的sequence number的順序(若非固定加1，表示有漏)
  • SPS(即NAL為0x67), PPS(即NAL為0x68)可以被取得，當Server回應RTSP/DESCRIBE

  • 正常的H264 NALU順序：
    SPS(NAL type = 0x07) -> PPS(0x8) -> I-Frame(0X5), P-Frame(0x1)
    把[SPS,PPS,IDR(即IFRAME)]當成一個Frame送到decoder
    圖片來源：Exploring H.264. Part 2: H.264 Bitstream format

    [h264 Stream]

  • 由回應中的sprop-parameter-sets可取得SPS,PPS相關參數

    [RTSP / Describe]

NAL Unit

  • There are two ways to pack a NAL unit for different systems, Packet-Transport System and Byte-Stream Format.
    • For Packet-Transport systems like RTP, the transport system protocol frames the coded data into different pieces.

  • boundaries of the NAL units.

    • three-byte or four-byte start code, 0x000001

  • nal unit syntax

    [NAL Unit Syntax]

    f(n): fixed-pattern bit string using n bits written
    u(n): unsinged interger using n bit
    b(8): byte having any pattern of bit string(8bit)

ex. 0x67 (0110 0111)
b[0]: forbidden_zero_bit = 0,
b[1-2]: nal_ref_idc = 3,
b[3-7]: nal_unit_type = 7

  • More about NAL Unit Type
nal type note
7 Sequence parameter set (SPS)
8 Picture parameter set (PPS)
  • NALU_TYPE_SLICE 1
  • NALU_TYPE_DPA 2
  • NALU_TYPE_DPB 3
  • NALU_TYPE_DPC 4
  • NALU_TYPE_IDR 5
  • NALU_TYPE_SEI 6
  • NALU_TYPE_SPS 7
  • NALU_TYPE_PPS 8
  • NALU_TYPE_AUD 9
  • NALU_TYPE_EOSEQ 10
  • NALU_TYPE_EOSTREAM 11
  • NALU_TYPE_FILL 12

H264 Encoder Profile

實際收到的RTP資料

Camera送出的RTP資料

  • Camera送送出的RTP資料如下[RTP Data]
seq time marker padload note
1 0 1 0x67 nal type =7, SPS
2 0 1 0x68 nal type =8, PPS
3 0 0 0x7c,0x85 nal type =28,FU-A,S=1, (nal type=5)
4 0 0 0x7c,0x05 nal type =28
5 0 0 0x7c,0x05 nal type =28
22 0 1 0x7c,0x45 nal type=28,FU-A,E=1 (nal type=5)
23 5940 1 0x41 nal type = 1
24 11880 0 0x5c,0x81 nal type=28,FU-A,S=1, (nal type=1)
25 11880 1 0x5c,0x41 nal type=28,FU-A,E=1, (nal type=1)
26 17820 0 0x5c,0x81 nal type=28,FU-A,S=1
  • 程式處理流程
    • 取得seq1，即SPS(type=7)，增加start-code及nal-head(type=7)，之後進入decoder
    • 取得seq2,即PPS(type=8)，增加start-code及nal-head(type=8)，之後進入decoder
    • 取得seq3，即FU-A，開始組合frame,由seq3-seq22(即FU-A結束)，為完整一包
      增加start-code及nal-head(即type=5)，之後進入decoder

DTV Server送出的RTP資料

  • DTV Server送出的RTP資料如下[DTV Data] [DTV Data]
seq time marker padload note
27895 1876090015 0 0x7c,0x85 nal type =28,FU-A,S=1, (nal type=5)
27896 1876090015 0 0x7c,0x05 nal type =28
27897 1876090015 0 0x7c,0x05 nal type =28
27986 1876090015 1 0x7c,0x45 nal type=28,FU-A,E=1 (nal type=5)
27987 1876090075 0 0x5c,0x81 nal type=28,FU-A,S=1, (nal type=1)
27988 11880 0 0x5c,0x01 nal type=28,FU-A

注意
在DTV傳送的RTP資料中，無SPS及PPS因此在送次decoder之前
需要由RTSP describe回應中的sprop-parameter-set中解出SPS及PPS參數

1
a=fmtp:96 profile-level-id=676400;packetization-mode=1;sprop-parameter-sets=Z2QAKKzoB4AiflQ=,aO48MA==

H264 Emulation Prevention Bytes

  • Start codes work because the four byte sequences 0x000000, 0x000001, 0x000002 and 0x000003 are illegal within a non-RBSP NALU.
  • H264的RAW Data是無定義長度，因此需要用start code來當區隔(即分開不同type)

  • 但有可能此值(0x000001)，存在bitstream裡面，因此需要跳脫字元(esc,這邊使用Emulation Prevention Bytes, 0x03)
    因此為在建立nal包時會去檢查
    0x000000,0x0000001,0x000002,0x000003
    就它變成
    0x0000<03>00,0x0000<03>01,0x0000<03>02,0x0000<03>03
    This ensures that no sequence of consecutive byte-aligned bytes in the NAL unit contains a start code prefix.
    防止start code在存於nal padload裡面

  • 原始的SPS資料如下
    00 00 00 01
    67 64 00 0A AC 72 84 44 26 84 00 00 00 04 00 00 00 CA 3C 48 96 11 80

  • 在encode資料時會變成如下，遇到0x00000 (3byte start code)，變成0x00000300
    裡面有兩個Prevention Bytes(0x03)
    00 00 00 01
    67 64 00 0A AC 72 84 44 26 84 00 00 03 00 04 00 00 03 00 CA 3C 48 96 11 80

  • emulation prevention byte
    A byte equal to 0x03 that may be present within a NAL unit.
    The presence of emulation prevention bytes ensures that no sequence of consecutive byte aligned bytes in the NAL unit contains a start code prefix.

  • start code prefix
    A unique sequence of three bytes equal to 0x000001 embedded in the byte stream as a prefix to each NAL unit.
    The location of a start code prefix can be used by a decoder to identify the beginning of a new NAL unit and the end of a previous NAL unit.
    Emulation of start code prefixes is prevented within NAL units by the inclusion of emulation prevention bytes.

參考來源：Introduction to H.264: (1) NAL Unit,ITU-T Rec. H.264 pdf

參考