h264

Rendering mp4 video not silky, why?

by

The original HD mp4 file 'movie.mp4' plays very smoothly. I picked out the h264 stream saved as 'movie.264' from it, then using mp4v2 converted 'movie.264' to a new mp4 file 'output.mp4'. However, experience of watching 'output.mp4' was very bad. Why?

I debugged the 'ClipTrack' function in 'mp4v2/test/OLD/mp4clip.cpp', found a key argument 'renderingOffset' of function 'MP4WriteSample'. All examples searched from the web about writing h264 to mp4 through mp4v2 set 'renderingOffset' to 0, while in 'ClipTrack', 'renderingOffset' of the function 'MP4WriteSample' is directly set to the value from 'renderingOffset' of the function 'MP4ReadSample'.

Here we make a concept clear, the 'sample' entity in mp4 file format specification is equal to a frame which can completely rendering as a picture. A sample consists of several nalus, a nalu whose start code is 0x00, 0x00, 0x00, 0x01 indicates a new sample, while a nalu which starts with 0x00, 0x00, 0x01 is among a sample, and a sample needs not to be a IDR key frame.

Through carrying 'renderingOffset' value at beginning of each sample in 'movie.264', I did generate an 'output.mp4' which displays as smoothly as the original one.

Did 'renderingOffset' mean time stamp of a sample? No, at least not exactly. By watching log, I found its value was not increasing as time went by.

I continued debugging mp4v2, then saw a key word 'ctts' atom, and searched a link from the web,

https://blog.csdn.net/cheyo809775692/article/details/100924523

I now understand, the decoding order of h264 samples is slightly different from the rendering order of them.

For example, in stts atom,

1
2
3
4
5

'decoding order'	'sample count'		'sample duration'
1,2,3,4			4			20
5			1			40
6,7,8			3			20
9			1			60

in ctts atom,

1
2
3
4
5
6
7
8

'decoding order'	'sample count'		'sample offset'
1,2,3			3			20
4			1			40
5			1			0
6			1			40
7			1			0
8			1			40
9			1			0

final rendering timing is

1
2
3

'decoding order'	1	2	3	4	5	6	7	8	9
'duration'		20	20	20	20	40	20	20	20	60
'rending order'			1	2	3	5	5,4	7	6	9	9,8	9

Terms,

1
2
3
4
5
6
7
8
9
10
11
12

mvhd: movie header
tkhd: track header
mdhd: media header
stsd: sample descriptions
stts: time to sample
ctts: composition offset
stss: sync sample offset
stsc: sample to chunk
stsz: sample size
stco/stco64: chunk offset
dts: decoding timestamp
pts: presentation timestamp

NDK硬解h264

by

有几点须注意，

  1. 不要让硬解码器自已绘制表面，也就是AMediaCodec_configure传入的surface应设为空。否则运行时间长了，会出现“weak global reference table overflow”的崩机错误。
  2. 我之前文章里也说过了，不要静态链编libmediandk.so，老的安卓机上没有这个库，要动态检测加载，详细过程见之后代码。
  3. AMediaCodec_dequeueInputBuffer不是每一次都返回成功，如果失败了而丢弃h264数据帧，则最终的播放效果就时不时出现花屏，正确处理见之后代码。
  4. 硬解码器解出的数据默认可能是NV12格式也可能是YV12格式，不同的手机不一样，预先配置解码器的"color-format"为19，也没用。为了与ffmpeg解出的数据一致为YU12格式，统一地我们自已转。

以下是源码，摘选自EuhatRtsp开源软件，经历过长时间拷机运行没问题。其网址是：http://euhat.com/rtsp.html

DecoderHard.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

#pragma once
 
#include "DecodeOp.h"
 
class AMediaCodec;
 
class EuhatDecoderHard : public EuhatDecoderBase
{
    int updateSpsAndPps();
 
    int spsChanged_;
    int ppsChanged_;
    char *sps_;
    char *pps_;
    int width_;
    int height_;
    int oriHeight_;
    int colorFormat_;
    int stride_;
    int sliceHeight_;
    int cropTop_;
    int cropBottom_;
    int cropLeft_;
    int cropRight_;
    int isCodecInited_;
    void *surface_;
    int surfaceChanged_;
 
public:
    EuhatDecoderHard();
    virtual ~EuhatDecoderHard();
 
    virtual int init(EuhatDecoderCallback callback, void *context);
    virtual int fini();
    virtual int updateSps(const char *sps);
    virtual int updatePps(const char *pps);
    virtual int updateWH(int width, int height);
    virtual int updateSurface(void *surface);
    virtual int decode(char *frame, int frameLen);
 
    static int canWork();
 
    AMediaCodec *mediaCodec_;
};

Read more