GStreamer memo

https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html

https://cloud.tencent.com/developer/article/1820526
https://blog.csdn.net/csdnhuaong/article/details/80026433
https://blog.csdn.net/qq_42711516/article/details/123921984
https://blog.csdn.net/han2529386161/article/details/102724856

https://blog.csdn.net/m0_51004308/article/details/121357638
https://blog.csdn.net/Aidam_Bo/article/details/109772430

gstreamer之RTSP Server一个进程提供多路不同视频
https://blog.51cto.com/u_13161667/3310768

全网首发:gstreamer如何接入RTSP流(IP摄像头)的代码范例
https://blog.51cto.com/u_13161667/3310521

GstMP4Mux caps
DeepStream/Gstreamer queue实现预缓存功能
https://blog.csdn.net/qq_41632852/article/details/124959601

gstreamer中tee如何实现动态增减支路(预览+截图+录像)
https://blog.csdn.net/qq_41563600/article/details/121343927

【gstreamer opencv::Mat】将opencv的cv::Mat数据转换成MP4视频
https://blog.csdn.net/weixin_44495869/article/details/121900517

#播放并存储为h264
gst-launch-1.0 -e -v  udpsrc port=10001 !  "application/x-rtp, media=video, clock-rate=90000, encoding-name=H264"  ! rtph264depay ! tee name=t  t. ! queue ! h264parse ! "video/x-h264, stream-format=byte-stream" ! filesink location=./aa.h264 t. !  queue ! avdec_h264 ! glimagesink

#混流+缩放
./gst-launch-1.0  intervideosrc !  video/x-raw,framerate=(fraction)10/1,width=3200,height=2400 ! videomixer name=mix sink_1::xpos=0 sink_2::xpos=1000 sink_2::width=100 sink_2::height=100 ! videoconvert ! glimagesink filesrc location=./ttt.mov ! decodebin ! video/x-raw,width=1280,height=720 ! videoscale ! video/x-raw,width=100  ! mix.  videotestsrc ! video/x-raw ! mix.

gst-launch-1.0.exe videotestsrc ! queue ! d3d11videosink

http://t.zoukankan.com/missmzt-p-10918216.html

GStreamer插件:appsrc 解决播放实时视频流高延迟问题
https://blog.csdn.net/zzs0829/article/details/111562443

https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html

g_setenv("GST_DEBUG_DUMP_DOT_DIR", "D:/work/gstreamer/gstdot", true);
gst_init(NULL, NULL);

GST_DEBUG_BIN_TO_DOT_FILE(GST_BIN(pipeline), GST_DEBUG_GRAPH_SHOW_ALL, "capture1234");

g_setenv("GST_DEBUG_FILE", "D:/work/gstreamer/gstdot/output.log", true);
g_setenv("G_MESSAGES_DEBUG","all",TRUE);
gst_debug_set_default_threshold(GST_LEVEL_TRACE);

http://www.graphviz.org/doc/info/command.html

1
2
3
4
push
	upstream element calls downstream sink pads' gst_pad_push
pull
	downstream element calls upstream source pads' gst_pad_pull_range

gst_rtspsrc_loop_interleaved
gst_rtspsrc_stream_configure_manager
gst_qt_mux_sink_event_pre_queue

rtp_session_process_sr
https://stackoverflow.com/questions/53169699/gstreamer-calculate-delay-in-received-video-frames-buffers-to-detect-communicat

Deep learning memo

yolo pdf,

EfficientDet	https://arxiv.org/pdf/2011.08036.pdf
YOLOv1		https://arxiv.org/pdf/1506.02640.pdf
YOLOv2		https://arxiv.org/pdf/1612.08242.pdf
YOLOv3		https://arxiv.org/pdf/2011.08036.pdf
		https://pjreddie.com/media/files/papers/YOLOv3.pdf
YOLOv4		https://github.com/AlexeyAB/darknet
YOLOv4-Scaled	https://github.com/WongKinYiu/ScaledYOLOv4
YOLO-PPv2	https://arxiv.org/pdf/2104.10419.pdf
YOLOv5		https://arxiv.org/pdf/2104.10419.pdf
YOLOX		https://github.com/Megvii-BaseDetection/YOLOX
YOLOR		https://github.com/WongKinYiu/yolor
YOLOF		https://arxiv.org/pdf/2103.09460.pdf
YOLOS		https://arxiv.org/pdf/2106.00666.pdf
YOLOP		https://arxiv.org/pdf/2108.11250.pdf
YOLOV6		https://mp.weixin.qq.com/s/RrQCP4pTSwpTmSgvly9evg
		https://github.com/meituan/YOLOv6 
		https://zhuanlan.zhihu.com/p/353697121 
 
YOLOV7		https://arxiv.org/abs/2207.02696 
		https://github.com/WongKinYiu/yolov7

https://github.com/yael-vinker/CLIPasso
https://github.com/openai/CLIP

refer to:
AI有啥用
https://blog.csdn.net/PercentageC/article/details/126744534

Speed up pip install

1
2
3
4
5
6
7
8
9
python3 -m pip install opencv-python==4.5.3.56 -i https://pypi.tuna.tsinghua.edu.cn/simple
 
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
 
http://mirrors.aliyun.com/pypi/simple/ 
https://pypi.mirrors.ustc.edu.cn/simple/ 
https://pypi.douban.com/simple/ 
https://pypi.tuna.tsinghua.edu.cn/simple/ 
http://pypi.mirrors.ustc.edu.cn/simple/

refer to:
https://www.cnblogs.com/huangguifeng/p/12002049.html

pragma pack vs. cplusplus align

TAlignedBytes definition is picked from ue4,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
typedef unsigned char uint8;
typedef int int32;
typedef unsigned int uint32;
 
template<int32 Size, uint32 Alignment>
struct TAlignedBytes; // this intentionally won't compile, we don't support the requested alignment
 
/** Unaligned storage. */
template<int32 Size>
struct TAlignedBytes<Size, 1>
{
	uint8 Pad[Size];
};
 
#define GCC_ALIGN(_align)
#define MS_ALIGN(n) __declspec(align(n))
 
// C++/CLI doesn't support alignment of native types in managed code, so we enforce that the element
// size is a multiple of the desired alignment
#ifdef __cplusplus_cli
#define IMPLEMENT_ALIGNED_STORAGE(Align) \
		template<int32 Size>        \
		struct TAlignedBytes<Size,Align> \
		{ \
			uint8 Pad[Size]; \
			static_assert(Size % Align == 0, "CLR interop types must not be aligned."); \
		};
#else
/** A macro that implements TAlignedBytes for a specific alignment. */
#define IMPLEMENT_ALIGNED_STORAGE(Align) \
	template<int32 Size>        \
	struct TAlignedBytes<Size,Align> \
	{ \
		struct MS_ALIGN(Align) TPadding \
		{ \
			uint8 Pad[Size]; \
		} GCC_ALIGN(Align); \
		TPadding Padding; \
	};
#endif
 
// Implement TAlignedBytes for these alignments.
IMPLEMENT_ALIGNED_STORAGE(16);
IMPLEMENT_ALIGNED_STORAGE(8);
IMPLEMENT_ALIGNED_STORAGE(4);
IMPLEMENT_ALIGNED_STORAGE(2);
 
#undef IMPLEMENT_ALIGNED_STORAGE

My test code is in vc2019 x64,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
struct TTask
{
	char ch2;
	char ch3;
	double d4;
	__int64 i64;
};
 
#pragma pack(push, 1)
struct TTaskB
{
	char ch2;
	alignas(16) char ch3;
	double d4;
	__int64 i64;
};
struct AAa
{
	char ch1;
	TAlignedBytes<sizeof(TTask), alignof(TTask)> TaskStorage;
};
struct AAb
{
	char ch1;
	TTask TaskStorage;
};
#pragma pack(pop)
 
void CMFCApplication1Dlg::OnBnClickedButton1()
{
	__int64 sizeTTask = alignof(TTask);			// sizeTTask is 8
	__int64 sizeTTaskB = alignof(TTaskB);			// sizeTTaskB is 16
 
	AAa aa;
	TTask& Task = *(TTask*)&aa.TaskStorage;
	__int64 gapAAa = (char*)&Task - (char*)&aa;		// gapAAa is 8
 
	AAb bb;
	TTask& TaskB = *(TTask*)&bb.TaskStorage;
	__int64 gapAAb = (char*)&TaskB - (char*)&bb;		// gapAAb is 1
}

Let cpu fans be quiet when UE4 Editor is in front

Two methods to solve this,

  • Modify source code,
    D:\ue\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsPlatformApplicationMisc.cpp

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    
    #include <io.h>
    int doesFileExist(const char* filePath)
    {
    	int result = _access(filePath, 0);
    	if (result < 0)
    		return 0;
    	return 1;
    }
     
    bool FWindowsPlatformApplicationMisc::IsThisApplicationForeground()
    {
    	if (doesFileExist("d:/ue/not_focus.txt"))
    	{
    		return false;
    	}
     
    	uint32 ForegroundProcess;
    	::GetWindowThreadProcessId(GetForegroundWindow(), (::DWORD *)&ForegroundProcess);
    	return (ForegroundProcess == GetCurrentProcessId());
    }

    If d:\ue\not_focus.txt exists, whether UE4 Editor is active, UE4 will not occupy a cpu kernel thoroughly.

  • In D:\ue\Engine\Config\ConsoleVariables.ini, or in console, type

    1
    
    t.MaxFPS 5

refer to:
なんとなく日誌

Navigate to source location by writing code

UE4 has a functionality of "Goto Definition",

1
FSourceCodeNavigationImpl::NavigateToFunctionSource

which uses two apis from dbghelp.dll,

1
2
SymGetSymFromName64
SymGetLineFromAddr64

Chrome memo

To prevent inactive tab from using cpu resources, in address bar, visit

1
chrome://discards

toggle restless pages 'Urgent Discard' to grey under 'Actions' column.

chrome://flags#

enable-parallel-downloading
enable-force-dark

refer to:
https://www.askvg.com/tip-enable-tab-freeze-or-tab-suspend-feature-in-google-chrome/
https://blog.csdn.net/ythuiyi/article/details/113506238

UE4 blueprint memo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
Level Viewport
	bp_test
		Details
			Actor
				Tags
	Box Trigger
		Actor Hidden in Game
 
 
Level Blueprint
	1
		DestroyActor
	Event Tick
		Get All Actors with Tag
			Get Actor Location
			Set Actor Location
	Add Event for Trigger Box 1
		OnActorBeginOverlap
			cast to ThirdPersonCharacter
				begin_event
		OnActorEndOverlap
 
bp_test: Event Graph
	Event BeginPlay
		Exec
			Sequence
				Delay
					Print String
				Set Timer by Event
					Event
						test_event
					Return Value
						Promote to variable
							New Var 1
			If
				New Var 0
					true
						Print String
					false
						Print String
	Event Destroyed
	4
		New Var 1
			Clear and Invalidate Timer by Handle
	5
		Array Var 2
			For Each Loop
			Reverse for Each Loop
	6
		Struct Var 3 : Struct Type 3
			Break Struct Type 3
	7
		Enum Var 4 : Enum Type 4
			Switch on Enum Type 4
	Add Timeline
		Lerp Vector
			Set World Location
 
	test_event: Custom Event
		Exec
			Draw Debug Point
ThirdPersonCharacter : Event Graph
	begin_event: Custom Event
	Event Dispatchers
		hitDemon
	1
		hitDemon
Demon1 : Event Graph
	Get Player Character
		cast to ThirdPersonCharacter
			bind event to hitDemon
				add custom event
Demon1
	Details
		Collision
			Collision Presets
				Custom...
ThirdPersonCharacter : Blueprint
	Components
		ThirdPersonCharacter
			Use Controller Rotation Yaw
		CameraBoom
			Use Pawn Control Rotation
		CharacterMovement
			Orient Rotation to Movement
	1
		Capsule Component
			Get Word Location as Start
			Get Forward Vector
				x 10000
					+ Get World Location
						as End
							Line Trace By Channel
Get Player Controller
	Show Mouse Cursor
	ConvertMouseLocationToWorldSpace
 
Level Blueprint
	Q
		CubeMesh
			Static Mesh Component
				Create Dynamic Material Instance
					Set Vector Parameter Value
						Parameter Name : cube_color4
 
Mat_cube asigned to CubeMesh
	cube_color4 : Constant3Vector
		Base Color

In Actor Blueprint, enable input event,

1
2
3
4
bp_test: Event Graph
	Event BeginPlay
		Get Player Controller
			Enable Input

Questions,

  • Why can't 'Delay' function be used in 'While Loop'?
  • In Game mode, can't control previous character while incarnating into a new character?

    1
    2
    3
    4
    5
    
    ThirdPersonCharactor1
    	Details
    		Pawn
    			Auto Possess Player
    				Player 0

refer to:
我是一只好蛋YEAH
https://blog.csdn.net/sunday7279/article/details/88665577

Rendering mp4 video not silky, why?

The original HD mp4 file 'movie.mp4' plays very smoothly. I picked out the h264 stream saved as 'movie.264' from it, then using mp4v2 converted 'movie.264' to a new mp4 file 'output.mp4'. However, experience of watching 'output.mp4' was very bad. Why?

I debugged the 'ClipTrack' function in 'mp4v2/test/OLD/mp4clip.cpp', found a key argument 'renderingOffset' of function 'MP4WriteSample'. All examples searched from the web about writing h264 to mp4 through mp4v2 set 'renderingOffset' to 0, while in 'ClipTrack', 'renderingOffset' of the function 'MP4WriteSample' is directly set to the value from 'renderingOffset' of the function 'MP4ReadSample'.

Here we make a concept clear, the 'sample' entity in mp4 file format specification is equal to a frame which can completely rendering as a picture. A sample consists of several nalus, a nalu whose start code is 0x00, 0x00, 0x00, 0x01 indicates a new sample, while a nalu which starts with 0x00, 0x00, 0x01 is among a sample, and a sample needs not to be a IDR key frame.

Through carrying 'renderingOffset' value at beginning of each sample in 'movie.264', I did generate an 'output.mp4' which displays as smoothly as the original one.

Did 'renderingOffset' mean time stamp of a sample? No, at least not exactly. By watching log, I found its value was not increasing as time went by.

I continued debugging mp4v2, then saw a key word 'ctts' atom, and searched a link from the web,

https://blog.csdn.net/cheyo809775692/article/details/100924523

I now understand, the decoding order of h264 samples is slightly different from the rendering order of them.

For example, in stts atom,

1
2
3
4
5
'decoding order'	'sample count'		'sample duration'
1,2,3,4			4			20
5			1			40
6,7,8			3			20
9			1			60

in ctts atom,

1
2
3
4
5
6
7
8
'decoding order'	'sample count'		'sample offset'
1,2,3			3			20
4			1			40
5			1			0
6			1			40
7			1			0
8			1			40
9			1			0

final rendering timing is

1
2
3
'decoding order'	1	2	3	4	5	6	7	8	9
'duration'		20	20	20	20	40	20	20	20	60
'rending order'			1	2	3	5	5,4	7	6	9	9,8	9

Terms,

1
2
3
4
5
6
7
8
9
10
11
12
mvhd: movie header
tkhd: track header
mdhd: media header
stsd: sample descriptions
stts: time to sample
ctts: composition offset
stss: sync sample offset
stsc: sample to chunk
stsz: sample size
stco/stco64: chunk offset
dts: decoding timestamp
pts: presentation timestamp