Programming on Kyle's Blog

The Theory of Generalization: From Break Points To The Vc Inequality

Mon, 01 Sep 2025 00:00:00 +0000

The Explanation of Function B

$B(N, k)$ denotes the maximum number of dichotomies on $N$ points, with break point $k$. For example, $B(1000, 4)$ denotes the number of dichotomies that a 2-dimensional perceptron can make on 1000 data points.

Note: A 2-dimensional perceptron has a break point of 4, which means a 2-dimensional perceptron cannot offer all dichotomies on 4 points.

The Recursive Function B

For a dataset with $N$ data points, the total number of possible dichotomies is $2^N$, because each data point has two possible classifications. In this case, the growth rate of the number of dichotomies is an exponential function of $N$. However, due to the existence of a break point, the actual growth rate becomes a polynomial function of $N$, with the degree of the polynomial related to $k$.

Reasoning Process: We divide the $B(N,k)$ dichotomies into two categories based on the following rule: if we ignore the last data point (the $N$-th data point), are the dichotomies of the first $N-1$ data points unique?

Dichotomies that are unique go into Class A.
Dichotomies that are repeated go into Class B.

For example, suppose a dataset has three data points and an algorithm generates four classification methods: 1,0,1, 1,0,0, 0,0,1, 0,1,1. Ignoring the last data point, there are two repeated patterns (1,0) and two unique patterns (0,0 and 0,1).

By definition, Classes A and B are disjoint, and their union is the original set (because each pattern is either unique or repeated).

Let $\alpha$ be the number of dichotomies in Class A.
Since each dichotomy in Class B is repeated exactly twice (the $N$-th data point has two possibilities), let the number of dichotomies in B be $2\beta$.
Thus:

$$ B(N,k) = \alpha + 2\beta $$

Also, $\alpha + \beta$ represents the total number of unique dichotomies for the first $N-1$ data points, so:

$$ \alpha + \beta \le B(N-1,k) $$

The break point of B is $k-1$. This is because if there were $k-1$ columns in B that cover all possible binary patterns, then adding the $N$-th data point would create $k$ columns that cover all possible binary patterns.

For example, if $k=4$, meaning that among all dichotomies, any 4 columns cannot form all binary patterns, and suppose B contains 3 columns:

After adding one more data point, we can find 4 columns that cover all possible binary patterns:

Thus, B cannot have $k-1$ columns covering all binary patterns, meaning the break point is at most $k-1$.

Therefore:

$$ \beta \le B(N-1, k-1) $$

Combining the above, we obtain:

$$ B(N,k) = \alpha + 2\beta \le B(N-1,k) + B(N-1, k-1) $$

The Implementation of Recursive Function B

Show/Hide the code

1
2
3
4
5
6
7
8
9


def B(N, k):

 if k <= 0 or N <= 0:
 return 0
 if k == 1:
 return 1
 if N == 1:
 return 2
 return B(N - 1, k) + B(N - 1, k - 1)

The Table of Function B

Show/Hide the code

1
2
3
4
5
6
7
8
9


print("\t", end="")
for i in range(1, 11):
 print(f"k={i}", end="\t")
print("\n")
for i in range(1, 11):
 print(f"N={i}", end='\t')
 for j in range(1, 11):
 print(B(i, j), end="\t")
 print("\n")

 k=1 k=2 k=3 k=4 k=5 k=6 k=7 k=8 k=9 k=10 

N=1 1 2 2 2 2 2 2 2 2 2 

N=2 1 3 4 4 4 4 4 4 4 4 

N=3 1 4 7 8 8 8 8 8 8 8 

N=4 1 5 11 15 16 16 16 16 16 16 

N=5 1 6 16 26 31 32 32 32 32 32 

N=6 1 7 22 42 57 63 64 64 64 64 

N=7 1 8 29 64 99 120 127 128 128 128 

N=8 1 9 37 93 163 219 247 255 256 256 

N=9 1 10 46 130 256 382 466 502 511 512 

N=10 1 11 56 176 386 638 848 968 1013 1023

A Theorem

$$ B(N,k) \le \sum_{i=0}^{k-1}{\binom{N}{i}} $$

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


import math
import numpy as np


def binomial(N, i):
 if N < i:
 return 0
 return int(math.factorial(N) / (math.factorial(i) * math.factorial(N - i)))


def B(N, k):
 return np.sum([binomial(N, i) for i in range(0, k)])

Goal:

$$ \sum_{i=0}^{k-1}{\binom{N}{i}}=\sum_{i=0}^{k-1}{\binom{N-1}{i}}+\sum_{i=0}^{k-2}{\binom{N-1}{i}} $$

Steps:

$$ \begin{align} &\sum_{i=0}^{k-1}{\binom{N-1}{i}}+\sum_{i=0}^{k-2}{\binom{N-1}{i}} \nonumber \\ &=1 + \sum_{i=1}^{k-1}{\binom{N-1}{i}}+\sum_{i=1}^{k-2}{\binom{N-1}{i-1}} \nonumber \\ &=1 + \sum_{i=1}^{k-1}{\left[\binom{N-1}{i}+\binom{N-1}{i-1}\right]} \qquad\text{(1)} \nonumber \\ &=1 + \sum_{i=1}^{k-1}{\binom{N}{i}} \qquad\text{(2)} \nonumber \\ &=\sum_{i=0}^{k-1}{\binom{N}{i}} \nonumber \end{align} $$

Note: An intuitive way to understand how (1) becomes to (2) is to consider the essence of the combination formula. The term $\binom{N}{i}$ represents the number of ways to choose $i$ people from a total of $N$. There are two mutually exclusive scenarios for any particular person: either they are chosen, or they are not.

If the person is not chosen, then we need to select all $i$ people from the remaining $N-1$, giving $\binom{N-1}{i}$ combinations. If the person is chosen, we have already selected one, so we only need to choose the remaining $i-1$ people from the other $N-1$, resulting in $\binom{N-1}{i-1}$ combinations. Together, these two cases account for all possible selections.

自动切换 SumatraPDF 主题

Tue, 15 Jul 2025 17:01:11 +0800

SumatraPDF 小巧，快速，对于浏览 PDF 来说再合适不过。唯一的缺点就是功能有些许缺失，例如：虽然有更换主题功能，但是不能跟随 Windows 主题一起变化，并且界面颜色改变时，文档的颜色不改变。于是乎每当在夜晚打开一份 PDF 都仿佛打开了太阳……

微软的应用商店 Microsoft Store 中有一款软件 Auto Dark Mode 可以自动切换 Windows 的暗色模式，并且支持在切换时运行脚本。正巧 SumatraPDF 可以通过配置文件配置，实际上对于一些高级设置（例如字体颜色和背景颜色）只支持使用配置文件设置。因此一个简单的方案就是：编写一个脚本根据参数不同自动修改 SumatraPDF 的配置文件，并且设置 Auto Dark Mode 使其在切换 Windows 主题时自动运行此脚本。

Auto Dark Mode 配置：

Show/Hide the code

1
2
3
4
5
6
7
8
9


Enabled: true
Component:
 Scripts:
 - Name: SumatraPDF
 Command: python
 WorkingDirectory: C:\Users\kyle\AppData\Local\SumatraPDF
 ArgsLight: [toggle_theme.py, light]
 ArgsDark: [toggle_theme.py, dark]
 AllowedSources: [Any]

Python 脚本：

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


import sys
import os
import re

if len(sys.argv) != 2 or sys.argv[1] not in ['light', 'dark']:
 print("Usage: python toggle_sumatra_theme.py [light|dark]")
 sys.exit(1)

mode = sys.argv[1]

config_path = os.path.join(os.getcwd(), "SumatraPDF-settings.txt")

if not os.path.isfile(config_path):
 print(f"Error: Config file not found: {config_path}")
 sys.exit(1)

if mode == 'light':
 theme_val = 'Light'
 text_color = '#262626'
 background_color = '#f8f3e9'
else:
 theme_val = 'Darker'
 text_color = '#bbbbbb'
 background_color = '#17191a'

with open(config_path, 'r', encoding='utf-8') as f:
 config = f.read()

config = re.sub(r'Theme\s*=\s*\w+', f'Theme = {theme_val}', config)
config = re.sub(r'TextColor\s*=\s*#[0-9a-fA-F]+', f'TextColor = {text_color}', config)
config = re.sub(r'BackgroundColor\s*=\s*#[0-9a-fA-F]+', f'BackgroundColor = {background_color}', config)

with open(config_path, 'w', encoding='utf-8') as f:
 f.write(config)

Convert All Mp4 to Mp3 in One Command

Fri, 13 Jun 2025 12:37:35 +0800

I get a directory like following:

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


 .
 ├── S01
 │   ├── TBBTS01E01.mp4
 │   ├── TBBTS01E01.mp4
 │   ├── TBBTS01E01.mp4
 │   ...
 ├── S02
 │   ├── TBBTS02E01.mp4
 │   ├── TBBTS02E02.mp4
 │   ├── TBBTS02E03.mp4
 │   ...
 ├── S03
 │   ├── TBBTS03E01.mp4
 │   ├── TBBTS03E02.mp4
 │   ├── TBBTS03E03.mp4
 │   ...
 ├── S04
 │   ├── TBBTS04E01.mp4
 │   ├── TBBTS04E02.mp4
 │   ├── TBBTS04E03.mp4
 │   ...
 ├── S05
 │   ├── TBBTS05E01.mp4
 │   ├── TBBTS05E02.mp4
 │   ├── TBBTS05E03.mp4
 │   ├── TBBTS05E04.mp4
 │  ...
 └── S06
	 ├── TBBTS06E01.mp4
	 ├── TBBTS06E02.mp4
	 ├── TBBTS06E03.mp4
	 ...

I want to transform all mp4 files to mp3 and move them to a new directory called mp3 in current directory. here is the command:

Show/Hide the code

1

 find . -name "*.mp4" -type f -print | parallel -j20 "ffmpeg -i {} ./mp3/{/.}.mp3"

where {/.} represents a clear file name that stripped extension name and path

Convert Jupyter Notebook to PDF With Reporting Format

Fri, 13 Jun 2025 00:00:00 +0000

Install nbconvert via pip

Show/Hide the code

1

pip install nbconvert

Now, the command

Show/Hide the code

1

jupyter nbconvert --to pdf --template-file report filename.ipynb

should work well, producing a document with a table of contents and properly formatted code blocks. However, an unnecessary counter appears before each code block, wasting space and serving no purpose. To remove it, a custom template is required.

The template consists of two files: assignment.tex.j2 and style_assignment.tex.j2, which are modified duplicates of existing files. Template files are located in share\jupyter\nbconvert\templates\latex. Duplicate report.tex.j2 and modify the second line to

Show/Hide the code

1

((* set cell_style = 'style_assignment.tex.j2' *))

The file style_assignment.tex.j2 is a duplicate of style_jupyter.tex.j2 with the line

Show/Hide the code

1

{\ttfamily\llap{{\color{#2}[#3]:\hspace{3pt}#4}}\vspace{-\baselineskip}}

replaced by

Show/Hide the code

1

{\ttfamily\llap{{\color{#2}\hspace{3pt}#4}}}

Moreover, in order to support Chinese, the code below should be added

Show/Hide the code

1
2


\usepackage{xeCJK}
\setCJKmainfont{SimSun}

Now that custom files are used, the command is

Show/Hide the code

1

jupyter nbconvert --to pdf --template-file assignment filename.ipynb

or dictate author names and the document title as converting:

Show/Hide the code

1

jupyter nbconvert --to pdf --template-file assignment ./Journal.ipynb --LatexPreprocessor.author_names="Name" --LatexPreprocessor.title="Title"

The complete file is shown as follows:

assignment.tex.j2

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


% Default to the notebook output style
((* if not cell_style is defined *))
 ((* set cell_style = 'style_assignment.tex.j2' *))
((* endif *))

% Inherit from the specified cell style.
((* extends cell_style *))


%===============================================================================
% Latex Book
%===============================================================================

((* block predoc *))
 ((( super() )))
 ((* block tableofcontents *))\tableofcontents((* endblock tableofcontents *))
((* endblock predoc *))

((* block docclass *))
\documentclass{report}
((* endblock docclass *))

((* block markdowncell scoped *))
((( cell.source | citation2latex | strip_files_prefix | convert_pandoc('markdown+tex_math_double_backslash', 'json',extra_args=[]) | resolve_references | convert_pandoc('json','latex', extra_args=["--top-level-division=chapter"]) )))
((* endblock markdowncell *))

style_assignment.tex.j2

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180


((=- IPython input/output style -=))
((*- extends 'base.tex.j2' -*))

((*- block packages -*))
	\usepackage{xeCJK}
	\setCJKmainfont{SimSun}
 \usepackage[breakable]{tcolorbox}
 \usepackage{parskip} % Stop auto-indenting (to mimic markdown behaviour)
 ((( super() )))
((*- endblock packages -*))

((*- block definitions -*))
 ((( super() )))
% Pygments definitions
 (((- resources.latex.pygments_definitions )))

 % For linebreaks inside Verbatim environment from package fancyvrb.
 \makeatletter
 \newbox\Wrappedcontinuationbox
 \newbox\Wrappedvisiblespacebox
 \newcommand*\Wrappedvisiblespace {\textcolor{red}{\textvisiblespace}}
 \newcommand*\Wrappedcontinuationsymbol {\textcolor{red}{\llap{\tiny$\m@th\hookrightarrow$}}}
 \newcommand*\Wrappedcontinuationindent {3ex }
 \newcommand*\Wrappedafterbreak {\kern\Wrappedcontinuationindent\copy\Wrappedcontinuationbox}
 % Take advantage of the already applied Pygments mark-up to insert
 % potential linebreaks for TeX processing.
 % {, <, #, %, $, ' and ": go to next line.
 % _, }, ^, &, >, - and ~: stay at end of broken line.
 % Use of \textquotesingle for straight quote.
 \newcommand*\Wrappedbreaksatspecials {%
 \def\PYGZus{\discretionary{\char`\_}{\Wrappedafterbreak}{\char`\_}}%
 \def\PYGZob{\discretionary{}{\Wrappedafterbreak\char`\{}{\char`\{}}%
 \def\PYGZcb{\discretionary{\char`\}}{\Wrappedafterbreak}{\char`\}}}%
 \def\PYGZca{\discretionary{\char`\^}{\Wrappedafterbreak}{\char`\^}}%
 \def\PYGZam{\discretionary{\char`\&}{\Wrappedafterbreak}{\char`\&}}%
 \def\PYGZlt{\discretionary{}{\Wrappedafterbreak\char`\<}{\char`\<}}%
 \def\PYGZgt{\discretionary{\char`\>}{\Wrappedafterbreak}{\char`\>}}%
 \def\PYGZsh{\discretionary{}{\Wrappedafterbreak\char`\#}{\char`\#}}%
 \def\PYGZpc{\discretionary{}{\Wrappedafterbreak\char`\%}{\char`\%}}%
 \def\PYGZdl{\discretionary{}{\Wrappedafterbreak\char`\$}{\char`\$}}%
 \def\PYGZhy{\discretionary{\char`\-}{\Wrappedafterbreak}{\char`\-}}%
 \def\PYGZsq{\discretionary{}{\Wrappedafterbreak\textquotesingle}{\textquotesingle}}%
 \def\PYGZdq{\discretionary{}{\Wrappedafterbreak\char`\"}{\char`\"}}%
 \def\PYGZti{\discretionary{\char`\~}{\Wrappedafterbreak}{\char`\~}}%
 }
 % Some characters . , ; ? ! / are not pygmentized.
 % This macro makes them "active" and they will insert potential linebreaks
 \newcommand*\Wrappedbreaksatpunct {%
 \lccode`\~`\.\lowercase{\def~}{\discretionary{\hbox{\char`\.}}{\Wrappedafterbreak}{\hbox{\char`\.}}}%
 \lccode`\~`\,\lowercase{\def~}{\discretionary{\hbox{\char`\,}}{\Wrappedafterbreak}{\hbox{\char`\,}}}%
 \lccode`\~`\;\lowercase{\def~}{\discretionary{\hbox{\char`\;}}{\Wrappedafterbreak}{\hbox{\char`\;}}}%
 \lccode`\~`\:\lowercase{\def~}{\discretionary{\hbox{\char`\:}}{\Wrappedafterbreak}{\hbox{\char`\:}}}%
 \lccode`\~`\?\lowercase{\def~}{\discretionary{\hbox{\char`\?}}{\Wrappedafterbreak}{\hbox{\char`\?}}}%
 \lccode`\~`\!\lowercase{\def~}{\discretionary{\hbox{\char`\!}}{\Wrappedafterbreak}{\hbox{\char`\!}}}%
 \lccode`\~`\/\lowercase{\def~}{\discretionary{\hbox{\char`\/}}{\Wrappedafterbreak}{\hbox{\char`\/}}}%
 \catcode`\.\active
 \catcode`\,\active
 \catcode`\;\active
 \catcode`\:\active
 \catcode`\?\active
 \catcode`\!\active
 \catcode`\/\active
 \lccode`\~`\~
 }
 \makeatother

 \let\OriginalVerbatim=\Verbatim
 \makeatletter
 \renewcommand{\Verbatim}[1][1]{%
 %\parskip\z@skip
 \sbox\Wrappedcontinuationbox {\Wrappedcontinuationsymbol}%
 \sbox\Wrappedvisiblespacebox {\FV@SetupFont\Wrappedvisiblespace}%
 \def\FancyVerbFormatLine ##1{\hsize\linewidth
 \vtop{\raggedright\hyphenpenalty\z@\exhyphenpenalty\z@
 \doublehyphendemerits\z@\finalhyphendemerits\z@
 \strut ##1\strut}%
 }%
 % If the linebreak is at a space, the latter will be displayed as visible
 % space at end of first line, and a continuation symbol starts next line.
 % Stretch/shrink are however usually zero for typewriter font.
 \def\FV@Space {%
 \nobreak\hskip\z@ plus\fontdimen3\font minus\fontdimen4\font
 \discretionary{\copy\Wrappedvisiblespacebox}{\Wrappedafterbreak}
 {\kern\fontdimen2\font}%
 }%

 % Allow breaks at special characters using \PYG... macros.
 \Wrappedbreaksatspecials
 % Breaks at punctuation characters . , ; ? ! and / need catcode=\active
 \OriginalVerbatim[#1,codes*=\Wrappedbreaksatpunct]%
 }
 \makeatother

 % Exact colors from NB
 ((*- block style_colors *))
 \definecolor{incolor}{HTML}{303F9F}
 \definecolor{outcolor}{HTML}{D84315}
 \definecolor{cellborder}{HTML}{CFCFCF}
 \definecolor{cellbackground}{HTML}{F7F7F7}
 ((*- endblock style_colors *))

 % prompt
 \makeatletter
 \newcommand{\boxspacing}{\kern\kvtcb@left@rule\kern\kvtcb@boxsep}
 \makeatother
 ((*- block style_prompt *))
 \newcommand{\prompt}[4]{
 {\ttfamily\llap{{\color{#2}\hspace{3pt}#4}}}
 }
 ((* endblock style_prompt *))

((*- endblock definitions -*))

%===============================================================================
% Input
%===============================================================================

((* block input scoped *))
 ((( draw_cell(cell.source | highlight_code(strip_verbatim=True), cell, 'In', 'incolor', '\\boxspacing') )))
((* endblock input *))


%===============================================================================
% Output
%===============================================================================

((*- if charlim is not defined -*))
 ((* set charlim = 80 *))
((*- endif -*))

((* block execute_result scoped *))
 ((*- for type in output.data | filter_data_type -*))
 ((*- if type in ['text/plain']*))
 ((( draw_cell(output.data['text/plain'] | wrap_text(charlim) | escape_latex | ansi2latex, cell, 'Out', 'outcolor', '\\boxspacing') )))
 ((* else -*))
 ((( " " )))
 ((( draw_prompt(cell, 'Out', 'outcolor','') )))((( super() )))
 ((*- endif -*))
 ((*- endfor -*))
((* endblock execute_result *))

((* block stream *))
 \begin{Verbatim}[commandchars=\\\{\}]
((( output.text | wrap_text(charlim) | escape_latex | strip_trailing_newline | ansi2latex )))
 \end{Verbatim}
((* endblock stream *))

%==============================================================================
% Support Macros
%==============================================================================

% Name: draw_cell
% Purpose: Renders an output/input prompt
((*- if draw_cell is not defined -*)) % Required to allow overriding.
((* macro draw_cell(text, cell, prompt, prompt_color, extra_space) -*))
((*- if prompt == 'In' -*))
((*- set style = "breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder"-*))
((*- else -*))((*- set style = "breakable, size=fbox, boxrule=.5pt, pad at break*=1mm, opacityfill=0"-*))((*- endif -*))

\begin{tcolorbox}[((( style )))]
(((- draw_prompt(cell, prompt, prompt_color, extra_space) )))
\begin{Verbatim}[commandchars=\\\{\}]
((( text )))
\end{Verbatim}
\end{tcolorbox}
((*- endmacro *))
((*- endif -*))

% Name: draw_prompt
% Purpose: Renders an output/input prompt
((* macro draw_prompt(cell, prompt, prompt_color, extra_space) -*))
 ((*- if cell.execution_count is defined -*))
 ((*- set execution_count = "" ~ (cell.execution_count | replace(None, " ")) -*))
 ((*- else -*))((*- set execution_count = " " -*))((*- endif *))

 ((*- if (resources.global_content_filter.include_output_prompt and prompt == 'Out')
 or (resources.global_content_filter.include_input_prompt and prompt == 'In' ) *))
\prompt{(((prompt)))}{(((prompt_color)))}{(((execution_count)))}{(((extra_space)))}
 ((*- endif -*))
((*- endmacro *))

Furthermore, if you want to hide code:

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181


((=- IPython input/output style -=))
((*- extends 'base.tex.j2' -*))

((*- block packages -*))
	\usepackage{xeCJK}
	\setCJKmainfont{SimSun}
 \usepackage[breakable]{tcolorbox}
 \usepackage{parskip} % Stop auto-indenting (to mimic markdown behaviour)
 ((( super() )))
((*- endblock packages -*))

((*- block definitions -*))
 ((( super() )))
% Pygments definitions
 (((- resources.latex.pygments_definitions )))

 % For linebreaks inside Verbatim environment from package fancyvrb.
 \makeatletter
 \newbox\Wrappedcontinuationbox
 \newbox\Wrappedvisiblespacebox
 \newcommand*\Wrappedvisiblespace {\textcolor{red}{\textvisiblespace}}
 \newcommand*\Wrappedcontinuationsymbol {\textcolor{red}{\llap{\tiny$\m@th\hookrightarrow$}}}
 \newcommand*\Wrappedcontinuationindent {3ex }
 \newcommand*\Wrappedafterbreak {\kern\Wrappedcontinuationindent\copy\Wrappedcontinuationbox}
 % Take advantage of the already applied Pygments mark-up to insert
 % potential linebreaks for TeX processing.
 % {, <, #, %, $, ' and ": go to next line.
 % _, }, ^, &, >, - and ~: stay at end of broken line.
 % Use of \textquotesingle for straight quote.
 \newcommand*\Wrappedbreaksatspecials {%
 \def\PYGZus{\discretionary{\char`\_}{\Wrappedafterbreak}{\char`\_}}%
 \def\PYGZob{\discretionary{}{\Wrappedafterbreak\char`\{}{\char`\{}}%
 \def\PYGZcb{\discretionary{\char`\}}{\Wrappedafterbreak}{\char`\}}}%
 \def\PYGZca{\discretionary{\char`\^}{\Wrappedafterbreak}{\char`\^}}%
 \def\PYGZam{\discretionary{\char`\&}{\Wrappedafterbreak}{\char`\&}}%
 \def\PYGZlt{\discretionary{}{\Wrappedafterbreak\char`\<}{\char`\<}}%
 \def\PYGZgt{\discretionary{\char`\>}{\Wrappedafterbreak}{\char`\>}}%
 \def\PYGZsh{\discretionary{}{\Wrappedafterbreak\char`\#}{\char`\#}}%
 \def\PYGZpc{\discretionary{}{\Wrappedafterbreak\char`\%}{\char`\%}}%
 \def\PYGZdl{\discretionary{}{\Wrappedafterbreak\char`\$}{\char`\$}}%
 \def\PYGZhy{\discretionary{\char`\-}{\Wrappedafterbreak}{\char`\-}}%
 \def\PYGZsq{\discretionary{}{\Wrappedafterbreak\textquotesingle}{\textquotesingle}}%
 \def\PYGZdq{\discretionary{}{\Wrappedafterbreak\char`\"}{\char`\"}}%
 \def\PYGZti{\discretionary{\char`\~}{\Wrappedafterbreak}{\char`\~}}%
 }
 % Some characters . , ; ? ! / are not pygmentized.
 % This macro makes them "active" and they will insert potential linebreaks
 \newcommand*\Wrappedbreaksatpunct {%
 \lccode`\~`\.\lowercase{\def~}{\discretionary{\hbox{\char`\.}}{\Wrappedafterbreak}{\hbox{\char`\.}}}%
 \lccode`\~`\,\lowercase{\def~}{\discretionary{\hbox{\char`\,}}{\Wrappedafterbreak}{\hbox{\char`\,}}}%
 \lccode`\~`\;\lowercase{\def~}{\discretionary{\hbox{\char`\;}}{\Wrappedafterbreak}{\hbox{\char`\;}}}%
 \lccode`\~`\:\lowercase{\def~}{\discretionary{\hbox{\char`\:}}{\Wrappedafterbreak}{\hbox{\char`\:}}}%
 \lccode`\~`\?\lowercase{\def~}{\discretionary{\hbox{\char`\?}}{\Wrappedafterbreak}{\hbox{\char`\?}}}%
 \lccode`\~`\!\lowercase{\def~}{\discretionary{\hbox{\char`\!}}{\Wrappedafterbreak}{\hbox{\char`\!}}}%
 \lccode`\~`\/\lowercase{\def~}{\discretionary{\hbox{\char`\/}}{\Wrappedafterbreak}{\hbox{\char`\/}}}%
 \catcode`\.\active
 \catcode`\,\active
 \catcode`\;\active
 \catcode`\:\active
 \catcode`\?\active
 \catcode`\!\active
 \catcode`\/\active
 \lccode`\~`\~
 }
 \makeatother

 \let\OriginalVerbatim=\Verbatim
 \makeatletter
 \renewcommand{\Verbatim}[1][1]{%
 %\parskip\z@skip
 \sbox\Wrappedcontinuationbox {\Wrappedcontinuationsymbol}%
 \sbox\Wrappedvisiblespacebox {\FV@SetupFont\Wrappedvisiblespace}%
 \def\FancyVerbFormatLine ##1{\hsize\linewidth
 \vtop{\raggedright\hyphenpenalty\z@\exhyphenpenalty\z@
 \doublehyphendemerits\z@\finalhyphendemerits\z@
 \strut ##1\strut}%
 }%
 % If the linebreak is at a space, the latter will be displayed as visible
 % space at end of first line, and a continuation symbol starts next line.
 % Stretch/shrink are however usually zero for typewriter font.
 \def\FV@Space {%
 \nobreak\hskip\z@ plus\fontdimen3\font minus\fontdimen4\font
 \discretionary{\copy\Wrappedvisiblespacebox}{\Wrappedafterbreak}
 {\kern\fontdimen2\font}%
 }%

 % Allow breaks at special characters using \PYG... macros.
 \Wrappedbreaksatspecials
 % Breaks at punctuation characters . , ; ? ! and / need catcode=\active
 \OriginalVerbatim[#1,codes*=\Wrappedbreaksatpunct]%
 }
 \makeatother

 % Exact colors from NB
 ((*- block style_colors *))
 \definecolor{incolor}{HTML}{303F9F}
 \definecolor{outcolor}{HTML}{D84315}
 \definecolor{cellborder}{HTML}{CFCFCF}
 \definecolor{cellbackground}{HTML}{F7F7F7}
 ((*- endblock style_colors *))

 % prompt
 \makeatletter
 \newcommand{\boxspacing}{\kern\kvtcb@left@rule\kern\kvtcb@boxsep}
 \makeatother
 ((*- block style_prompt *))
 \newcommand{\prompt}[4]{
 {\ttfamily\llap{{\color{#2}\hspace{3pt}#4}}}
 }
 ((* endblock style_prompt *))

((*- endblock definitions -*))

%===============================================================================
% Input
%===============================================================================

((* block input scoped *))
 % input hidden
((* endblock input *))



%===============================================================================
% Output
%===============================================================================

((*- if charlim is not defined -*))
 ((* set charlim = 80 *))
((*- endif -*))

((* block execute_result scoped *))
 ((*- for type in output.data | filter_data_type -*))
 ((*- if type in ['text/plain']*))
 ((( draw_cell(output.data['text/plain'] | wrap_text(charlim) | escape_latex | ansi2latex, cell, 'Out', 'outcolor', '\\boxspacing') )))
 ((* else -*))
 ((( " " )))
 ((( draw_prompt(cell, 'Out', 'outcolor','') )))((( super() )))
 ((*- endif -*))
 ((*- endfor -*))
((* endblock execute_result *))

((* block stream *))
 \begin{Verbatim}[commandchars=\\\{\}]
((( output.text | wrap_text(charlim) | escape_latex | strip_trailing_newline | ansi2latex )))
 \end{Verbatim}
((* endblock stream *))

%==============================================================================
% Support Macros
%==============================================================================

% Name: draw_cell
% Purpose: Renders an output/input prompt
((*- if draw_cell is not defined -*)) % Required to allow overriding.
((* macro draw_cell(text, cell, prompt, prompt_color, extra_space) -*))
((*- if prompt == 'In' -*))
((*- set style = "breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder"-*))
((*- else -*))((*- set style = "breakable, size=fbox, boxrule=.5pt, pad at break*=1mm, opacityfill=0"-*))((*- endif -*))

\begin{tcolorbox}[((( style )))]
(((- draw_prompt(cell, prompt, prompt_color, extra_space) )))
\begin{Verbatim}[commandchars=\\\{\}]
((( text )))
\end{Verbatim}
\end{tcolorbox}
((*- endmacro *))
((*- endif -*))

% Name: draw_prompt
% Purpose: Renders an output/input prompt
((* macro draw_prompt(cell, prompt, prompt_color, extra_space) -*))
 ((*- if cell.execution_count is defined -*))
 ((*- set execution_count = "" ~ (cell.execution_count | replace(None, " ")) -*))
 ((*- else -*))((*- set execution_count = " " -*))((*- endif *))

 ((*- if (resources.global_content_filter.include_output_prompt and prompt == 'Out')
 or (resources.global_content_filter.include_input_prompt and prompt == 'In' ) *))
\prompt{(((prompt)))}{(((prompt_color)))}{(((execution_count)))}{(((extra_space)))}
 ((*- endif -*))
((*- endmacro *))

Merge Anki Audios

Fri, 13 Jun 2025 00:00:00 +0000

exporting all selected notes with media references

extract media paths from the exported file
write a python script merging all selected audios into one

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


import os
import subprocess
import pandas as pd
from concurrent.futures import ThreadPoolExecutor, as_completed
from tqdm import tqdm
import random

AUDIO_DIRECTORY = r"C:\Users\kyle\AppData\Roaming\Anki2\Eng\collection.media"
SILENT = r"C:\Users\momo\Home\Projects\Anki\silent.mp3"

list_file_path = "../notes.txt"
df = pd.read_csv(
 list_file_path,
 sep="\t",
 header=None,
 names=[
 "word", "lemma", "context", "sentence", "examples", "symbol", "voc", "def",
 "word_sound", "sentence_sound"
 ],
)

word_audio_names = df["word_sound"].apply(lambda x: x.strip()[7:-1]).tolist()
df["sentence_sound"] = df["sentence_sound"].fillna("")
sentence_audio_names = df["sentence_sound"].apply(lambda x: x.strip()[7:-1]).tolist()

audio_paths = [os.path.join(AUDIO_DIRECTORY, item) for item in word_audio_names] + [
 os.path.join(AUDIO_DIRECTORY, item) for item in sentence_audio_names if item != ""
]


def normalize_audio(input_file, output_file):
 try:
 command = [
 "ffmpeg", "-loglevel", "quiet", "-y", "-i", input_file, "-ar", "24000",
 "-ab", "96k", output_file
 ]
 subprocess.run(command, check=True)
 return output_file
 except subprocess.CalledProcessError as e:
 return None


def normalize_audio_multithreaded(file_list, output_dir, max_workers=16):
 if not os.path.exists(output_dir):
 os.makedirs(output_dir)
 print("start normalizing audios")
 with ThreadPoolExecutor(max_workers=max_workers) as executor:
 futures = [
 executor.submit(normalize_audio, input_file,
 os.path.join(output_dir, os.path.basename(input_file)))
 for input_file in file_list
 ]
 for _ in tqdm(as_completed(futures), total=len(file_list)):
 pass
 else:
 print("files has prepared well")


normalize_audio_multithreaded(audio_paths, "temp")

audio_paths = [(w, s) for w, s in zip(word_audio_names, sentence_audio_names)]
random.shuffle(audio_paths)

with open("file_list.txt", "w") as f:
 for w, s in audio_paths:
 f.write(f"file '{os.path.join('temp', w)}'\n")
 f.write(f"file silent.mp3\n")
 f.write(f"file '{os.path.join('temp', w)}'\n")
 f.write(f"file silent.mp3\n")
 if s != "":
 f.write(f"file '{os.path.join('temp', s)}'\n")
 f.write(f"file silent.mp3\n")
 f.write(f"file silent.mp3\n")

command = [
 "ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", "file_list.txt", "-c", "copy",
 "output.mp3"
]
subprocess.run(command, check=True)

Prune a Dictionary File(.mdx)

Fri, 13 Jun 2025 00:00:00 +0000

Introduction

Dictionaries such as “Oxford” and “Longman” are used as the sources for the definitions in my flashcards. However, they sometimes contain too much unnecessary information, therefore making it unwieldy and grandiose, leading to a low loading speed and distractions.

Since an MDX file utilizes the HTML format to save entries, which closely resembles the tree structure, I use the term “prune” to describe the course of removing branches, i.e. html nodes that contain superfluous elements.

Each group of three lines in a mdx file constitutes an entry of a word where the first and the third lines are delimiters while the middle line is html content. Since typically MDX files are quite large, a division step is required for parallel processing. Thus, the breakdown procedures are as follows:

split a gigantic mdx file into several small files. Note: the number of lines in each file must be a multiple of 3.
process such files parallel.
merge files into one mdx file.

Code

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


import sys
import os


def split_file_by_lines(file_path):
 file_name, file_extension = os.path.splitext(file_path)
 lines_per_file = 28311
 folder_path = "divided_files"

 if not os.path.exists(folder_path):
 os.makedirs(folder_path)
 else:
 raise BaseException("directory exists")

 with open(file_path, "r", encoding="utf-8") as f:
 file_count = 0
 lines = []
 for line in f:
 lines.append(line)
 if len(lines) == lines_per_file:
 new_file_name = f"{file_name}_{file_count:02d}{file_extension}"
 with open(
 os.path.join(folder_path, new_file_name), "w", encoding="utf-8"
 ) as chunk_file:
 chunk_file.writelines(lines)
 lines = []
 file_count += 1
 if lines:
 new_file_name = f"{file_name}_{file_count}{file_extension}"
 with open(
 os.path.join(folder_path, new_file_name), "w", encoding="utf-8"
 ) as chunk_file:
 chunk_file.writelines(lines)
 file_count += 1


def merge_files(files, output_file="res.txt"):
 with open(output_file, "w") as f:
 pass
 for file in files:
 if os.path.isfile(file):
 with open(file, "r", encoding="utf-8") as f:
 content = f.read()
 with open(output_file, "a", encoding="utf-8") as f:
 f.write(content)

split_file_by_lines("./mdx.txt")
os.system(
 r'ls ./divided_files | parallel -j16 "python3 ./prune.py ./divided_files/{} ./divided_files/new_{}"'
)
merge_files([
 os.path.join("./divided_files", f) for f in os.listdir("./divided_files")
 if os.path.isfile(os.path.join("./divided_files", f))
 and f.startswith("new")
])
import shutil


shutil.rmtree("./divided_files")

detailed pruning code:

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


#!/usr/bin/python
import sys
from bs4 import BeautifulSoup
def remove_tags_with_class(html_content):
 soup = BeautifulSoup(html_content, features="html.parser")
 [tag.decompose() for tag in soup.find_all("idm-gs-blk")]
 [tag.decompose() for tag in soup.find_all("pv-gs-blk")]
 def_tags = soup.find_all("def")
 for def_tag in def_tags:
 xhtml_a_texts = [a_tag.unwrap() for a_tag in def_tag.find_all("xhtml:a")]
 [tag.decompose() for tag in def_tag.find_all("chn")]
 new_soup = BeautifulSoup(
 r'<html><body></body></html>',
 "html.parser",
 )
 for tag in def_tags:
 new_soup.body.append(tag)
 for def_tag in def_tags:
 new_tag = soup.new_tag("p")
 new_tag.string = def_tag.get_text().strip()
 def_tag.replace_with(new_tag)
 return str(new_soup)
def process_html_file(input_file, output_file):
 with open(input_file, "r", encoding="utf-8") as file:
 lines = file.readlines()
 with open(output_file, "w", encoding="utf-8") as outfile:
 for line in lines:
 if line.strip().startswith("<head><link"):
 new_line = remove_tags_with_class(line)
 outfile.write(new_line + "\n")
 else:
 outfile.write(line)
input_file = sys.argv[1]
output_file = sys.argv[2]
process_html_file(input_file, output_file)

GNU Parallel 的妙用

Thu, 08 Jun 2023 00:00:00 +0000

Android QQ 的图片文件是以下面这种结构保存的，以Cache_开头的文件就是图片。这种结构在浏览时要分别点进每个文件夹才能看到图片，很不方便。

Show/Hide the code

1
2
3
4
5
6
7
8
9


├── 0a0
│   ├── Cache_2783644ca99260a0
│   └── Cache_-4dfb57e9a7cec0a0
├── 0a6
│   ├── Cache_373e35b7b8a290a6
│   ├── Cache_-489f6e9c62ea10a6
│   └── Cache_57776dc1e17110a6
└── 0a7
 └── Cache_25d732f84bd7d0a7

使用 GNU Parallel 配合mv可以快速的 “解散” 每个二级文件夹，变成以下这种结构。

Show/Hide the code

1
2
3
4
5
6
7


.
├── Cache_25d732f84bd7d0a7
├── Cache_2783644ca99260a0
├── Cache_373e35b7b8a290a6
├── Cache_-489f6e9c62ea10a6
├── Cache_-4dfb57e9a7cec0a0
└── Cache_57776dc1e17110a6

命令如下:

Show/Hide the code

1

find . -name "*" -type f -print | parallel mv {} ./{/}

参考: https://www.myfreax.com/gnu-parallel/

使用 Shell 批量修改文件后缀和批量格式转换

Tue, 06 Jun 2023 00:00:00 +0000

shell 脚本是处理这类工作最为简单，快捷的方式。

修改后缀

Show/Hide the code

1

for file in *.原后缀; do mv "$file" "`echo $file | sed s/.原后缀/.新后缀/`"; done

格式转换

heic 转 jpg

首先，安装转换工具。

Show/Hide the code

1

yay -S libheif

然后可以使用 heif-convert 命令转换。

Show/Hide the code

1

heif-convert input.heic output.jpg

批量转换，命令如下。

Show/Hide the code

1

for file in *.heic; do heif-convert "$file" ${file/%.heic/.jpg} && rm "$file"; done

这条命令的原理是：根据当前文件夹下的 .heic 文件生成 .jpg 文件，如果成功生成则删除原 .heic 文件，如果未成功则不会删除原文件。此外，如果未能生成 .jpg 文件的原因是 Input file 'filename.heic' is a JPEG image ，那么可以使用批量修改后缀的方法直接将文件的后缀改为 .jpg。

flac 转 mp3

首先，安装转换工具：

Show/Hide the code

1

yay -S ffmpeg

然后可以使用 ffmpeg 命令转换，除了格式转换，ffmpeg 还支持很多功能，具体可以查阅文档。

Show/Hide the code

1

ffmpeg -i 'input.flac' -ab 320k -map_metadata 0 -id3v2_version 3 'output.mp3'

批量转换，命令如下：

Show/Hide the code

1

for file in *.flac; do ffmpeg -i "$file" -ab 320k -map_metadata 0 -id3v2_version 3 ${file/%.flac/.mp3} && rm "$file"; done

原理同上。

速度优化

使用 for 循环的方式，所有文件排着队一个一个的被 ffmpeg 处理。这种方式固然可以，但效率太低了。我的运行环境如下:

Show/Hide the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


 -` momo@momo-arch
 .o+` --------------
 `ooo/ OS: Arch Linux x86_64
 `+oooo: Host: 81AC Lenovo ideapad 720S-15IKB
 `+oooooo: Kernel: 6.3.6-arch1-1
 -+oooooo+: Uptime: 6 hours, 59 mins
 `/:-:++oooo+: Packages: 975 (pacman), 8 (flatpak)
 `/++++/+++++++: Shell: bash 5.1.16
 `/++++++++++++++: Resolution: 1920x1080
 `/+++ooooooooooooo/` DE: Plasma 5.27.5
 ./ooosssso++osssssso+` WM: kwin
 .oossssso-````/ossssss+` Theme: [Plasma], Layan-Light-Solid [GTK2/3]
 -osssssso. :ssssssso. Icons: Papirus [Plasma], Papirus [GTK2/3]
 :osssssss/ osssso+++. Terminal: konsole
 /ossssssss/ +ssssooo/- CPU: Intel i5-7300HQ (4) @ 3.500GHz
 `/ossssso+/:- -:/+osssso+- GPU: Intel HD Graphics 630
 `+sso+:-` `.-/+oso: Memory: 7216MiB / 19554MiB
 `++:. `-/+/
 .` `/

经过测试，处理 10 个 .flac 文件所需时间为 59 秒。速度慢的原因是目前使用的是单进程串行的处理方式，为了提高速度，可以同时启动多个进程并行处理。

使用终端工具GNU Parallel，命令如下：

Show/Hide the code

1

ls *.flac | parallel -j4 "ffmpeg -i {} -ab 320k -map_metadata 0 -id3v2_version 3 {.}.mp3 && rm {}"

parallel 将从标准输入中读取文件列表，-j 指定了并行进程数，{} 代表输入文件名，{.} 代表没有后缀的输入文件名，其他用法可以 man parallel 查看。

经过测试，在同样环境下处理同样的 .flac 文件仅需 17 秒。