fanyang

Memo

共 45 条

docker-compose.yml 合集

本篇文章收录了鄙人使用的各种服务的 docker-compose.yml

Milvus

services:
  etcd:
    container_name: milvus-etcd
    image: registry.example.com/quay.io/coreos/etcd:v3.5.18
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: registry.example.com/minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: registry.example.com/milvusdb/milvus:v2.5.12
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus

解决 PostgreSQL `could not open shared memory segment` 错误

解决 PostgreSQL could not open shared memory segment 错误。

打开 journalctl -u postgresql.service 可以看到：

Jun 15 11:55:52 fanyang-legion postgres[322924]: 2025-06-15 11:55:52.856 CST [322924] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 11:56:52 fanyang-legion postgres[322973]: 2025-06-15 11:56:52.914 CST [322973] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 11:57:52 fanyang-legion postgres[323026]: 2025-06-15 11:57:52.975 CST [323026] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 11:58:53 fanyang-legion postgres[323076]: 2025-06-15 11:58:53.034 CST [323076] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 11:59:53 fanyang-legion postgres[323127]: 2025-06-15 11:59:53.042 CST [323127] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 12:00:53 fanyang-legion postgres[323181]: 2025-06-15 12:00:53.099 CST [323181] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 12:01:53 fanyang-legion postgres[323229]: 2025-06-15 12:01:53.155 CST [323229] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 12:02:53 fanyang-legion postgres[323286]: 2025-06-15 12:02:53.193 CST [323286] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 12:03:53 fanyang-legion postgres[323334]: 2025-06-15 12:03:53.219 CST [323334] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 12:04:53 fanyang-legion postgres[323385]: 2025-06-15 12:04:53.232 CST [323385] ERROR:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory
Jun 15 12:05:04 fanyang-legion postgres[323386]: 2025-06-15 12:05:04.501 CST [323386] FATAL:  could not open shared memory segment "/PostgreSQL.2816709408": No such file or directory

解决方法：

PostgreSQL 11 Shared Memory Error: could not open shared memory segment "/PostgreSQL.XXXXXXXX": No such file or directory

# sudo vim /etc/systemd/logind.conf
RemoveIPC=no

systemctl restart systemd-logind.service

一键清理空的 Github Repo

在 GitHub 上创建了一堆 repository 但是又没空写，怎么把这些空的 repository 都删了呢？

写了一个小脚本 find-empty-github-repo 把他们都删了。

使用方法

  1. 点击这里创建一个 token，并且具有 repo 权限。

  2. 创建一个 .env 把 token 填进去，然后运行脚本。

详细的操作参考 README.md

vLLM 部署为 systemd 服务

配置 vLLM 环境，并且让他随着系统自启动

安装环境：

mkdir my-vllm
cd my-vllm

uv init
uv add vllm

编写 service 文件：

# vim /etc/systemd/system/vllm.service

[Unit]
Description=vLLM
After=network.target
Wants=network.target

[Service]
Type=simple
Environment=HF_ENDPOINT=https://hf-mirror.com
WorkingDirectory=/opt/vllm
ExecStart=uv run vllm serve "Qwen/Qwen3-Embedding-0.6B"
Restart=always
RestartSec=10
User=user
Group=user

[Install]
WantedBy=multi-user.target

vLLM，启动！

systemctl daemon-reload
systemctl enable --now vllm

默认端口

常用 Go modules

鄙人常用的一些 go mod，备查

存数据

  • github.com/cockroachdb/pebble
  • github.com/mattn/go-sqlite3
  • github.com/spf13/afero
  • github.com/syndtr/goleveldb
  • gorm.io/gorm
  • gorm.io/driver/sqlite

错误处理

  • github.com/cockroachdb/errors

哈希

  • github.com/cespare/xxhash

测试

  • github.com/negrel/assert
  • github.com/stretchr/testify

命令行

  • github.com/urfave/cli/v3
  • github.com/schollz/progressbar/v3
  • github.com/joho/godotenv

日志

  • github.com/rs/zerolog
  • github.com/fanyang89/pzlog
  • github.com/outrigdev/outrig

字符串

  • github.com/gobwas/glob

缓存

  • github.com/maypok86/otter

Oracle Linux Shell Script to Calculate Values Recommended Linux HugePages Configuration (Doc ID 401749.1)

Oracle Doc ID 401749.1 的脚本，自动配置 hugepage

#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
# on Oracle Linux
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
#
# This script is provided by Doc ID 401749.1 from My Oracle Support
# http://support.oracle.com
# Welcome text
echo "
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
* For ASM instance, it needs to configure ASMM instead of AMM.
* The 'pga_aggregate_target' is outside the SGA and
 you should accommodate this while calculating the overall size.
* In case you changes the DB SGA size,
 as the new SGA will not fit in the previous HugePages configuration,
 it had better disable the whole HugePages,
 start the DB with new SGA size and run the script again.
And make sure that:
* Oracle Database instance(s) are up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not setup
 (See Doc ID 749851.1)
* The shared memory segments can be listed by command:
 # ipcs -m
Press Enter to proceed..."
read
# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
if [ -z "$HPG_SZ" ];then
 echo "The hugepages may not be supported in the system where the script is being
executed."
 exit 1
fi
# Initialize the counter
NUM_PG=0
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
do
 MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
 if [ $MIN_PG -gt 0 ]; then
 NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
 fi
done
RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`
# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [ $RES_BYTES -lt 100000000 ]; then
 echo "***********"
 echo "** ERROR **"
 echo "***********"
 echo "Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:
 # ipcs -m
of a size that can match an Oracle Database SGA. Please make sure that:
* Oracle Database instance is up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not configured"
 exit 1
fi
# Finish with results
case $KERN in
 '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
 echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
 '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
 '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
 '3.10') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
 '4.1') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
 '4.14') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
 '4.18') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
 '5.4') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
 *) echo "Kernel version $KERN is not supported by this script (yet). Exiting." ;;
esac
# End

破解 Delete All Google Photos 插件

自从有了 NAS 之后，就想把所有云的会员都取消了。但上云容易下云难

上传到 Google Photos 的时候爽的不行，结果现在会员到期了，空间直接给我爆了，还不让我发邮件。

一番搜索，发现了一个插件，可以自动化删除所有的照片：Delete All Google Photos

扩展 ID：bebhhjmapjadpdkkhbkpnpbjhkhndofl

试用版每天只能删 500 张，根本不好使啊。我直接反手一个 js 的读。

找到下面的代码段：

const R = 1e3,
  M = 1000000;
function E(i) {
  return new Promise((e) => setTimeout(e, i));
}

M 改成很大的值。

if (
  !!P &&
  (console.log("removing undo delete button and continuing", new Date()),
  P.remove(),
  (r += b),
  localStorage.setItem("num_deleted", 0),
  localStorage.setItem("last_deleted_at", new Date().getTime()),
  !(o != null && o.paid) && r >= M)
) {
  t(0, (p = !0));
  return;
}

每次 r += b 然后设置 num_deleted。改成每次 num_deleted 都设置为 0

然后开发人员模式加载扩展，直接一把梭删完，爽！

ESXi 7.0 安装时移除 VMFSL

在 ESXi 7.0 之后，安装的时候会自动创建一个很大的 VMFSL 分区：

vmfsl-example.png

安装 ESXi 时，在引导后按 Shift+O，并输入

autoPartitionOSDataSize=8192

然后回车，正常安装即可。

Seagate TurboBoost and Advanced Write Caching

传统的 HDD 以较慢的响应时间来应对较高的工作负载

  • 在 200 IOPS 时，硬盘阵列可能提供 5 毫秒的响应
  • 在 600 IOPS 时，延迟可能飙升至 40 毫秒甚至更高。

TurboBoost

TurboBoost 是在传统硬盘中加入 NAND 闪存，集成了

  • 传统的多段式缓存
  • 少量的 eMLC NAND，其特点是比消费级 MLC 的耐用性要强得多

适量的 NAND 能够满足价值需求，以足够低的成本提供最大的性能优势

76d4e14c25b23afca9840d8ce905fbd3.png

NVC：non-volatile cache

Back electromotive force（back EMF）指与电机线圈在磁场中运动产生的电流方向相反的电压。HDD 可以在断电后立即利用反电动势，将 DRAM 中的数据写入 NVC Flash 中。有了 NVC 保护的写缓存，在享受到性能好处的同时，仍然可以与关闭 write cache 同等的保护。

内置的 NOR Flash 大小：2M

5b267ab31853330f53d9e3be9dab501f.png

dm-delay 模拟慢速磁盘

速查

# 创建一个 10G 大小的 ram disk
sudo modprobe brd rd_nr=1 rd_size=1048576
sudo blockdev --getsize /dev/ram0

# 创建 delayed dm，延迟为 500ms
export RAM_SIZE=$(blockdev --getsize /dev/ram0)
echo "0 $RAM_SIZE delay /dev/ram0 0 500" | sudo dmsetup create delayed

# 重新加载参数
echo "0 $RAM_SIZE delay /dev/ram0 0 500" | sudo dmsetup reload delayed

delayed 参数

<device> <offset> <delay> [<write_device> <write_offset> <write_delay> [<flush_device> <flush_offset> <flush_delay>]]

暂停 I/O

sudo dmsetup suspend /dev/dm-0
sudo dmsetup resume  /dev/dm-0

检查 delayed 设备的 I/O 延迟

fio --name a --filename=/dev/dm-0 --bs=4k --rw=randread --ioengine=libaio --direct=1 --iodepth=1 --numjobs=1 --time_based=1 --runtime=10
fio --name a --filename=/dev/dm-0 --bs=4k --rw=randread --ioengine=sync --direct=1 --iodepth=1 --numjobs=1 --time_based=1 --runtime=10
Memo