JotMyNotes

2025年5月10日星期六

2025 Amazon S3 AWS Storage

Reference

https://blog.csdn.net/jiuhebaobao/article/details/137256664

2025年3月29日星期六

2025-03-29 Debian 12 Promox 8.3.5 IOMMU

# reference

https://cloud.tencent.com/developer/article/2414840

# before

root@box02:~# dmesg | grep -E "DMAR|IOMMU"

echo nothing

# after reboot: IOMMU

root@box02:~# dmesg | grep -E "DMAR|IOMMU"

[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA

[ 0.004754] ACPI: DMAR 0x0000000072312000 000088 (v01 INTEL EDK2 00000002 01000013)

[ 0.004783] ACPI: Reserving DMAR table memory at [mem 0x72312000-0x72312087]

[ 0.068253] DMAR: IOMMU enabled

[ 0.160656] DMAR: Host address width 39

[ 0.160656] DMAR: DRHD base: 0x000000fed90000 flags: 0x0

[ 0.160661] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e

[ 0.160662] DMAR: DRHD base: 0x000000fed91000 flags: 0x1

[ 0.160665] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da

[ 0.160666] DMAR: RMRR base: 0x0000007c000000 end: 0x000000807fffff

[ 0.160668] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1

[ 0.160669] DMAR-IR: HPET id 0 under DRHD base 0xfed91000

[ 0.160669] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.

[ 0.162202] DMAR-IR: Enabled IRQ remapping in x2apic mode

[ 0.358227] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics

[ 0.417403] DMAR: No ATSR found

[ 0.417404] DMAR: No SATC found

[ 0.417405] DMAR: IOMMU feature fl1gp_support inconsistent

[ 0.417406] DMAR: IOMMU feature pgsel_inv inconsistent

[ 0.417406] DMAR: IOMMU feature nwfs inconsistent

[ 0.417407] DMAR: IOMMU feature dit inconsistent

[ 0.417408] DMAR: IOMMU feature sc_support inconsistent

[ 0.417408] DMAR: IOMMU feature dev_iotlb_support inconsistent

[ 0.417409] DMAR: dmar0: Using Queued invalidation

[ 0.417411] DMAR: dmar1: Using Queued invalidation

[ 0.418166] DMAR: Intel(R) Virtualization Technology for Directed I/O

# verify vfio

box02:~# dmesg | grep -i vfio

[ 1.783340] VFIO - User Level meta-driver version: 0.3

[ 7.532731] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[ 7.533818] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[ 7.534253] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

# verify interrupt mapping

box02:~# dmesg | grep 'remapping'

[ 0.160669] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.

[ 0.162202] DMAR-IR: Enabled IRQ remapping in x2apic mode

# grub

# update-initramfs -k all -u

update-initramfs: Generating /boot/initrd.img-6.8.12-8-pve

Running hook script 'zz-proxmox-boot'..

Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..

No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

Removable bootloader found at '/boot/efi/EFI/BOOT/BOOTX64.efi', but GRUB packages not set up to update it!

Run the following command:

echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u

Then reinstall GRUB with 'apt install --reinstall grub-efi-amd64'

# before

$ sudo lspci

00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)

# after

$ sudo lspci | grep VGA

00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)

00:10.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce GTX 1650] (rev a1)

# driver

sudo apt-get remove --purge '^nvidia-.*'

sudo apt install nvidia-driver-550 nvidia-dkms-550

sudo reboot

nvidia-smi

$ nvidia-smi

Sat Mar 29 17:30:35 2025

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 |

|-----------------------------------------+------------------------+----------------------+

| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |

| | | MIG M. |

|=========================================+========================+======================|

| 0 NVIDIA GeForce GTX 1650 Off | 00000000:00:10.0 Off | N/A |

| 26% 37C P8 9W / 100W | 6MiB / 4096MiB | 0% Default |

| | | N/A |

+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+

| Processes: |

| GPU GI CI PID Type Process name GPU Memory |

| ID ID Usage |

|=========================================================================================|

| 0 N/A N/A 1638 G /usr/lib/xorg/Xorg 4MiB |

+-----------------------------------------------------------------------------------------+

# more

:~$ nvidia-smi --query-gpu=driver_version --format=csv

driver_version

550.120

# desktop manager

sudo nvidia-settings

# also

nano /etc/modprobe.d/pve-blacklist.conf

blacklist radeon

blacklist nouveau

blacklist nvidia

# something

nano /etc/modprobe.d/pve-blacklist.conf

blacklist radeon

blacklist nouveau

blacklist nvidia

touch /etc/modprobe.d/vfio.conf

nano /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1f0a,10de:10f9

update-initramfs -u

lsmod | grep vfio

# lsmod | grep vfio

vfio_pci 16384 2

vfio_pci_core 86016 1 vfio_pci

irqbypass 12288 3 vfio_pci_core,kvm

vfio_iommu_type1 49152 1

vfio 65536 9 vfio_pci_core,vfio_iommu_type1,vfio_pci

iommufd 94208 1 vfio

# also 2

https://www.mereith.com/post/116#%E8%99%9A%E6%8B%9F%E6%9C%BA%E8%AE%BE%E7%BD%AE

# also 3

https://upsangel.com/htpc-nas/iommu-vfio-gpo-proxmox-pve-pcie%E7%9B%B4%E9%80%9A%E5%92%8C%E6%A0%B8%E9%A1%AFhdmi%E7%9B%B4%E9%80%9A%E8%A8%AD%E7%BD%AE%E5%8F%83%E6%95%B8%E8%A7%A3%E9%87%8B/

vfio.conf 没有 disable_vga=1，有的删掉！

# also 4

https://blog.csdn.net/qq_34419607/article/details/139872906

vga=none! then win10 ok

# revisit

# 01 file

vi /etc/default/grub

# original

# GRUB_CMDLINE_LINUX_DEFAULT="quiet"

# original to updated

# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream video=efifb:off,vesa

fb:off vfio-pci.ids=10de:1f0a,10de:10f9"

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream"

# 02 file

cat /etc/modprobe.d/blacklist.conf

# amd

blacklist radeon

blacklist amdgpu

# nvidia

blacklist nouveau

blacklist nvidia

blacklist nvidiafb

blacklist nvidia_drm

# options

options vfio_iommu_type1 allow_unsafe_interrupts=1

# 03 file

# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

# 04 execute

update-initramfs -u -k all

# 05 virtual machine

32GB

16 core

BIOS UEFI

Display none

Machine i440fx

scsi2 /dev/disk/by-id/nvme-Samsung_SSD_990

hostpci0 01:00

hostpci1 00:14

2025年3月23日星期日

2025 Old Server Hardware

# Chipset

Intel 7 Series X79 2011Q4 65nm PCIe2.0 8x1 4x11x4 USB 2.0 VT-d

Intel 9 Series X99 2014Q3 32nm PCIe2.0 x1 x2 x8 USB 3.0 VT-d

# bench AM4

3900x 12C 24T Single 2705 Multithread 32580 AM4 105W 2019

3700x 8C 16T Single 2658 Multithread 22493 AM4 2019

# hardware basic

DDR3 ECC 240 pin

DDR4 ECC 288 pin

HPE ProLiant Gen10 suppors Intel Optane Persistent memory

# hardware price

DDR3 32G $52

DDR4 16G $60

DDR4 32G $150

Intel Optane 128G DDR4 2666 $200

HP DL360G10 $1800

M2 250G $70

HSF $20

Case $20

PSU 400W $50

Display $25

# network card

Mellanox Technologies MT27800 Family [ConnectX-5]

# server choice

Dell R730xd $550 2U C602 24x DDR4 1866MT/s RDIMM LRDIMM 768GB H330 H730 H830 PCIe 3.0

Dell R720xd $380 2U C610 24x DDR3 2133MT/s RDIMM LRDIMM UDIMM 768GB H310 H710 H810 PCIe 3.0

Dell R630 1U C610 24x DDR4 2400MT/s (24x64GB=1.5TB)

Dell R620 1U C602

# bench X79 (Single 32GB DDR3 ECC)

E5 1650 v2 6C 12T Single 2049 Multithread 9329 FCLGA2011 130W 2013

# 14nm Gold Silver

Gold 6342 24C 48T Single 2396 Multithread 47076 FCLGA4189 230W 2021

Gold 6240 18C 36T Single 2294 Multithread 47999 FCLGA3647 150W 2019

Gold 6132 14C 28T Single 2125 Multithread 22247 FCLGA3647 140W 2018 ($100)

Silver 4314 16C 32T Single 2187 Multithread 28795 FCLGA4189 135W 2021

Platinum 8160 24C 48T Single 2077 Multithread 28825 FCLGA3647 150W 2017

# 14nm E5

E5 2696v4 22C 44T Single 2132 Multithread 24879 FCLGA2011-3 150W 2016 ($899)

E5 2686v4 18C 36T Single 1801 Multithread 20159 FCLGA2011-3 145W 2016 ($150)

E5 2680v4 14C 28T Single 1934 Multithread 17504 FCLGA2011-3 120W 2016 ($60)

HSF $55

# 22nm

E5 2698Bv3 16C 32T Single 1666 Multithread 16287 LGA2011-v3 135W 2014

E5 2690v3 12C 24T Single 1925 Multithread 16051 LGA2011-v3 135W 2014

E5 2678v3 12C 24T Single 1764 Multithread 14755 LGA2011-v3 120W 2015

E5 2676v3 12C 24T Single 1629 Multithread 13457 LGA2011-v3 120W 2015

E5 2673v3 12C 24T Single 1745 Multithread 14253 LGA2011-v3 110W 2015

E5 2666v3 10C 20T Single 1969 Multithread 14167 LGA2011-v3 135W 2015 ($70)

E5 2643v3 6C 12T Single 2079 Multithread 10434 LGA2011-v3 135W 2015

E5 2630v3 8C 16T Single 1751 Multithread 10333 LGA2011-v3 85W 2014

E5 2696v2 12C 24T Single 1695 Multithread 14048 LGA2011 120W 2013

E5 2690v2 10C 20T Single 1874 Multithread 13423 FCLGA2011 130W 2013

E5 2680v2 10C 20T Single 1790 Multithread 12697 FCLGA2011 115W 2013

E5 2670v2 10C 20T Single 1632 Multithread 11369 FCLGA2011 115W 2014

E5 2650v2 8C 16T Single 1680 Multithread 9847 FCLGA2011 95W 2013

E5 4620v3 10C 20T Single 1486 Multithread 10525 FCLGA2011-3 105W 2019

E5 4640v2 10C 20T Single 1373 Multithread 19028 LGA2011 2015

# 32nm

E5 1650 6C 12T Single 1727 Multithread 8085 LGA2011 130W 2012

E5 1620 4C 8T Single 1756 Multithread 5848 LGA2011 130W 2012

E5 2690 8C 16T Single 1660 Multithread 9755 LGA2011 135W 2012

E5 2689 8C 16T Single 1574 Multithread 9588 LGA2011 115W 2013

E5 2670 8C 16T Single 1474 Multithread 8974 LGA2011 115W 2012

E5 2665 8C 16T Single 1379 Multithread 8233 LGA2011 115W 2012

E5 2660 8C 16T Single 1393 Multithread 8096 LGA2011 95W 2012

E5 2650 8C 16T Single 1222 Multithread 7368 LGA2011 95W 2012

E5 2640 6C 12T Single 1344 Multithread 6280 LGA2011 95W 2012

E5 2620 6C 12T Single 1291 Multithread 6250 FCLGA2011 80W 2013

E5 4640 8C 16T Single 1233 Multithread 7012 LGA2011 95W 2012

E5 2450 8C 16T Single 1180 Multithread 7534 LGA1356 95W 2015

# Example - X99 Set

$265 E5 2680v4 14C 28T + X99-ZD4 (DDR4 max 128GB)

$180 E5 2676v3 12C 24T + X99 (4x DDR4)

$180 E5 2673v3 12C 24T + X99

$209 E5 2666v3 10C 20T + AngXun X99-P3 (4x DDR3 Max 128GB)

$200 E5 2666v3 10C 20T + X99 (4x DDR4)

$159 E5 2673v3 12C 24T + X99 (DDR3 DDR4)

$150 E5 4620v3 10C 20T + AngXun X99 v500 (DDR4/DDR3 Max 64GB) (choice)

# Example - X79 Set

$210 E5 2670v2 10C 20T + MSI x79 (M.2 yes) + 64GB ECC $210

$130 E5 2670 8C 16T + X79F - 2011JK2 2 slots (M.2 no) + 16GB ECC

$120 E5 1650v2 6C 12T + jginyue x79M

$138 E5 2665 8C 16T + X79 (M.2 yes) + 32GB

$70 E5 2650 8C 16T + X79

$100 E5 2689 8C 16T + X79

$138 E5 2620v2 + X79 + 16GB

$120 E5 2650v2 8C 16T + X79

$115 E5 1620 + X79 + 16GB

$90 E5 4640

$90 E5-4640 v2 + X79 + 16G

$121 E5 2680v2 10C 20T + X79-2011 (M.2) (choice)

$150 E5 2690v2 10C 20T + X79 (M.2) (choice)

$110 E5 2670v2 10C 20T + X79 (M.2) (choice)

2025年3月17日星期一

2025 My Hardware Personal Computer Used

Check
Ubunut
sudo dmidecode -t 2
sudo lsblk -o NAME,FSTYPE,LABEL,MOUNTPOINT,SIZE,MODEL

2025

Intel(R) Core(TM) i9-14900 HKD$4300 @ 2025
24C 32T 65W Single 4411 Multi 46288

Samsung SSD 990 EVO 1TB PCIe 4.0 x4 HKD$500 @2025

ASUSTeK COMPUTER INC.
PRIME Z790M-PLUS
Intel® Z790 Chipset
3x M.2 PCIe 4.0 x4
4x SATA 6Gb/s
1GbE

$ lspci
00:00.0 Host bridge: Intel Corporation Device a700 (rev 01)
00:01.0 PCI bridge: Intel Corporation Device a70d (rev 01)
00:06.0 PCI bridge: Intel Corporation Device a74d (rev 01)
00:0a.0 Signal processing controller: Intel Corporation Device a77d (rev 01)
00:0e.0 RAID bus controller: Intel Corporation Device a77f
00:14.0 USB controller: Intel Corporation Device 7a60 (rev 11)
00:14.2 RAM memory: Intel Corporation Device 7a27 (rev 11)
00:15.0 Serial bus controller: Intel Corporation Device 7a4c (rev 11)
00:15.1 Serial bus controller: Intel Corporation Device 7a4d (rev 11)
0:15.2 Serial bus controller: Intel Corporation Device 7a4e (rev 11)
00:16.0 Communication controller: Intel Corporation Device 7a68 (rev 11)
00:17.0 SATA controller: Intel Corporation Device 7a62 (rev 11)
00:1a.0 PCI bridge: Intel Corporation Device 7a48 (rev 11)
00:1b.0 PCI bridge: Intel Corporation Device 7a40 (rev 11)
00:1c.0 PCI bridge: Intel Corporation Device 7a38 (rev 11)
00:1d.0 PCI bridge: Intel Corporation Device 7a30 (rev 11)
00:1d.4 PCI bridge: Intel Corporation Device 7a34 (rev 11)
00:1f.0 ISA bridge: Intel Corporation Device 7a04 (rev 11)
00:1f.3 Audio device: Intel Corporation Device 7a50 (rev 11)
00:1f.4 SMBus: Intel Corporation Device 7a23 (rev 11)
00:1f.5 Serial bus controller: Intel Corporation Device 7a24 (rev 11)
00:1f.6 Ethernet controller: Intel Corporation Device 0dc8 (rev 11)
01:00.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce GTX 1650] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU106 High Definition Audio Controller (rev a1)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a80d
07:00.0 Non-Volatile memory controller: Micron Technology Inc Device 5416 (rev 01)

2022-2023

AMD Ryzen 3900x
12C 24T

INTEL SSDPEKKF256G8L

B450M MORTAR MAX

2021

AMD Ryzen 7 3700x
8C 16T 65W Single 2658 Multi 22494

ADATA SX8200PNP PCIe Gen3 x4 HKD$580 @2021

ASUSTeK COMPUTER INC.
TUF GAMING B550M-PLUS
AMD B550
2x M.2 PCIe 3.0 x4
4x SATA 6Gb/s

2025年3月13日星期四

2025 NVME Disk Speed Test

# tool
sudo apt install nvme-cli

# mytest 01 3.6 GB/s Samsung SSD 990 EVO 1TB HKD$500
native Asus onboard nvme
sudo nvme list
Samsung SSD 990 EVO 1TB
$ time dd if=/dev/zero of=testfile bs=16k count=128k
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.599403 s, 3.6 GB/s
# if promox ubuntu pcie device passthrough, also 3.8GB/s. no loss.

harddrivebenchmark says 4,547 MBytes/Sec

# mytest 02 112 MB/s FreeNas NVME HKD$430
NVME over FreeNAS and TPLINK 1Gbe Switch
sudo nvme list
ADATA SX8200PNP
$ time dd if=/dev/zero of=/mnt/ricky/testfile bs=16k count=128k
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 19.0907 s, 112 MB/s

# mytest 03 3.9 GB/s Crucial 1TB

Crucial 1TB attached to Z790M-PLUS M.2_3 PCIe 4.0
sudo nvme list
CT1000P3PSSD8
$ time dd if=/dev/zero of=testfile bs=16k count=128k
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.545956 s, 3.9 GB/s

harddrivebenchmark says 4,214 MBytes/Sec

# mytest 04 2.0 GB/s INTEL SSDPEKKF256G8L

2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.545956 s, 3.9 GB/s

3900x Mortar Max B450
sudo nvme list
INTEL SSDPEKKF256G8L
$ time dd if=/dev/zero of=testfile bs=16k count=128k

harddrivebenchmark says 2,430 MBytes/Sec

# mytest 05 292 MB/s WDC WD30PURX 3TB SATA

2147483648 bytes (2.1 GB, 2.0 GiB) copied, 7.36371 s, 292 MB/s

78P6ZY0 89P6ZY0

# test

TEST: Samsung 990 EVO 1TB M.2 SSD
Read speed 5000 MB/s
Write speed 4200 MB/s
Interface PCIe 4.0 x4

TEST: ADATA XPG SX8200 Pro 1TB M.2 SSD
Read speed 3500 MB/s
Write speed 2800 MB/s
Interface PCIe 3.0 x4

# Network

1 Gbps = 125 MB/s
2.5 Gbps = 312.5 MB/s
10 Gbps = 1250 MB/s = 1.25 GB/s
25 Gbps = 3125 MB/s = 3.125 GB/s
40 Gbps = 5000 MB/s = 5 GB/s

# NVME

https://ssd-tester.com/m2_ssd_test.php?sort=&sort_interface=PCIe+5.0+x4

The following speed transfer rates are possible: Theoretical Practical

SATA 3 6 GBit/s (=750 MB/s) ~550 MB/s
PCIe 2.0 x2 10 GBit/s (=1.250 MB/s) ~800 MB/s
PCIe 2.0 x4 20 GBit/s (=2.500 MB/s) ~1,600 MB/s
PCIe 3.0 x2 16 GBit/s (=2.000 MB/s) ~1,600 MB/s
PCIe 3.0 x4 32 GBit/s (=4.000 MB/s) ~3,500 MB/s
PCIe 4.0 x4 64 GBit/s (=8.000 MB/s) ~7,500 MB/s
PCIe 5.0 x4 128 GBit/s (=16.000 MB/s) ~15,000 MB/s

2025年1月22日星期三

202501 Debian USB LAN Card Bridge

ChatGPT Question

I have a machine running debian 12 with a LAN port using a cable to connect to my office and can obtain below ip address correctly

2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether f0:2f:74:2e:57:ea brd ff:ff:ff:ff:ff:ff
inet 10.20.15.101/24 brd 10.20.15.255 scope global dynamic noprefixroute enp6s0
valid_lft 82884sec preferred_lft 82884sec
inet6 fe80::f22f:74ff:fe2e:57ea/64 scope link noprefixroute
valid_lft forever preferred_lft forever

Now i want to add another NIC enx8cb0e9c91446 to that machine and allow another client machine to connect to this newly added NIC so that that client machines can also obtain ip address from that NIC as if that client machine is connect to the same subnet. How?

ChatGPT Answer (does not work, do not directly use)

To allow another client machine to connect to your Debian 12 machine via the newly added NIC enx8cb0e9c91446 and obtain an IP address from the same subnet, you can set up the new NIC as a bridge interface. This will enable the client machine to be on the same subnet and communicate with other devices in the network. Here's how you can achieve this:

Identify the New NIC: First, ensure that the new NIC enx8cb0e9c91446 is properly identified and recognized by your Debian 12 system.
Install Bridge Utilities (if needed): If the bridge utilities package is not already installed on your system, you can install it by running:
```
sudo apt update
sudo apt install bridge-utils
```
Configure the Bridge Interface: Create a bridge interface that includes both enp6s0 and enx8cb0e9c91446. You can create a bridge configuration file in the /etc/network/interfaces.d/ directory. Here's an example configuration for the bridge interface:

~~Add the following configuration to a new file, for example, /etc/network/interfaces.d/br0:~~
```
auto enp6s0
iface enp6s0 inet manual

auto enx8cb0e9c91446
iface enx8cb0e9c91446 inet manual

auto br0
iface br0 inet dhcp
    bridge_ports enp6s0 enx8cb0e9c91446
```
Restart Networking Service:
After saving the configuration, you can restart the networking service to apply the changes:
```
sudo systemctl restart networking
```
Verify the Bridge Configuration:
You can verify that the bridge interface is created and both NICs are part of it by running:
```
ip addr show br0
```
Connect the Client Machine:
Connect the client machine to the new NIC enx8cb0e9c91446. The client machine should now be able to obtain an IP address from the same subnet as 10.20.15.101 and communicate with other devices on the network.

By setting up a bridge interface in Debian 12, you are allowing the new client machine to be on the same subnet as your original NIC enp6s0, enabling seamless communication within the network. If you have any questions or need further assistance, feel free to ask.

Exercise

before

cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"

ls /etc/network/interfaces.d

echo nothing

$ ip -br a

lo UNKNOWN 127.0.0.1/8 ::1/128

enp6s0 UP 10.20.15.101/24 fe80::f22f:74ff:fe2e:57ea/64

docker0 DOWN 172.17.0.1/16 fe80::42:fdff:feea:3e0f/64

enx8cb0e9c91446 DOWN

reference

https://wiki.debian.org/BridgeNetworkConnections

action (will work)

sudo touch /etc/network/interfaces.d/br0

sudo nano /etc/network/interfaces.d/br0

cat /etc/network/interfaces.d/br0

auto lo

iface lo inet loopback

iface enp6s0 inet manual

iface enx8cb0e9c91446 inet manual

auto br0

iface br0 inet dhcp

bridge_ports enp6s0 enx8cb0e9c91446

sudo systemctl restart networking

after

ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp6s0 UP 10.20.15.101/24 fe80::f22f:74ff:fe2e:57ea/64
docker0 DOWN 172.17.0.1/16 fe80::42:fdff:feea:3e0f/64
enx8cb0e9c91446 DOWN
br0 UP 10.20.15.127/24 fe80::384e:eaff:fe87:beca/64

how to use have the original ip address 10.20.15.101 for br0?

Ask ChatGPT

my br0 now has an ip address 10.20.15.127 from dhcp. now i want to use keepalived so that it attach an virtual ip address 10.20.15.101 so that other users using 10.20.15.101:24000 will be redirect to 10.20.15.127:24000. how?

sudo apt update
sudo apt install keepalived

$ cat /etc/keepalived/keepalived.conf

global_defs {

router_id keepalived_node

}

vrrp_instance VI_1 {

state MASTER

interface br0

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass your_password

}

virtual_ipaddress {

10.20.15.101

}

virtual_server 10.20.15.101 24000 {

delay_loop 2

lb_algo rr

lb_kind NAT

persistence_timeout 0

protocol TCP

real_server 10.20.15.127 24000 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

sudo systemctl restart keepalived

$ ip -br a

lo UNKNOWN 127.0.0.1/8 ::1/128

enp6s0 UP

docker0 DOWN 172.17.0.1/16 fe80::42:fdff:feea:3e0f/64

enx8cb0e9c91446 DOWN

br0 UP 10.20.15.127/24 10.20.15.101/32 fe80::384e:eaff:fe87:beca/64

enx00e04c534458 DOWN

2023年11月13日星期一

2025 2023 Promox on Morefine N6000 16GB 512GB

2025 Feb Continue

# reference

https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm

# Install a standard Debian Bookworm (amd64)
gnome
openssh server
standard system utilities

# useful
cat /etc/apt/sources.list
deb http://deb.debian.org/debian bookworm main
deb http://deb.debian.org/debian bookworm-updates main
deb http://deb.debian.org/debian-security bookworm-security main
https://extensions.gnome.org/extension/1160/dash-to-panel/

# cockpit
sudo apt-get install cockpit
http://192.168.1.60:9090

# check
cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm

# hosts
cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# This host address
127.0.1.1 n6000
192.168.1.61 n6000.proxmox.com n6000

# check hostname
hostname --ip-address
127.0.1.1 192.168.1.61

# Adapt your sources.list
sudo su root
echo "deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
sha512sum /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
7da6fe34168adc6e479327ba517796d4702fa2f8b4f0a9833f5ea6e6b48f6507a6da403a274fe201595edc86a84463d50383d07f64bdde2e3658108db7d6dc87 /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
apt update && apt full-upgrade

# Install the Proxmox VE Kernel
apt install proxmox-default-kernel
systemctl reboot

# may need below #1 (n6000)
# see https://forum.proxmox.com/threads/solved-pve-firmware-package-error-when-installing-proxmox-8-over-debian-12-bookworm.132671/
sudo dpkg --purge --force-all firmware-ath9k-htc
sudo apt --fix-broken install
# may need above #1

# may need below #2 (3700x, n6000)
# dpkg: error processing archive /var/cache/apt/archives/pve-firmware_3.14-3_all.deb (--unpack):
# trying to overwrite '/lib/firmware/rtl_bt/rtl8723cs_xx_config.bin', # which is also in package firmware-realtek-rtl8723cs-bt 20181104-2
# https://free-pmx.pages.dev/guides/error-dpkg-conflict/
apt install -fo Dpkg::Options::="--force-overwrite"
# may need above #2

# Install the Proxmox VE packages
sudo su root
apt install proxmox-ve postfix open-iscsi chrony
choose local only

# Remove the Debian Kernel
# local only
# leave the system name as is
apt remove linux-image-amd64 'linux-image-6.1*'
update-grub

# Recommended: Remove the os-prober Package
apt remove os-prober

# reboot
echo last chance to make sure static ip address
sudo reboot -h now

# comment
on 2025-02-16, failed after reboot. machine status unknown. cannot connect. 10.20.15.101.
on 2025-02-25, n6000 can

# if not dhcp address, change from
cat /etc/network/interfaces
source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
# to
# to unknown
# auto enp0s3
# iface enp0s3 inet dhcp
# sudo ifdown enp0s3
# sudo ifup enp0s3
# ip -br a
# lo UNKNOWN 127.0.0.1/8 ::1/128
# enp0s3 UP 10.20.15.101/24 # fe80::a00:27ff:fec1:1a95/64

# comment
successful! has desktop, cockpit port 9090 working. promox port 8006 working.

# iso
cd /var/lib/vz/template/iso/
scp ricky@10.20.15.101:/var/lib/vz/template/iso/ubuntu-22.04.4-desktop-amd64.iso .

# if 1024x768 only
https://askubuntu.com/questions/206283/how-can-i-uninstall-a-nvidia-driver-completely
dpkg -l | grep -i nvidia
sudo apt-get remove --purge '^nvidia-.*'
sudo reboot -h now

# Usual config
Options > Start at boot > No to Yes
Options > QEMU Guest Agent > Default Disabled to Enabled

# install guest agent
user@guest:~#
sudo apt update
sudo apt-get -y install qemu-guest-agent
sudo systemctl enable qemu-guest-agent
sudo systemctl start qemu-guest-agent
sudo systemctl status qemu-guest-agent

# ksm sharing

https://forum.proxmox.com/threads/ksm-sharing.75774/
root@promox-ve:~#
apt-get install -y ksmtuned
systemctl enable ksmtuned
systemctl start ksmtuned
systemctl enable --now ksm.service
systemctl status ksmtuned

# nvidia driver
https://askubuntu.com/questions/841876/how-to-disable-nouveau-kernel-driver
https://www.reddit.com/r/Proxmox/comments/1coquzo/did_anyone_succesfully_installed_nvidiadriver_on/
sudo nano /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
sudo update-initramfs -u
sudo reboot
sudo apt install -y dkms pve-headers wget
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.78/NVIDIA-Linux-x86_64-550.78.run
chmod +x NVIDIA-Linux-x86_64-550.78.run
sudo ./NVIDIA-Linux-x86_64-550.78.run --dkms
# comment: smooth. excellent. on 2025-03-04

# openwrt on promox
https://i12bretro.github.io/tutorials/0405.html
cd /var/lib/vz/images/100/
wget https://downloads.openwrt.org/releases/24.10.0/targets/x86/64/openwrt-24.10.0-x86-64-generic-ext4-combined.img.gz
qm importdisk 100 openwrt.raw local --format raw
unused1: successfully imported disk 'local:100/vm-100-disk-3.raw'

# TrueNAS Scale
https://blog.csdn.net/weixin_44924918/article/details/142478446
https://blog.csdn.net/luoqingyong/article/details/139901002#:~:text=TrueNAS%20SCALE%E7%9A%84%E6%9C%80%E4%BD%8E%E7%A1%AC%E4%BB%B6,8%20GB%E5%86%85%E5%AD%98%EF%BC%88%E6%8E%A8%E8%8D%9016GB%EF%BC%89
TrueNas-SCALE-24.10.2.iso
4vCPU 8GB

# assign a disk to vm
#if 14900
$ ls -larth /dev/disk/by-id/nvme-Samsung_SSD_990_EVO_1TB_S7GCNS0X108543R
lrwxrwxrwx 1 root root 13 Mar 19 13:35 /dev/disk/by-id/nvme-Samsung_SSD_990_EVO_1TB_S7GCNS0X108543R -> ../../nvme1n1
qm set 100 -scsi2 /dev/disk/by-id/nvme-Samsung_SSD_990_EVO_1TB_S7GCNS0X108543R
# elif n6000
Choose "Raw Device 0000:02:00.0" as "PCI Device (hostpci0)"
# else
qm set 100 -scsi2 none
qm set 100 -scsi2 /dev/nvme0n1
# or
qm set 100 -vertio0 /dev/nvme0n1

# vmbr0

# iface vmbr0 inet dhcpworks. but remember to re-attach nic to vm.

cat /etc/network/interfaces

source /etc/network/interfaces.d/*

auto lo

iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0

iface vmbr0 inet dhcp

bridge-ports enp1s0

bridge-stp off

bridge-fd 0

# vmbr1

# comment: vm can use wifi nat

auto wlo1

iface wlo1 inet dhcp

auto vmbr1

iface vmbr1 inet static

address 10.10.10.1/24

bridge-ports none

bridge-stp off

bridge-fd 0

post-up echo 1 > /proc/sys/net/ipv4/ip_forward

post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o wlo1 -j MASQUE>

post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o wlo1 -j MASQUE>

#internal 10.10.10.1

# win10

10.10.10.2

255.255.255.0

10.10.10.1

8.8.8.8

ping 10.10.10.1

ping 8.8.8.8

# host route

# can ping if

ricky@n6000:~$ ping 10.20.15.159

PING 10.20.15.159 (10.20.15.159) 56(84) bytes of data.

64 bytes from 10.20.15.159: icmp_seq=1 ttl=61 time=22.1 ms

ip route show

default via 192.168.137.1 dev vmbr0

default via 192.168.1.1 dev wlo1 proto dhcp src 192.168.1.60 metric 600

169.254.0.0/16 dev wlo1 scope link metric 1000

192.168.1.0/24 dev wlo1 proto kernel scope link src 192.168.1.60 metric 600

192.168.137.0/24 dev vmbr0 proto kernel scope link src 192.168.137.162

sudo ip route del 169.254.0.0/16

ip -br a

lo UNKNOWN 127.0.0.1/8 ::1/128

enp1s0 UP

wlo1 DOWN 169.254.9.171/16

vmbr0 UP 192.168.137.162/24 fe80::200:ff:fe00:13/64

tap100i0 UNKNOWN

fwbr100i0 UP

fwpr100p0@fwln100i0 UP

fwln100i0@fwpr100p0 UP

2023 Promox on Morefine N6000 16GB 512GB (failed)

Software

Etcher 100MB (not but can be rufus-4.3.exe 1.4MB)
Proxmox VE 7.4 ISO Installer (1st ISO release) 1GB

Console

root
Sky4dots

Web

https://192.168.1.60:8006

Reference

Google "proxmox 安裝教學"

https://post.smzdm.com/p/agqw24zw/

訂閱：文章 (Atom)

JotMyNotes

2025年5月10日星期六

2025 Amazon S3 AWS Storage

2025年3月29日星期六

2025-03-29 Debian 12 Promox 8.3.5 IOMMU

2025年3月23日星期日

2025 Old Server Hardware

2025 Old Server Hardware

2025年3月17日星期一

2025 My Hardware Personal Computer Used

2025年3月13日星期四

2025 NVME Disk Speed Test

2025年1月22日星期三

202501 Debian USB LAN Card Bridge

2023年11月13日星期一

2025 2023 Promox on Morefine N6000 16GB 512GB

2025 Amazon S3 AWS Storage

檢舉濫用情形

標籤

2025年5月10日 星期六

2025年3月29日 星期六

2025年3月23日 星期日

2025 Old Server Hardware

2025年3月17日 星期一

2025年3月13日 星期四

2025年1月22日 星期三

2023年11月13日 星期一

2025年5月10日星期六

2025年3月29日星期六

2025年3月23日星期日

2025年3月17日星期一

2025年3月13日星期四

2025年1月22日星期三

2023年11月13日星期一