dualfroz/proxmox-git-pve-storage
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dualfroz/proxmox-git-pve-storage:ab748c8530925ab4f068ce90e5ebd5db8948eaa4
dualfroz/proxmox-git-pve-storage:main
..
compare: dualfroz/proxmox-git-pve-storage:b7d36d0ecaa9e0656de1b66625e851c6a78132c3
dualfroz/proxmox-git-pve-storage:main
10 Commits
ab748c8530 .. b7d36d0eca
Author SHA1 Message Date
dualfroz b7d36d0eca Describe this repository 2026-10-22 21:06:24 +02:00
dualfroz b77120f913 fix #6184: import: ovf: add CD drives 
The OVF parser only looks at disk drives, so the CD drives of a guest
are dropped on import. Appliances shipped as OVA sometimes come with
an ISO image in a CD drive, for example to install the guest on first
boot. Such a guest then does not boot after the import, and the user
is not told about the image at all.

Add the CD drives of the manifest as empty drives on their controller,
after the disks in the boot order, like the ESXi import does. As CD
images cannot be imported, report a referenced image with the existing
'cdrom-image-ignored' warning, so that the import wizard tells the user
to configure the drive in the 'Advanced' tab.

As the address only takes the position on the controller into account,
a CD drive can end up with the address of a disk, for example with a
disk on the first IDE controller and the CD drive on the second one.
Such a CD drive gets the next free address on its bus.

Signed-off-by: Michal Fox <me@dualfroz.com>
 2026-10-03 02:00:08 +02:00
upstream 095cd282f3 Upstream proxmox-git/pve-storage state of the touched files before 1860d190cb 2026-10-03 02:00:08 +02:00
dualfroz ab149ea5d1 fix #6405: disks: smart: parse ATA attributes with additional flags 
In its brief output format, smartctl prints the attribute flags as six
characters, followed by a seventh one which is a '+' if any other flag
bits are set and a space otherwise, for example:

  5 Reallocated_Sector_Ct   ------+  100   100   050    -    0

The regex for the ATA attributes only accepted exactly six non-space
characters for the flags, so such attribute lines were skipped. For
drives that set those other bits on all attributes, like the SATA SSD
from the bug report, no SMART attributes showed up at all and the
wearout could not be determined either.

Accept an optional '+' after the flags and add the output from the
bug report as test case.

Signed-off-by: Michal Fox <me@dualfroz.com>
 2026-10-02 16:56:44 +02:00
upstream 4265fdd04a Upstream proxmox-git/pve-storage state of the touched files before 458cafb446 2026-10-02 16:33:12 +02:00
dualfroz dc7402d9ff fix #6441: btrfs: fix cloning templates with non-raw disk images 
Cloning a template with a disk image that is neither raw nor a
subvolume, for example a qcow2 image, failed on a BTRFS storage with:

  clone failed: Undefined subroutine &PVE::Storage::DirPlugin::clone_image called

For such images, the BTRFS plugin forwards the call to the DirPlugin.
But the DirPlugin does not implement clone_image itself, it inherits it
from the base plugin, and calling it as a plain function does not go
through the inheritance.

Call the implementation of the base plugin directly instead, like it is
already done for create_base.

Fixes: af50c2e ("add BTRFS storage plugin")
Signed-off-by: Michal Fox <me@dualfroz.com>
 2026-10-02 16:30:32 +02:00
upstream fa2737e0ae Upstream proxmox-git/pve-storage state of the touched files before 6d611c0324 2026-10-02 16:30:32 +02:00
dualfroz 467e6d641e fix #6389: api: disks: zfs: parse the message of hot spares in use 
Hot spares have no error counters in the output of 'zpool status', so a
spare that is in use only has a message after its state:

  nvme10n1      INUSE     currently in use

The parser took the three words of that message as READ, WRITE and
CKSUM counters, so the API returned zero for all of them and lost the
message, while logging warnings like:

  Argument "currently" isn't numeric in addition (+)

As 'zpool status' is called with '-p', the counters are always plain
numbers, so only accept digits for them. This way, the message of a
spare in use ends up in 'msg', like for any other vdev.

Signed-off-by: Michal Fox <me@dualfroz.com>
 2026-10-02 16:17:54 +02:00
dualfroz 6320a3a111 fix #6640, #6938: api: disks: zfs: return exact error counters 
Without '-p', 'zpool status' shortens the READ, WRITE and CKSUM counters
of a vdev to a value with a unit suffix once they reach 1024, for
example 1130 read errors are shown as '1.10K'. The parser just adds
zero to those values, so the API returned 1.1 read errors in that case
and logged a warning like:

  Argument "1.10K" isn't numeric in addition (+)

Request exact values with '-p'. For the non-JSON output of 'zpool
status' it only affects those counters, the sizes in the scan line stay
human readable.

Signed-off-by: Michal Fox <me@dualfroz.com>
 2026-10-02 16:15:39 +02:00
upstream e47d23f3c7 Upstream proxmox-git/pve-storage state of the touched files before 9fddd8a43e 2026-10-02 16:15:39 +02:00
17 changed files with 3695 additions and 49 deletions
Expand all files Collapse all files

No files matched your search

README.md
+6
View File
@@ -0,0 +1,6 @@
# Contributions to proxmox-git/pve-storage
This repository holds only my commits to the upstream project, with their original author, date and diff.
Each of my commits is preceded by a snapshot of the upstream files it touched, so every diff shows exactly what I changed.
src/PVE/API2/Disks/ZFS.pm
+655
View File
@@ -0,0 +1,655 @@
package PVE::API2::Disks::ZFS;
use strict;
use warnings;
use PVE::Diskmanage;
use PVE::JSONSchema qw(get_standard_option parse_property_string);
use PVE::Systemd;
use PVE::API2::Storage::Config;
use PVE::Storage;
use PVE::Tools qw(run_command lock_file trim);
use PVE::RPCEnvironment;
use PVE::RESTHandler;
use base qw(PVE::RESTHandler);
my $ZPOOL = '/sbin/zpool';
my $ZFS = '/sbin/zfs';
sub get_pool_data {
die "zfsutils-linux not installed\n" if !-f $ZPOOL;
my $propnames = [qw(name size alloc free frag dedup health)];
my $numbers = {
size => 1,
alloc => 1,
free => 1,
frag => 1,
dedup => 1,
};
my $pools = [];
run_command(
[$ZPOOL, 'list', '-HpPLo', join(',', @$propnames)],
outfunc => sub {
my ($line) = @_;
my @props = split('\s+', trim($line));
my $pool = {};
for (my $i = 0; $i < scalar(@$propnames); $i++) {
if ($numbers->{ $propnames->[$i] }) {
$pool->{ $propnames->[$i] } = $props[$i] + 0;
} else {
$pool->{ $propnames->[$i] } = $props[$i];
}
}
push @$pools, $pool;
},
);
return $pools;
}
__PACKAGE__->register_method({
name => 'index',
path => '',
method => 'GET',
proxyto => 'node',
protected => 1,
permissions => {
check => ['perm', '/', ['Sys.Audit']],
},
description => "List Zpools.",
parameters => {
additionalProperties => 0,
properties => {
node => get_standard_option('pve-node'),
},
},
returns => {
type => 'array',
items => {
type => 'object',
properties => {
name => {
type => 'string',
description => "",
},
size => {
type => 'integer',
description => "",
},
alloc => {
type => 'integer',
description => "",
},
free => {
type => 'integer',
description => "",
},
frag => {
type => 'integer',
description => "",
},
dedup => {
type => 'number',
description => "",
},
health => {
type => 'string',
description => "",
},
},
},
links => [{ rel => 'child', href => "{name}" }],
},
code => sub {
my ($param) = @_;
return get_pool_data();
},
});
sub preparetree {
my ($el) = @_;
delete $el->{lvl};
if ($el->{children} && scalar(@{ $el->{children} })) {
$el->{leaf} = 0;
foreach my $child (@{ $el->{children} }) {
preparetree($child);
}
} else {
$el->{leaf} = 1;
}
}
__PACKAGE__->register_method({
name => 'detail',
path => '{name}',
method => 'GET',
proxyto => 'node',
protected => 1,
permissions => {
check => ['perm', '/', ['Sys.Audit']],
},
description => "Get details about a zpool.",
parameters => {
additionalProperties => 0,
properties => {
node => get_standard_option('pve-node'),
name => get_standard_option('pve-storage-id'),
},
},
returns => {
type => 'object',
properties => {
name => {
type => 'string',
description => 'The name of the zpool.',
},
state => {
type => 'string',
description => 'The state of the zpool.',
},
status => {
optional => 1,
type => 'string',
description => 'Information about the state of the zpool.',
},
action => {
optional => 1,
type => 'string',
description => 'Information about the recommended action to fix the state.',
},
scan => {
optional => 1,
type => 'string',
description => 'Information about the last/current scrub.',
},
errors => {
type => 'string',
description => 'Information about the errors on the zpool.',
},
children => {
type => 'array',
description =>
"The pool configuration information, including the vdevs for each section (e.g. spares, cache), may be nested.",
items => {
type => 'object',
properties => {
name => {
type => 'string',
description => 'The name of the vdev or section.',
},
state => {
optional => 1,
type => 'string',
description => 'The state of the vdev.',
},
read => {
optional => 1,
type => 'number',
},
write => {
optional => 1,
type => 'number',
},
cksum => {
optional => 1,
type => 'number',
},
msg => {
type => 'string',
description => 'An optional message about the vdev.',
},
},
},
},
},
},
code => sub {
my ($param) = @_;
if (!-f $ZPOOL) {
die "zfsutils-linux not installed\n";
}
my $cmd = [$ZPOOL, 'status', '-P', '-p', $param->{name}];
my $pool = {
lvl => 0,
};
my $curfield;
my $config = 0;
my $stack = [$pool];
my $curlvl = 0;
run_command(
$cmd,
outfunc => sub {
my ($line) = @_;
if ($line =~ m/^\s*(\S+): (\S+.*)$/) {
$curfield = $1;
$pool->{$curfield} = $2;
$config = 0 if $curfield eq 'errors';
} elsif (!$config && $line =~ m/^\s+(\S+.*)$/) {
$pool->{$curfield} .= " " . $1;
} elsif (!$config && $line =~ m/^\s*config:/) {
$config = 1;
} elsif (
$config
&& $line =~ m/^(\s+)(\S+)\s*(\S+)?(?:\s+(\d+)\s+(\d+)\s+(\d+))?\s*(.*)$/
) {
my ($space, $name, $state, $read, $write, $cksum, $msg) =
($1, $2, $3, $4, $5, $6, $7);
if ($name ne "NAME") {
my $lvl = int(length($space) / 2) + 1; # two spaces per level
my $vdev = {
name => $name,
msg => $msg,
lvl => $lvl,
};
$vdev->{state} = $state if defined($state);
$vdev->{read} = $read + 0 if defined($read);
$vdev->{write} = $write + 0 if defined($write);
$vdev->{cksum} = $cksum + 0 if defined($cksum);
my $cur = pop @$stack;
if ($lvl > $curlvl) {
$cur->{children} = [$vdev];
} elsif ($lvl == $curlvl) {
$cur = pop @$stack;
push @{ $cur->{children} }, $vdev;
} else {
while ($lvl <= $cur->{lvl} && $cur->{lvl} != 0) {
$cur = pop @$stack;
}
push @{ $cur->{children} }, $vdev;
}
push @$stack, $cur;
push @$stack, $vdev;
$curlvl = $lvl;
}
}
},
);
# change treenodes for extjs tree
$pool->{name} = delete $pool->{pool};
preparetree($pool);
return $pool;
},
});
my $draid_config_format = {
spares => {
type => 'integer',
minimum => 0,
description => 'Number of dRAID spares.',
},
data => {
type => 'integer',
minimum => 1,
description => 'The number of data devices per redundancy group. (dRAID)',
},
};
__PACKAGE__->register_method({
name => 'create',
path => '',
method => 'POST',
proxyto => 'node',
protected => 1,
permissions => {
description =>
"Requires additionally 'Datastore.Allocate' on /storage when setting 'add_storage'",
check => ['perm', '/', ['Sys.Modify']],
},
description => "Create a ZFS pool.",
parameters => {
additionalProperties => 0,
properties => {
node => get_standard_option('pve-node'),
name => get_standard_option('pve-storage-id'),
raidlevel => {
type => 'string',
description => 'The RAID level to use.',
enum => [
'single',
'mirror',
'raid10',
'raidz',
'raidz2',
'raidz3',
'draid',
'draid2',
'draid3',
],
},
devices => {
type => 'string',
format => 'string-list',
description => 'The block devices you want to create the zpool on.',
},
'draid-config' => {
type => 'string',
format => $draid_config_format,
optional => 1,
},
ashift => {
type => 'integer',
minimum => 9,
maximum => 16,
optional => 1,
default => 12,
description => 'Pool sector size exponent.',
},
compression => {
type => 'string',
description => 'The compression algorithm to use.',
enum => ['on', 'off', 'gzip', 'lz4', 'lzjb', 'zle', 'zstd'],
optional => 1,
default => 'on',
},
add_storage => {
description => "Configure storage using the zpool.",
type => 'boolean',
optional => 1,
default => 0,
},
},
},
returns => { type => 'string' },
code => sub {
my ($param) = @_;
my $rpcenv = PVE::RPCEnvironment::get();
my $user = $rpcenv->get_user();
my $name = $param->{name};
my $node = $param->{node};
my $devs = [PVE::Tools::split_list($param->{devices})];
my $raidlevel = $param->{raidlevel};
my $compression = $param->{compression} // 'on';
my $draid_config;
if (exists $param->{'draid-config'}) {
die "draid-config set without using dRAID level\n" if $raidlevel !~ m/^draid/;
$draid_config =
parse_property_string($draid_config_format, $param->{'draid-config'});
}
for my $dev (@$devs) {
$dev = PVE::Diskmanage::verify_blockdev_path($dev);
PVE::Diskmanage::assert_disk_unused($dev);
}
my $storage_params = {
type => 'zfspool',
pool => $name,
storage => $name,
content => 'rootdir,images',
nodes => $node,
};
my $verify_params = [qw(pool)];
if ($param->{add_storage}) {
$rpcenv->check($user, "/storage", ['Datastore.Allocate']);
# reserve the name and add as disabled, will be enabled below if creation works out
PVE::API2::Storage::Config->create_or_update(
$name, $node, $storage_params, $verify_params, 1,
);
}
my $pools = get_pool_data();
die "pool '${name}' already exists on node '${node}'\n"
if grep { $_->{name} eq $name } @{$pools};
my $numdisks = scalar(@$devs);
my $mindisks = {
single => 1,
mirror => 2,
raid10 => 4,
raidz => 3,
raidz2 => 4,
raidz3 => 5,
draid => 3,
draid2 => 4,
draid3 => 5,
};
# sanity checks
die "raid10 needs an even number of disks\n"
if $raidlevel eq 'raid10' && $numdisks % 2 != 0;
die "please give only one disk for single disk mode\n"
if $raidlevel eq 'single' && $numdisks > 1;
die "$raidlevel needs at least $mindisks->{$raidlevel} disks\n"
if $numdisks < $mindisks->{$raidlevel};
# draid checks
if ($raidlevel =~ m/^draid/) {
# bare minimum would be two drives: one for parity & one for data, but forbid that
# because it makes no sense in practice, at least one spare disk should be used
my $draid_min = $mindisks->{$raidlevel} - 2;
if ($draid_config) {
$draid_min += $draid_config->{data} || 0;
$draid_min += $draid_config->{spares} || 0;
}
die "At least $draid_min disks needed for current dRAID config\n"
if $numdisks < $draid_min;
}
my $code = sub {
for my $dev (@$devs) {
PVE::Diskmanage::assert_disk_unused($dev);
my $is_partition = PVE::Diskmanage::is_partition($dev);
if ($is_partition) {
eval {
PVE::Diskmanage::change_parttype(
$dev,
'6a898cc3-1dd2-11b2-99a6-080020736631',
);
};
warn $@ if $@;
}
my $sysfsdev = $is_partition ? PVE::Diskmanage::get_blockdev($dev) : $dev;
$sysfsdev =~ s!^/dev/!/sys/block/!;
if ($is_partition) {
my $part = $dev =~ s!^/dev/!!r;
$sysfsdev .= "/${part}";
}
my $udevinfo = PVE::Diskmanage::get_udev_info($sysfsdev);
$dev = $udevinfo->{by_id_link} if defined($udevinfo->{by_id_link});
}
# create zpool with desired raidlevel
my $ashift = $param->{ashift} // 12;
my $cmd = [$ZPOOL, 'create', '-o', "ashift=$ashift", $name];
if ($raidlevel eq 'raid10') {
for (my $i = 0; $i < @$devs; $i += 2) {
push @$cmd, 'mirror', $devs->[$i], $devs->[$i + 1];
}
} elsif ($raidlevel eq 'single') {
push @$cmd, $devs->[0];
} elsif ($raidlevel =~ m/^draid/) {
my $draid_cmd = $raidlevel;
$draid_cmd .= ":$draid_config->{data}d" if $draid_config->{data};
$draid_cmd .= ":$draid_config->{spares}s" if $draid_config->{spares};
push @$cmd, $draid_cmd, @$devs;
} else {
push @$cmd, $raidlevel, @$devs;
}
print "# ", join(' ', @$cmd), "\n";
run_command($cmd);
$cmd = [$ZFS, 'set', "compression=$compression", $name];
print "# ", join(' ', @$cmd), "\n";
run_command($cmd);
if (-e '/lib/systemd/system/zfs-import@.service') {
my $importunit =
'zfs-import@' . PVE::Systemd::escape_unit($name, undef) . '.service';
$cmd = ['systemctl', 'enable', $importunit];
print "# ", join(' ', @$cmd), "\n";
run_command($cmd);
}
PVE::Diskmanage::udevadm_trigger($devs->@*);
if ($param->{add_storage}) {
PVE::API2::Storage::Config->create_or_update(
$name, $node, $storage_params, $verify_params,
);
}
};
return $rpcenv->fork_worker(
'zfscreate',
$name,
$user,
sub {
PVE::Diskmanage::locked_disk_action($code);
},
);
},
});
__PACKAGE__->register_method({
name => 'delete',
path => '{name}',
method => 'DELETE',
proxyto => 'node',
protected => 1,
permissions => {
description =>
"Requires additionally 'Datastore.Allocate' on /storage when setting 'cleanup-config'",
check => ['perm', '/', ['Sys.Modify']],
},
description => "Destroy a ZFS pool.",
parameters => {
additionalProperties => 0,
properties => {
node => get_standard_option('pve-node'),
name => get_standard_option('pve-storage-id'),
'cleanup-config' => {
description =>
"Marks associated storage(s) as not available on this node anymore "
. "or removes them from the configuration (if configured for this node only).",
type => 'boolean',
optional => 1,
default => 0,
},
'cleanup-disks' => {
description => "Also wipe disks so they can be repurposed afterwards.",
type => 'boolean',
optional => 1,
default => 0,
},
},
},
returns => { type => 'string' },
code => sub {
my ($param) = @_;
my $rpcenv = PVE::RPCEnvironment::get();
my $user = $rpcenv->get_user();
$rpcenv->check($user, "/storage", ['Datastore.Allocate']) if $param->{'cleanup-config'};
my $name = $param->{name};
my $node = $param->{node};
my $worker = sub {
PVE::Diskmanage::locked_disk_action(sub {
my $to_wipe = [];
if ($param->{'cleanup-disks'}) {
# Using -o name does not only output the name in combination with -v.
run_command(
['zpool', 'list', '-vHPL', $name],
outfunc => sub {
my ($line) = @_;
my ($name) = PVE::Tools::split_list($line);
return if $name !~ m|^/dev/.+|;
my $dev = PVE::Diskmanage::verify_blockdev_path($name);
my $wipe = $dev;
$dev =~ s|^/dev/||;
my $info = PVE::Diskmanage::get_disks($dev, 1, 1);
die "unable to obtain information for disk '$dev'\n"
if !$info->{$dev};
# Wipe whole disk if usual ZFS layout with partition 9 as ZFS reserved.
my $parent = $info->{$dev}->{parent};
if ($parent && scalar(keys $info->%*) == 3) {
$parent =~ s|^/dev/||;
my $info9 = $info->{"${parent}9"};
$wipe = $info->{$dev}->{parent} # need leading /dev/
if $info9
&& $info9->{used}
&& $info9->{used} =~ m/^ZFS reserved/;
}
push $to_wipe->@*, $wipe;
},
);
}
if (-e '/lib/systemd/system/zfs-import@.service') {
my $importunit =
'zfs-import@' . PVE::Systemd::escape_unit($name) . '.service';
run_command(['systemctl', 'disable', $importunit]);
}
run_command(['zpool', 'destroy', $name]);
my $config_err;
if ($param->{'cleanup-config'}) {
my $match = sub {
my ($scfg) = @_;
return $scfg->{type} eq 'zfspool' && $scfg->{pool} eq $name;
};
eval {
PVE::API2::Storage::Config->cleanup_storages_for_node($match, $node);
};
warn $config_err = $@ if $@;
}
eval { PVE::Diskmanage::wipe_blockdev($_) for $to_wipe->@*; };
my $err = $@;
PVE::Diskmanage::udevadm_trigger($to_wipe->@*);
die "cleanup failed - $err" if $err;
die "config cleanup failed - $config_err" if $config_err;
});
};
return $rpcenv->fork_worker('zfsremove', $name, $user, $worker);
},
});
1;
src/PVE/Diskmanage.pm
+996
View File
@@ -0,0 +1,996 @@
package PVE::Diskmanage;
use strict;
use warnings;
use PVE::ProcFSTools;
use Data::Dumper;
use Cwd qw(abs_path);
use Fcntl ':mode';
use File::Basename;
use File::stat;
use JSON;
use PVE::Tools
qw(extract_param run_command file_get_contents file_read_firstline dir_glob_regex dir_glob_foreach trim);
my $SMARTCTL = "/usr/sbin/smartctl";
my $ZPOOL = "/sbin/zpool";
my $SGDISK = "/sbin/sgdisk";
my $PVS = "/sbin/pvs";
my $LVS = "/sbin/lvs";
my $LSBLK = "/bin/lsblk";
my sub strip_dev : prototype($) {
my ($devpath) = @_;
$devpath =~ s|^/dev/||;
return $devpath;
}
sub check_bin {
my ($path) = @_;
return -x $path;
}
sub verify_blockdev_path {
my ($rel_path) = @_;
die "missing path" if !$rel_path;
my $path = abs_path($rel_path);
die "failed to get absolute path to $rel_path\n" if !$path;
die "got unusual device path '$path'\n" if $path !~ m|^/dev/(.*)$|;
$path = "/dev/$1"; # untaint
assert_blockdev($path);
return $path;
}
sub assert_blockdev {
my ($dev, $noerr) = @_;
if ($dev !~ m|^/dev/| || !(-b $dev)) {
return if $noerr;
die "not a valid block device\n";
}
return 1;
}
sub init_disk {
my ($disk, $uuid) = @_;
assert_blockdev($disk);
# we should already have checked these in the api call, but we check again for safety
die "$disk is a partition\n" if is_partition($disk);
die "disk $disk is already in use\n" if disk_is_used($disk);
my $id = $uuid || 'R';
run_command([$SGDISK, $disk, '-U', $id]);
return 1;
}
sub disk_is_used {
my ($disk) = @_;
my $dev = $disk;
$dev =~ s|^/dev/||;
my $disklist = get_disks($dev, 1, 1);
die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev});
return 1 if $disklist->{$dev}->{used};
return 0;
}
sub get_smart_data {
my ($disk, $healthonly) = @_;
assert_blockdev($disk);
my $smartdata = {};
my $type;
my $cmd = [$SMARTCTL, '-H'];
push @$cmd, '-A', '-f', 'brief' if !$healthonly;
push @$cmd, $disk;
my $returncode = eval {
run_command(
$cmd,
noerr => 1,
timeout => 10,
outfunc => sub {
my ($line) = @_;
# ATA SMART attributes, e.g.:
# ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
# 1 Raw_Read_Error_Rate POSR-K 100 100 000 - 0
# 5 Reallocated_Sector_Ct ------+ 100 100 050 - 0
#
# SAS and NVME disks, e.g.:
# Data Units Written: 5,584,952 [2.85 TB]
# Accumulated start-stop cycles: 34
if (
defined($type)
&& $type eq 'ata'
&& $line =~
m/^([ \d]{2}\d)\s+(\S+)\s+(\S{6}\+?)\s+(\d+)\s+(\d+)\s+(\S+)\s+(\S+)\s+(.*)$/
) {
my $entry = {};
$entry->{name} = $2 if defined $2;
$entry->{flags} = $3 if defined $3;
# the +0 makes a number out of the strings
# FIXME: 'value' is depreacated by 'normalized'; remove with PVE 7.0
$entry->{value} = $4 + 0 if defined $4;
$entry->{normalized} = $4 + 0 if defined $4;
$entry->{worst} = $5 + 0 if defined $5;
# some disks report the default threshold as --- instead of 000
if (defined($6) && $6 eq '---') {
$entry->{threshold} = 0;
} else {
$entry->{threshold} = $6 + 0 if defined $6;
}
$entry->{fail} = $7 if defined $7;
$entry->{raw} = $8 if defined $8;
$entry->{id} = $1 if defined $1;
push @{ $smartdata->{attributes} }, $entry;
} elsif ($line =~ m/(?:Health Status|self\-assessment test result): (.*)$/) {
$smartdata->{health} = $1;
} elsif ($line =~ m/Vendor Specific SMART Attributes with Thresholds:/) {
$type = 'ata';
delete $smartdata->{text};
} elsif ($line =~ m/=== START OF (READ )?SMART DATA SECTION ===/) {
$type = 'text';
} elsif (defined($type) && $type eq 'text') {
$smartdata->{text} = '' if !defined $smartdata->{text};
$smartdata->{text} .= "$line\n";
# extract wearout from nvme/sas text, allow for decimal values
if ($line =~
m/Percentage Used(?: endurance indicator)?:\s*(\d+(?:\.\d+)?)\%/i
) {
$smartdata->{wearout} = 100 - $1;
}
} elsif ($line =~ m/SMART Disabled/) {
$smartdata->{health} = "SMART Disabled";
}
},
);
};
die "Error getting S.M.A.R.T. data: $@\n" if $@;
# bit 0 and 1 mark a fatal error, other bits are for disk status -> ignore (see man 8 smartctl)
if (defined($returncode) && ($returncode & 0b00000011)) {
die "Error getting S.M.A.R.T. data: Exit code: $returncode\n";
}
$smartdata->{type} = $type;
return $smartdata;
}
sub get_lsblk_info {
my $cmd = [$LSBLK, '--json', '-o', 'path,parttype,fstype'];
my $output = "";
eval {
run_command($cmd, outfunc => sub { $output .= "$_[0]\n"; });
};
warn "$@\n" if $@;
return {} if $output eq '';
my $parsed = eval { decode_json($output) } // {};
warn "$@\n" if $@;
my $list = $parsed->{blockdevices} // [];
return {
map {
$_->{path} => {
parttype => $_->{parttype},
fstype => $_->{fstype},
}
} @{$list}
};
}
my sub get_devices_by_partuuid {
my ($lsblk_info, $uuids, $res) = @_;
$res = {} if !defined($res);
foreach my $dev (sort keys %{$lsblk_info}) {
my $uuid = $lsblk_info->{$dev}->{parttype};
next if !defined($uuid) || !defined($uuids->{$uuid});
$res->{$dev} = $uuids->{$uuid};
}
return $res;
}
sub get_zfs_devices {
my ($lsblk_info) = @_;
my $res = {};
return {} if !check_bin($ZPOOL);
# use zpool and parttype uuid, because log and cache do not have zfs type uuid
eval {
run_command(
[$ZPOOL, 'list', '-HPLv'],
outfunc => sub {
my ($line) = @_;
if ($line =~ m|^\t([^\t]+)\t|) {
$res->{$1} = 1;
}
},
);
};
# only warn here, because maybe zfs tools are not installed
warn "$@\n" if $@;
my $uuids = {
"6a898cc3-1dd2-11b2-99a6-080020736631" => 1, # apple
"516e7cba-6ecf-11d6-8ff8-00022d09712b" => 1, # bsd
};
$res = get_devices_by_partuuid($lsblk_info, $uuids, $res);
return $res;
}
sub get_lvm_devices {
my ($lsblk_info) = @_;
my $res = {};
eval {
run_command(
[$PVS, '--noheadings', '--readonly', '-o', 'pv_name'],
outfunc => sub {
my ($line) = @_;
$line = trim($line);
if ($line =~ m|^/dev/|) {
$res->{$line} = 1;
}
},
);
};
# if something goes wrong, we do not want to give up, but indicate an error has occurred
warn "$@\n" if $@;
my $uuids = {
"e6d6d379-f507-44c2-a23c-238f2a3df928" => 1,
};
$res = get_devices_by_partuuid($lsblk_info, $uuids, $res);
return $res;
}
sub get_ceph_journals {
my ($lsblk_info) = @_;
my $res = {};
my $uuids = {
'45b0969e-9b03-4f30-b4c6-b4b80ceff106' => 1, # journal
'30cd0809-c2b2-499c-8879-2d6b78529876' => 2, # db
'5ce17fce-4087-4169-b7ff-056cc58473f9' => 3, # wal
'cafecafe-9b03-4f30-b4c6-b4b80ceff106' => 4, # block
};
$res = get_devices_by_partuuid($lsblk_info, $uuids, $res);
return $res;
}
# reads the lv_tags and matches them with the devices
sub get_ceph_volume_infos {
my $result = {};
my $cmd = [
$LVS,
'-S',
'lv_name=~^osd-',
'-o',
'devices,lv_name,lv_tags',
'--noheadings',
'--readonly',
'--separator',
';',
];
run_command(
$cmd,
outfunc => sub {
my $line = shift;
$line =~ s/(?:^\s+)|(?:\s+$)//g; # trim whitespaces
my $fields = [split(';', $line)];
# lvs syntax is /dev/sdX(Y) where Y is the start (which we do not need)
my ($dev) = $fields->[0] =~ m|^(/dev/[a-z]+[^(]*)|;
if ($fields->[1] =~ m|^osd-([^-]+)-|) {
my $type = $1;
# $result autovivification is wanted, to not creating empty hashes
if (
($type eq 'block' || $type eq 'data')
&& $fields->[2]
&& $fields->[2] =~ m/ceph.osd_id=([^,]+)/
) {
$result->{$dev}->{osdid} = $1;
if (!defined($result->{$dev}->{'osdid-list'})) {
$result->{$dev}->{'osdid-list'} = [];
}
push($result->{$dev}->{'osdid-list'}->@*, $1);
$result->{$dev}->{bluestore} = ($type eq 'block');
if ($fields->[2] =~ m/ceph\.encrypted=1/) {
$result->{$dev}->{encrypted} = 1;
}
} else {
# undef++ becomes '1' (see `perldoc perlop`: Auto-increment)
$result->{$dev}->{$type}++;
}
}
},
);
return $result;
}
sub get_udev_info {
my ($dev) = @_;
my $info = "";
my $data = {};
eval {
run_command(
['udevadm', 'info', '-p', $dev, '--query', 'all'],
outfunc => sub {
my ($line) = @_;
$info .= "$line\n";
},
);
};
warn $@ if $@;
return if !$info;
return if $info !~ m/^E: DEVTYPE=(disk|partition)$/m;
return if $info =~ m/^E: ID_CDROM/m;
# we use this, because some disks are not simply in /dev e.g. /dev/cciss/c0d0
if ($info =~ m/^E: DEVNAME=(\S+)$/m) {
$data->{devpath} = $1;
}
return if !defined($data->{devpath});
$data->{serial} = 'unknown';
$data->{serial} = $1 if $info =~ m/^E: ID_SERIAL_SHORT=\s*(\S+)$/m;
$data->{gpt} = $info =~ m/^E: ID_PART_TABLE_TYPE=gpt$/m ? 1 : 0;
$data->{rpm} = -1;
$data->{rpm} = $1 if $info =~ m/^E: ID_ATA_ROTATION_RATE_RPM=(\d+)$/m; # detects SSD implicit
$data->{usb} = 1 if $info =~ m/^E: ID_BUS=usb$/m;
$data->{model} = $1 if $info =~ m/^E: ID_MODEL=(.+)$/m;
$data->{wwn} = 'unknown';
$data->{wwn} = $1 if $info =~ m/^E: ID_WWN=(.*)$/m;
if ($info =~ m/^E: DEVLINKS=(.+)$/m) {
my @devlinks = grep(m#^/dev/disk/by-id/(ata|scsi|nvme(?!-eui))#, split(/ /, $1));
$data->{by_id_link} = $devlinks[0] if defined($devlinks[0]);
}
return $data;
}
sub get_sysdir_size {
my ($sysdir) = @_;
my $size = file_read_firstline("$sysdir/size");
return if !$size;
# linux always considers sectors to be 512 bytes, independently of real block size
return $size * 512;
}
sub get_sysdir_info {
my ($sysdir) = @_;
return if !-d "$sysdir/device";
my $data = {};
$data->{size} = get_sysdir_size($sysdir) or return;
# dir/queue/rotational should be 1 for hdd, 0 for ssd
$data->{rotational} = file_read_firstline("$sysdir/queue/rotational") // -1;
$data->{vendor} = file_read_firstline("$sysdir/device/vendor") || 'unknown';
$data->{model} = file_read_firstline("$sysdir/device/model") || 'unknown';
return $data;
}
sub get_wear_leveling_info {
my ($smartdata) = @_;
my $attributes = $smartdata->{attributes};
if (defined($smartdata->{wearout})) {
return $smartdata->{wearout};
}
my $wearout;
# Common register names that represent percentage values of potential failure indicators used
# in drivedb.h of smartmontool's. Order matters, as some drives may have multiple definitions
my @wearoutregisters = (
"Media_Wearout_Indicator",
"SSD_Life_Left",
"Wear_Leveling_Count",
"Perc_Write\/Erase_Ct_BC",
"Perc_Rated_Life_Remain",
"Remaining_Lifetime_Perc",
"Percent_Lifetime_Remain",
"Lifetime_Left",
"PCT_Life_Remaining",
"Lifetime_Remaining",
"Percent_Life_Remaining",
"Percent_Lifetime_Used",
"Perc_Rated_Life_Used",
);
# Search for S.M.A.R.T. attributes for known register
foreach my $register (@wearoutregisters) {
last if defined $wearout;
foreach my $attr (@$attributes) {
next if $attr->{name} !~ m/$register/;
$wearout = $attr->{value};
last;
}
}
return $wearout;
}
sub dir_is_empty {
my ($dir) = @_;
my $dh = IO::Dir->new($dir);
return 1 if !$dh;
while (defined(my $tmp = $dh->read)) {
next if $tmp eq '.' || $tmp eq '..';
$dh->close;
return 0;
}
$dh->close;
return 1;
}
sub is_iscsi {
my ($sysdir) = @_;
if (-l $sysdir && readlink($sysdir) =~ m|host[^/]*/session[^/]*|) {
return 1;
}
return 0;
}
my sub is_ssdlike {
my ($type) = @_;
return $type eq 'ssd' || $type eq 'nvme';
}
sub mounted_blockdevs {
my $mounted = {};
my $mounts = PVE::ProcFSTools::parse_proc_mounts();
foreach my $mount (@$mounts) {
next if $mount->[0] !~ m|^/dev/|;
$mounted->{ abs_path($mount->[0]) } = $mount->[1];
}
return $mounted;
}
# returns hashmap of abs mount path -> first part of /proc/mounts (what)
sub mounted_paths {
my $mounted = {};
my $mounts = PVE::ProcFSTools::parse_proc_mounts();
foreach my $mount (@$mounts) {
$mounted->{ abs_path($mount->[1]) } = $mount->[0];
}
return $mounted;
}
sub get_disks {
my ($disks, $nosmart, $include_partitions) = @_;
my $disklist = {};
my $mounted = mounted_blockdevs();
my $lsblk_info = get_lsblk_info();
my $journalhash = get_ceph_journals($lsblk_info);
my $ceph_volume_infos = get_ceph_volume_infos();
my $zfshash = get_zfs_devices($lsblk_info);
my $lvmhash = get_lvm_devices($lsblk_info);
my $disk_regex = ".*";
if (defined($disks)) {
if (!ref($disks)) {
$disks = [$disks];
} elsif (ref($disks) ne 'ARRAY') {
die "disks is not a string or array reference\n";
}
# we get cciss/c0d0 but need cciss!c0d0
$_ =~ s|cciss/|cciss!| for @$disks;
if ($include_partitions) {
# Proper blockdevice is needed for the regex, use parent for partitions.
for my $disk ($disks->@*) {
next if !is_partition("/dev/$disk");
$disk = strip_dev(get_blockdev("/dev/$disk"));
}
}
$disk_regex = "(?:" . join('|', @$disks) . ")";
}
dir_glob_foreach(
'/sys/block',
$disk_regex,
sub {
my ($dev) = @_;
# whitelisting following devices
# - hdX ide block device
# - sdX scsi/sata block device
# - vdX virtIO block device
# - xvdX: xen virtual block device
# - nvmeXnY: nvme devices
# - cciss!cXnY cciss devices
return
if $dev !~ m/^(h|s|x?v)d[a-z]+$/
&& $dev !~ m/^nvme\d+n\d+$/
&& $dev !~ m/^cciss\!c\d+d\d+$/;
my $data = get_udev_info("/sys/block/$dev") // return;
my $devpath = $data->{devpath};
my $sysdir = "/sys/block/$dev";
# we do not want iscsi devices
return if is_iscsi($sysdir);
my $sysdata = get_sysdir_info($sysdir);
return if !defined($sysdata);
my $type = 'unknown';
if ($sysdata->{rotational} == 0) {
$type = 'ssd';
$type = 'nvme' if $dev =~ m/^nvme\d+n\d+$/;
$data->{rpm} = 0;
} elsif ($sysdata->{rotational} == 1) {
if ($data->{rpm} != -1) {
$type = 'hdd';
} elsif ($data->{usb}) {
$type = 'usb';
$data->{rpm} = 0;
}
}
my ($health, $wearout) = ('UNKNOWN', 'N/A');
if (!$nosmart) {
eval {
my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
$health = $smartdata->{health} if $smartdata->{health};
if (is_ssdlike($type)) { # if we have an ssd we try to get the wearout indicator
my $wear_level = get_wear_leveling_info($smartdata);
$wearout = $wear_level if defined($wear_level);
}
};
}
# we replaced cciss/ with cciss! above, but in the result we need cciss/ again because the
# caller might want to check the result again with the original parameter
if ($dev =~ m|^cciss!|) {
$dev =~ s|^cciss!|cciss/|;
}
$disklist->{$dev} = {
vendor => $sysdata->{vendor},
model => $data->{model} || $sysdata->{model},
size => $sysdata->{size},
serial => $data->{serial},
gpt => $data->{gpt},
rpm => $data->{rpm},
type => $type,
wwn => $data->{wwn},
health => $health,
devpath => $devpath,
wearout => $wearout,
};
$disklist->{$dev}->{mounted} = 1 if exists $mounted->{$devpath};
my $by_id_link = $data->{by_id_link};
$disklist->{$dev}->{by_id_link} = $by_id_link if defined($by_id_link);
my ($osdid, $bluestore, $osdencrypted) = (-1, 0, 0);
my $osdid_list;
my ($journal_count, $db_count, $wal_count) = (0, 0, 0);
my $partpath = $devpath;
# remove trailing part to get the partition base path, e.g. /dev/cciss/c0d0 -> /dev/cciss
$partpath =~ s/\/[^\/]+$//;
my $determine_usage = sub {
my ($devpath, $sysdir, $is_partition) = @_;
return 'LVM' if $lvmhash->{$devpath};
return 'ZFS' if $zfshash->{$devpath};
my $info = $lsblk_info->{$devpath} // {};
if (defined(my $parttype = $info->{parttype})) {
return 'BIOS boot' if $parttype eq '21686148-6449-6e6f-744e-656564454649';
return 'EFI' if $parttype eq 'c12a7328-f81f-11d2-ba4b-00a0c93ec93b';
return 'ZFS reserved'
if $parttype eq '6a945a3b-1dd2-11b2-99a6-080020736631';
}
return "$info->{fstype}" if defined($info->{fstype});
return 'mounted' if $mounted->{$devpath};
return if !$is_partition;
# for devices, this check is done explicitly later
return 'Device Mapper' if !dir_is_empty("$sysdir/holders");
return; # unused partition
};
my $collect_ceph_info = sub {
my ($devpath) = @_;
my $ceph_volume = $ceph_volume_infos->{$devpath} or return;
$journal_count += $ceph_volume->{journal} // 0;
$db_count += $ceph_volume->{db} // 0;
$wal_count += $ceph_volume->{wal} // 0;
if (defined($ceph_volume->{osdid})) {
$osdid = $ceph_volume->{osdid};
$osdid_list = $ceph_volume->{'osdid-list'};
$bluestore = 1 if $ceph_volume->{bluestore};
$osdencrypted = 1 if $ceph_volume->{encrypted};
}
my $result = { %{$ceph_volume} };
$result->{journals} = delete $result->{journal} if $result->{journal};
return $result;
};
my $partitions = {};
dir_glob_foreach(
"$sysdir",
"$dev.+",
sub {
my ($part) = @_;
$partitions->{$part} = $collect_ceph_info->("$partpath/$part");
my $lvm_based_osd = defined($partitions->{$part});
$partitions->{$part}->{devpath} = "$partpath/$part";
$partitions->{$part}->{parent} = "$devpath";
$partitions->{$part}->{mounted} = 1 if exists $mounted->{"$partpath/$part"};
$partitions->{$part}->{gpt} = $data->{gpt};
$partitions->{$part}->{type} = 'partition';
$partitions->{$part}->{size} = get_sysdir_size("$sysdir/$part") // 0;
$partitions->{$part}->{used} =
$determine_usage->("$partpath/$part", "$sysdir/$part", 1);
$partitions->{$part}->{osdid} //= -1;
$partitions->{$part}->{'osdid-list'} //= undef;
# avoid counting twice (e.g. partition with the LVM for the DB OSD is in $journalhash)
return if $lvm_based_osd;
# Legacy handling for non-LVM based OSDs
if (my $mp = $mounted->{"$partpath/$part"}) {
if ($mp =~ m|^/var/lib/ceph/osd/ceph-(\d+)$|) {
$osdid = $1;
$partitions->{$part}->{osdid} = $osdid;
$osdid_list = [$1]; # assuming only one OSD per disk
$partitions->{$part}->{'osdid-list'} = $osdid_list;
}
}
if (my $journal_part = $journalhash->{"$partpath/$part"}) {
$journal_count++ if $journal_part == 1;
$db_count++ if $journal_part == 2;
$wal_count++ if $journal_part == 3;
$bluestore = 1 if $journal_part == 4;
$partitions->{$part}->{journals} = 1 if $journal_part == 1;
$partitions->{$part}->{db} = 1 if $journal_part == 2;
$partitions->{$part}->{wal} = 1 if $journal_part == 3;
$partitions->{$part}->{bluestore} = 1 if $journal_part == 4;
}
},
);
my $used = $determine_usage->($devpath, $sysdir, 0);
if (!$include_partitions) {
foreach my $part (sort keys %{$partitions}) {
$used //= $partitions->{$part}->{used};
}
} else {
# fstype might be set even if there are partitions, but showing that is confusing
$used = 'partitions' if scalar(keys %{$partitions});
}
$used //= 'partitions' if scalar(keys %{$partitions});
# multipath, software raid, etc.
# this check comes in last, to show more specific info
# if we have it
$used //= 'Device Mapper' if !dir_is_empty("$sysdir/holders");
$disklist->{$dev}->{used} = $used if $used;
$collect_ceph_info->($devpath);
$disklist->{$dev}->{osdid} = $osdid;
$disklist->{$dev}->{'osdid-list'} = $osdid_list;
$disklist->{$dev}->{journals} = $journal_count if $journal_count;
$disklist->{$dev}->{bluestore} = $bluestore if $osdid != -1;
$disklist->{$dev}->{osdencrypted} = $osdencrypted if $osdid != -1;
$disklist->{$dev}->{db} = $db_count if $db_count;
$disklist->{$dev}->{wal} = $wal_count if $wal_count;
if ($include_partitions) {
$disklist->{$_} = $partitions->{$_} for keys %{$partitions};
}
},
);
return $disklist;
}
sub get_partnum {
my ($part_path) = @_;
my $st = stat($part_path);
die "error detecting block device '$part_path'\n"
if !$st || !$st->mode || !S_ISBLK($st->mode) || !$st->rdev;
my $major = PVE::Tools::dev_t_major($st->rdev);
my $minor = PVE::Tools::dev_t_minor($st->rdev);
my $partnum_path = "/sys/dev/block/$major:$minor/";
my $partnum = file_read_firstline("${partnum_path}partition");
die "Partition does not exist\n" if !defined($partnum);
die "Failed to get partition number\n" if $partnum !~ m/(\d+)/; # untaint
$partnum = $1;
die "Partition number $partnum is invalid\n" if $partnum > 128;
return $partnum;
}
sub get_blockdev {
my ($part_path) = @_;
my ($dev, $block_dev);
if ($part_path =~ m|^/dev/(.*)$|) {
$dev = $1;
my $link = readlink "/sys/class/block/$dev";
$block_dev = $1 if $link =~ m|([^/]*)/$dev$|;
}
die "Can't parse parent device\n" if !defined($block_dev);
die "No valid block device\n" if index($dev, $block_dev) == -1;
$block_dev = "/dev/$block_dev";
die "Block device does not exists\n" if !(-b $block_dev);
return $block_dev;
}
sub is_partition {
my ($dev_path) = @_;
return defined(eval { get_partnum($dev_path) });
}
sub locked_disk_action {
my ($sub) = @_;
my $res = PVE::Tools::lock_file('/run/lock/pve-diskmanage.lck', undef, $sub);
die $@ if $@;
return $res;
}
sub assert_disk_unused {
my ($dev) = @_;
die "device '$dev' is already in use\n" if disk_is_used($dev);
return;
}
sub append_partition {
my ($dev, $size) = @_;
my $devname = $dev;
$devname =~ s|^/dev/||;
my $newpartid = 1;
dir_glob_foreach(
"/sys/block/$devname",
qr/\Q$devname\E.*?(\d+)/,
sub {
my ($part, $partid) = @_;
if ($partid >= $newpartid) {
$newpartid = $partid + 1;
}
},
);
$size = PVE::Tools::convert_size($size, 'b' => 'mb');
run_command(
[$SGDISK, '-n', "$newpartid:0:+${size}M", $dev],
errmsg => "error creating partition '$newpartid' on '$dev'",
);
my $partition;
# loop again to detect the real partition device which does not always follow
# a strict $devname$partition scheme like /dev/nvme0n1 -> /dev/nvme0n1p1
dir_glob_foreach(
"/sys/block/$devname",
qr/\Q$devname\E.*$newpartid/,
sub {
my ($part) = @_;
$partition = "/dev/$part";
},
);
return $partition;
}
# Check if a disk or any of its partitions has a holder.
# Can also be called with a partition.
# Expected to be called with a result of verify_blockdev_path().
sub has_holder {
my ($devpath) = @_;
my $dev = strip_dev($devpath);
return $devpath if !dir_is_empty("/sys/class/block/${dev}/holders");
my $found;
dir_glob_foreach(
"/sys/block/${dev}",
"${dev}.+",
sub {
my ($part) = @_;
$found = "/dev/${part}" if !dir_is_empty("/sys/class/block/${part}/holders");
},
);
return $found;
}
# Basic check if a disk or any of its partitions is mounted.
# Can also be called with a partition.
# Expected to be called with a result of verify_blockdev_path().
sub is_mounted {
my ($devpath) = @_;
my $mounted = mounted_blockdevs();
return $devpath if $mounted->{$devpath};
my $dev = strip_dev($devpath);
my $found;
dir_glob_foreach(
"/sys/block/${dev}",
"${dev}.+",
sub {
my ($part) = @_;
my $partpath = "/dev/${part}";
$found = $partpath if $mounted->{$partpath};
},
);
return $found;
}
# Currently only supports GPT-partitioned disks.
sub change_parttype {
my ($partpath, $parttype) = @_;
my $err = "unable to change partition type for $partpath";
my $partnum = get_partnum($partpath);
my $blockdev = get_blockdev($partpath);
my $dev = strip_dev($blockdev);
my $info = get_disks($dev, 1);
die "$err - unable to get disk info for '$blockdev'\n" if !defined($info->{$dev});
die "$err - disk '$blockdev' is not GPT partitioned\n" if !$info->{$dev}->{gpt};
run_command(['sgdisk', "-t${partnum}:${parttype}", $blockdev], errmsg => $err);
}
# Wipes all labels and the first 200 MiB of a disk/partition (or the whole if it is smaller).
# If called with a partition, also sets the partition type to 0x83 'Linux filesystem'.
# Expected to be called with a result of verify_blockdev_path().
sub wipe_blockdev {
my ($devpath) = @_;
my $devname = basename($devpath);
my $dev_size = PVE::Tools::file_get_contents("/sys/class/block/$devname/size");
($dev_size) = $dev_size =~ m|(\d+)|; # untaint $dev_size
die "Couldn't get the size of the device $devname\n" if !defined($dev_size);
my $size = ($dev_size * 512 / 1024 / 1024);
my $count = ($size < 200) ? $size : 200;
my $to_wipe = [];
dir_glob_foreach(
"/sys/class/block/${devname}",
"${devname}.+",
sub {
my ($part) = @_;
push $to_wipe->@*, "/dev/${part}" if -b "/dev/${part}";
},
);
if (scalar($to_wipe->@*) > 0) {
print "found child partitions to wipe: " . join(', ', $to_wipe->@*) . "\n";
}
push $to_wipe->@*, $devpath; # put actual device last
print "wiping block device ${devpath}\n";
run_command(['wipefs', '--all', $to_wipe->@*], errmsg => "error wiping '${devpath}'");
run_command(
['dd', 'if=/dev/zero', "of=${devpath}", 'bs=1M', 'conv=fdatasync', "count=${count}"],
errmsg => "error wiping '${devpath}'",
);
if (is_partition($devpath)) {
eval { change_parttype($devpath, '8300'); };
warn $@ if $@;
}
}
# FIXME: Remove once we depend on systemd >= v249.
# Work around udev bug https://github.com/systemd/systemd/issues/18525 ensuring database is updated.
sub udevadm_trigger {
my @devs = @_;
return if scalar(@devs) == 0;
eval { run_command(['udevadm', 'trigger', @devs]); };
warn $@ if $@;
}
1;
src/PVE/GuestImport/OVF.pm
+42 -19
View File
@@ -369,24 +369,6 @@ ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType", $controller_id,
};
}
# the address is only unique per controller, so disks on different controllers of the same
# type can end up with the same address, move those to the next free one of that bus
my $used_addresses = { map { $_->{disk_address} => 1 } @disks };
my $assigned_addresses = {};
for my $disk (@disks) {
if (
$assigned_addresses->{ $disk->{disk_address} }
&& $disk->{disk_address} =~ m/^([a-z]+)\d+$/
) {
my $bus = $1;
my $index = 0;
$index++ while $used_addresses->{"$bus$index"};
$disk->{disk_address} = "$bus$index";
$used_addresses->{ $disk->{disk_address} } = 1;
}
$assigned_addresses->{ $disk->{disk_address} } = 1;
}
my $nic_id = dtmf_name_to_id('Ethernet Adapter');
my $xpath_find_nics =
"/ovf:Envelope/ovf:VirtualSystem/ovf:VirtualHardwareSection/ovf:Item[rasd:ResourceType=${nic_id}]";
@@ -403,7 +385,46 @@ ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType", $controller_id,
$net_count++;
}
return { qm => $qm, disks => \@disks, net => $net };
# CD drives are added empty, as images cannot be imported, but the referenced image is kept
# so the user can be told about it
my $cdrom_id = dtmf_name_to_id('CD Drive');
my $xpath_find_cdroms =
"/ovf:Envelope/ovf:VirtualSystem/ovf:VirtualHardwareSection/ovf:Item[rasd:ResourceType=${cdrom_id}]";
my @cdrom_items = $xpc->findnodes($xpath_find_cdroms);
my $addresses_in_use = { map { $_->{disk_address} => 1 } @disks };
my @cdroms;
for my $item_node (@cdrom_items) {
my $controller_id = $xpc->findvalue('rasd:Parent', $item_node);
my $xpath_find_parent_type = sprintf(
"/ovf:Envelope/ovf:VirtualSystem/ovf:VirtualHardwareSection/\
ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType", $controller_id,
);
my $bus = id_to_pve($xpc->findvalue($xpath_find_parent_type));
if (!$bus) {
warn "invalid or missing controller for CD drive, skipping\n";
next;
}
my $index = $xpc->findvalue('rasd:AddressOnParent', $item_node);
$index = 0 if $index !~ m/^\d+$/;
$index++ while $addresses_in_use->{"$bus$index"};
my $address = "$bus$index";
$addresses_in_use->{$address} = 1;
my $image;
my $host_resource = $xpc->findvalue('rasd:HostResource', $item_node);
if ($host_resource =~ m|^(?:ovf:)?/file/([[:alnum:]\-\._~]+)$|) {
my $xpath_find_image =
sprintf("/ovf:Envelope/ovf:References/ovf:File[\@ovf:id='%s']/\@ovf:href", $1);
$image = $xpc->findvalue($xpath_find_image) || undef;
}
$qm->{$address} = 'none,media=cdrom';
push @cdroms, { disk_address => $address, image => $image };
}
return { qm => $qm, disks => \@disks, net => $net, cdroms => \@cdroms };
}
my sub resolve_disks {
@@ -454,6 +475,8 @@ my sub resolve_disks {
push @$boot_order, $pve_disk_address;
}
push @$boot_order, map { $_->{disk_address} } $qm->{cdroms}->@*;
$qm->{qm}->{boot} = "order=" . join(';', @$boot_order) if scalar(@$boot_order) > 0;
$qm->{disks} = \@disks;
src/PVE/Storage/BTRFSPlugin.pm
+1005
View File
File diff suppressed because it is too large. Load diff
src/PVE/Storage/DirPlugin.pm
+336
View File
@@ -0,0 +1,336 @@
package PVE::Storage::DirPlugin;
use strict;
use warnings;
use Cwd;
use Encode qw(decode encode);
use File::Path;
use File::Spec;
use IO::File;
use POSIX;
use PVE::Storage::Plugin;
use PVE::GuestImport::OVF;
use PVE::JSONSchema qw(get_standard_option);
use base qw(PVE::Storage::Plugin);
# Configuration
sub type {
return 'dir';
}
sub plugindata {
return {
content => [
{
images => 1,
rootdir => 1,
vztmpl => 1,
iso => 1,
backup => 1,
snippets => 1,
none => 1,
import => 1,
},
{ images => 1, rootdir => 1 },
],
format => [{ raw => 1, qcow2 => 1, vmdk => 1, subvol => 1 }, 'raw'],
'sensitive-properties' => {},
};
}
sub properties {
return {
path => {
description => "File system path.",
type => 'string',
format => 'pve-storage-path',
},
mkdir => {
description =>
"Create the directory if it doesn't exist and populate it with default sub-dirs."
. " NOTE: Deprecated, use the 'create-base-path' and 'create-subdirs' options instead.",
type => 'boolean',
default => 'yes',
},
'create-base-path' => {
description => "Create the base directory if it doesn't exist.",
type => 'boolean',
default => 'yes',
},
'create-subdirs' => {
description => "Populate the directory with the default structure.",
type => 'boolean',
default => 'yes',
},
is_mountpoint => {
description => "Assume the given path is an externally managed mountpoint "
. "and consider the storage offline if it is not mounted. "
. "Using a boolean (yes/no) value serves as a shortcut to using the target path in this field.",
type => 'string',
default => 'no',
},
bwlimit => get_standard_option('bwlimit'),
};
}
sub options {
return {
path => { fixed => 1 },
'content-dirs' => { optional => 1 },
nodes => { optional => 1 },
shared => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
'max-protected-backups' => { optional => 1 },
content => { optional => 1 },
format => { optional => 1 },
mkdir => { optional => 1 },
'create-base-path' => { optional => 1 },
'create-subdirs' => { optional => 1 },
is_mountpoint => { optional => 1 },
bwlimit => { optional => 1 },
preallocation => { optional => 1 },
'snapshot-as-volume-chain' => { optional => 1, fixed => 1 },
};
}
# Storage implementation
#
# NOTE: should ProcFSTools::is_mounted accept an optional cache like this?
sub path_is_mounted {
my ($mountpoint, $mountdata) = @_;
$mountpoint = Cwd::realpath($mountpoint); # symlinks
return 0 if !defined($mountpoint); # path does not exist
$mountdata = PVE::ProcFSTools::parse_proc_mounts() if !$mountdata;
return 1 if grep { $_->[1] eq $mountpoint } @$mountdata;
return undef;
}
sub parse_is_mountpoint {
my ($scfg) = @_;
my $is_mp = $scfg->{is_mountpoint};
return undef if !defined $is_mp;
if (defined(my $bool = PVE::JSONSchema::parse_boolean($is_mp))) {
return $bool ? $scfg->{path} : undef;
}
return $is_mp; # contains a path
}
# FIXME move into 'get_volume_attribute' when removing 'get_volume_notes'
my $get_volume_notes_impl = sub {
my ($class, $scfg, $storeid, $volname, $timeout) = @_;
my ($vtype) = $class->parse_volname($volname);
return if $vtype ne 'backup';
my $path = $class->filesystem_path($scfg, $volname);
$path .= $class->SUPER::NOTES_EXT;
if (-f $path) {
my $data = PVE::Tools::file_get_contents($path);
return eval { decode('UTF-8', $data, 1) } // $data;
}
return '';
};
# FIXME remove on the next APIAGE reset.
# Deprecated, use get_volume_attribute instead.
sub get_volume_notes {
my ($class, $scfg, $storeid, $volname, $timeout) = @_;
return $get_volume_notes_impl->($class, $scfg, $storeid, $volname, $timeout);
}
# FIXME move into 'update_volume_attribute' when removing 'update_volume_notes'
my $update_volume_notes_impl = sub {
my ($class, $scfg, $storeid, $volname, $notes, $timeout) = @_;
my ($vtype) = $class->parse_volname($volname);
die "only backups can have notes\n" if $vtype ne 'backup';
my $path = $class->filesystem_path($scfg, $volname);
$path .= $class->SUPER::NOTES_EXT;
if (defined($notes) && $notes ne '') {
my $encoded = encode('UTF-8', $notes);
PVE::Tools::file_set_contents($path, $encoded);
} else {
unlink $path or $! == ENOENT or die "could not delete notes - $!\n";
}
return;
};
# FIXME remove on the next APIAGE reset.
# Deprecated, use update_volume_attribute instead.
sub update_volume_notes {
my ($class, $scfg, $storeid, $volname, $notes, $timeout) = @_;
return $update_volume_notes_impl->($class, $scfg, $storeid, $volname, $notes, $timeout);
}
sub get_volume_attribute {
my ($class, $scfg, $storeid, $volname, $attribute) = @_;
if ($attribute eq 'notes') {
return $get_volume_notes_impl->($class, $scfg, $storeid, $volname);
}
my ($vtype) = $class->parse_volname($volname);
return if $vtype ne 'backup';
if ($attribute eq 'protected') {
my $path = $class->filesystem_path($scfg, $volname);
return -e PVE::Storage::protection_file_path($path) ? 1 : 0;
}
return;
}
sub update_volume_attribute {
my ($class, $scfg, $storeid, $volname, $attribute, $value) = @_;
if ($attribute eq 'notes') {
return $update_volume_notes_impl->($class, $scfg, $storeid, $volname, $value);
}
my ($vtype) = $class->parse_volname($volname);
die "only backups support attribute '$attribute'\n" if $vtype ne 'backup';
if ($attribute eq 'protected') {
my $path = $class->filesystem_path($scfg, $volname);
my $protection_path = PVE::Storage::protection_file_path($path);
return if !((-e $protection_path) xor $value); # protection status already correct
if ($value) {
my $fh = IO::File->new($protection_path, O_CREAT, 0644)
or die "unable to create protection file '$protection_path' - $!\n";
close($fh);
} else {
unlink $protection_path
or $! == ENOENT
or die "could not delete protection file '$protection_path' - $!\n";
}
return;
}
die "attribute '$attribute' is not supported for storage type '$scfg->{type}'\n";
}
sub status {
my ($class, $storeid, $scfg, $cache) = @_;
if (defined(my $mp = parse_is_mountpoint($scfg))) {
$cache->{mountdata} = PVE::ProcFSTools::parse_proc_mounts()
if !$cache->{mountdata};
return undef if !path_is_mounted($mp, $cache->{mountdata});
}
return $class->SUPER::status($storeid, $scfg, $cache);
}
sub activate_storage {
my ($class, $storeid, $scfg, $cache) = @_;
my $path = $scfg->{path};
my $mp = parse_is_mountpoint($scfg);
if (defined($mp) && !path_is_mounted($mp, $cache->{mountdata})) {
die "unable to activate storage '$storeid' - "
. "directory is expected to be a mount point but is not mounted: '$mp'\n";
}
$class->config_aware_base_mkdir($scfg, $path);
$class->SUPER::activate_storage($storeid, $scfg, $cache);
}
sub check_config {
my ($self, $sectionId, $config, $create, $skipSchemaCheck) = @_;
my $opts =
PVE::SectionConfig::check_config($self, $sectionId, $config, $create, $skipSchemaCheck);
return $opts if !$create;
if ($opts->{path} !~ m|^/[-/a-zA-Z0-9_.@]+$|) {
die "illegal path for directory storage: $opts->{path}\n";
}
# remove trailing slashes from path
$opts->{path} = File::Spec->canonpath($opts->{path});
return $opts;
}
sub get_import_metadata {
my ($class, $scfg, $volname, $storeid) = @_;
my ($vtype, $name, undef, undef, undef, undef, $fmt) = $class->parse_volname($volname);
die "invalid content type '$vtype'\n" if $vtype ne 'import';
die "invalid format\n" if $fmt ne 'ova' && $fmt ne 'ovf';
# NOTE: all types of warnings must be added to the return schema of the import-metadata API endpoint
my $warnings = [];
my $isOva = 0;
if ($fmt =~ m/^ova/) {
$isOva = 1;
push @$warnings, { type => 'ova-needs-extracting' };
}
my $path = $class->path($scfg, $volname, $storeid, undef);
my $res = PVE::GuestImport::OVF::parse_ovf($path, $isOva);
my $disks = {};
for my $disk ($res->{disks}->@*) {
my $id = $disk->{disk_address};
my $size = $disk->{virtual_size};
my $path = $disk->{relative_path};
my $volid;
if ($isOva) {
$volid = "$storeid:$volname/$path";
} else {
$volid = "$storeid:import/$path",;
}
$disks->{$id} = {
volid => $volid,
defined($size) ? (size => $size) : (),
};
}
for my $cdrom ($res->{cdroms}->@*) {
next if !defined($cdrom->{image});
push @$warnings,
{
type => 'cdrom-image-ignored',
key => $cdrom->{disk_address},
value => $cdrom->{image},
};
}
if (defined($res->{qm}->{bios}) && $res->{qm}->{bios} eq 'ovmf') {
$disks->{efidisk0} = 1;
push @$warnings, { type => 'efi-state-lost', key => 'bios', value => 'ovmf' };
}
return {
type => 'vm',
source => $volname,
'create-args' => $res->{qm},
'disks' => $disks,
warnings => $warnings,
net => $res->{net},
};
}
sub volume_qemu_snapshot_method {
my ($class, $storeid, $scfg, $volname) = @_;
my $format = ($class->parse_volname($volname))[6];
return 'storage' if $format ne 'qcow2';
return $scfg->{'snapshot-as-volume-chain'} ? 'mixed' : 'qemu';
}
1;
src/test/disk_tests/ssd_smart/disklist
+6
View File
@@ -0,0 +1,6 @@
sda
sdb
sdc
sdd
sde
sdf
src/test/disk_tests/ssd_smart/disklist_expected.json
+92
View File
@@ -0,0 +1,92 @@
{
"sda" : {
"serial" : "000000000000",
"vendor" : "ATA",
"rpm" : 0,
"gpt" : 1,
"health" : "PASSED",
"wearout" : "100",
"osdid" : -1,
"osdid-list" : null,
"size" : 512000,
"type" : "ssd",
"devpath" : "/dev/sda",
"model" : "Crucial_CT500MX200SSD1",
"wwn" : "0x0000000000000000"
},
"sdb" : {
"model" : "INTEL_SSDSC2BB080G6",
"devpath" : "/dev/sdb",
"osdid" : -1,
"osdid-list" : null,
"type" : "ssd",
"size" : 512000,
"wwn" : "0x0000000000000000",
"gpt" : 1,
"rpm" : 0,
"vendor" : "ATA",
"serial" : "000000000000000000",
"wearout" : "97",
"health" : "PASSED"
},
"sdc" : {
"wwn" : "0x0000000000000000",
"devpath" : "/dev/sdc",
"model" : "Samsung SSD 850 PRO 512GB",
"osdid" : -1,
"osdid-list" : null,
"type" : "ssd",
"size" : 512000,
"wearout" : "99",
"health" : "PASSED",
"gpt" : 1,
"rpm" : 0,
"vendor" : "ATA",
"serial" : "000000000000"
},
"sdd" : {
"rpm" : 0,
"gpt" : 1,
"serial" : "000000000000",
"vendor" : "ATA",
"wearout" : "100",
"health" : "PASSED",
"devpath" : "/dev/sdd",
"model" : "SanDisk SD8SB8U1T001122",
"size" : 512000,
"osdid" : -1,
"osdid-list" : null,
"type" : "ssd",
"wwn" : "0x0000000000000000"
},
"sde" : {
"type" : "ssd",
"osdid" : -1,
"osdid-list" : null,
"size" : 512000,
"model" : "KINGSTON SHFS37A120G",
"devpath" : "/dev/sde",
"wwn" : "0x0000000000000000",
"vendor" : "ATA",
"serial" : "000000000000",
"gpt" : 1,
"rpm" : 0,
"health" : "PASSED",
"wearout" : "91"
},
"sdf" : {
"type" : "ssd",
"osdid" : -1,
"osdid-list" : null,
"size" : 512000,
"model" : "NT-256 2242",
"devpath" : "/dev/sdf",
"wwn" : "0x0000000000000000",
"vendor" : "ATA",
"serial" : "000000000000",
"gpt" : 1,
"rpm" : 0,
"health" : "PASSED",
"wearout" : "100"
}
}
src/test/disk_tests/ssd_smart/sdf/device/vendor
+1
View File
@@ -0,0 +1 @@
ATA
src/test/disk_tests/ssd_smart/sdf/queue/rotational
+1
View File
@@ -0,0 +1 @@
0
src/test/disk_tests/ssd_smart/sdf/size
+1
View File
@@ -0,0 +1 @@
1000
src/test/disk_tests/ssd_smart/sdf_smart
+31
View File
@@ -0,0 +1,31 @@
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-10-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART Attributes Data Structure revision number: 20
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
5 Reallocated_Sector_Ct ------+ 100 100 050 - 0
9 Power_On_Hours ------+ 100 100 000 - 93
12 Power_Cycle_Count ------+ 100 100 000 - 11
167 Unknown_Attribute ------+ 100 100 000 - 0
168 Unknown_Attribute ------+ 100 100 000 - 0
169 Unknown_Attribute ------+ 100 100 010 - 196614
173 Unknown_Attribute ------+ 200 200 000 - 4296343561
175 Program_Fail_Count_Chip ------+ 100 100 010 - 0
180 Unused_Rsvd_Blk_Cnt_Tot ------+ 100 100 000 - 27
192 Power-Off_Retract_Count ------+ 100 100 000 - 7
194 Temperature_Celsius ------+ 040 040 000 - 40 (Min/Max 27/46)
231 Unknown_SSD_Attribute ------+ 100 100 005 - 0
233 Media_Wearout_Indicator ------+ 100 100 000 - 455
234 Unknown_Attribute ------+ 100 100 005 - 159998
241 Total_LBAs_Written ------+ 100 100 000 - 262
242 Total_LBAs_Read ------+ 100 100 000 - 68
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
src/test/disk_tests/ssd_smart/sdf_smart_expected.json
+182
View File
@@ -0,0 +1,182 @@
{
"attributes" : [
{
"id" : " 5",
"name" : "Reallocated_Sector_Ct",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 50,
"fail" : "-",
"raw" : "0"
},
{
"id" : " 9",
"name" : "Power_On_Hours",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "93"
},
{
"id" : " 12",
"name" : "Power_Cycle_Count",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "11"
},
{
"id" : "167",
"name" : "Unknown_Attribute",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "0"
},
{
"id" : "168",
"name" : "Unknown_Attribute",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "0"
},
{
"id" : "169",
"name" : "Unknown_Attribute",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 10,
"fail" : "-",
"raw" : "196614"
},
{
"id" : "173",
"name" : "Unknown_Attribute",
"flags" : "------+",
"value" : 200,
"normalized" : 200,
"worst" : 200,
"threshold" : 0,
"fail" : "-",
"raw" : "4296343561"
},
{
"id" : "175",
"name" : "Program_Fail_Count_Chip",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 10,
"fail" : "-",
"raw" : "0"
},
{
"id" : "180",
"name" : "Unused_Rsvd_Blk_Cnt_Tot",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "27"
},
{
"id" : "192",
"name" : "Power-Off_Retract_Count",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "7"
},
{
"id" : "194",
"name" : "Temperature_Celsius",
"flags" : "------+",
"value" : 40,
"normalized" : 40,
"worst" : 40,
"threshold" : 0,
"fail" : "-",
"raw" : "40 (Min/Max 27/46)"
},
{
"id" : "231",
"name" : "Unknown_SSD_Attribute",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 5,
"fail" : "-",
"raw" : "0"
},
{
"id" : "233",
"name" : "Media_Wearout_Indicator",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "455"
},
{
"id" : "234",
"name" : "Unknown_Attribute",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 5,
"fail" : "-",
"raw" : "159998"
},
{
"id" : "241",
"name" : "Total_LBAs_Written",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "262"
},
{
"id" : "242",
"name" : "Total_LBAs_Read",
"flags" : "------+",
"value" : 100,
"normalized" : 100,
"worst" : 100,
"threshold" : 0,
"fail" : "-",
"raw" : "68"
}
],
"health" : "PASSED",
"type" : "ata"
}
src/test/disk_tests/ssd_smart/sdf_udevadm
+10
View File
@@ -0,0 +1,10 @@
E: DEVNAME=/dev/sdf
E: DEVTYPE=disk
E: ID_ATA_ROTATION_RATE_RPM=0
E: ID_BUS=ata
E: ID_MODEL=NT-256 2242
E: ID_PART_TABLE_TYPE=gpt
E: ID_SERIAL=NT-256_2242_000000000000
E: ID_SERIAL_SHORT=000000000000
E: ID_TYPE=disk
E: ID_WWN=0x0000000000000000
src/test/ovf_manifests/cdrom-image.ovf
+116
View File
@@ -0,0 +1,116 @@
<?xml version="1.0" encoding="UTF-8"?>
<Envelope vmw:buildId="build-348481" xmlns="http://schemas.dmtf.org/ovf/envelope/1" xmlns:cim="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<References>
<File ovf:href="disk1.vmdk" ovf:id="file1" ovf:size="73216"/>
<File ovf:href="installer.iso" ovf:id="file2" ovf:size="650809344"/>
<File ovf:href="disk2.vmdk" ovf:id="file3" ovf:size="68096"/>
</References>
<DiskSection>
<Info>Virtual disk information</Info>
<Disk ovf:capacity="8" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="293011456"/>
<Disk ovf:capacity="1" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk2" ovf:fileRef="file3" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
</DiskSection>
<NetworkSection>
<Info>The list of logical networks</Info>
<Network ovf:name="VM Network">
<Description>Service Port</Description>
</Network>
</NetworkSection>
<VirtualSystem ovf:id="cdrom-image">
<Info>A virtual machine</Info>
<Name>cdrom-image</Name>
<OperatingSystemSection ovf:id="102" vmw:osType="otherGuest64">
<Info>The kind of installed guest operating system</Info>
</OperatingSystemSection>
<VirtualHardwareSection ovf:transport="iso">
<Info>Virtual hardware requirements</Info>
<System>
<vssd:ElementName>Virtual Hardware Family</vssd:ElementName>
<vssd:InstanceID>0</vssd:InstanceID>
<vssd:VirtualSystemIdentifier>cdrom-image</vssd:VirtualSystemIdentifier>
<vssd:VirtualSystemType>vmx-07</vssd:VirtualSystemType>
</System>
<Item>
<rasd:AllocationUnits>hertz * 10^6</rasd:AllocationUnits>
<rasd:Description>Number of Virtual CPUs</rasd:Description>
<rasd:ElementName>1 virtual CPU(s)</rasd:ElementName>
<rasd:InstanceID>1</rasd:InstanceID>
<rasd:ResourceType>3</rasd:ResourceType>
<rasd:VirtualQuantity>1</rasd:VirtualQuantity>
</Item>
<Item>
<rasd:AllocationUnits>byte * 2^20</rasd:AllocationUnits>
<rasd:Description>Memory Size</rasd:Description>
<rasd:ElementName>2048MB of memory</rasd:ElementName>
<rasd:InstanceID>2</rasd:InstanceID>
<rasd:ResourceType>4</rasd:ResourceType>
<rasd:VirtualQuantity>2048</rasd:VirtualQuantity>
</Item>
<Item>
<rasd:Address>0</rasd:Address>
<rasd:Description>SCSI Controller</rasd:Description>
<rasd:ElementName>SCSI Controller 0</rasd:ElementName>
<rasd:InstanceID>3</rasd:InstanceID>
<rasd:ResourceSubType>lsilogic</rasd:ResourceSubType>
<rasd:ResourceType>6</rasd:ResourceType>
</Item>
<Item>
<rasd:Address>1</rasd:Address>
<rasd:Description>IDE Controller</rasd:Description>
<rasd:ElementName>VirtualIDEController 1</rasd:ElementName>
<rasd:InstanceID>4</rasd:InstanceID>
<rasd:ResourceType>5</rasd:ResourceType>
</Item>
<Item>
<rasd:Address>0</rasd:Address>
<rasd:Description>IDE Controller</rasd:Description>
<rasd:ElementName>VirtualIDEController 0</rasd:ElementName>
<rasd:InstanceID>5</rasd:InstanceID>
<rasd:ResourceType>5</rasd:ResourceType>
</Item>
<Item>
<rasd:AddressOnParent>0</rasd:AddressOnParent>
<rasd:ElementName>Hard Disk 1</rasd:ElementName>
<rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
<rasd:InstanceID>6</rasd:InstanceID>
<rasd:Parent>3</rasd:Parent>
<rasd:ResourceType>17</rasd:ResourceType>
</Item>
<Item>
<rasd:AddressOnParent>0</rasd:AddressOnParent>
<rasd:ElementName>Hard Disk 2</rasd:ElementName>
<rasd:HostResource>ovf:/disk/vmdisk2</rasd:HostResource>
<rasd:InstanceID>7</rasd:InstanceID>
<rasd:Parent>5</rasd:Parent>
<rasd:ResourceType>17</rasd:ResourceType>
</Item>
<Item ovf:required="false">
<rasd:AddressOnParent>0</rasd:AddressOnParent>
<rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
<rasd:Description>Floppy Drive</rasd:Description>
<rasd:ElementName>Floppy 1</rasd:ElementName>
<rasd:InstanceID>8</rasd:InstanceID>
<rasd:ResourceType>14</rasd:ResourceType>
</Item>
<Item>
<rasd:AddressOnParent>0</rasd:AddressOnParent>
<rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
<rasd:ElementName>CD-ROM 1</rasd:ElementName>
<rasd:HostResource>ovf:/file/file2</rasd:HostResource>
<rasd:InstanceID>9</rasd:InstanceID>
<rasd:Parent>4</rasd:Parent>
<rasd:ResourceType>15</rasd:ResourceType>
</Item>
<Item>
<rasd:AddressOnParent>9</rasd:AddressOnParent>
<rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
<rasd:Connection>VM Network</rasd:Connection>
<rasd:Description>E1000 ethernet adapter on &quot;VM Network&quot;</rasd:Description>
<rasd:ElementName>Ethernet 0</rasd:ElementName>
<rasd:InstanceID>10</rasd:InstanceID>
<rasd:ResourceSubType>E1000</rasd:ResourceSubType>
<rasd:ResourceType>10</rasd:ResourceType>
</Item>
</VirtualHardwareSection>
</VirtualSystem>
</Envelope>
src/test/run_ovf_tests.pl
+34 -29
View File
@@ -38,15 +38,6 @@ if (my $err = $@) {
ok('parse win10 no default rasd NS');
}
my $two_controllers =
eval { PVE::GuestImport::OVF::parse_ovf("$test_manifests/two-scsi-controllers.ovf") };
if (my $err = $@) {
fail('parse two scsi controllers');
warn("error: $err\n");
} else {
ok('parse two scsi controllers');
}
my $xee = eval { PVE::GuestImport::OVF::parse_ovf("$test_manifests/XEE.ovf") };
if (my $err = $@) {
ok("parsing XEE.ovf failed as expected");
@@ -54,6 +45,14 @@ if (my $err = $@) {
fail('parsing XEE.ovf should have failed!');
}
my $cdromImage = eval { PVE::GuestImport::OVF::parse_ovf("$test_manifests/cdrom-image.ovf") };
if (my $err = $@) {
fail('parse cdrom image');
warn("error: $err\n");
} else {
ok('parse cdrom image');
}
print "testing disks\n";
is(
@@ -104,45 +103,49 @@ is(
'single disk vm (no default rasd NS) has the correct size',
);
is_deeply(
[map { $_->{disk_address} } $two_controllers->{disks}->@*],
['scsi0', 'scsi2', 'scsi1'],
'disks with the same address on different controllers get unique addresses',
);
is(
$two_controllers->{disks}->[1]->{backing_file},
"$test_manifests/Win10-Liz-disk1.vmdk",
'disk on the second controller has the correct backing device',
);
print "testing nics\n";
is($win2008->{net}->{net0}->{model}, 'e1000', 'win2008 has correct nic model');
is($win10->{net}->{net0}->{model}, 'e1000e', 'win10 has correct nic model');
is($win10noNs->{net}->{net0}->{model}, 'e1000e',
'win10 (no default rasd NS) has correct nic model');
print "testing cdroms\n";
is_deeply(
$win2008->{cdroms},
[{ disk_address => 'sata1', image => undef }],
'win2008 has an empty cdrom',
);
is($win2008->{qm}->{sata1}, 'none,media=cdrom', 'win2008 cdrom is added to the config');
is_deeply(
$cdromImage->{cdroms},
[{ disk_address => 'ide1', image => 'installer.iso' }],
'cdrom with image gets the next free address on its bus',
);
is($cdromImage->{qm}->{ide1}, 'none,media=cdrom', 'cdrom with image is added empty to the config');
is_deeply(
[map { $_->{disk_address} } $cdromImage->{disks}->@*],
['scsi0', 'ide0'],
'cdrom with image vm has the correct disks',
);
print "\ntesting vm.conf extraction\n";
is($win2008->{qm}->{boot}, 'order=scsi0;scsi1', 'win2008 VM boot is correct');
is($win2008->{qm}->{boot}, 'order=scsi0;scsi1;sata1', 'win2008 VM boot is correct');
is($win2008->{qm}->{name}, 'Win2008-R2x64', 'win2008 VM name is correct');
is($win2008->{qm}->{memory}, '2048', 'win2008 VM memory is correct');
is($win2008->{qm}->{cores}, '1', 'win2008 VM cores are correct');
is($win2008->{qm}->{ostype}, 'win7', 'win2008 VM ostype is correcty');
is(
$two_controllers->{qm}->{boot},
'order=scsi0;scsi2;scsi1',
'VM with two scsi controllers boot is correct',
);
is($win10->{qm}->{boot}, 'order=scsi0', 'win10 VM boot is correct');
is($win10->{qm}->{boot}, 'order=scsi0;sata1', 'win10 VM boot is correct');
is($win10->{qm}->{name}, 'Win10-Liz', 'win10 VM name is correct');
is($win10->{qm}->{memory}, '6144', 'win10 VM memory is correct');
is($win10->{qm}->{cores}, '4', 'win10 VM cores are correct');
# older esxi/ovf standard used 'other' for windows10
is($win10->{qm}->{ostype}, 'other', 'win10 VM ostype is correct');
is($win10noNs->{qm}->{boot}, 'order=scsi0', 'win10 VM (no default rasd NS) boot is correct');
is(
$win10noNs->{qm}->{boot}, 'order=scsi0;sata1', 'win10 VM (no default rasd NS) boot is correct',
);
is($win10noNs->{qm}->{name}, 'Win10-Liz', 'win10 VM (no default rasd NS) name is correct');
is($win10noNs->{qm}->{memory}, '6144', 'win10 VM (no default rasd NS) memory is correct');
is($win10noNs->{qm}->{cores}, '4', 'win10 VM (no default rasd NS) cores are correct');
@@ -150,4 +153,6 @@ is($win10noNs->{qm}->{cores}, '4', 'win10 VM (no default rasd NS) cores are corr
is($win10noNs->{qm}->{ostype}, 'other', 'win10 VM (no default rasd NS) ostype is correct');
is($win10noNs->{qm}->{bios}, 'ovmf', 'win10 VM (no default rasd NS) bios is correct');
is($cdromImage->{qm}->{boot}, 'order=scsi0;ide0;ide1', 'cdrom with image VM boot is correct');
done_testing();
src/test/zfs_pool_detail_test.pm
+181 -1
View File
@@ -18,7 +18,7 @@ $zfs_api_module->mock(
my $cmdline = join(' ', @$cmd);
die "unexpected run_command call: '$cmdline'\n"
if $cmdline ne '/sbin/zpool status -P tank';
if $cmdline ne '/sbin/zpool status -P -p tank';
$param{outfunc}->($_) for split(/\n/, $zpool_status_output);
@@ -148,6 +148,186 @@ EOF
],
},
},
{
description => 'error counters above 1023',
status => <<"EOF",
pool: tank
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
\tattempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
\tusing 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 1.38M in 00:41:15 with 0 errors on Sun Sep 13 01:05:15 2026
config:
\tNAME STATE READ WRITE CKSUM
\ttank ONLINE 0 0 0
\t mirror-0 ONLINE 0 0 0
\t /dev/disk/by-id/ata-DISK_A-part1 ONLINE 1130 0 0
\t /dev/disk/by-id/ata-DISK_B-part1 ONLINE 0 0 15565
errors: No known data errors
EOF
expected => {
name => 'tank',
state => 'ONLINE',
status => 'One or more devices has experienced an unrecoverable error. An'
. ' attempt was made to correct the error. Applications are unaffected.',
action => 'Determine if the device needs to be replaced, and clear the errors'
. " using 'zpool clear' or replace the device with 'zpool replace'.",
see => 'https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P',
scan =>
'scrub repaired 1.38M in 00:41:15 with 0 errors on Sun Sep 13 01:05:15 2026',
errors => 'No known data errors',
leaf => 0,
children => [
{
name => 'tank',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 0,
children => [
{
name => 'mirror-0',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 0,
children => [
{
name => '/dev/disk/by-id/ata-DISK_A-part1',
state => 'ONLINE',
read => 1130,
write => 0,
cksum => 0,
msg => '',
leaf => 1,
},
{
name => '/dev/disk/by-id/ata-DISK_B-part1',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 15565,
msg => '',
leaf => 1,
},
],
},
],
},
],
},
},
{
description => 'hot spare in use',
status => <<"EOF",
pool: tank
state: ONLINE
scan: resilvered 725M in 00:00:00 with 0 errors on Mon May 12 16:50:25 2025
config:
\tNAME STATE READ WRITE CKSUM
\ttank ONLINE 0 0 0
\t mirror-0 ONLINE 0 0 0
\t spare-0 ONLINE 0 0 0
\t /dev/disk/by-id/ata-DISK_A-part1 ONLINE 0 0 0
\t /dev/disk/by-id/ata-DISK_C-part1 ONLINE 0 0 0
\t /dev/disk/by-id/ata-DISK_B-part1 ONLINE 0 0 0
\tspares
\t /dev/disk/by-id/ata-DISK_C-part1 INUSE currently in use
errors: No known data errors
EOF
expected => {
name => 'tank',
state => 'ONLINE',
scan => 'resilvered 725M in 00:00:00 with 0 errors on Mon May 12 16:50:25 2025',
errors => 'No known data errors',
leaf => 0,
children => [
{
name => 'tank',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 0,
children => [
{
name => 'mirror-0',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 0,
children => [
{
name => 'spare-0',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 0,
children => [
{
name => '/dev/disk/by-id/ata-DISK_A-part1',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 1,
},
{
name => '/dev/disk/by-id/ata-DISK_C-part1',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 1,
},
],
},
{
name => '/dev/disk/by-id/ata-DISK_B-part1',
state => 'ONLINE',
read => 0,
write => 0,
cksum => 0,
msg => '',
leaf => 1,
},
],
},
],
},
{
name => 'spares',
msg => '',
leaf => 0,
children => [
{
name => '/dev/disk/by-id/ata-DISK_C-part1',
state => 'INUSE',
msg => 'currently in use',
leaf => 1,
},
],
},
],
},
},
];
plan tests => scalar @$tests;