Translations of this page:
Show pagesource
AdminRecent ChangesSitemap

It looks scrubing degraded ZFS pools can't be imported by another systems?

It's a long story, I imported a RAIDZ pool which consisted of 4 HDDs and was created by FreeBSD to a ZoL environment as a degraded pool consisted of 3 HDDs.

Although scrub was automatically running, I cancelled it and exported it, then tried to import the entire 4 HDDs on the BSD system, but it couldn't.

# zpool import zdata
cannot import 'zdata': no such pool available

A such error happened. I sweated it because the pool could get corrupted by come-and-go the pool between the BSD and Linux with degraded.

On the ZoL environment, I found the degraded pool was succeeded to import. After the pool got back to 4 HDDs and finished to resilver, it could be imported on the FreeBSD system.

It appears a degraded pool can't be imported by other systems once it is imported by the first system until its reliability will be recovered, I guess. I don't know this behaviour is normal or a special case occurred by come-and-go the pool between FreeBSD (Legacy ZFS) and the ZoL.

If it is right behaviour, the pool virtually will get unavailable when the PC will break up, so I guess it may not be right behaviour. I log this phenomenon as it actually happened.

ConnectX-3 VFs are not working on pfSense 2.4.5 on PVE 6.3

The project builds up stout router with SR-IOV is now starting up!

That being so I tried to create a pfSense 2.4.5-p1 VM on Proxmox VE 6.3-2 with ConnectX-3 VFs by PCI pass-through, but it didn't recognise them properly…😇

Some error logs are recorded in dmesg such as “pcib1: failed to allocate initial I/O port window: 0xd000-0xdfff” and “pcib1: Failed to allocate interrupt for PCI-e events”, and the VFs aren't listed up with pciconf -lv command. That means device probing fails anyways, I guess.

The pfSense 2.4 is based on FreeBSD 11.3-RELEASE. It is old-ish, so I tried newer pfSense 2.5 which is under development and is based on FreeBSD 12-STABLE.

The system recognises the VFs properly and mlxen devices are created though the former error occurs same as usual. I don't know why two mlxen are identified as mlxen0 and mlxen2 even though I pass through sequence number of VFs. It's weird, but I'll settle for this for now.

It seems mlx4 modules are embedded into a kernel of pfSense 2.4.5 or 2.5.

It is out of the question if NIC is out of order, so I decided to use developing pfSense 2.5. There is no doubt stable pfSense 2.5 will be released at some future date.

How to boot a FreeBSD system in any partitions with loader.efi

In about FreeBSD 12.0-RELEASE, loader.efi has been used as UEFI bootloader instead of prior boot.efi.

Both loaders can boot a FreeBSD system from ZFS or UFS file system. The loader.efi only looks for the file system from the storage which the loader is loaded from though the boot1.efi does from multiple ones. Briefly speaking, the former can't boot the system from other HDDs. Well, it can do it if I operate to set the boot devices manually in loader prompt each time, but it is not realistic.

I thought that it somehow got be able to automatically boot the system, then I read some documents and googled, but there was no idea. After I reluctantly saw the source codes of loader.efi, I found a way using a loader.env file and a rootdev variable.

Regardless of loader.efi or boot1.efi, it eventually uses the value of currdev variable as a boot target. In loader.efi, the 'currdev' is set from a 'rootdev' variable unquestionably if it exists.

And then, it seems the rootdev is set by /efi/freebsd/loader.env file in ESP. This function is developed relatively recently, then FreeBSD 12.2-RELEASE and over support it.

Add a following line to the file to specify the file system path corresponding to root directory in ZFS. The value becomes like disk0p1 in UFS. Suffix colon is not wrong, it's necessary!


They are undocumented as of 9th January 2021, so they may change in the future and also as-is.

Of course, I simply use prior boot1.efi with no need to such a chore way.

boot1.efi has no longer been used in FreeBSD 12 as UEFI bootloader

It is perceived that boot sequence steps of FreeBSD (x64) in UEFI environment are as below. It is pedigreed process written on manpages.

    • A UEFI bootloader executed when system starts up.
  2. boot1.efi (man)
    • The 1st stage bootloader looking for a freebsd-zfs or freebsd-ufs partition and booting a next stage. The look-up proceeds from a device which the loader is loaded to another ones in order of an UEFI's boot order setting.
  3. The final stage bootloader: loader.efi (man)
    • A bootloader executing a FreeBSD kernel in the storage specified by an environment variable of currdev or loaddev.
  4. kernel

The UEFI firmware gets BOOTX64.EFI up in EFI system partition. Next it runs the boot1.efi, then the loader.efi is lunched, and finally the kernel boots up. Actually the process is supposed to move ahead as BOOTX64.EFI → laoder.efi → kernel because boot1.efi is copied as BOOTX64.EFI

The reason why I say ambiguously is because, as you know, it is different between the implementation and manpages' description. It looks like that the loader.efi has been used as BOOTX64.EFI in FreeBSD 12.0-RELEASE. (Corresponding commit.) It seems that these implementations are now work-in-progress, so there is a moderate improved patch to generate an ESP.

You can find the same hash values between BOOTX64.EFI and loader.efi when mounting the ESP generated by FreeBSD 12.2-RELEASE installer.

So, the final stage bootloader goes straight skipping over the 1st stage in fact.

This way is no problem in most environments, but it probably can't boot the system placed on another storage because the loader.efi seems to only look for system partitions from the storage which it is loaded from. That's a fine how-do-you-do!

Workaround is to replace the BOOTX64.EFI with boot1.efi or to set currdev and load the kernel and zfs.ko manually in bootloader prompt remaining the loader.efi as EFI bootloader. The former is clearly easier than the latter.

According to source codes of the loader.efi, it looks like the loader can boot the system by an undocumented way. I will try it after.


FreeBSD's "cp" doesn't copy extended attributes (in FreeBSD 12 currently)

FreeBSD's cp command doesn't copy extended attributes at all. I checked it in FreeBSD 12.0-RELEASE-p4. Thus, at this time, extended attributes attached to a file will be lost when copying the file with cp command on FreeBSD.

Someone who thinks “Use -p option, won't you?” is naive. The option just preserves file's mtime, atime, flags, permissions, ACL, UID and GID as manpage says.

For instance, there is a file hasxattr.txt which has two extended attribute (OpusMetaInformation and DOSATTRIB.)

$ ls -al
total 42
drwxr-xr-x   2 Decomo  Decomo    3 10月 22 00:11 .
drwxr-xr-x  56 Decomo  Decomo  190 10月 21 23:43 ..
-rwxr--r--   1 Decomo  Decomo    0 10月 21 23:49 hasxattr.txt
$ lsextattr user *
hasxattr.txt    OpusMetaInformation     DOSATTRIB

Then, copy the file with cp -p.

$ cp -p hasxattr.txt hasxattr_cp-p.txt

…and check extended attributes.

$ lsextattr user *
hasxattr.txt    OpusMetaInformation     DOSATTRIB

\(^o^)/ < They've gone.

The destination file's extended attributes surely have been lost when I checked them as a color label with DirectoryOpus filer. (OpusMetaInformation stored the label information.)

As a one of EA lover, it's a pity.

I tried GNU cp provided by sysutils/coreutils ports, but his reply was loveless…

$ gcp --preserve=xattr hasxattr.txt hasxattr_gcp.txt
gcp: 拡張属性を保護できません。cp が xattr サポートなしで作成されています

It seems that rsync -X and mv preserve extended file attributes.

$ rsync -X hasxattr.txt hasxattr_rsync.txt
$ mv hasxattr.txt hasxattr_mv.txt
$ lsextattr user *
hasxattr_mv.txt OpusMetaInformation     DOSATTRIB
hasxattr_rsync.txt      OpusMetaInformation     DOSATTRIB

I found a wonderful page which described a support status of extended attributes of each command on FreeBSD, Linux and MacOS: Extended attributes: the good, the not so good, the bad.

At any rate, I wish the coreutils ports could choice a option whether handling extended attributes or not.

en/start.txt · Last modified: 2021-02-02 05:37 by Decomo
CC Attribution-Noncommercial-Share Alike 4.0 International Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0