TL;DR: This was resolved by disabling my $PERL_LOCAL_LIB_ROOT folder. I should have tried that earlier, instead of ruling it out just because the seemingly failing module was not there.
Here are mostly merely my notes from a problem I encountered this morning. Not really a proper blog post, but published anyway as I know at least one other person has found it interesting.
After upgrading one of my machines from OpenBSD 7.6 to 7.7 I could no longer run (some of) my perl scripts. A trivial hello world script worked, but this extremely stripped down version of an actual script didn't:
#!/usr/bin/env perl
use DateTime;
My expectation of running the above would be that it essentially should do nothing. However what actually happened was that perl returned 1 and gave the following error message as its output:
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xf280000, needed 0xf180000)
That seems correct, but it is not phrased in a super-helpful way. Given that
the script only contains one line, I assumed that the mismatch was between perl
and the DateTime module. That one does indeed contain a .so-file which may
depend on the perl version. These were both installed the OpenBSD way. As
mentioned in the release notes, OpenBSD comes with version 5.40.1 of
Perl. On my system DateTime came from package p5-DateTime-1.66v0, installed
through ports using pkg_add.
Today was not my first time to encounter this error message. I kind of knew what it was all about, on some level. The devil is always in the details, and I have not previously saved any notes. There is an issue, which I have visited in the past and found again today. It is titled "loadable library and perl binaries are mismatched" not informative and is a long-read. It now appears to ended up in better error handling got added as recently as a couple of weeks ago, with the merge of #22719. Great news, for anyone encountering this with a future version of Perl!
When I ran pkg_info -v p5-DateTime-1.66v0 | grep signify it appeared the
package (built 2025-04-11T19:55:16Z) was older than OpenBSD 7.7 (released April
28, 2025). I'm not fully understanding the release process though, so it is not
impossible the package was built for a release candidate or whatever. When
trying to analyze the package on technical merit, I got lost a little. These
however seem reasonble to guess having to do with ABI versioning or something:
@wantlib c.100.3 @wantlib perl.25.0
The only file installed by the package which seems to be affected by versions
is a shared library. Merely running file on it gave:
/usr/local/libdata/perl5/site_perl/amd64-openbsd/auto/DateTime/DateTime.so: ELF 64-bit LSB shared object, x86-64, version 1
Some comments to the existing bug report suggests examining which perl version
the object depends on using ldd. However attempting to do so on OpenBSD,
where the command takes no options, only gave:
/usr/local/libdata/perl5/site_perl/amd64-openbsd/auto/DateTime/DateTime.so:
Start End Type Open Ref GrpRef Name
00000092cf403000 00000092cf40a000 dlib 1 0 0 /usr/local/libdata/perl5/site_perl/amd64-openbsd/auto/DateTime/DateTime.so
I'm uncertain of how to dig deeper in order to verify it is indeed the culprit.
One might suspect that perl is picking up a version of the module installed
with cpanm, perhaps? The output of running find "$PERL_LOCAL_LIB_ROOT" -name "*DateTime*" suggest that is not the case though. Maybe the packaged version
is actually in a broken state?
In an attempt to mitigate the issue, I did pkg_delete p5-DateTime, updated my
ports tree and ran make install in devel/p5-DateTime. The error message
about versions remained even after doing that, suggesting my hypothesis of the
port being to blame was faulty, and the issue to be a bit more complex than
what I hoped for.
After running mv "$PERL_LOCAL_LIB_ROOT" "$PERL_LOCAL_LIB_ROOT-$( date +%Y%m%d )" the issue finally got resolved. In retrospect I should have tried this
sooner. Already knowing the DateTime module was not there, one now wonders what
the cause could be. Could it be that the DateTime module for some reason was
using a locally installed module for something which in some way got installed
in some other manner with the OpenBSD upgrade? Analyzing stuff just slightly
further, revealed that the failing module was Package::Stash::XS. We
recognize XS from the error message. I have not went down the rabbit hole, but
it has to do with a foreign function interface for perl. In other words,
the issue is not really with the specific module at all. It is rather some kind
of problem with module loading in general. Replacing use DateTime; with e.g.
use Moose; results in an identical error message.
What is to learn? Not much which wasn't already known. Installing software should ideally be done using system tools. Using tools like cpanm, cargo, gem, luarocks, pipx, et cetera such should ideally be avoided. Yet as we know, one really can't. So the only real take away for my part is that an eight year old bug, originally reported against 5.24.0, recently got a patch accepted and will likely be in a future release of Perl. Maybe in 5.42.0? I've stopped myself short of trying to understand Perl's release schedule. Fingers crossed, it lands before the release of OpenBSD 7.8.