perl5240cdelta - what is new for cperl v5.24.0
This document describes perl-only differences between the cperl 5.22.3 release and the cperl 5.24.0 release.
Keep the lexical topic feature
Undo the removal of lexical topic. We fixed that in 5.22.2c already, and it is critically needed for given/when, smartmatch. Had to change pp_entergiven a bit, storing NULL for defgv_save in the givwhen block. See [cperl #168] and [cperl #137].
Proper signatures
Support builtin and efficient signatures, with types and references (call-by-ref). goto to a signatured function is now a true tail call, reusing the old padframe variables without copying.
See "Signatures" in perlsub and "prototype" in perlfunc.
The prototype function returns the parsed string of a declared signature if no prototype attribute overrides it.
The @_ array inside a function body with a signature is empty. With default parameters there's is currently no way to find out how many actual arguments were provided, and how many were filled out with defaults. [cperl #7] [cperl #134]
Many subroutines of the form my ($args...) = @_; are automatically converted via fake_signatures to use signatures internally.
Proper tailcalls
goto to a subroutine with signature is now a proper tailcall, and does not create a new pad frame.
All 4 variants of tailcalls, no-sigs from and to sigs are covered.
Less experimental needed
Remove the previously required use experimental for lexical_topic, smartmatch. given/when is now standard cperl syntax.
Note that postderef and autoderef were removed by perl5.
Improved fast arithmetic
The compile-time u_{add,subtract,multiply} op variants for constant folding have now proper overflowing behaviour, and are now also used with 32bit. [cperl #2]
keep CxFOREACHDEF removed with perl-5.24.0
API macro to check for a loop with default var (gv or pad $_). Added with 5.9.3 by Robin Houston when he added given/when. Deleted with 5.24.0, without any deprecation or mention.
2 new OP members
op_rettype was added and op_typechecked taken from a spare bit.
many new public symbols needed for the compiler
unshare_hek, mro_isa_changed_in, sv_clean_objs, win32_wait_for_children, PerlIO_cleanup, PerlIO_destruct
Fixed the generated perldll.def (makedef.pl)
better -Dt tracing
Many ops print now their arguments: signature, padrange, entersub, enterxssub, method_named, goto.
auto-created coretypes on demand
The parser now knows about our coretypes, and auto-creates them on demand. This saves initialization time and memory a bit, and avoids regression errors when walking our protected readonly packages, even in code which does not use coretypes. [cperl #133]
-flto support with ~10-20% performance win
i.e. -Dcc='gcc-6 -O2 -flto' or even clang-3.9 -O2 -flto=thin is 38% faster than normal perl5, compared to only 27% without -flto.
Added the d_attribute_used probe from [perl #125096].
However clang-mp-3.9 -fsanitize=cfi -flto=thin -fvisibility=hidden fails in setenv() when linking miniperl still. gcc-6 -flto=thin -fvisibility=hidden fails with many hidden symbols. And on darwin wrong compiler warnings are thrown, like ld: warning: object file (/var/.../cc-502b6c.o) was built for newer OSX version (10.11) than being linked (0.5.180).
Backported some fixes from p5p, merged with perl-5.24.0. [cperl #137]
~20% performance improvements with a CX context system rewrite
DESTROY cache moved back from the stash to mro_meta
So we don't need our special B::SvSTASH fixes anymore.
Replace all % characters in user-controlled library filenames, passed via the system dl_error call verbatim to printf, without any arguments on the stack, which could lead to execution of arbitrary stack code. No CVE. This affects all systems with dynamic loading where the attacker can cause a dynamic loading error.
CVSSv2 Severity: 7.2 (AV:L/AC:L/Au:N/C:C/I:C/A:C/E:U/RL:OF/RC:C/CDP:MH/TD:H/CR:H/IR:H/AR:ND)
In deviation from Unicode 1.1 we treat the two HANGUL FILLER characters +UFFA0 and +U3164 not as valid ID_Start and ID_Continue characters for perl identifiers. Variable and package names may not begin with them and may not contain them.
They are usually rendered as whitespace, and would lead to classic TR39 confusables. See https://github.com/jagracey/Awesome-Unicode#user-content-variable-identifiers-can-effectively-include-whitespace and http://www.unicode.org/reports/tr39/.
In a more Korean friendly environment, we could check for a ID_Start Hangul filler if the next character is a valid Hangul ID_Continue character, and allow it then. Ditto for a ID_Continue Hangul filler if the previous and next character is a valid Hangul ID_Start or ID_Continue character, and allow it then. But those fillers should be treated as whitespace, and should be ignored.
http://www.unicode.org/L2/L2006/06310-hangul-decompose9.pdf explains:
The two other hangul fillers HANGUL CHOSEONG FILLER (Lf), i.e. lead filler, and HANGUL JUNGSEONG FILLER (Vf) are used as placeholders for missing letters, where there should be at least one letter.
... that leaves the (HALFWIDTH) HANGUL FILLERs useless. Indeed, they should not be rendered at all, despite that they have been given the property Lo. Note that these FILLERs are also given the property of Default_Ignorable_Codepoint.
Note that the standard normal forms NFKD and NFKC ... return (in all views) incorrect results for strings containing these characters.
Upstream XSLoader 0.22 (perl 5.26) fixed a minor security problem with XSLoader within eval or with a #line directive, which can load a local relative shared library, which is not in @INC. See [cpan #115808].
cperl XSLoader was already protected against the eval case since 5.22, when being rewritten in C. cperl-5.24.0 fixed now also ignoring a relative filename in a #line directive, when the relative path is not in @INC.
Prohibit the automatic conversion to fake signatures via no fake_signatures, and document the new conversion of argument handling to fake signatures. See fake_signatures.
Many internal core modules are now half-way "modernized", i.e. use typed signatures, and catch errors at compile-time. [cperl #97]
But no classes, methods and roles yet.
Fixed the #line relativefilename part of the [cpan #115808] security problem, the (eval 1) part was already fixed in the cperl rewrite as XS.
Ensure that the local stash filename part is absolute and ends with /, the part before "auto/" and before the fallback to the DynaLoader search.
Fixed utf8 handling, suse [bnc#493978]
Note that libnet has hundreds of more open tickets
Favor nroff over groff, suse perl-nroff.diff [bnc#463444]
Silence some CC warnings
Fix my to our $XS_VERSION
Check for max length before derefing by length (jhi) [cpan #111707]
See "DynaLoader format string hardening" above.
Many Windows and cygwin fixes, for the old MSVC compiler, and PERL_CORE test integration. Use many new core exports needed for windows.
Update perlcc to 2.21, handle multiple -I and -L arguments.
Handle cperl OP_SIGNATURE.
Fix refcount of cop hints hashes with 5.22-nt [cperl #220]
Better UVX L and U suffices.
Handle shared IV and UV xpv structs, fixed 5.24 specific assertions, with shared xpviv/xpvuv. fixed wrong 32 bit ptr offset. re-enabled all 32bit tests.
Handle changed subdirs-test target with newer EUMM.
Add more cperl-specific ops: aelem*_u, u_{add,subtract,multiply}, {i,n,s}_aelem{,_u}
Based on 1.001014, not Test2 as in 1.302022 yet. Test2 breaks too many modules still, and I need to time to properly modernize it (as v2.x)
Modernized and stricter type checks. skip does the switched argument check at compile-time. See the FAQ at http://perl11.github.io/cperl/STATUS.html why we enforce stricter types there.
Enable t/is_deeply_with_threads.t
Modernized. Enforce many str types.
Modernized. Enforce many str types.
Enable XS prototypes and suppress warnings. [cperl #152]
Disallow the optional getcwd argument (5.8.5 problem). Cannot be called as method.
Modernized
Modernized
Modernized
Modernized. Fixed deprecated usage of ::binf, ::bnan as function.
Modernized
Fixed tests. Type skip calls and File-Path args.
Change FirstTime yaml_module default from YAML to YAML::XS and prohibit the unsupported CPAN::Meta::YAML.
Type File-Path args.
Fix ExtUtils::Liblist::Kid for systemlib symlinks, esp. on darwin.
Modernized. str type for skip
fix Encode test for modernized Test-Simple, stringify the 1st skip arg
type the skip calls
type the skip calls
Spiffy and Test::Base 0.88c fixes for fake signatures.
Spiffy uses a source filter to expand many methods, but this does not work with fake_signatures. Expand the compile-time state of Test::Base to avoid source filters. Also add . to @INC in some tests.
utf-8 handling stills throws many warnings.
Fixed 1 test for signature errmsg
Fixed 1 test for signature errmsg
Skip tests for changed dynamic prototype() with sigs. Fix tests with -DDEBUG_LEAKING_SCALARS
add B::OP::rettype, B::OP::typechecked, B::CV::SIGOP and support for OP_SIGNATURE with B::UNOP_AUX::aux_list and B::UNOP_AUX::string.
Note: With threads in B::UNOP_AUX::aux_list the padoffsets are returned as such, and not as SV. This is different to upstream perl5, a perl5 bug.
fix test for short CV_NAME_NOMAIN. silence a C compiler warning.
fix test for short CV_NAME_NOMAIN. run-time load IO::Handle, broke miniperl. handle calls to signatures via goto, and add tests.
Add category types.
Disable 3 experimental warnings which are now default with cperl.
Added t/09_roundtrip.t
Various minor fixes
cperl patches. Add tests for 5.22 and 5.24. Support some multi-line keys: ccflags ldflags lddlflags Changed tests to use done_testing().
- Improve test 20_unknown for older JSON::PP (madsen)
- Fix wrong test 117 (pghmcfc)
- TODO the fragile mojo boolean interop test.
- Improve error message with class based method calls, when forgetting ->new. [cperl #66]
- Fix a off-by-one IV_MIN -> NV overflow in decode_json. [cperl #67] (xdg)
- Avoid encode_sv SEGV with -Dusequadmath [cperl #62] Fix quadmath NV stringification.
Skip checking the return status of the api calls.
Support the latest versions.
cperl specific: Minor -Dfortify_inc fixes, kept our Hv macros, fix __attribute__((warn_unused_result)) on windows.
Added some methods.
Sync with upstream, they fixed [RT #112827]
Some fixes. Kept our cperl-specific fallbacks to Cpanel::JSON::XS and boolean interop.
Removed the deprecated alias_code subroutine. Added some LOCALE_CODE_* UN and GENC constants.
Many changes. Fixed a skip count/why mixup in t/mbimbf.inc, detected by cperl.
No changes.
Many changes.
Fixed the t/extract-version.t TODO tests.
Keeping our darwin performance fix for hostname
Now based on Pod::Simple, not on Pod::Parser anymore. dos2unix
Whitespace only to please the new Pod::Checker
Remove the Pod::Find test, using pod_where. dos2unix
Handle fallback/syslog.h properly. honor HAVE_SETLOCALE fix windows fallback. Exclude the t/facilities-routing.t test requiring POE. Skip failing udp tests when the server is overloaded.
Minor changes from 5.22.3c. Improve t/nanosleep.t to show smoker errors.
Replace threads::yield with thread->join in t/07_lock.t.
Updated from 2.06. Improve 2 tests
No functional changes
Improve Makefile.PL, esp. for Windows. See https://github.com/jonathanstowe/TermReadKey/pull/15
Improved set_subname for ' package seperators. ' is counted as seperator only if it appears at first, before any ::. If seen, do the same magic as parse_gv_stash_name, converting all seperators over to ::. Taken with improvements from Sub::Name.
Fix high isASCII chars for EBCDIC, [cpan #12282], on CPAN as 2.161.
Fix [perl #128524], wrong indentation for utf-8 key hash elements.
This modules are currently not ready to be updated.
Broken for -Dfortify_inc, no "." in @INC. Broken new "_" logic when merged with our 'c' suffix logic.
We need some time to modernize it, as done with the previous version.
Our Test::Simple and Test::More versions were bumped to 1.4 to prohibit automatic installation of Test2 from CPAN, to use our improved versions instead.
Broken new "_" logic when merged with our 'c' suffix logic.
The following additions or changes have been made to diagnostic output, including warnings and fatal error messages. For the complete list of diagnostic messages, see perldiag.
(F) The function requires more arguments than you specified.
This error might be detected at compile-time or run-time.
If the called function is declared with a signature, and the error message includes "Want: \d, but got: 0", it might be that the call &$sub without () needs to be replaced with a goto \⊂. See "Changed calls to signatures" in perlcperl.
(F) The function cannot handle the declared type of the argument. E.g. tie or bless cannot use a typed arg with a coretype, as a coretype declaration guarantees no magic.
(F) In a subroutine signature, you put a = after a reference \ parameter. A reference parameter cannot be assigned default values on subroutine entries.
(F) A subroutine signature had more than 32767 individual parameter entries.. In the unlikely event that you need so many parameters, use a slurpy array instead.
(W types) The type of the argument does not match the user-defined type of the function declaration.
(W types) The compiler found an argument with a declared type, which the needs to be converted at run-time. This warning usually points to a wrong type usage.
(W misc) The ":caller" attribute has no effect in pure perl subroutine declarations. You applied it to a subroutine via attributes.pm. This is only valid as negated variant :-caller on subroutines.
Work is ongoing for perlcc compiler support on Win32. It is now usable, but not as good as on other platforms.
With -DDEBUGGING dsymutil is called automatically on all created binaries for easier debugging.
Several fixes were added: silence time64.h compilation warnings, fixup archname for 64bit, set d_eofnblk correctly for non-win95, define _GNU_SOURCE to declare memmem.
See RT #128358, RT #128359 and RT #127708.
The git_describe Config key and PERL_PATCHNUM define was changed to use git --long --tags, which leads to more relevant values with cperl. I.e. before v5.23.0-3113-g43962f1, now cperl-5.24.0-0-g43962f1.
Known bug upstream, not fixed there. This problem appears more often with cperl with its protected coretypes than upstream.
Fedora Patch 37
Fedora Patch36
Fedora Patch35
Fedora Patch34
Fedora Patch32
E.g. when use re 'strict'; is used.
Fedora Patch31
Many Configure and linux hints enhancements, esp for lib64, probe fixes, gdbm and ODBM fixes, gnu readline integration with the debugger. See https://build.opensuse.org/package/show/devel:languages:perl/perl
With many backref groups (>I32)
Handle get magic with globs in the regex compiler. Correctly restore context, esp. when loading unicode swashes. Reported at 5.12, patched for suse 5.14, still ignored with 5.24.
The pattern /\X*(?0)/ and various flavours thereof did not work correctly due to RExC_open_paren[0] being inappropriately adjusted. Broken in cperl-5.24.0-RC1-2 and perl-5.24.0 upstream.
See RT #128109 and RT #128085.
A race condition which occurred when computing "$!" with threads activated has been fixed. This showed up only on Darwin platforms. A related problem on Cygwin platforms involving UTF-8 strings has also been fixed. See RT #127708
See RT #128204
See RT #128313
See RT #63244 or http://bugs.debian.org/164615, crashes from 5.8.8 to blead.
env PERLIO=stdio perl -e'open(F, "<&STDOUT")' 1</dev/null
See RT #128597.
open my $fh, ">", \$buf; my $sub = eval q|sub {die}|; $sub->()' crashes from 5.22 to 5.24.1.
See RT #128532.
delete $My::{"Foo::"}; \&My::Foo::foo crashes from 5.18 to 5.24.1.
cperl 5.24.0 represents approximately 14 weeks of development since cperl 5.22.2c and contains approximately 1,100,000 lines of changes across 1,600 files from 59 authors.
Excluding auto-generated files, documentation and release tools, there were approximately 130,000 lines of changes to 700 .pm, .t, .c and .h files.
The following people are known to have contributed the improvements that became cperl 5.24.0:
Karl Williamson, Reini Urban, David Mitchell, Jarkko Hietaniemi, Andy Broad, Tony Cook, Daniel Dragan, Lukas Mai, Craig A. Berry, Yves Orton, Aaron Crane, Father Chrysostomos, Ricardo Signes, Dagfinn Ilmari Mannsåker, Steve Hay, H.Merijn Brand, James E Keenan, Matthew Horsfall, Aristotle Pagaltzis, Karen Etheridge, Abigail, Chris 'BinGOs' Williams, Shlomi Fish, Doug Bell, Dan Collins, Stevan Little, Tom Hukins, Rafael Garcia-Suarez, Andy Dougherty, Ed J, Vincent Pit, Todd Rinaldo, Nicholas Clark, Leon Timmermans, Thomas Sibley, Achim Gratz, Vladimir Timofeev, Stanislaw Pusep, Aaron Priven, Dr.Ruud, Pip Cet, Hugo van der Sanden, Mohammed El-Afifi, Mattia Barbon, John SJ Anderson, Victor Adam, Alex Vandiver, John Peacock, Niko Tyni, Alexander D'Archangel, Ludovic E. R. Tolhurst-Cleaver, Zachary Storer, Herbert Breunung, Ed Avis, Sisyphus, Max Maischein, Ivan Pozdeev, Chas. Owens, Peter Martini.
The list above is almost certainly incomplete as it is automatically generated from version control history including the perl and cperl repos. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the Perl bug tracker and the cperl github issues.
Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.
For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.
If you find what you think is a bug, you might check the articles recently posted to the comp.lang.perl.misc newsgroup and the perl bug database at https://rt.perl.org/ . There may also be information at http://www.perl.org/ , the Perl Home Page.
If you believe you have an unreported bug, please run the perlbug program included with your release. Be sure to trim your bug down to a tiny but sufficient test case. Your bug report, along with the output of perl -V, will be sent off to perlbug@perl.org to be analysed by the Perl porting team.
If you think it's a cperl specific bug or trust the cperl developers more please file an issue at https://github.com/perl11/cperl/issues.
If the bug you are reporting has security implications, which make it inappropriate to send to a publicly archived mailing list, then please send it to perl5-security-report@perl.org. This points to a closed subscription unarchived mailing list, which includes all the p5p core committers, who will be able to help assess the impact of issues, figure out a resolution, and help co-ordinate the release of patches to mitigate or fix the problem across all platforms on which Perl is supported. Please only use this address for security issues in the Perl core, not for modules independently distributed on CPAN.
If you trust the cperl developers more, please send an email to them. The p5p security teams skips many security issues, or are unwilling to fix them.
The Changes file for an explanation of how to view exhaustive details on what changed.
The INSTALL file for how to build Perl.
The README file for general stuff.
The Artistic and Copying files for copyright information.