perl5252cdelta - what is new for cperl v5.25.2
This document describes the differences between the cperl 5.25.1 and the cperl 5.25.2 development releases.
In order to avoid TR39 confusable security hacks, we add the following unicode rules for identifiers and literals with mixed script properties:
The 'Common', 'Latin' and 'Inherited' scripts are always allowed and don't need to be declared.
The first non-default unicode script for an identifier is the only allowed one. This qualifies as single-script. More scripts lead to parsers errors.
Additional unicode scripts can and should be declared via use utf8 'Greek', 'script-name2'.... This allows mixed scripts in identifiers. This can be scoped in blocks.
See http://www.unicode.org/reports/tr39/#Mixed_Script_Detection and [cperl #229]
This holds for all identifiers (i.e. all names: package, gv, sub, variables) and literal numbers.
Currently there exist 131 scripts, see "Valid scripts" in utf8.
All stored utf8 names, identifiers and literals are parsed and stored as normalized NFC unicode, which prevents from various TR39 and TR36 unicode confusable and spoofing security problems.
However, dynamically created symbols via string refs are not normalized. ${"$decomposed"} stays decomposed.
Note that even perl6 stores different names for confusables, which match each other due to their NFG rules on their string matchers. perl5 matches strictly binary, which leads to confusable and spoofing security problems.
See [cperl #228], http://www.unicode.org/reports/tr36/, http://www.unicode.org/reports/tr39, http://www.unicode.org/reports/tr31/ and the Python 3 discussion 2007 on PEP 3131 https://docs.python.org/3/reference/lexical_analysis.html#identifiers.
Python 3 normalizes to NFKC (Compatibility Decomposition, followed by Canonical Composition), cperl uses both canonical transformations. See http://unicode.org/reports/tr15/#Norm_Forms for the difference. Basically NFKC transforms to shorter ligatures. NFC is recommended by TR15.
Fallback to the secure behvaiour as before v5.16 and strip symbol names of everything after the first \0 character. This protects from creating binary symbols as with no strict 'refs'; ${"a\0\hidden"}, which were especially problematic for package names, which were mapped 1:1 to filenames. With the default warning 'security' in effect, a warning is produced by the "warn_security" in perlapi API, same as for unsafe syscalls since 5.20.
See "Invalid \0 character in string for SYMBOL: %s" in perldiag and [cperl #233].
cperl5.22.2 added a restraint to expose the internal hash secret seed via the environment variable PERL_HASH_SEED_DEBUG=1 to be hidden in taint mode. See [cperl #114] and "Core Enhancements" in perl5222cdelta.
PERL_HASH_SEED_DEBUG=1 cperl5.22.2 -e1 =>
HASH_FUNCTION = FNV1A HASH_SEED = 0xecfb00eb PERTURB_KEYS = 0 (TOP)
PERL_HASH_SEED_DEBUG=1 cperl5.22.2 -t -e1 => emptyBut unfortunately not many perl services are actually protected with -t, even if cperl fixed taint mode to be actually secure. The seed exposure is only needed for a debugging perl, and actually is security relevant.
So PERL_HASH_SEED_DEBUG=1 will now hide the seed value in non-DEBUGGING builds.
PERL_HASH_SEED_DEBUG=1 cperl5.25.2 -e1 =>
HASH_FUNCTION = FNV1A HASH_SEED = <hidden> PERTURB_KEYS = 0 (TOP)Note that the seed is still trivially exposable via other means if a local script can be executed, as the seed value is readable from a fixed memory offset via unpack "P". That's why cperl fixed hash table security via proper means in the collision resolution, not via a slow hash function, and not via order hiding as perl5 believes in.
More discussion at https://github.com/google/highwayhash/issues/28 and https://github.com/google/highwayhash/issues/29.
Fixed heap-buffer-overflow in chdir without argument. make chdir allocate the stack it needs for the result, overwriting one stack entry. Note that the perl stack is on the heap, so it's not that dramatic. [perl #129130]
Those macros clash on darwin XTools with the system iostream _OutputIterator methods. We need to use the fullname Perl_do_open and Perl_do_close functions whenever perl needs to be embedded into C++ projects.
With the system C++ compiler on darwin do_open, do_close are now undefined. See [cperl #227]
Made something like sub foo'bar; a syntax error. ' is not replaced by :: anymore when used as package seperator. This was deprecated 10 years ago.
cperl fixed the "c2ph" core utility using this last remaining perl4'ism, and removed the isn't method from Test::More. In a later versions ' can be reenabled as proper IDContinue character for identifiers, e.g. for Test::More isn't.
See [cperl #217].
Most NULL ops are now removed in the peephole optimizer. Check for #if defined(PERL_REMOVE_OP_NULL) in your XS module if you hardcoded any NULL-sensitive op-tree structure. See how many with -Dk.
-DPERL_FAKE_SIGNATURE is now default, making most function calls 2x faster. See "fake_signatures"
The new compiler option support allows generation of much faster code. I.e. clang-4.0 with -flto or zapcc produce ~20% faster code.
Add aliases ansi16 through ansi255 and on_ansi16 through on_ansi255 (plus the corresponding constants) for the grey and rgb colors so that one can refer to all of the 256 ANSI colors with consistent names. These are aliases; the colors returned by uncolor will still use the grey and rgb names. (#118267)
Add the t/data/snippets tests.
Use Pod::Simple's logic to determine the native code points for NO BREAK SPACE and SOFT HYPHEN instead of hard-coding the ASCII values. Hopefully fixes the case of mysterious disappearing open brackets on EBCDIC systems. (#118240)
Added Czech republic aliases back in
Better 09_roundtrip.t tests.
Support 5.25.6 split optimization
Fix -Wc++11-compat warnings, and various minor issues.
Fix -Wc++11-compat warnings
Fix -Wc++11-compat warnings
Fix -Wc++11-compat warnings
Fix -Wc++11-compat warnings
Fix -Wc++11-compat warnings.
Warn and strip \0 in names. [cperl #233]
Fix -Wc++11-compat warnings
Fix -Wc++11-compat warnings
Less runtime memory: demand-load Carp, Config.
Fix -Wc++11-compat warnings Keep our better C++ fixes Keep our t/usleep.t, t/alarm.t, t/utime.t fixes. Keep our do_openn improvements in typemap.
from upstream:
- El Capitan compatibility - use CLOCK_REALTIME for clock_nanosleep scan - include file consistency in scans - use clockid_t consistently - use hv_fetchs() - scan for clockid_t (needed for macos Sierra) - darwin lacks clockid_t [rt.cpan.org #129789]
Fix -Wc++11-compat warnings
various upstream fixes. plus g++-6 -fpermissive and -Wc++11-compat fixes. our local make -s silent patches and various others are now all upstream.
Fix -Wc++11-compat warnings in generated const-xs.inc code.
Fix visibility declaration of XS_EXTERNAL for -flto and -fvisibility=hidden.
Removed the deprecated isn't method, using the ' package seperator.
Fixed CPAN::Shell-expand("Module", "Bundle::...")> error with Bundles, a regression from 1.63 and prev. cperl releases up to 5.25.0c. cperl only. cpan . in local directory still broken (upstream).
Fixed build dependency for dlboot.c. No excessive rebuilds anymore.
- Fixes for g++-6, stricter -fpermissive and -Wc++11-compat.
- Added tests for ill-formed utf8 sequences from Encode.
- modfl() mingw 4.0 runtime bug [perl #125924]
- Tested with the comprehensive JSON decode spectests from http://seriot.ch/parsing_json.html. Not added to core. #72
- decode with BOM: UTF-8, UTF-16, or UTF-32.
- fixed detection of final \0 as illegal non-whitespace garbage. Fixes spectest 'n_number_then_00'. #72
- warn with unicode noncharacters as in core when not in relaxed mode. #74
- fail decode of non-unicode raw characters above U+10FFFF when not in relaxed mode.
Fixed IsCOW savepvn, store the last cowrefcnt.
Fixed wrong savepvn length, failing with asan.
Optimized mro_isa_changed_in initialization.
Fixed missing _unpack_sockaddr_in family, which took AF_INET6 for a AF_INET addr in t/500_ping_icmp.t and t/500_ping_icmp_ttl.t. Use now a proper default. Detected by the new gitlab ci.
Fixed _pack_sockaddr_in for a proper 2nd argument type, hash or packed address.
Improved 500_ping_icmp.t to try sudo -n for tests requiring root, plus adding -n fir fixing [RT #118451]. Relaxed more tests failing with firewalled icmp on localhost. [RT #118441]
Fixed ping_external argument type, either packed ip or hostname. [RT #113825]
Fixed wrong skip message in t/020_external.t
Improved parallel testing, relative testdir/test$$.lib. See [RT #118416].
Many new locales. Some major fixes.
Merge latest version with the '_' lyon concensus with the cperl extension of the optional final 'c' suffix. Extend version::regex for cperl. Now also parse the 'c' natively.
Added new cperl v5.25.2c restrictions for "Identifier parsing" in perldata: No undeclared mixed scripts, normalization to NFC and no binary symbols.
The following additions or changes have been made to diagnostic output, including warnings and fatal error messages. For the complete list of diagnostic messages, see perldiag.
A new -Dmv debugging mode for verbose arena memory debugging was added, similar to -Dm and env PERL_MEM_LOG=s.
Malformed UTF-8 character in compose (empty string)
Malformed UTF-8 character in decompose (empty string)
Malformed UTF-8 character in reorder (empty string)
New unicode identifier normalization errors.
Removed all the old ' package seperators, and lexicalized most internal variables from c2ph. But it is not yet completely strict safe. See "Removed ' as package seperator" and [cperl #217].
With the SunPro cc compiler we use now the __global declaration for exported functions, similar to Windows __declspec(dllexport).
This is hidden under the new GLOBAL macro.
Probe for arflags, which do support D for deterministic static archives without member timestamps. On darwin we currently only have llvm-ar-mp-3.9 (since 3.7) which does support this.
ranlib is probed for the -D flag for reproducible build determinism.
With Intel long double it didn't clean random excess garbage bytes after the 10th byte.
Added the following release scripts to Porting: do-conf-clean do-conf-cperl-release do-make-cperl-release do-make-srctarball perl_version for Linux (debian and rpm) and Darwin.
Those builds are now reproducible, see below.
cperl has now support for automatic reproducible builds on most platforms. A new cf_epoch config key was added.
The config key cf_time is now based on: 1. SOURCE_DATE_EPOCH, 2. with .git the newest file in the repository, or 3. the newest file in the MANIFEST.
Builds are done with LC_ALL=C and PERL_HASH_SEED=0, but builds are still LANGUAGE or compiler specific.
Those builds are reproducible when done on the same machine and user. Otherwise set the keys: cf_by, cf_email, mydomain, myhostname, myuname also.
See [cperl #169].
This suspicious Config key was removed from cperl. If you have a NIS database use ypcat passwd. passcat is not used in any public CPAN module.
Ask for fake_signatures being compiled in as default or not. Defaults to yes with cperl, no without. Sets $Config{fake_signatures} and defines PERL_FAKE_SIGNATURE.
Probe for llabs() needed for PERL_IABS on 32bit with -Duse64bitint, the default on mingw/cygwin. Defines HAS_LLABS.
Probe for setenv() needed on some platforms with strict linkage or -fvisibility=hidden.
Probe for __attribute__((always_inline)), which is helpful with clang -flto=thin for exported mathoms (b) and inlined functions.
The problem is that __attribute__((used)) functions are not inlined. With always_inline + global visibility, but not __attribute__((used)) we get inlined variants plus exported copies for the API. Add PERL_MATHOM_CALLCONV to use it.
Added t/porting/embedcpp.t to check for perl.h C++ compatibility with a modern C++ compiler. There must be no fatal compilation errors in the -c step from C++ incompatibilities in any perl header file.
Note that Microsoft Visual C++ still throws errors. You cannot use that yet. See [cperl #227].
This is related to the new -Wc++11-compat and -fpermissive fixes.
Support for clang -flto and the new -flto=thin optimization was added, via GLOBAL declaration and __attribute__global__ for global visibility for all exported API functions, even if not used, and -DLTO. Note that is not needed for gcc -flto, and the clang variant produces slower code.
Rudimentary support for -fsanitize=cfi was also added, which is safer than the old -fstack-protector-strong, but this is not yet finished.
ExtUtils::ParseXS adds now a correct visibility declaration of XS_EXTERNAL for -flto and -fvisibility=hidden, which is needed for -fsanitize=cfi support.
Perl_xs_handshake is now properly exported, which is needed for clang -flto=thin.
XS_EXTERNAL and XSPROTO were simplified to use the new GLOBAL declaration and __attribute__global__ attribute, for easier platform abstractions.
Added many OP read-write field accessor macros, like OpFIRST, OpLAST, OpOTHER, OpKIDS, OpSPECIAL, OpSTACKED, OpDEREF, OpWANT_VOID, OpWANT_SCALAR, OpWANT_LIST. And shorter type checks: IS_TYPE, ISNT_TYPE, NO_OP_TYPE_OR_WASNT.
rpeep uses now consistently the local o, and not the global PL_op variable. See [cperl #219].
Fixed the range unicode bug. When the right side of the range is a UTF-8 encoded string, but the left side not, downgrade the right side to native octets. E.g.
my $r = chr 255; utf8::upgrade $r; my $num = ("a" .. $r);
print $numshould print 26 but does 702, because the utf-8 repr. of \x{ff} is "\303\277" [UTF8 "\x{ff}"], and the range was incremented from "a" to "\x{c3}\x{bf}" instead. See [cperl #218].
Fixed several issues with -Duseshrplib, a shared libcperl.$so:
install it (!!), fix ExtUtils::Embed and B-C compilation and testing, fix tests on darwin, fix configuration probe of Term::ReadKey.
Fixed sv_dump of fbm-magic strings which did previously contain the wrong "VALID,EVALED" string for a flag which is either VALID or EVALED. cperl only.
Fixed a cperl-only failure in signatures with default blocks introducing a new lexical variable. As in sub t151($a,$b=do{my $f},$c=1){} t151($x,$x,$x). This failure was only fatal on 32bit + -Duse64bitint systems.
SIGNATURE_arg_default_op does not have a items arg. See [cperl #164]. and [cperl #213].
v-strings with a 'c' suffix can now be parsed natively, in scan_vstring(). See [cperl #211].
For open cperl problems see [issues].
Some of these fixes also can to be backported from perl5.25.x upstream.
The perl debugger doesn't yet work with signatures. It doesn't step into such functions and it may fail. See e.g. [cperl #167]
t/op/taint.t contained a test with signatures and 6 default arguments, which on some 32 bit systems led to random "Reference parameter cannot take default value at op/taint.t line 2461" compile-time failures. This test has been rewritten to ony use 4 arguments.
See [cperl #164]
clang -flto=thin and on some systems even gcc -flto with -O3 or -finline leads to invisible symbols which were inlined and not exported, even if they should be declared as public API. Work is ongoing in the feature/gh186-lto-thin branch, but there the inlining is disabled by the used attribute, leading to a 10% performance regression. On the other hand a working clang-3.9 -flto leads to 20% performance improvements.
until ($x = 1) { ... } and ... until $x = 1 should warn when syntax warnings are enabled. [perl #127333]
${\vec %h, 0, 1} and ${\substr %h, 0} should not segfault, rather the lvalue context should be propagated, and list context properly handled. [perl #128260]
cperl 5.25.2 represents approximately 2 months of development since cperl 5.25.1c and contains approximately 44,000 lines of changes across 400 files from 5 authors.
Excluding auto-generated files, documentation and release tools, there were approximately 31,000 lines of changes to 260 .pm, .t, .c and .h files.
The following people are known to have contributed the improvements that became Perl 5.25.2:
Reini Urban, Tony Cook, Yves Orton, Hugo van der Sanden, Karl Williamson.
The list above is almost certainly incomplete as it is automatically generated from version control history. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the Perl bug tracker.
Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.
For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.
Generated with:
cperl Porting/acknowledgements.pl cperl-5.25.1..HEADIf you find what you think is a bug, you might check the articles recently posted to the comp.lang.perl.misc newsgroup and the perl bug database at https://rt.perl.org/ . There may also be information at http://www.perl.org/ , the Perl Home Page.
If you believe you have an unreported bug, please run the cperlbug program included with your release. Be sure to trim your bug down to a tiny but sufficient test case. Your bug report, along with the output of perl -V, will be sent off to perlbug@perl.org to be analysed by the Perl porting team.
If you think it's a cperl specific bug or trust the cperl developers more please file an issue at https://github.com/perl11/cperl/issues.
If the bug you are reporting has security implications which make it inappropriate to send to a publicly archived mailing list, then see "SECURITY VULNERABILITY CONTACT INFORMATION" in perlsec For details of how to report the issue.
The Changes file for an explanation of how to view exhaustive details on what changed.
The INSTALL file for how to build Perl.
The README file for general stuff.
The Artistic and Copying files for copyright information.