From patchwork Mon May 27 14:43:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Schwab X-Patchwork-Id: 1105867 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=sourceware.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=libc-alpha-return-102277-incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.b="Zx8T66o7"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45CKTZ32kKz9s55 for ; Tue, 28 May 2019 00:43:13 +1000 (AEST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:mime-version :content-type; q=dns; s=default; b=PIW8F8Sn7nkpKkXKkRyNk32daEcUC 4JLP9vxlfrc+CpENd7Dg/6SxZ8nhg4BTpkZuKdM78Lgenh1rsk+2c8mAHkJRicj3 58g0DTgmQp7muXtoRfReYLQEpPBlvLdpb0FF4KZEQCVB2afqc444j+Dhgu1DOyum Ku0Zw8HGkJleck= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:mime-version :content-type; s=default; bh=811PIbAJ9YCg8m+c1inM2my0v9Q=; b=Zx8 T66o7UT6bYbvNcFqnHYEQv7pQia63bHPm4PzaR6HbFY2U9tlUP9vmGzIBePAEiSl 4Rz0lUpsMMkW4Dp6bP/L26HkAk1pHwNHrqbp1VoopfgssOG9lxQT79gMRaZXvxcK e/HuDzDE/meJ/94XvOVcQsXyNOqN0vvasmFaz9jk= Received: (qmail 76364 invoked by alias); 27 May 2019 14:43:07 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 76356 invoked by uid 89); 27 May 2019 14:43:07 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-15.3 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, SPF_PASS autolearn=ham version=3.3.1 spammy=1207, upto, stdio.h, schwab@suse.de X-HELO: mx1.suse.de From: Andreas Schwab To: libc-alpha@sourceware.org Subject: [PATCH] Fix iconv buffer handling with IGNORE error handler (bug #18830) X-Yow: Look DEEP into the OPENINGS!! Do you see any ELVES or EDSELS... or a HIGHBALL??... Date: Mon, 27 May 2019 16:43:01 +0200 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 [BZ #18830] * iconv/skeleton.c (FUNCTION_NAME): Use RESET_INPUT_BUFFER only if no irreversible characters occurred. * iconv/gconv_simple.c (internal_ucs4_loop) (internal_ucs4_loop_unaligned, internal_ucs4_loop_single) (ucs4_internal_loop, ucs4_internal_loop_unaligned) (ucs4_internal_loop_single, internal_ucs4le_loop) (internal_ucs4le_loop_unaligned, internal_ucs4le_loop_single) (ucs4le_internal_loop, ucs4le_internal_loop_unaligned) (ucs4le_internal_loop_single): Add const to outend. * sysdeps/s390/multiarch/gconv_simple.c (internal_ucs4le_loop) (ucs4_internal_loop, ucs4le_internal_loop): Likewise. * iconv/Makefile (tests): Add tst-iconv7. * iconv/tst-iconv7.c: New file. Reviewed-by: Carlos O'Donell --- iconv/Makefile | 2 +- iconv/gconv_simple.c | 32 ++++++++----- iconv/skeleton.c | 28 ++++++++--- iconv/tst-iconv7.c | 67 +++++++++++++++++++++++++++ sysdeps/s390/multiarch/gconv_simple.c | 6 +-- 5 files changed, 112 insertions(+), 23 deletions(-) create mode 100644 iconv/tst-iconv7.c diff --git a/iconv/Makefile b/iconv/Makefile index f6631e861d..74cd9bf860 100644 --- a/iconv/Makefile +++ b/iconv/Makefile @@ -44,7 +44,7 @@ CFLAGS-linereader.c += -DNO_TRANSLITERATION CFLAGS-simple-hash.c += -I../locale tests = tst-iconv1 tst-iconv2 tst-iconv3 tst-iconv4 tst-iconv5 tst-iconv6 \ - tst-iconv-mt + tst-iconv7 tst-iconv-mt others = iconv_prog iconvconfig install-others-programs = $(inst_bindir)/iconv diff --git a/iconv/gconv_simple.c b/iconv/gconv_simple.c index 35aaa8aacd..75ce8fb1f4 100644 --- a/iconv/gconv_simple.c +++ b/iconv/gconv_simple.c @@ -76,7 +76,7 @@ __attribute ((always_inline)) internal_ucs4_loop (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, const unsigned char *outend, size_t *irreversible) { const unsigned char *inptr = *inptrp; @@ -120,7 +120,8 @@ internal_ucs4_loop_unaligned (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { const unsigned char *inptr = *inptrp; @@ -169,7 +170,8 @@ internal_ucs4_loop_single (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { mbstate_t *state = step_data->__statep; @@ -231,7 +233,7 @@ __attribute ((always_inline)) ucs4_internal_loop (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, const unsigned char *outend, size_t *irreversible) { int flags = step_data->__flags; @@ -298,7 +300,8 @@ ucs4_internal_loop_unaligned (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { int flags = step_data->__flags; @@ -368,7 +371,8 @@ ucs4_internal_loop_single (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { mbstate_t *state = step_data->__statep; @@ -443,7 +447,7 @@ __attribute ((always_inline)) internal_ucs4le_loop (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, const unsigned char *outend, size_t *irreversible) { const unsigned char *inptr = *inptrp; @@ -488,7 +492,8 @@ internal_ucs4le_loop_unaligned (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { const unsigned char *inptr = *inptrp; @@ -540,7 +545,8 @@ internal_ucs4le_loop_single (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { mbstate_t *state = step_data->__statep; @@ -601,7 +607,7 @@ __attribute ((always_inline)) ucs4le_internal_loop (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, const unsigned char *outend, size_t *irreversible) { int flags = step_data->__flags; @@ -671,7 +677,8 @@ ucs4le_internal_loop_unaligned (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { int flags = step_data->__flags; @@ -745,7 +752,8 @@ ucs4le_internal_loop_single (struct __gconv_step *step, struct __gconv_step_data *step_data, const unsigned char **inptrp, const unsigned char *inend, - unsigned char **outptrp, unsigned char *outend, + unsigned char **outptrp, + const unsigned char *outend, size_t *irreversible) { mbstate_t *state = step_data->__statep; diff --git a/iconv/skeleton.c b/iconv/skeleton.c index cc39fdcc70..d43fac7683 100644 --- a/iconv/skeleton.c +++ b/iconv/skeleton.c @@ -597,6 +597,10 @@ FUNCTION_NAME (struct __gconv_step *step, struct __gconv_step_data *data, inptr = *inptrp; /* The outbuf buffer is empty. */ outstart = outbuf; +#ifdef RESET_INPUT_BUFFER + size_t loop_irreversible + = lirreversible + (irreversible ? *irreversible : 0); +#endif #ifdef SAVE_RESET_STATE SAVE_RESET_STATE (1); @@ -671,8 +675,16 @@ FUNCTION_NAME (struct __gconv_step *step, struct __gconv_step_data *data, if (__glibc_unlikely (outerr != outbuf)) { #ifdef RESET_INPUT_BUFFER - RESET_INPUT_BUFFER; -#else + if (loop_irreversible + == lirreversible + (irreversible ? *irreversible : 0)) + { + /* RESET_INPUT_BUFFER can only work if there + were no irreversible characters during the + last loop. */ + RESET_INPUT_BUFFER; + goto done_reset; + } +#endif /* We have a problem in one of the functions below. Undo the conversion upto the error point. */ size_t nstatus __attribute__ ((unused)); @@ -682,9 +694,9 @@ FUNCTION_NAME (struct __gconv_step *step, struct __gconv_step_data *data, outbuf = outstart; /* Restore the state. */ -# ifdef SAVE_RESET_STATE +#ifdef SAVE_RESET_STATE SAVE_RESET_STATE (0); -# endif +#endif if (__glibc_likely (!unaligned)) { @@ -701,7 +713,7 @@ FUNCTION_NAME (struct __gconv_step *step, struct __gconv_step_data *data, lirreversiblep EXTRA_LOOP_ARGS); } -# if POSSIBLY_UNALIGNED +#if POSSIBLY_UNALIGNED else { if (FROM_DIRECTION) @@ -720,7 +732,7 @@ FUNCTION_NAME (struct __gconv_step *step, struct __gconv_step_data *data, lirreversiblep EXTRA_LOOP_ARGS); } -# endif +#endif /* We must run out of output buffer space in this rerun. */ @@ -731,9 +743,11 @@ FUNCTION_NAME (struct __gconv_step *step, struct __gconv_step_data *data, the invocation counter. */ if (__glibc_unlikely (outbuf == outstart)) --data->__invocation_counter; -#endif /* reset input buffer */ } +#ifdef RESET_INPUT_BUFFER + done_reset: +#endif /* Change the status. */ status = result; } diff --git a/iconv/tst-iconv7.c b/iconv/tst-iconv7.c new file mode 100644 index 0000000000..ee793d2f3e --- /dev/null +++ b/iconv/tst-iconv7.c @@ -0,0 +1,67 @@ +/* Test iconv buffer handling with the IGNORE error handler. + Copyright (C) 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* Derived from BZ #18830 */ +#include +#include +#include + + +static int +do_test (void) +{ + iconv_t cd = iconv_open ("ASCII//IGNORE", "ASCII"); + if (cd == (iconv_t) -1) + { + puts ("iconv_open failed"); + return 1; + } + + char input[5 + 3] = { 0, 0, 0, 0, 0, '1', '\200', '2' }; + char *inptr = input; + size_t insize = sizeof (input); + char output[5]; + char *outptr = output; + size_t outsize = sizeof (output); + + size_t ret = iconv (cd, &inptr, &insize, &outptr, &outsize); + if (ret != (size_t) -1) + { + puts ("iconv succeeded"); + return 1; + } + if (errno != E2BIG) + { + puts ("iconv did not set errno to E2BIG"); + return 1; + } + if (inptr != input + sizeof (output) - outsize) + { + printf ("iconv consumed %td characters\n", inptr - input); + return 1; + } + + if (iconv_close (cd) == -1) + { + puts ("iconv_close failed"); + return 1; + } + return 0; +} + +#include diff --git a/sysdeps/s390/multiarch/gconv_simple.c b/sysdeps/s390/multiarch/gconv_simple.c index ce7bcf541e..2861b6dacb 100644 --- a/sysdeps/s390/multiarch/gconv_simple.c +++ b/sysdeps/s390/multiarch/gconv_simple.c @@ -404,7 +404,7 @@ ICONV_VX_NAME (internal_ucs4le_loop) (struct __gconv_step *step, const unsigned char **inptrp, const unsigned char *inend, unsigned char **outptrp, - unsigned char *outend, + const unsigned char *outend, size_t *irreversible) { const unsigned char *inptr = *inptrp; @@ -504,7 +504,7 @@ ICONV_VX_NAME (ucs4_internal_loop) (struct __gconv_step *step, const unsigned char **inptrp, const unsigned char *inend, unsigned char **outptrp, - unsigned char *outend, + const unsigned char *outend, size_t *irreversible) { int flags = step_data->__flags; @@ -631,7 +631,7 @@ ICONV_VX_NAME (ucs4le_internal_loop) (struct __gconv_step *step, const unsigned char **inptrp, const unsigned char *inend, unsigned char **outptrp, - unsigned char *outend, + const unsigned char *outend, size_t *irreversible) { int flags = step_data->__flags;