From patchwork Tue Jan 2 13:21:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janne Blomqvist X-Patchwork-Id: 854541 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gcc.gnu.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=gcc-patches-return-469959-incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b="PeW9YSf7"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z9vqk3vH4z9s9Y for ; Wed, 3 Jan 2018 00:22:25 +1100 (AEDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id; q=dns; s=default; b=MYojyjCZdIEc B3QKVjbffocPcOJuzwW/Lth/HZjQJ+m33KE1snA2joKWoeBTEWwdquGa95irJ5Gx tzE8KZJyrFgHfMWyPOK2Kz6rFhm/fJ7pNzBBEzXLpfTNF7WyV+donff3kYboOI2v oJMmMR47vnJcRckzWQcNZ1HulkapfaI= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id; s=default; bh=49/N5nVCXsURLKSopL RXj8lrKtQ=; b=PeW9YSf7vajTvObMeP7PTX26i8LSHWfv6ea+MX9w3RwDVQnRw9 GyXK8zs/WUzM+ox4GH9MAky4DR3JihTJ2FYnmblD53MGqHjlo3g2FR1ywu7Oa4eT ouDc74J1P6MQGiWFcBKpTmuUaoq7NU4QIT+PVQYrqIgeoLsMVwB+EPLSI= Received: (qmail 28171 invoked by alias); 2 Jan 2018 13:22:11 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 28147 invoked by uid 89); 2 Jan 2018 13:22:10 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=AWL, BAYES_00, FREEMAIL_FROM, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=Os, outright X-Spam-User: qpsmtpd, 2 recipients X-HELO: mail-lf0-f46.google.com Received: from mail-lf0-f46.google.com (HELO mail-lf0-f46.google.com) (209.85.215.46) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Jan 2018 13:22:09 +0000 Received: by mail-lf0-f46.google.com with SMTP id h140so4843060lfg.1; Tue, 02 Jan 2018 05:22:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=KbxYCK2D5kX7jm87wKB4BsneFPfzGwIx36lrt78GkHc=; b=opfI7by04AUfo8MU3SlTJVT7Q7KDZMqcTvPDRPrcbaD0z3FoSjJWTEroohIdv/D/xQ EYYvuJ6yKaNGAcVVzrhySNjq45PoZzxuEjOQjWyPwueBGKx3QXNfvV4Mc2xcrSPlLxqn vsoPUEc+Y84Z7v5U7ZMO4Ro/o7Bd13doXHF7BJ+U4gWnFPKQROX3Qr0n+ug0/3GaIuc2 5k6nYYgJDRQJY5zIIOmUUTdbVo/dsTkIksA/h01Uj5OzzMFzD1/k61hMonMMTGZJX+nQ 8sqEwzcEanJzTAHk8v3Qna9b1vg869+L8VMDbg+b6uiI/OUrGJqFSbTeVRKlHeHAFGoC lydg== X-Gm-Message-State: AKGB3mLCkPjmBXVKnQdcXLFRdZZf/P440Km/Ui1pTVmFRTsolZ6IPh2Q H+rk8QLLVQ79X+7i1Io35wAvHw== X-Google-Smtp-Source: ACJfBospjFuT8CE6S4m2rV84BF/8TKjFcqgveCI68Shg4doYQxrQuj8zpnMQoZStiHz9ihFMnRx77A== X-Received: by 10.25.35.76 with SMTP id j73mr22451433lfj.122.1514899326286; Tue, 02 Jan 2018 05:22:06 -0800 (PST) Received: from dt.lan (a88-114-247-254.elisa-laajakaista.fi. [88.114.247.254]) by smtp.gmail.com with ESMTPSA id i18sm8561382ljb.45.2018.01.02.05.22.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Jan 2018 05:22:05 -0800 (PST) From: Janne Blomqvist To: fortran@gcc.gnu.org, gcc-patches@gcc.gnu.org Cc: Janne Blomqvist Subject: [PATCH] PR libgfortran/83649 Chunk large reads and writes Date: Tue, 2 Jan 2018 15:21:57 +0200 Message-Id: <1514899317-11668-1-git-send-email-blomqvist.janne@gmail.com> It turns out that Linux never reads or writes more than 2147479552 bytes in a single syscall. For writes this is not a problem as libgfortran already contains a loop around write() to handle short writes. But for reads we cannot do this, since then read will hang if we have a short read when reading from the terminal. Also, there are reports that macOS fails I/O's larger than 2 GB. Thus, to work around these issues do large reads/writes in chunks. The testcase from the PR program largewr integer(kind=1) :: a(2_8**31+1) a = 0 a(size(a, kind=8)) = 1 open(10, file="largewr.dat", access="stream", form="unformatted") write (10) a close(10) a(size(a, kind=8)) = 2 open(10, file="largewr.dat", access="stream", form="unformatted") read (10) a if (a(size(a, kind=8)) == 1) then print *, "All is well" else print *, "Oh no" end if end program largewr fails on trunk but works with the patch. Regtested on x86_64-pc-linux-gnu, committed to trunk. libgfortran/ChangeLog: 2018-01-02 Janne Blomqvist PR libgfortran/83649 * io/unix.c (MAX_CHUNK): New define. (raw_read): For reads larger than MAX_CHUNK, loop. (raw_write): Write no more than MAX_CHUNK bytes per iteration. --- libgfortran/io/unix.c | 50 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 8 deletions(-) diff --git a/libgfortran/io/unix.c b/libgfortran/io/unix.c index a07a3c9..7a982b3 100644 --- a/libgfortran/io/unix.c +++ b/libgfortran/io/unix.c @@ -292,18 +292,49 @@ raw_flush (unix_stream *s __attribute__ ((unused))) return 0; } +/* Write/read at most 2 GB - 4k chunks at a time. Linux never reads or + writes more than this, and there are reports that macOS fails for + larger than 2 GB as well. */ +#define MAX_CHUNK 2147479552 + static ssize_t raw_read (unix_stream *s, void *buf, ssize_t nbyte) { /* For read we can't do I/O in a loop like raw_write does, because that will break applications that wait for interactive I/O. We - still can loop around EINTR, though. */ - while (true) + still can loop around EINTR, though. This however causes a + problem for large reads which must be chunked, see comment above. + So assume that if the size is larger than the chunk size, we're + reading from a file and not the terminal. */ + if (nbyte <= MAX_CHUNK) { - ssize_t trans = read (s->fd, buf, nbyte); - if (trans == -1 && errno == EINTR) - continue; - return trans; + while (true) + { + ssize_t trans = read (s->fd, buf, nbyte); + if (trans == -1 && errno == EINTR) + continue; + return trans; + } + } + else + { + ssize_t bytes_left = nbyte; + char *buf_st = buf; + while (bytes_left > 0) + { + ssize_t to_read = bytes_left < MAX_CHUNK ? bytes_left: MAX_CHUNK; + ssize_t trans = read (s->fd, buf_st, to_read); + if (trans == -1) + { + if (errno == EINTR) + continue; + else + return trans; + } + buf_st += trans; + bytes_left -= trans; + } + return nbyte - bytes_left; } } @@ -317,10 +348,13 @@ raw_write (unix_stream *s, const void *buf, ssize_t nbyte) buf_st = (char *) buf; /* We must write in a loop since some systems don't restart system - calls in case of a signal. */ + calls in case of a signal. Also some systems might fail outright + if we try to write more than 2 GB in a single syscall, so chunk + up large writes. */ while (bytes_left > 0) { - trans = write (s->fd, buf_st, bytes_left); + ssize_t to_write = bytes_left < MAX_CHUNK ? bytes_left: MAX_CHUNK; + trans = write (s->fd, buf_st, to_write); if (trans == -1) { if (errno == EINTR)