From patchwork Wed Feb 22 15:46:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabrice Fontaine X-Patchwork-Id: 731183 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3vT1vN0vr9z9s65 for ; Thu, 23 Feb 2017 02:46:56 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JdigM3nu"; dkim-atps=neutral Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9A844C1AF1; Wed, 22 Feb 2017 15:46:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F0xs5ARyRIy3; Wed, 22 Feb 2017 15:46:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 120C9C1ACC; Wed, 22 Feb 2017 15:46:53 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id C35991C00A4 for ; Wed, 22 Feb 2017 15:46:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id C0BA48B210 for ; Wed, 22 Feb 2017 15:46:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tZFA9OxFUlDo for ; Wed, 22 Feb 2017 15:46:47 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by hemlock.osuosl.org (Postfix) with ESMTPS id 5F0638B1E2 for ; Wed, 22 Feb 2017 15:46:47 +0000 (UTC) Received: by mail-wm0-f67.google.com with SMTP id u63so1106393wmu.2 for ; Wed, 22 Feb 2017 07:46:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dOISvj1Q48yi0hxtvgdinXhlF/zGRrgISzw+qY3hYY0=; b=JdigM3nuP7HFO77cgKp0dau6x+vVthZCMCnbEsS/PxIn3wxV1dODJN3DYCXf3y50JP gwAXzvBLXAUTEES076M/FZqC2eg5jRJlwAE4fssUcB5fuOmlfmzOzIqzqRCBzvTom7+z CO6I/BHSLr9PBwuX25NJq6gWpd/fEwLxotOf57Fp1x1fdPYaJC1hAACrVgnIM1Gmty/W fO2VbpPBdHGdzKiu+6d1l7W2FmzfFBTYe48wYm2MgKHWAb1bXtHK8mybpTXgTde57sfp tBdLNB1y7WaDVIydqhWUHtOta7X+EzIKI5E+g1xIEW6BQlMfU9K6p4ISB3krqwq55f+e vG5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dOISvj1Q48yi0hxtvgdinXhlF/zGRrgISzw+qY3hYY0=; b=BRrhAMk/OPbB6QSVmUBAnqNqTgzMVIJptEwRS73BUJ0OzTbVcJDABRmK+oArFKgHJB HDuwgoa9Hy4k8TSOagK38A1t/tOkaf1kbkmr8CgvAt49fLrzoYh4sjK99V12fwRBISi3 UMEVvk3AUCqZ4vVFHqlyEduyHJMmsz6i9ywiHSfYSrXjSdczrzXJ4FGRnOf8eNTdbNqJ hphkvD48GRHAIItJePZwCZXOzqnc2Poh2Y+mb/sVmBpIGz+t1+0WGpri1ejR0Bgyn3T0 FCQT6A7QhpLRhxe4u2ps3asRHYwxXSsRUbHSNUcsxags5t0lkX48CfQP7btIp3HasKlU FJpw== X-Gm-Message-State: AMke39nrKSu6CvR+2z4SonBHrNgD7n3JZBR+xnlo2MRJEFDQ8iXPruIUiPIyxKajrFeg9w== X-Received: by 10.28.147.72 with SMTP id v69mr3138008wmd.51.1487778405743; Wed, 22 Feb 2017 07:46:45 -0800 (PST) Received: from localhost.localdomain (ARennes-656-1-283-247.w109-218.abo.wanadoo.fr. [109.218.203.247]) by smtp.gmail.com with ESMTPSA id g23sm2544574wme.27.2017.02.22.07.46.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 22 Feb 2017 07:46:45 -0800 (PST) From: Fabrice Fontaine To: buildroot@buildroot.org Date: Wed, 22 Feb 2017 16:46:25 +0100 Message-Id: <1487778385-98470-3-git-send-email-fontaine.fabrice@gmail.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1487778385-98470-1-git-send-email-fontaine.fabrice@gmail.com> References: <1487778385-98470-1-git-send-email-fontaine.fabrice@gmail.com> Cc: Fabrice Fontaine Subject: [Buildroot] [PATCH 3/3] docker-engine: bump to version 1.13.1 X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" - Remove patch as it has been applied upstream since 1.13 - go packages are now in vendor instead of vendor/src so update slightly the configure and build commands Signed-off-by: Fabrice Fontaine --- ...-issues-with-tailing-rotated-jsonlog-file.patch | 297 --------------------- package/docker-engine/docker-engine.hash | 2 +- package/docker-engine/docker-engine.mk | 7 +- 3 files changed, 5 insertions(+), 301 deletions(-) delete mode 100644 package/docker-engine/0001-Fix-issues-with-tailing-rotated-jsonlog-file.patch diff --git a/package/docker-engine/0001-Fix-issues-with-tailing-rotated-jsonlog-file.patch b/package/docker-engine/0001-Fix-issues-with-tailing-rotated-jsonlog-file.patch deleted file mode 100644 index 413cfd6..0000000 --- a/package/docker-engine/0001-Fix-issues-with-tailing-rotated-jsonlog-file.patch +++ /dev/null @@ -1,297 +0,0 @@ -From 8d6f2e3fe8851b581309da25fc4c32f8be675932 Mon Sep 17 00:00:00 2001 -From: Brian Goff -Date: Mon, 11 Jul 2016 16:31:42 -0400 -Subject: [PATCH] Fix issues with tailing rotated jsonlog file - -Fixes a race where the log reader would get events for both an actual -rotation as we from fsnotify (`fsnotify.Rename`). -This issue becomes extremely apparent when rotations are fast, for -example: - -``` -$ docker run -d --name test --log-opt max-size=1 --log-opt max-file=2 -busybox sh -c 'while true; do echo hello; usleep 100000; done' -``` - -With this change the log reader for jsonlogs can handle rotations that -happen as above. - -Instead of listening for both fs events AND rotation events -simultaneously, potentially meaning we see 2 rotations for only a single -rotation due to channel buffering, only listen for fs events (like -`Rename`) and then wait to be notified about rotation by the logger. -This makes sure that we don't see 2 rotations for 1, and that we don't -start trying to read until the logger is actually ready for us to. - -Signed-off-by: Brian Goff - -This commit is pending upstream commit fixing broken log tailing. The -original commit can be found in the PR here: - - - https://github.com/docker/docker/pull/24514 - -Signed-off-by: Christian Stewart ---- - daemon/logger/jsonfilelog/read.go | 180 +++++++++++++++++++++++++------------- - 1 file changed, 119 insertions(+), 61 deletions(-) - -diff --git a/daemon/logger/jsonfilelog/read.go b/daemon/logger/jsonfilelog/read.go -index bea83dd..0cb44af 100644 ---- a/daemon/logger/jsonfilelog/read.go -+++ b/daemon/logger/jsonfilelog/read.go -@@ -3,11 +3,14 @@ package jsonfilelog - import ( - "bytes" - "encoding/json" -+ "errors" - "fmt" - "io" - "os" - "time" - -+ "gopkg.in/fsnotify.v1" -+ - "github.com/Sirupsen/logrus" - "github.com/docker/docker/daemon/logger" - "github.com/docker/docker/pkg/filenotify" -@@ -44,6 +47,10 @@ func (l *JSONFileLogger) ReadLogs(config logger.ReadConfig) *logger.LogWatcher { - func (l *JSONFileLogger) readLogs(logWatcher *logger.LogWatcher, config logger.ReadConfig) { - defer close(logWatcher.Msg) - -+ // lock so the read stream doesn't get corrupted do to rotations or other log data written while we read -+ // This will block writes!!! -+ l.mu.Lock() -+ - pth := l.writer.LogPath() - var files []io.ReadSeeker - for i := l.writer.MaxFiles(); i > 1; i-- { -@@ -61,6 +68,7 @@ func (l *JSONFileLogger) readLogs(logWatcher *logger.LogWatcher, config logger.R - latestFile, err := os.Open(pth) - if err != nil { - logWatcher.Err <- err -+ l.mu.Unlock() - return - } - -@@ -80,6 +88,7 @@ func (l *JSONFileLogger) readLogs(logWatcher *logger.LogWatcher, config logger.R - if err := latestFile.Close(); err != nil { - logrus.Errorf("Error closing file: %v", err) - } -+ l.mu.Unlock() - return - } - -@@ -87,7 +96,6 @@ func (l *JSONFileLogger) readLogs(logWatcher *logger.LogWatcher, config logger.R - latestFile.Seek(0, os.SEEK_END) - } - -- l.mu.Lock() - l.readers[logWatcher] = struct{}{} - l.mu.Unlock() - -@@ -128,92 +136,142 @@ func tailFile(f io.ReadSeeker, logWatcher *logger.LogWatcher, tail int, since ti - } - } - -+func watchFile(name string) (filenotify.FileWatcher, error) { -+ fileWatcher, err := filenotify.New() -+ if err != nil { -+ return nil, err -+ } -+ -+ if err := fileWatcher.Add(name); err != nil { -+ logrus.WithField("logger", "json-file").Warnf("falling back to file poller due to error: %v", err) -+ fileWatcher.Close() -+ fileWatcher = filenotify.NewPollingWatcher() -+ -+ if err := fileWatcher.Add(name); err != nil { -+ fileWatcher.Close() -+ logrus.Debugf("error watching log file for modifications: %v", err) -+ return nil, err -+ } -+ } -+ return fileWatcher, nil -+} -+ - func followLogs(f *os.File, logWatcher *logger.LogWatcher, notifyRotate chan interface{}, since time.Time) { - dec := json.NewDecoder(f) - l := &jsonlog.JSONLog{} - -- fileWatcher, err := filenotify.New() -+ name := f.Name() -+ fileWatcher, err := watchFile(name) - if err != nil { - logWatcher.Err <- err -+ return - } - defer func() { - f.Close() - fileWatcher.Close() - }() -- name := f.Name() - -- if err := fileWatcher.Add(name); err != nil { -- logrus.WithField("logger", "json-file").Warnf("falling back to file poller due to error: %v", err) -- fileWatcher.Close() -- fileWatcher = filenotify.NewPollingWatcher() -+ var retries int -+ handleRotate := func() error { -+ f.Close() -+ fileWatcher.Remove(name) - -+ // retry when the file doesn't exist -+ for retries := 0; retries <= 5; retries++ { -+ f, err = os.Open(name) -+ if err == nil || !os.IsNotExist(err) { -+ break -+ } -+ } -+ if err != nil { -+ return err -+ } - if err := fileWatcher.Add(name); err != nil { -- logrus.Debugf("error watching log file for modifications: %v", err) -- logWatcher.Err <- err -- return -+ return err - } -+ dec = json.NewDecoder(f) -+ return nil - } - -- var retries int -- for { -- msg, err := decodeLogLine(dec, l) -- if err != nil { -- if err != io.EOF { -- // try again because this shouldn't happen -- if _, ok := err.(*json.SyntaxError); ok && retries <= maxJSONDecodeRetry { -- dec = json.NewDecoder(f) -- retries++ -- continue -+ errRetry := errors.New("retry") -+ errDone := errors.New("done") -+ waitRead := func() error { -+ select { -+ case e := <-fileWatcher.Events(): -+ switch e.Op { -+ case fsnotify.Write: -+ dec = json.NewDecoder(f) -+ return nil -+ case fsnotify.Rename, fsnotify.Remove: -+ <-notifyRotate -+ if err := handleRotate(); err != nil { -+ return err - } -- -- // io.ErrUnexpectedEOF is returned from json.Decoder when there is -- // remaining data in the parser's buffer while an io.EOF occurs. -- // If the json logger writes a partial json log entry to the disk -- // while at the same time the decoder tries to decode it, the race condition happens. -- if err == io.ErrUnexpectedEOF && retries <= maxJSONDecodeRetry { -- reader := io.MultiReader(dec.Buffered(), f) -- dec = json.NewDecoder(reader) -- retries++ -- continue -+ return nil -+ } -+ return errRetry -+ case err := <-fileWatcher.Errors(): -+ logrus.Debug("logger got error watching file: %v", err) -+ // Something happened, let's try and stay alive and create a new watcher -+ if retries <= 5 { -+ fileWatcher, err = watchFile(name) -+ if err != nil { -+ return err - } -- -- return -+ retries++ -+ return errRetry - } -+ return err -+ case <-logWatcher.WatchClose(): -+ fileWatcher.Remove(name) -+ return errDone -+ } -+ } - -- select { -- case <-fileWatcher.Events(): -- dec = json.NewDecoder(f) -- continue -- case <-fileWatcher.Errors(): -- logWatcher.Err <- err -- return -- case <-logWatcher.WatchClose(): -- fileWatcher.Remove(name) -- return -- case <-notifyRotate: -- f.Close() -- fileWatcher.Remove(name) -- -- // retry when the file doesn't exist -- for retries := 0; retries <= 5; retries++ { -- f, err = os.Open(name) -- if err == nil || !os.IsNotExist(err) { -- break -- } -+ handleDecodeErr := func(err error) error { -+ if err == io.EOF { -+ for err := waitRead(); err != nil; { -+ if err == errRetry { -+ // retry the waitRead -+ continue - } -+ return err -+ } -+ return nil -+ } -+ // try again because this shouldn't happen -+ if _, ok := err.(*json.SyntaxError); ok && retries <= maxJSONDecodeRetry { -+ dec = json.NewDecoder(f) -+ retries++ -+ return nil -+ } -+ // io.ErrUnexpectedEOF is returned from json.Decoder when there is -+ // remaining data in the parser's buffer while an io.EOF occurs. -+ // If the json logger writes a partial json log entry to the disk -+ // while at the same time the decoder tries to decode it, the race condition happens. -+ if err == io.ErrUnexpectedEOF && retries <= maxJSONDecodeRetry { -+ reader := io.MultiReader(dec.Buffered(), f) -+ dec = json.NewDecoder(reader) -+ retries++ -+ return nil -+ } -+ return err -+ } - -- if err = fileWatcher.Add(name); err != nil { -- logWatcher.Err <- err -- return -- } -- if err != nil { -- logWatcher.Err <- err -+ // main loop -+ for { -+ msg, err := decodeLogLine(dec, l) -+ if err != nil { -+ if err := handleDecodeErr(err); err != nil { -+ if err == errDone { - return - } -- -- dec = json.NewDecoder(f) -- continue -+ // we got an unrecoverable error, so return -+ logWatcher.Err <- err -+ return - } -+ // ready to try again -+ continue - } - - retries = 0 // reset retries since we've succeeded --- -2.7.3 - diff --git a/package/docker-engine/docker-engine.hash b/package/docker-engine/docker-engine.hash index 5e09453..bc1b846 100644 --- a/package/docker-engine/docker-engine.hash +++ b/package/docker-engine/docker-engine.hash @@ -1,2 +1,2 @@ # Locally calculated -sha256 0413f3513c2a6842ed9cf837154c8a722e9b34cb36b33430348489baa183707e docker-engine-v1.12.6.tar.gz +sha256 2730e7cc15492de8f1d6f9510c64620fc9004c8afc1410bf3ebac9fc3f9f83c6 docker-engine-v1.13.1.tar.gz diff --git a/package/docker-engine/docker-engine.mk b/package/docker-engine/docker-engine.mk index d575662..85ef34e 100644 --- a/package/docker-engine/docker-engine.mk +++ b/package/docker-engine/docker-engine.mk @@ -4,8 +4,8 @@ # ################################################################################ -DOCKER_ENGINE_VERSION = v1.12.6 -DOCKER_ENGINE_COMMIT = 78d18021ecba00c00730dec9d56de6896f9e708d +DOCKER_ENGINE_VERSION = v1.13.1 +DOCKER_ENGINE_COMMIT = 092cba3727bb9b4a2f0e922cd6c0f93ea270e363 DOCKER_ENGINE_SITE = $(call github,docker,docker,$(DOCKER_ENGINE_VERSION)) DOCKER_ENGINE_LICENSE = Apache-2.0 @@ -66,6 +66,7 @@ DOCKER_ENGINE_BUILD_TAGS += exclude_graphdriver_vfs endif define DOCKER_ENGINE_CONFIGURE_CMDS + mkdir -p $(DOCKER_ENGINE_GOPATH)/src/github.com/docker ln -fs $(@D) $(DOCKER_ENGINE_GOPATH)/src/github.com/docker/docker cd $(@D) && \ GITCOMMIT="$$(echo $(DOCKER_ENGINE_COMMIT) | head -c7)" \ @@ -100,7 +101,7 @@ define DOCKER_ENGINE_BUILD_CMDS -o $(@D)/bin/$(target) \ -tags "$(DOCKER_ENGINE_BUILD_TAGS)" \ -ldflags "$(DOCKER_ENGINE_GLDFLAGS)" \ - ./cmd/$(target) + github.com/docker/docker/cmd/$(target) ) endef