From patchwork Tue Dec 27 14:02:16 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avi Kivity X-Patchwork-Id: 133317 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [140.186.70.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 9BDF2B701D for ; Wed, 28 Dec 2011 01:02:52 +1100 (EST) Received: from localhost ([::1]:35049 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfXc0-0008H4-Lz for incoming@patchwork.ozlabs.org; Tue, 27 Dec 2011 09:02:40 -0500 Received: from eggs.gnu.org ([140.186.70.92]:40210) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfXbr-0008Gx-MU for qemu-devel@nongnu.org; Tue, 27 Dec 2011 09:02:33 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RfXbp-0006FO-U0 for qemu-devel@nongnu.org; Tue, 27 Dec 2011 09:02:31 -0500 Received: from mx1.redhat.com ([209.132.183.28]:18274) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfXbp-0006F6-Jw for qemu-devel@nongnu.org; Tue, 27 Dec 2011 09:02:29 -0500 Received: from int-mx12.intmail.prod.int.phx2.redhat.com (int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id pBRE2RKX025108 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 27 Dec 2011 09:02:27 -0500 Received: from cleopatra.tlv.redhat.com (cleopatra.tlv.redhat.com [10.35.255.11]) by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id pBRE2Q2w020207; Tue, 27 Dec 2011 09:02:26 -0500 Received: from s01.tlv.redhat.com (s01.tlv.redhat.com [10.35.255.8]) by cleopatra.tlv.redhat.com (Postfix) with ESMTP id 6D026250B9E; Tue, 27 Dec 2011 16:02:23 +0200 (IST) From: Avi Kivity To: Anthony Liguori , qemu-devel@nongnu.org Date: Tue, 27 Dec 2011 16:02:16 +0200 Message-Id: <1324994536-3955-1-git-send-email-avi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.25 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH] Fix qapi code generation wrt parallel build X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Make's multiple output syntax x.c x.h: x.template gen < x.template actually invokes the command once for x.c and once for x.h (with differing $@ in each invocation). During a parallel build, the two commands may be invoked in parallel; this opens up a race, where the second invocation trashes a file supposedly produced during the first, and now in use by a dependent command. The various qapi code generators are susceptible to this; fix by making them generate just one file per invocation. Signed-off-by: Avi Kivity --- Makefile | 20 +++++++++++--------- scripts/qapi-commands.py | 30 +++++++++++++++++++++++++++--- scripts/qapi-types.py | 27 ++++++++++++++++++++++++--- scripts/qapi-visit.py | 27 ++++++++++++++++++++++++--- 4 files changed, 86 insertions(+), 18 deletions(-) diff --git a/Makefile b/Makefile index 2c03055..5958289 100644 --- a/Makefile +++ b/Makefile @@ -172,35 +172,37 @@ qapi-dir := $(BUILD_DIR)/qapi-generated test-qmp-input-visitor.o test-qmp-output-visitor.o test-qmp-commands.o qemu-ga$(EXESUF): QEMU_CFLAGS += -I $(qapi-dir) qemu-ga$(EXESUF): LIBS = $(LIBS_QGA) +gen-out-type = $(subst .,-,$@) + $(qapi-dir)/test-qapi-types.c $(qapi-dir)/test-qapi-types.h :\ $(SRC_PATH)/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py -o "$(qapi-dir)" -p "test-" < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py $(gen-out-type) -o "$(qapi-dir)" -p "test-" < $<, " GEN $@") $(qapi-dir)/test-qapi-visit.c $(qapi-dir)/test-qapi-visit.h :\ $(SRC_PATH)/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-visit.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py -o "$(qapi-dir)" -p "test-" < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py $(gen-out-type) -o "$(qapi-dir)" -p "test-" < $<, " GEN $@") $(qapi-dir)/test-qmp-commands.h $(qapi-dir)/test-qmp-marshal.c :\ $(SRC_PATH)/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-commands.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py -o "$(qapi-dir)" -p "test-" < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py $(gen-out-type) -o "$(qapi-dir)" -p "test-" < $<, " GEN $@") $(qapi-dir)/qga-qapi-types.c $(qapi-dir)/qga-qapi-types.h :\ $(SRC_PATH)/qapi-schema-guest.json $(SRC_PATH)/scripts/qapi-types.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py -o "$(qapi-dir)" -p "qga-" < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py $(gen-out-type) -o "$(qapi-dir)" -p "qga-" < $<, " GEN $@") $(qapi-dir)/qga-qapi-visit.c $(qapi-dir)/qga-qapi-visit.h :\ $(SRC_PATH)/qapi-schema-guest.json $(SRC_PATH)/scripts/qapi-visit.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py -o "$(qapi-dir)" -p "qga-" < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py $(gen-out-type) -o "$(qapi-dir)" -p "qga-" < $<, " GEN $@") $(qapi-dir)/qga-qmp-commands.h $(qapi-dir)/qga-qmp-marshal.c :\ $(SRC_PATH)/qapi-schema-guest.json $(SRC_PATH)/scripts/qapi-commands.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py -o "$(qapi-dir)" -p "qga-" < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py $(gen-out-type) -o "$(qapi-dir)" -p "qga-" < $<, " GEN $@") qapi-types.c qapi-types.h :\ $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py -o "." < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py $(gen-out-type) -o "." < $<, " GEN $@") qapi-visit.c qapi-visit.h :\ $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py -o "." < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py $(gen-out-type) -o "." < $<, " GEN $@") qmp-commands.h qmp-marshal.c :\ $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py -m -o "." < $<, " GEN $@") + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py $(gen-out-type) -m -o "." < $<, " GEN $@") test-qmp-output-visitor.o: $(addprefix $(qapi-dir)/, test-qapi-types.c test-qapi-types.h test-qapi-visit.c test-qapi-visit.h) $(qapi-obj-y) test-qmp-output-visitor: test-qmp-output-visitor.o $(qobject-obj-y) $(qapi-obj-y) $(tools-obj-y) $(qapi-dir)/test-qapi-visit.o $(qapi-dir)/test-qapi-types.o diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py index 54d1f5d..bd7b207 100644 --- a/scripts/qapi-commands.py +++ b/scripts/qapi-commands.py @@ -372,7 +372,9 @@ def gen_command_def_prologue(prefix="", proxy=False): try: - opts, args = getopt.gnu_getopt(sys.argv[1:], "p:o:m", ["prefix=", "output-dir=", "type=", "middle"]) + opts, args = getopt.gnu_getopt(sys.argv[1:], "chp:o:m", + ["source", "header", "prefix=", + "output-dir=", "type=", "middle"]) except getopt.GetoptError, err: print str(err) sys.exit(1) @@ -384,6 +386,9 @@ c_file = 'qmp-marshal.c' h_file = 'qmp-commands.h' middle_mode = False +do_c = False +do_h = False + for o, a in opts: if o in ("-p", "--prefix"): prefix = a @@ -393,10 +398,29 @@ for o, a in opts: dispatch_type = a elif o in ("-m", "--middle"): middle_mode = True + elif o in ("-c", "--source"): + do_h = True + elif o in ("-h", "--header"): + do_c = True + +if not do_c and not do_h: + do_c = True + do_h = True c_file = output_dir + prefix + c_file h_file = output_dir + prefix + h_file +def maybe_open(really, name, opt): + class Null(object): + def write(self, str): + pass + def read(self): + return '' + if really: + return open(name, opt) + else: + return Null() + try: os.makedirs(output_dir) except os.error, e: @@ -408,8 +432,8 @@ commands = filter(lambda expr: expr.has_key('command'), exprs) commands = filter(lambda expr: not expr.has_key('gen'), commands) if dispatch_type == "sync": - fdecl = open(h_file, 'w') - fdef = open(c_file, 'w') + fdecl = maybe_open(do_h, h_file, 'w') + fdef = maybe_open(do_c, c_file, 'w') ret = gen_command_decl_prologue(header=basename(h_file), guard=guardname(h_file), prefix=prefix) fdecl.write(ret) ret = gen_command_def_prologue(prefix=prefix) diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py index 267cb49..ae644bc 100644 --- a/scripts/qapi-types.py +++ b/scripts/qapi-types.py @@ -163,7 +163,8 @@ void qapi_free_%(type)s(%(c_type)s obj) try: - opts, args = getopt.gnu_getopt(sys.argv[1:], "p:o:", ["prefix=", "output-dir="]) + opts, args = getopt.gnu_getopt(sys.argv[1:], "chp:o:", + ["source", "header", "prefix=", "output-dir="]) except getopt.GetoptError, err: print str(err) sys.exit(1) @@ -173,11 +174,22 @@ prefix = "" c_file = 'qapi-types.c' h_file = 'qapi-types.h' +do_c = False +do_h = False + for o, a in opts: if o in ("-p", "--prefix"): prefix = a elif o in ("-o", "--output-dir"): output_dir = a + "/" + elif o in ("-c", "--source"): + do_h = True + elif o in ("-h", "--header"): + do_c = True + +if not do_c and not do_h: + do_c = True + do_h = True c_file = output_dir + prefix + c_file h_file = output_dir + prefix + h_file @@ -188,8 +200,17 @@ except os.error, e: if e.errno != errno.EEXIST: raise -fdef = open(c_file, 'w') -fdecl = open(h_file, 'w') +def maybe_open(really, name, opt): + class Null(object): + def write(self, str): + pass + def read(self): + return '' + if really: + return open(name, opt) + +fdef = maybe_open(do_c, c_file, 'w') +fdecl = maybe_open(do_h, h_file, 'w') fdef.write(mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */ diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py index 62de83d..e9d0584 100644 --- a/scripts/qapi-visit.py +++ b/scripts/qapi-visit.py @@ -139,7 +139,8 @@ void visit_type_%(name)s(Visitor *m, %(name)s * obj, const char *name, Error **e name=name) try: - opts, args = getopt.gnu_getopt(sys.argv[1:], "p:o:", ["prefix=", "output-dir="]) + opts, args = getopt.gnu_getopt(sys.argv[1:], "chp:o:", + ["source", "header", "prefix=", "output-dir="]) except getopt.GetoptError, err: print str(err) sys.exit(1) @@ -149,11 +150,22 @@ prefix = "" c_file = 'qapi-visit.c' h_file = 'qapi-visit.h' +do_c = False +do_h = False + for o, a in opts: if o in ("-p", "--prefix"): prefix = a elif o in ("-o", "--output-dir"): output_dir = a + "/" + elif o in ("-c", "--source"): + do_h = True + elif o in ("-h", "--header"): + do_c = True + +if not do_c and not do_h: + do_c = True + do_h = True c_file = output_dir + prefix + c_file h_file = output_dir + prefix + h_file @@ -164,8 +176,17 @@ except os.error, e: if e.errno != errno.EEXIST: raise -fdef = open(c_file, 'w') -fdecl = open(h_file, 'w') +def maybe_open(really, name, opt): + class Null(object): + def write(self, str): + pass + def read(self): + return '' + if really: + return open(name, opt) + +fdef = maybe_open(do_c, c_file, 'w') +fdecl = maybe_open(do_h, h_file, 'w') fdef.write(mcgen(''' /* THIS FILE IS AUTOMATICALLY GENERATED, DO NOT MODIFY */