Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/257/?format=api
{ "id": 257, "url": "http://patchwork.ozlabs.org/api/patches/257/?format=api", "web_url": "http://patchwork.ozlabs.org/project/cbe-oss-dev/patch/200809112038.22459.adetsch@br.ibm.com/", "project": { "id": 1, "url": "http://patchwork.ozlabs.org/api/projects/1/?format=api", "name": "Cell Broadband Engine development", "link_name": "cbe-oss-dev", "list_id": "cbe-oss-dev.ozlabs.org", "list_email": "cbe-oss-dev@ozlabs.org", "web_url": null, "scm_url": null, "webscm_url": null, "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<200809112038.22459.adetsch@br.ibm.com>", "list_archive_url": null, "date": "2008-09-11T23:38:22", "name": "powerpc/spufs: Limit size of gangs to avoid starvation due to reserved spus", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": false, "hash": "1841bcd47503c76e21142ab14f9be4699c159cc3", "submitter": { "id": 93, "url": "http://patchwork.ozlabs.org/api/people/93/?format=api", "name": "Andre Detsch", "email": "adetsch@br.ibm.com" }, "delegate": { "id": 1, "url": "http://patchwork.ozlabs.org/api/users/1/?format=api", "username": "jk", "first_name": "Jeremy", "last_name": "Kerr", "email": "jk@ozlabs.org" }, "mbox": "http://patchwork.ozlabs.org/project/cbe-oss-dev/patch/200809112038.22459.adetsch@br.ibm.com/mbox/", "series": [], "comments": "http://patchwork.ozlabs.org/api/patches/257/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/257/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<cbe-oss-dev-bounces+patchwork=ozlabs.org@ozlabs.org>", "X-Original-To": [ "patchwork@ozlabs.org", "cbe-oss-dev@ozlabs.org" ], "Delivered-To": [ "patchwork@ozlabs.org", "cbe-oss-dev@ozlabs.org" ], "Received": [ "from ozlabs.org (localhost [127.0.0.1])\n\tby ozlabs.org (Postfix) with ESMTP id 18CCDDE516\n\tfor <patchwork@ozlabs.org>; Fri, 12 Sep 2008 09:40:35 +1000 (EST)", "from igw1.br.ibm.com (igw1.br.ibm.com [32.104.18.24])\n\t(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\n\t(Client did not present a certificate)\n\tby ozlabs.org (Postfix) with ESMTPS id 3E68BDE011;\n\tFri, 12 Sep 2008 09:39:16 +1000 (EST)", "from mailhub3.br.ibm.com (mailhub3 [9.18.232.110])\n\tby igw1.br.ibm.com (Postfix) with ESMTP id D9EF832C016;\n\tThu, 11 Sep 2008 20:08:34 -0300 (BRT)", "from d24av01.br.ibm.com (d24av01.br.ibm.com [9.18.232.46])\n\tby mailhub3.br.ibm.com (8.13.8/8.13.8/NCO v8.7) with ESMTP id\n\tm8BNdBM81167598; Thu, 11 Sep 2008 20:39:11 -0300", "from d24av01.br.ibm.com (loopback [127.0.0.1])\n\tby d24av01.br.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id\n\tm8BNd3aB024393; Thu, 11 Sep 2008 20:39:03 -0300", "from [9.8.10.86] ([9.8.10.86])\n\tby d24av01.br.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id\n\tm8BNd3Bw024383; Thu, 11 Sep 2008 20:39:03 -0300" ], "From": "Andre Detsch <adetsch@br.ibm.com>", "To": "cbe-oss-dev@ozlabs.org", "Date": "Thu, 11 Sep 2008 20:38:22 -0300", "User-Agent": "KMail/1.9.6", "References": "<200809111955.28780.adetsch@br.ibm.com>", "In-Reply-To": "<200809111955.28780.adetsch@br.ibm.com>", "MIME-Version": "1.0", "Content-Disposition": "inline", "Message-Id": "<200809112038.22459.adetsch@br.ibm.com>", "Cc": "LukeBrowning@us.ibm.com, Jeremy Kerr <jk@ozlabs.org>", "Subject": "[Cbe-oss-dev] [PATCH 09/11] powerpc/spufs: Limit size of gangs to\n\tavoid starvation due to reserved spus", "X-BeenThere": "cbe-oss-dev@ozlabs.org", "X-Mailman-Version": "2.1.11", "Precedence": "list", "List-Id": "Discussion about Open Source Software for the Cell Broadband Engine\n\t<cbe-oss-dev.ozlabs.org>", "List-Unsubscribe": "<https://ozlabs.org/mailman/options/cbe-oss-dev>,\n\t<mailto:cbe-oss-dev-request@ozlabs.org?subject=unsubscribe>", "List-Archive": "<http://ozlabs.org/pipermail/cbe-oss-dev>", "List-Post": "<mailto:cbe-oss-dev@ozlabs.org>", "List-Help": "<mailto:cbe-oss-dev-request@ozlabs.org?subject=help>", "List-Subscribe": "<https://ozlabs.org/mailman/listinfo/cbe-oss-dev>,\n\t<mailto:cbe-oss-dev-request@ozlabs.org?subject=subscribe>", "Content-Type": "text/plain; charset=\"us-ascii\"", "Content-Transfer-Encoding": "7bit", "Sender": "cbe-oss-dev-bounces+patchwork=ozlabs.org@ozlabs.org", "Errors-To": "cbe-oss-dev-bounces+patchwork=ozlabs.org@ozlabs.org" }, "content": "At context creation, determine the number of available spus for general\nscheduling to avoid creating a gang that can;t be placed by the scheduler.\nGangs require concurrent scheduling, so multiple spus have to be allocated\nat the same time. Similarly, prevent the reservation of an spu, if it\nwould result in the inability to schedule an existing job. A new data\nstructure is introduced to keep track of active gangs. It is designed to\nshow the size of the largest active gang and is coded to handle the\ndynamic addition and deletion of contexts within gangs. An array is\nallocated that has an element for each spus in the system. As contexts\nare added and removed, elements are incremented and decremented to show\nthe number of gangs of at least that size. For example, element 0\nrepresents the number of active gangs with at least 1 context. element 1\nis the number of gangs with at least 2 contexts, and so on. A high water\nmark is kept to keep track of the largest gang.\n\nSigned-off-by: Luke Browning <lukebrowning@us.ibm.com>\nSigned-off-by: Andre Detsch <adetsch@br.ibm.com>", "diff": "diff --git a/arch/powerpc/platforms/cell/spufs/context.c \nb/arch/powerpc/platforms/cell/spufs/context.c\nindex c472519..7ca787e 100644\n--- a/arch/powerpc/platforms/cell/spufs/context.c\n+++ b/arch/powerpc/platforms/cell/spufs/context.c\n@@ -32,6 +32,22 @@\n \n atomic_t nr_spu_contexts = ATOMIC_INIT(0);\n \n+static void inc_active_gangs(struct spu_gang *gang)\n+{\n+\tif (atomic_inc_return(&spu_active_gangs[gang->contexts]) == 1) {\n+\t\tatomic_set(&largest_active_gang, gang->contexts);\n+\t\tmb(); /* XXX atomic_set doesn't have a sync */\n+\t}\n+}\n+\n+static void dec_active_gangs(struct spu_gang *gang)\n+{\n+\tif (!atomic_dec_return(&spu_active_gangs[gang->contexts])) {\n+\t\tatomic_set(&largest_active_gang, gang->contexts);\n+\t\tmb(); /* XXX atomic_set doesn't have a sync */\n+\t}\n+}\n+\n struct spu_context *alloc_spu_context(struct spu_gang *gang)\n {\n \tstruct spu_context *ctx;\n@@ -57,6 +73,8 @@ struct spu_context *alloc_spu_context(struct spu_gang *gang)\n \tif (spu_init_csa(&ctx->csa))\n \t\tgoto out_free_gang;\n \n+\tinc_active_gangs(gang);\n+\n \t/* If the gang is running, it needs to be stopped, since we have a\n \t * new context that needs to be gang scheduled. Gangs are allowed\n \t * to grow and shrink over time, but they are unscheduled when it\n@@ -89,6 +107,7 @@ struct spu_context *alloc_spu_context(struct spu_gang \n*gang)\n \tctx->stats.util_state = SPU_UTIL_IDLE_LOADED;\n \n \tatomic_inc(&nr_spu_contexts);\n+\n \tgoto out;\n \n out_free_gang:\n@@ -123,6 +142,7 @@ void destroy_spu_context(struct kref *kref)\n \tif (ctx->prof_priv_kref)\n \t\tkref_put(ctx->prof_priv_kref, ctx->prof_priv_release);\n \tatomic_dec(&nr_spu_contexts);\n+\tdec_active_gangs(gang);\n \tkfree(ctx->switch_log);\n \tkfree(ctx);\n }\ndiff --git a/arch/powerpc/platforms/cell/spufs/inode.c \nb/arch/powerpc/platforms/cell/spufs/inode.c\nindex cf97761..c455a44 100644\n--- a/arch/powerpc/platforms/cell/spufs/inode.c\n+++ b/arch/powerpc/platforms/cell/spufs/inode.c\n@@ -263,6 +263,7 @@ spufs_mkdir(struct inode *dir, struct dentry *dentry, \nunsigned int flags,\n \tstruct inode *inode;\n \tstruct spu_context *ctx, *gang_ctx;\n \tstruct spu_gang *gang;\n+\tint node, avail_spus;\n \n \tret = -ENOSPC;\n \tinode = spufs_new_inode(dir->i_sb, mode | S_IFDIR);\n@@ -280,6 +281,27 @@ spufs_mkdir(struct inode *dir, struct dentry *dentry, \nunsigned int flags,\n \t\t}\n \t}\n \n+\tfor (node = 0, avail_spus = 0; node < MAX_NUMNODES; node++) {\n+\t\tavail_spus += cbe_spu_info[node].n_spus - atomic_read(\n+\t\t\t&cbe_spu_info[node].reserved_spus);\n+\t}\n+\n+\t/* Ensure there are enough available spus for scheduling. */\n+\tif (flags & SPU_CREATE_NOSCHED) {\n+\t\t/* Can't reserve an spu if it would starve an active gang */\n+\t\tif (avail_spus <= atomic_read(&largest_active_gang) + 1) {\n+\t\t\tret = -EPERM;\n+\t\t\tgoto out_iput;\n+\t\t}\n+\t}\n+\telse {\n+\t\t/* Can't create a gang too big either. */\n+\t\tif (!avail_spus || (gang && gang->contexts + 1 > avail_spus)) {\n+\t\t\tret = -EPERM;\n+\t\t\tgoto out_iput;\n+\t\t}\n+\t}\n+\n \tif (dir->i_mode & S_ISGID) {\n \t\tinode->i_gid = dir->i_gid;\n \t\tinode->i_mode &= S_ISGID;\ndiff --git a/arch/powerpc/platforms/cell/spufs/sched.c \nb/arch/powerpc/platforms/cell/spufs/sched.c\nindex f3dee8d..8326034 100644\n--- a/arch/powerpc/platforms/cell/spufs/sched.c\n+++ b/arch/powerpc/platforms/cell/spufs/sched.c\n@@ -90,6 +90,9 @@ static struct timer_list spusched_timer;\n static struct timer_list spuloadavg_timer;\n static void spu_unschedule(struct spu_gang *gang);\n \n+atomic_t *spu_active_gangs;\n+atomic_t largest_active_gang;\n+\n /*\n * Priority of a normal, non-rt, non-niced'd process (aka nice level 0).\n */\n@@ -1492,12 +1495,27 @@ static const struct file_operations spu_loadavg_fops = \n{\n int __init spu_sched_init(void)\n {\n \tstruct proc_dir_entry *entry;\n-\tint err = -ENOMEM, i;\n+\tint err = -ENOMEM, node, nspus, i;\n \n \tspu_prio = kzalloc(sizeof(struct spu_prio_array), GFP_KERNEL);\n \tif (!spu_prio)\n \t\tgoto out;\n \n+\t/*\n+\t * A gang cannot be larger than the number of spus in the system\n+\t * since they have to be scheduled at the same time. Allocate an\n+\t * array of that length to keep track of the size of active gangs.\n+\t * We need to limit the number of spus that can be reserved to\n+\t * the starvation of gangs. A reserved spus can be used by the\n+\t * scheduler.\n+\t */\n+\tfor (node = 0, nspus = 0; node < MAX_NUMNODES; node++)\n+\t\tnspus += cbe_spu_info[node].n_spus;\n+\tspu_active_gangs = kzalloc(sizeof(atomic_t) * nspus, GFP_KERNEL);\n+\tif (!spu_active_gangs)\n+\t\tgoto out_free_spu_prio;\n+\tatomic_set(&largest_active_gang, 0);\n+\n \tfor (i = 0; i < MAX_PRIO; i++) {\n \t\tINIT_LIST_HEAD(&spu_prio->runq[i]);\n \t\t__clear_bit(i, spu_prio->bitmap);\n@@ -1551,5 +1569,6 @@ void spu_sched_exit(void)\n \t\t\t}\n \t\tspin_unlock(&cbe_spu_info[node].list_lock);\n \t}\n+\tkfree(spu_active_gangs);\n \tkfree(spu_prio);\n }\ndiff --git a/arch/powerpc/platforms/cell/spufs/spufs.h \nb/arch/powerpc/platforms/cell/spufs/spufs.h\nindex de436f2..6afc514 100644\n--- a/arch/powerpc/platforms/cell/spufs/spufs.h\n+++ b/arch/powerpc/platforms/cell/spufs/spufs.h\n@@ -297,6 +297,17 @@ int put_spu_gang(struct spu_gang *gang);\n void spu_gang_remove_ctx(struct spu_gang *gang, struct spu_context *ctx);\n void spu_gang_add_ctx(struct spu_gang *gang, struct spu_context *ctx);\n \n+/*\n+ * Each element of the spu_active_gang[] identifies the number of active\n+ * gangs of at least that size. largest_active_gang identifies the size of\n+ * the largest aactive gang in the system. Array elements are incremented\n+ * as contexts are created and they are decremented as contexts are \ndestroyed.\n+ * The first context in a gang increments element[1], the second element[2],\n+ * and so on. largest_active_gang is set to the highest non-zero array \nelement.\n+ */\n+extern atomic_t largest_active_gang;\n+extern atomic_t *spu_active_gangs;\n+\n /* fault handling */\n", "prefixes": [] }