Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/690687/?format=api
{ "id": 690687, "url": "http://patchwork.ozlabs.org/api/patches/690687/?format=api", "web_url": "http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20161103051949.GC8368@fergus.ozlabs.ibm.com/", "project": { "id": 2, "url": "http://patchwork.ozlabs.org/api/projects/2/?format=api", "name": "Linux PPC development", "link_name": "linuxppc-dev", "list_id": "linuxppc-dev.lists.ozlabs.org", "list_email": "linuxppc-dev@lists.ozlabs.org", "web_url": "https://github.com/linuxppc/wiki/wiki", "scm_url": "https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git", "webscm_url": "https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/", "list_archive_url": "https://lore.kernel.org/linuxppc-dev/", "list_archive_url_format": "https://lore.kernel.org/linuxppc-dev/{}/", "commit_url_format": "https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id={}" }, "msgid": "<20161103051949.GC8368@fergus.ozlabs.ibm.com>", "list_archive_url": "https://lore.kernel.org/linuxppc-dev/20161103051949.GC8368@fergus.ozlabs.ibm.com/", "date": "2016-11-03T05:19:49", "name": "[1/4] powerpc/64: Make exception table clearer in __copy_tofrom_user_base", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "02fbfadb3efa1fd7290911c98a2abe4cbad680c3", "submitter": { "id": 67079, "url": "http://patchwork.ozlabs.org/api/people/67079/?format=api", "name": "Paul Mackerras", "email": "paulus@ozlabs.org" }, "delegate": null, "mbox": "http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20161103051949.GC8368@fergus.ozlabs.ibm.com/mbox/", "series": [], "comments": "http://patchwork.ozlabs.org/api/patches/690687/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/690687/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org>", "X-Original-To": [ "patchwork-incoming@ozlabs.org", "linuxppc-dev@lists.ozlabs.org" ], "Delivered-To": [ "patchwork-incoming@ozlabs.org", "linuxppc-dev@lists.ozlabs.org", "linuxppc-dev@ozlabs.org" ], "Received": [ "from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3t8YQK681Yz9t10\n\tfor <patchwork-incoming@ozlabs.org>;\n\tThu, 3 Nov 2016 16:27:53 +1100 (AEDT)", "from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3])\n\tby lists.ozlabs.org (Postfix) with ESMTP id 3t8YQK5JpHzDvp6\n\tfor <patchwork-incoming@ozlabs.org>;\n\tThu, 3 Nov 2016 16:27:53 +1100 (AEDT)", "from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby lists.ozlabs.org (Postfix) with ESMTPS id 3t8YK169RHzDvQZ\n\tfor <linuxppc-dev@lists.ozlabs.org>;\n\tThu, 3 Nov 2016 16:23:17 +1100 (AEDT)", "by ozlabs.org (Postfix)\n\tid 3t8YK15d95z9t2D; Thu, 3 Nov 2016 16:23:17 +1100 (AEDT)", "by ozlabs.org (Postfix, from userid 1003)\n\tid 3t8YK15JM9z9t3N; Thu, 3 Nov 2016 16:23:17 +1100 (AEDT)" ], "Date": "Thu, 3 Nov 2016 16:19:49 +1100", "From": "Paul Mackerras <paulus@ozlabs.org>", "To": "linuxppc-dev@ozlabs.org", "Subject": "[PATCH 1/4] powerpc/64: Make exception table clearer in\n\t__copy_tofrom_user_base", "Message-ID": "<20161103051949.GC8368@fergus.ozlabs.ibm.com>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=us-ascii", "Content-Disposition": "inline", "User-Agent": "Mutt/1.5.24 (2015-08-30)", "X-BeenThere": "linuxppc-dev@lists.ozlabs.org", "X-Mailman-Version": "2.1.23", "Precedence": "list", "List-Id": "Linux on PowerPC Developers Mail List\n\t<linuxppc-dev.lists.ozlabs.org>", "List-Unsubscribe": "<https://lists.ozlabs.org/options/linuxppc-dev>,\n\t<mailto:linuxppc-dev-request@lists.ozlabs.org?subject=unsubscribe>", "List-Archive": "<http://lists.ozlabs.org/pipermail/linuxppc-dev/>", "List-Post": "<mailto:linuxppc-dev@lists.ozlabs.org>", "List-Help": "<mailto:linuxppc-dev-request@lists.ozlabs.org?subject=help>", "List-Subscribe": "<https://lists.ozlabs.org/listinfo/linuxppc-dev>,\n\t<mailto:linuxppc-dev-request@lists.ozlabs.org?subject=subscribe>", "Errors-To": "linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org", "Sender": "\"Linuxppc-dev\"\n\t<linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org>" }, "content": "This aims to make the generation of exception table entries for the\nloads and stores in __copy_tofrom_user_base clearer and easier to\nverify. Instead of having a series of local labels on the loads\nand stores, with a series of corresponding labels later for the\nexception handlers, we now use macros to generate exception table\nentries at the point of each load and store that could potentially\ntrap. We do this with the macros extable, lex (load exception),\nand stex (store exception). These macros are used right before\nthe load or store to which they apply.\n\nSome complexity is introduced by the fact that we have some more work\nto do after hitting an exception. After an exception on a load, we\nneed to clear the rest of the destination buffer (if possible), and\nthen return the number of bytes not copied. After an exception on a\nstore, we only need to return the number of bytes not copied. In\neither case, the fixup code uses r3 as the current pointer into the\ndestination buffer, that is, the address of the first byte of the\ndestination that has not been modified. However, at various points\nin the copy loops, r3 can be 4, 8, 16 or 24 bytes behind that point.\n\nTo express this offset in an understandable way, we define a symbol\nr3_offset which is updated at various points so that it equal to the\ndifference between the address of the first unmodified byte of the\ndestination and the value at r3. (In fact it only needs to be\naccurate at the point of each lex or stex macro invocation.)\n\nThe rules for updating r3_offset are as follows:\n\n* It starts out at 0\n* An addi r3,r3,N instruction decreases r3_offset by N\n* A store instruction (stb, sth, stw, std) to N(r3)\n increases r3_offset by the width of the store (1, 2, 4, 8)\n* A store with update instruction (stbu, sthu, stwu, stdu) to N(r3)\n sets r3_offset to the width of the store.\n\nThere is some trickiness to the way that the lex and stex macros and\nthe associated exception handlers work. I would have liked to use\nthe current value of r3_offset in the name of the symbol used as\nthe exception handler, as in \"extable .Lld_exc_$(r3_offset)\" and then\nhave symbols .Lld_exc_0, .Lld_exc_8, .Lld_exc_16 etc. corresponding\nto the offsets that needed to be added to r3. However, I couldn't\nsee a way to do that with gas.\n\nInstead, the exception handler address is .Lld_exc - r3_offset or\n.Lst_exc - r3_offset, that is, the distance ahead of .Lld_exc/.Lst_exc\nthat we start executing is equal to the amount that we need to add to\nr3. This works because r3_offset is always a small multiple of 4,\nand our instructions are 4 bytes long. This means that before\n.Lld_exc and .Lst_exc, we have a sequence of instructions that\nincrements r3 by 4, 8, 16 or 24 depending on where we start. The\nsequence increments r3 by 4 per instruction (on average).\n\nWe also replace the exception table for the 4k copy loop by a\nmacro per load or store. These loads and stores all use exactly\nthe same exception handler, which simply resets the argument registers\nr3, r4 and r5 to there original values and re-does the whole copy\nusing the slower loop.\n\nSigned-off-by: Paul Mackerras <paulus@ozlabs.org>\n---\n arch/powerpc/lib/copyuser_64.S | 579 +++++++++++++++++------------------------\n 1 file changed, 239 insertions(+), 340 deletions(-)", "diff": "diff --git a/arch/powerpc/lib/copyuser_64.S b/arch/powerpc/lib/copyuser_64.S\nindex 60386b2..1c5247c 100644\n--- a/arch/powerpc/lib/copyuser_64.S\n+++ b/arch/powerpc/lib/copyuser_64.S\n@@ -18,6 +18,36 @@\n #define sHd sld\t\t/* Shift towards high-numbered address. */\n #endif\n \n+/*\n+ * These macros are used to generate exception table entries.\n+ * The exception handlers below use the original arguments\n+ * (stored on the stack) and the point where we're up to in\n+ * the destination buffer, i.e. the address of the first\n+ * unmodified byte. Generally r3 points into the destination\n+ * buffer, but the first unmodified byte is at a variable\n+ * offset from r3. In the code below, the symbol r3_offset\n+ * is set to indicate the current offset at each point in\n+ * the code. This offset is then used as a negative offset\n+ * from the exception handler code, and those instructions\n+ * before the exception handlers are addi instructions that\n+ * adjust r3 to point to the correct place.\n+ */\n+\t.macro\textable\thandler\n+100:\n+\t.section __ex_table,\"a\"\n+\t.align 3\n+\t.llong 100b,\\handler\n+\t.previous\n+\t.endm\n+\n+\t.macro\tlex\t\t/* exception handler for load */\n+\textable\t.Lld_exc - r3_offset\n+\t.endm\n+\n+\t.macro\tstex\t\t/* exception handler for store */\n+\textable\t.Lst_exc - r3_offset\n+\t.endm\n+\n \t.align\t7\n _GLOBAL_TOC(__copy_tofrom_user)\n BEGIN_FTR_SECTION\n@@ -26,7 +56,7 @@ FTR_SECTION_ELSE\n \tb\t__copy_tofrom_user_power7\n ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY)\n _GLOBAL(__copy_tofrom_user_base)\n-\t/* first check for a whole page copy on a page boundary */\n+\t/* first check for a 4kB copy on a 4kB boundary */\n \tcmpldi\tcr1,r5,16\n \tcmpdi\tcr6,r5,4096\n \tor\tr0,r3,r4\n@@ -55,6 +85,7 @@ ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \\\n \t\t CPU_FTR_UNALIGNED_LD_STD)\n .Ldst_aligned:\n \taddi\tr3,r3,-16\n+r3_offset = 16\n BEGIN_FTR_SECTION\n \tandi.\tr0,r4,7\n \tbne\t.Lsrc_unaligned\n@@ -62,57 +93,69 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n \tblt\tcr1,.Ldo_tail\t\t/* if < 16 bytes to copy */\n \tsrdi\tr0,r5,5\n \tcmpdi\tcr1,r0,0\n-20:\tld\tr7,0(r4)\n-220:\tld\tr6,8(r4)\n+lex;\tld\tr7,0(r4)\n+lex;\tld\tr6,8(r4)\n \taddi\tr4,r4,16\n \tmtctr\tr0\n \tandi.\tr0,r5,0x10\n \tbeq\t22f\n \taddi\tr3,r3,16\n+r3_offset = 0\n \taddi\tr4,r4,-16\n \tmr\tr9,r7\n \tmr\tr8,r6\n \tbeq\tcr1,72f\n-21:\tld\tr7,16(r4)\n-221:\tld\tr6,24(r4)\n+21:\n+lex;\tld\tr7,16(r4)\n+lex;\tld\tr6,24(r4)\n \taddi\tr4,r4,32\n-70:\tstd\tr9,0(r3)\n-270:\tstd\tr8,8(r3)\n-22:\tld\tr9,0(r4)\n-222:\tld\tr8,8(r4)\n-71:\tstd\tr7,16(r3)\n-271:\tstd\tr6,24(r3)\n+stex;\tstd\tr9,0(r3)\n+r3_offset = 8\n+stex;\tstd\tr8,8(r3)\n+r3_offset = 16\n+22:\n+lex;\tld\tr9,0(r4)\n+lex;\tld\tr8,8(r4)\n+stex;\tstd\tr7,16(r3)\n+r3_offset = 24\n+stex;\tstd\tr6,24(r3)\n \taddi\tr3,r3,32\n+r3_offset = 0\n \tbdnz\t21b\n-72:\tstd\tr9,0(r3)\n-272:\tstd\tr8,8(r3)\n+72:\n+stex;\tstd\tr9,0(r3)\n+r3_offset = 8\n+stex;\tstd\tr8,8(r3)\n+r3_offset = 16\n \tandi.\tr5,r5,0xf\n \tbeq+\t3f\n \taddi\tr4,r4,16\n .Ldo_tail:\n \taddi\tr3,r3,16\n+r3_offset = 0\n \tbf\tcr7*4+0,246f\n-244:\tld\tr9,0(r4)\n+lex;\tld\tr9,0(r4)\n \taddi\tr4,r4,8\n-245:\tstd\tr9,0(r3)\n+stex;\tstd\tr9,0(r3)\n \taddi\tr3,r3,8\n 246:\tbf\tcr7*4+1,1f\n-23:\tlwz\tr9,0(r4)\n+lex;\tlwz\tr9,0(r4)\n \taddi\tr4,r4,4\n-73:\tstw\tr9,0(r3)\n+stex;\tstw\tr9,0(r3)\n \taddi\tr3,r3,4\n 1:\tbf\tcr7*4+2,2f\n-44:\tlhz\tr9,0(r4)\n+lex;\tlhz\tr9,0(r4)\n \taddi\tr4,r4,2\n-74:\tsth\tr9,0(r3)\n+stex;\tsth\tr9,0(r3)\n \taddi\tr3,r3,2\n 2:\tbf\tcr7*4+3,3f\n-45:\tlbz\tr9,0(r4)\n-75:\tstb\tr9,0(r3)\n+lex;\tlbz\tr9,0(r4)\n+stex;\tstb\tr9,0(r3)\n 3:\tli\tr3,0\n \tblr\n \n .Lsrc_unaligned:\n+r3_offset = 16\n \tsrdi\tr6,r5,3\n \taddi\tr5,r5,-16\n \tsubf\tr4,r0,r4\n@@ -125,58 +168,69 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n \tadd\tr5,r5,r0\n \tbt\tcr7*4+0,28f\n \n-24:\tld\tr9,0(r4)\t/* 3+2n loads, 2+2n stores */\n-25:\tld\tr0,8(r4)\n+lex;\tld\tr9,0(r4)\t/* 3+2n loads, 2+2n stores */\n+lex;\tld\tr0,8(r4)\n \tsLd\tr6,r9,r10\n-26:\tldu\tr9,16(r4)\n+lex;\tldu\tr9,16(r4)\n \tsHd\tr7,r0,r11\n \tsLd\tr8,r0,r10\n \tor\tr7,r7,r6\n \tblt\tcr6,79f\n-27:\tld\tr0,8(r4)\n+lex;\tld\tr0,8(r4)\n \tb\t2f\n \n-28:\tld\tr0,0(r4)\t/* 4+2n loads, 3+2n stores */\n-29:\tldu\tr9,8(r4)\n+28:\n+lex;\tld\tr0,0(r4)\t/* 4+2n loads, 3+2n stores */\n+lex;\tldu\tr9,8(r4)\n \tsLd\tr8,r0,r10\n \taddi\tr3,r3,-8\n+r3_offset = 24\n \tblt\tcr6,5f\n-30:\tld\tr0,8(r4)\n+lex;\tld\tr0,8(r4)\n \tsHd\tr12,r9,r11\n \tsLd\tr6,r9,r10\n-31:\tldu\tr9,16(r4)\n+lex;\tldu\tr9,16(r4)\n \tor\tr12,r8,r12\n \tsHd\tr7,r0,r11\n \tsLd\tr8,r0,r10\n \taddi\tr3,r3,16\n+r3_offset = 8\n \tbeq\tcr6,78f\n \n 1:\tor\tr7,r7,r6\n-32:\tld\tr0,8(r4)\n-76:\tstd\tr12,8(r3)\n+lex;\tld\tr0,8(r4)\n+stex;\tstd\tr12,8(r3)\n+r3_offset = 16\n 2:\tsHd\tr12,r9,r11\n \tsLd\tr6,r9,r10\n-33:\tldu\tr9,16(r4)\n+lex;\tldu\tr9,16(r4)\n \tor\tr12,r8,r12\n-77:\tstdu\tr7,16(r3)\n+stex;\tstdu\tr7,16(r3)\n+r3_offset = 8\n \tsHd\tr7,r0,r11\n \tsLd\tr8,r0,r10\n \tbdnz\t1b\n \n-78:\tstd\tr12,8(r3)\n+78:\n+stex;\tstd\tr12,8(r3)\n+r3_offset = 16\n \tor\tr7,r7,r6\n-79:\tstd\tr7,16(r3)\n+79:\n+stex;\tstd\tr7,16(r3)\n+r3_offset = 24\n 5:\tsHd\tr12,r9,r11\n \tor\tr12,r8,r12\n-80:\tstd\tr12,24(r3)\n+stex;\tstd\tr12,24(r3)\n+r3_offset = 32\n \tbne\t6f\n \tli\tr3,0\n \tblr\n 6:\tcmpwi\tcr1,r5,8\n \taddi\tr3,r3,32\n+r3_offset = 0\n \tsLd\tr9,r9,r10\n \tble\tcr1,7f\n-34:\tld\tr0,8(r4)\n+lex;\tld\tr0,8(r4)\n \tsHd\tr7,r0,r11\n \tor\tr9,r7,r9\n 7:\n@@ -184,7 +238,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n #ifdef __BIG_ENDIAN__\n \trotldi\tr9,r9,32\n #endif\n-94:\tstw\tr9,0(r3)\n+stex;\tstw\tr9,0(r3)\n #ifdef __LITTLE_ENDIAN__\n \trotrdi\tr9,r9,32\n #endif\n@@ -193,7 +247,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n #ifdef __BIG_ENDIAN__\n \trotldi\tr9,r9,16\n #endif\n-95:\tsth\tr9,0(r3)\n+stex;\tsth\tr9,0(r3)\n #ifdef __LITTLE_ENDIAN__\n \trotrdi\tr9,r9,16\n #endif\n@@ -202,7 +256,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n #ifdef __BIG_ENDIAN__\n \trotldi\tr9,r9,8\n #endif\n-96:\tstb\tr9,0(r3)\n+stex;\tstb\tr9,0(r3)\n #ifdef __LITTLE_ENDIAN__\n \trotrdi\tr9,r9,8\n #endif\n@@ -210,47 +264,55 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n \tblr\n \n .Ldst_unaligned:\n+r3_offset = 0\n \tPPC_MTOCRF(0x01,r6)\t\t/* put #bytes to 8B bdry into cr7 */\n \tsubf\tr5,r6,r5\n \tli\tr7,0\n \tcmpldi\tcr1,r5,16\n \tbf\tcr7*4+3,1f\n-35:\tlbz\tr0,0(r4)\n-81:\tstb\tr0,0(r3)\n+\textable\t.Lld_exc_r7\n+\tlbz\tr0,0(r4)\n+\textable\t.Lst_exc_r7\n+\tstb\tr0,0(r3)\n \taddi\tr7,r7,1\n 1:\tbf\tcr7*4+2,2f\n-36:\tlhzx\tr0,r7,r4\n-82:\tsthx\tr0,r7,r3\n+\textable\t.Lld_exc_r7\n+\tlhzx\tr0,r7,r4\n+\textable\t.Lst_exc_r7\n+\tsthx\tr0,r7,r3\n \taddi\tr7,r7,2\n 2:\tbf\tcr7*4+1,3f\n-37:\tlwzx\tr0,r7,r4\n-83:\tstwx\tr0,r7,r3\n+\textable\t.Lld_exc_r7\n+\tlwzx\tr0,r7,r4\n+\textable\t.Lst_exc_r7\n+\tstwx\tr0,r7,r3\n 3:\tPPC_MTOCRF(0x01,r5)\n \tadd\tr4,r6,r4\n \tadd\tr3,r6,r3\n \tb\t.Ldst_aligned\n \n .Lshort_copy:\n+r3_offset = 0\n \tbf\tcr7*4+0,1f\n-38:\tlwz\tr0,0(r4)\n-39:\tlwz\tr9,4(r4)\n+lex;\tlwz\tr0,0(r4)\n+lex;\tlwz\tr9,4(r4)\n \taddi\tr4,r4,8\n-84:\tstw\tr0,0(r3)\n-85:\tstw\tr9,4(r3)\n+stex;\tstw\tr0,0(r3)\n+stex;\tstw\tr9,4(r3)\n \taddi\tr3,r3,8\n 1:\tbf\tcr7*4+1,2f\n-40:\tlwz\tr0,0(r4)\n+lex;\tlwz\tr0,0(r4)\n \taddi\tr4,r4,4\n-86:\tstw\tr0,0(r3)\n+stex;\tstw\tr0,0(r3)\n \taddi\tr3,r3,4\n 2:\tbf\tcr7*4+2,3f\n-41:\tlhz\tr0,0(r4)\n+lex;\tlhz\tr0,0(r4)\n \taddi\tr4,r4,2\n-87:\tsth\tr0,0(r3)\n+stex;\tsth\tr0,0(r3)\n \taddi\tr3,r3,2\n 3:\tbf\tcr7*4+3,4f\n-42:\tlbz\tr0,0(r4)\n-88:\tstb\tr0,0(r3)\n+lex;\tlbz\tr0,0(r4)\n+stex;\tstb\tr0,0(r3)\n 4:\tli\tr3,0\n \tblr\n \n@@ -258,48 +320,34 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n * exception handlers follow\n * we have to return the number of bytes not copied\n * for an exception on a load, we set the rest of the destination to 0\n+ * Note that the number of bytes of instructions for adjusting r3 needs\n+ * to equal the amount of the adjustment, due to the trick of using\n+ * .Lld_exc - r3_offset as the handler address.\n */\n \n-136:\n-137:\n+.Lld_exc_r7:\n \tadd\tr3,r3,r7\n-\tb\t1f\n-130:\n-131:\n+\tb\t.Lld_exc\n+\n+\t/* adjust by 24 */\n \taddi\tr3,r3,8\n-120:\n-320:\n-122:\n-322:\n-124:\n-125:\n-126:\n-127:\n-128:\n-129:\n-133:\n+\tnop\n+\t/* adjust by 16 */\n \taddi\tr3,r3,8\n-132:\n+\tnop\n+\t/* adjust by 8 */\n \taddi\tr3,r3,8\n-121:\n-321:\n-344:\n-134:\n-135:\n-138:\n-139:\n-140:\n-141:\n-142:\n-123:\n-144:\n-145:\n+\tnop\n \n /*\n- * here we have had a fault on a load and r3 points to the first\n- * unmodified byte of the destination\n+ * Here we have had a fault on a load and r3 points to the first\n+ * unmodified byte of the destination. We use the original arguments\n+ * and r3 to work out how much wasn't copied. Since we load some\n+ * distance ahead of the stores, we continue copying byte-by-byte until\n+ * we hit the load fault again in order to copy as much as possible.\n */\n-1:\tld\tr6,-24(r1)\n+.Lld_exc:\n+\tld\tr6,-24(r1)\n \tld\tr4,-16(r1)\n \tld\tr5,-8(r1)\n \tsubf\tr6,r6,r3\n@@ -310,9 +358,11 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n * first see if we can copy any more bytes before hitting another exception\n */\n \tmtctr\tr5\n+r3_offset = 0\n+\textable\t.Lclear_rest\n 43:\tlbz\tr0,0(r4)\n \taddi\tr4,r4,1\n-89:\tstb\tr0,0(r3)\n+stex;\tstb\tr0,0(r3)\n \taddi\tr3,r3,1\n \tbdnz\t43b\n \tli\tr3,0\t\t/* huh? all copied successfully this time? */\n@@ -321,13 +371,15 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n /*\n * here we have trapped again, need to clear ctr bytes starting at r3\n */\n-143:\tmfctr\tr5\n+.Lclear_rest:\n+\tmfctr\tr5\n \tli\tr0,0\n \tmr\tr4,r3\n \tmr\tr3,r5\t\t/* return the number of bytes not copied */\n 1:\tandi.\tr9,r4,7\n \tbeq\t3f\n-90:\tstb\tr0,0(r4)\n+\textable\t99f\n+\tstb\tr0,0(r4)\n \taddic.\tr5,r5,-1\n \taddi\tr4,r4,1\n \tbne\t1b\n@@ -337,133 +389,54 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n \tandi.\tr5,r5,7\n \tblt\tcr1,93f\n \tmtctr\tr9\n+\textable\t99f\n 91:\tstd\tr0,0(r4)\n \taddi\tr4,r4,8\n \tbdnz\t91b\n 93:\tbeqlr\n \tmtctr\tr5\t\n+\textable\t99f\n 92:\tstb\tr0,0(r4)\n \taddi\tr4,r4,1\n \tbdnz\t92b\n-\tblr\n+99:\tblr\n \n /*\n * exception handlers for stores: we just need to work\n * out how many bytes weren't copied\n+ * Note that the number of bytes of instructions for adjusting r3 needs\n+ * to equal the amount of the adjustment, due to the trick of using\n+ * .Lst_exc - r3_offset as the handler address.\n */\n-182:\n-183:\n+.Lst_exc_r7:\n \tadd\tr3,r3,r7\n-\tb\t1f\n-371:\n-180:\n+\tb\t.Lst_exc\n+\n+\t/* adjust by 24 */\n \taddi\tr3,r3,8\n-171:\n-177:\n-179:\n+\tnop\n+\t/* adjust by 16 */\n \taddi\tr3,r3,8\n-370:\n-372:\n-176:\n-178:\n+\tnop\n+\t/* adjust by 8 */\n \taddi\tr3,r3,4\n-185:\n+\t/* adjust by 4 */\n \taddi\tr3,r3,4\n-170:\n-172:\n-345:\n-173:\n-174:\n-175:\n-181:\n-184:\n-186:\n-187:\n-188:\n-189:\t\n-194:\n-195:\n-196:\n-1:\n+.Lst_exc:\n \tld\tr6,-24(r1)\n \tld\tr5,-8(r1)\n \tadd\tr6,r6,r5\n-\tsubf\tr3,r3,r6\t/* #bytes not copied */\n-190:\n-191:\n-192:\n-\tblr\t\t\t/* #bytes not copied in r3 */\n-\n-\t.section __ex_table,\"a\"\n-\t.align\t3\n-\t.llong\t20b,120b\n-\t.llong\t220b,320b\n-\t.llong\t21b,121b\n-\t.llong\t221b,321b\n-\t.llong\t70b,170b\n-\t.llong\t270b,370b\n-\t.llong\t22b,122b\n-\t.llong\t222b,322b\n-\t.llong\t71b,171b\n-\t.llong\t271b,371b\n-\t.llong\t72b,172b\n-\t.llong\t272b,372b\n-\t.llong\t244b,344b\n-\t.llong\t245b,345b\n-\t.llong\t23b,123b\n-\t.llong\t73b,173b\n-\t.llong\t44b,144b\n-\t.llong\t74b,174b\n-\t.llong\t45b,145b\n-\t.llong\t75b,175b\n-\t.llong\t24b,124b\n-\t.llong\t25b,125b\n-\t.llong\t26b,126b\n-\t.llong\t27b,127b\n-\t.llong\t28b,128b\n-\t.llong\t29b,129b\n-\t.llong\t30b,130b\n-\t.llong\t31b,131b\n-\t.llong\t32b,132b\n-\t.llong\t76b,176b\n-\t.llong\t33b,133b\n-\t.llong\t77b,177b\n-\t.llong\t78b,178b\n-\t.llong\t79b,179b\n-\t.llong\t80b,180b\n-\t.llong\t34b,134b\n-\t.llong\t94b,194b\n-\t.llong\t95b,195b\n-\t.llong\t96b,196b\n-\t.llong\t35b,135b\n-\t.llong\t81b,181b\n-\t.llong\t36b,136b\n-\t.llong\t82b,182b\n-\t.llong\t37b,137b\n-\t.llong\t83b,183b\n-\t.llong\t38b,138b\n-\t.llong\t39b,139b\n-\t.llong\t84b,184b\n-\t.llong\t85b,185b\n-\t.llong\t40b,140b\n-\t.llong\t86b,186b\n-\t.llong\t41b,141b\n-\t.llong\t87b,187b\n-\t.llong\t42b,142b\n-\t.llong\t88b,188b\n-\t.llong\t43b,143b\n-\t.llong\t89b,189b\n-\t.llong\t90b,190b\n-\t.llong\t91b,191b\n-\t.llong\t92b,192b\n-\t\n-\t.text\n+\tsubf\tr3,r3,r6\t/* #bytes not copied in r3 */\n+\tblr\n \n /*\n * Routine to copy a whole page of data, optimized for POWER4.\n * On POWER4 it is more than 50% faster than the simple loop\n * above (following the .Ldst_aligned label).\n */\n+\t.macro\texc\n+\textable\t.Labort\n+\t.endm\n .Lcopy_page_4K:\n \tstd\tr31,-32(1)\n \tstd\tr30,-40(1)\n@@ -482,86 +455,86 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n \tli\tr0,5\n 0:\taddi\tr5,r5,-24\n \tmtctr\tr0\n-20:\tld\tr22,640(4)\n-21:\tld\tr21,512(4)\n-22:\tld\tr20,384(4)\n-23:\tld\tr11,256(4)\n-24:\tld\tr9,128(4)\n-25:\tld\tr7,0(4)\n-26:\tld\tr25,648(4)\n-27:\tld\tr24,520(4)\n-28:\tld\tr23,392(4)\n-29:\tld\tr10,264(4)\n-30:\tld\tr8,136(4)\n-31:\tldu\tr6,8(4)\n+exc;\tld\tr22,640(4)\n+exc;\tld\tr21,512(4)\n+exc;\tld\tr20,384(4)\n+exc;\tld\tr11,256(4)\n+exc;\tld\tr9,128(4)\n+exc;\tld\tr7,0(4)\n+exc;\tld\tr25,648(4)\n+exc;\tld\tr24,520(4)\n+exc;\tld\tr23,392(4)\n+exc;\tld\tr10,264(4)\n+exc;\tld\tr8,136(4)\n+exc;\tldu\tr6,8(4)\n \tcmpwi\tr5,24\n 1:\n-32:\tstd\tr22,648(3)\n-33:\tstd\tr21,520(3)\n-34:\tstd\tr20,392(3)\n-35:\tstd\tr11,264(3)\n-36:\tstd\tr9,136(3)\n-37:\tstd\tr7,8(3)\n-38:\tld\tr28,648(4)\n-39:\tld\tr27,520(4)\n-40:\tld\tr26,392(4)\n-41:\tld\tr31,264(4)\n-42:\tld\tr30,136(4)\n-43:\tld\tr29,8(4)\n-44:\tstd\tr25,656(3)\n-45:\tstd\tr24,528(3)\n-46:\tstd\tr23,400(3)\n-47:\tstd\tr10,272(3)\n-48:\tstd\tr8,144(3)\n-49:\tstd\tr6,16(3)\n-50:\tld\tr22,656(4)\n-51:\tld\tr21,528(4)\n-52:\tld\tr20,400(4)\n-53:\tld\tr11,272(4)\n-54:\tld\tr9,144(4)\n-55:\tld\tr7,16(4)\n-56:\tstd\tr28,664(3)\n-57:\tstd\tr27,536(3)\n-58:\tstd\tr26,408(3)\n-59:\tstd\tr31,280(3)\n-60:\tstd\tr30,152(3)\n-61:\tstdu\tr29,24(3)\n-62:\tld\tr25,664(4)\n-63:\tld\tr24,536(4)\n-64:\tld\tr23,408(4)\n-65:\tld\tr10,280(4)\n-66:\tld\tr8,152(4)\n-67:\tldu\tr6,24(4)\n+exc;\tstd\tr22,648(3)\n+exc;\tstd\tr21,520(3)\n+exc;\tstd\tr20,392(3)\n+exc;\tstd\tr11,264(3)\n+exc;\tstd\tr9,136(3)\n+exc;\tstd\tr7,8(3)\n+exc;\tld\tr28,648(4)\n+exc;\tld\tr27,520(4)\n+exc;\tld\tr26,392(4)\n+exc;\tld\tr31,264(4)\n+exc;\tld\tr30,136(4)\n+exc;\tld\tr29,8(4)\n+exc;\tstd\tr25,656(3)\n+exc;\tstd\tr24,528(3)\n+exc;\tstd\tr23,400(3)\n+exc;\tstd\tr10,272(3)\n+exc;\tstd\tr8,144(3)\n+exc;\tstd\tr6,16(3)\n+exc;\tld\tr22,656(4)\n+exc;\tld\tr21,528(4)\n+exc;\tld\tr20,400(4)\n+exc;\tld\tr11,272(4)\n+exc;\tld\tr9,144(4)\n+exc;\tld\tr7,16(4)\n+exc;\tstd\tr28,664(3)\n+exc;\tstd\tr27,536(3)\n+exc;\tstd\tr26,408(3)\n+exc;\tstd\tr31,280(3)\n+exc;\tstd\tr30,152(3)\n+exc;\tstdu\tr29,24(3)\n+exc;\tld\tr25,664(4)\n+exc;\tld\tr24,536(4)\n+exc;\tld\tr23,408(4)\n+exc;\tld\tr10,280(4)\n+exc;\tld\tr8,152(4)\n+exc;\tldu\tr6,24(4)\n \tbdnz\t1b\n-68:\tstd\tr22,648(3)\n-69:\tstd\tr21,520(3)\n-70:\tstd\tr20,392(3)\n-71:\tstd\tr11,264(3)\n-72:\tstd\tr9,136(3)\n-73:\tstd\tr7,8(3)\n-74:\taddi\tr4,r4,640\n-75:\taddi\tr3,r3,648\n+exc;\tstd\tr22,648(3)\n+exc;\tstd\tr21,520(3)\n+exc;\tstd\tr20,392(3)\n+exc;\tstd\tr11,264(3)\n+exc;\tstd\tr9,136(3)\n+exc;\tstd\tr7,8(3)\n+\taddi\tr4,r4,640\n+\taddi\tr3,r3,648\n \tbge\t0b\n \tmtctr\tr5\n-76:\tld\tr7,0(4)\n-77:\tld\tr8,8(4)\n-78:\tldu\tr9,16(4)\n+exc;\tld\tr7,0(4)\n+exc;\tld\tr8,8(4)\n+exc;\tldu\tr9,16(4)\n 3:\n-79:\tld\tr10,8(4)\n-80:\tstd\tr7,8(3)\n-81:\tld\tr7,16(4)\n-82:\tstd\tr8,16(3)\n-83:\tld\tr8,24(4)\n-84:\tstd\tr9,24(3)\n-85:\tldu\tr9,32(4)\n-86:\tstdu\tr10,32(3)\n+exc;\tld\tr10,8(4)\n+exc;\tstd\tr7,8(3)\n+exc;\tld\tr7,16(4)\n+exc;\tstd\tr8,16(3)\n+exc;\tld\tr8,24(4)\n+exc;\tstd\tr9,24(3)\n+exc;\tldu\tr9,32(4)\n+exc;\tstdu\tr10,32(3)\n \tbdnz\t3b\n 4:\n-87:\tld\tr10,8(4)\n-88:\tstd\tr7,8(3)\n-89:\tstd\tr8,16(3)\n-90:\tstd\tr9,24(3)\n-91:\tstd\tr10,32(3)\n+exc;\tld\tr10,8(4)\n+exc;\tstd\tr7,8(3)\n+exc;\tstd\tr8,16(3)\n+exc;\tstd\tr9,24(3)\n+exc;\tstd\tr10,32(3)\n 9:\tld\tr20,-120(1)\n \tld\tr21,-112(1)\n \tld\tr22,-104(1)\n@@ -581,7 +554,8 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n * on an exception, reset to the beginning and jump back into the\n * standard __copy_tofrom_user\n */\n-100:\tld\tr20,-120(1)\n+.Labort:\n+\tld\tr20,-120(1)\n \tld\tr21,-112(1)\n \tld\tr22,-104(1)\n \tld\tr23,-96(1)\n@@ -597,79 +571,4 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)\n \tld\tr4,-16(r1)\n \tli\tr5,4096\n \tb\t.Ldst_aligned\n-\n-\t.section __ex_table,\"a\"\n-\t.align\t3\n-\t.llong\t20b,100b\n-\t.llong\t21b,100b\n-\t.llong\t22b,100b\n-\t.llong\t23b,100b\n-\t.llong\t24b,100b\n-\t.llong\t25b,100b\n-\t.llong\t26b,100b\n-\t.llong\t27b,100b\n-\t.llong\t28b,100b\n-\t.llong\t29b,100b\n-\t.llong\t30b,100b\n-\t.llong\t31b,100b\n-\t.llong\t32b,100b\n-\t.llong\t33b,100b\n-\t.llong\t34b,100b\n-\t.llong\t35b,100b\n-\t.llong\t36b,100b\n-\t.llong\t37b,100b\n-\t.llong\t38b,100b\n-\t.llong\t39b,100b\n-\t.llong\t40b,100b\n-\t.llong\t41b,100b\n-\t.llong\t42b,100b\n-\t.llong\t43b,100b\n-\t.llong\t44b,100b\n-\t.llong\t45b,100b\n-\t.llong\t46b,100b\n-\t.llong\t47b,100b\n-\t.llong\t48b,100b\n-\t.llong\t49b,100b\n-\t.llong\t50b,100b\n-\t.llong\t51b,100b\n-\t.llong\t52b,100b\n-\t.llong\t53b,100b\n-\t.llong\t54b,100b\n-\t.llong\t55b,100b\n-\t.llong\t56b,100b\n-\t.llong\t57b,100b\n-\t.llong\t58b,100b\n-\t.llong\t59b,100b\n-\t.llong\t60b,100b\n-\t.llong\t61b,100b\n-\t.llong\t62b,100b\n-\t.llong\t63b,100b\n-\t.llong\t64b,100b\n-\t.llong\t65b,100b\n-\t.llong\t66b,100b\n-\t.llong\t67b,100b\n-\t.llong\t68b,100b\n-\t.llong\t69b,100b\n-\t.llong\t70b,100b\n-\t.llong\t71b,100b\n-\t.llong\t72b,100b\n-\t.llong\t73b,100b\n-\t.llong\t74b,100b\n-\t.llong\t75b,100b\n-\t.llong\t76b,100b\n-\t.llong\t77b,100b\n-\t.llong\t78b,100b\n-\t.llong\t79b,100b\n-\t.llong\t80b,100b\n-\t.llong\t81b,100b\n-\t.llong\t82b,100b\n-\t.llong\t83b,100b\n-\t.llong\t84b,100b\n-\t.llong\t85b,100b\n-\t.llong\t86b,100b\n-\t.llong\t87b,100b\n-\t.llong\t88b,100b\n-\t.llong\t89b,100b\n-\t.llong\t90b,100b\n-\t.llong\t91b,100b\n EXPORT_SYMBOL(__copy_tofrom_user)\n", "prefixes": [ "1/4" ] }