| Message ID | 7bfe2625-725d-d1bb-7177-f2d31ce09e9c@landley.net |
|---|---|
| State | Not Applicable |
| Headers | show |
Hi, Rob Landley wrote, > On 12/08/2016 03:11 PM, Rich Felker wrote: > > On Tue, Dec 06, 2016 at 11:52:29PM -0600, Rob Landley wrote: > >> Index: src/setjmp/arm/setjmp.S > >> =================================================================== > >> --- src/setjmp/arm/setjmp.S (revision 0) > >> +++ src/setjmp/arm/setjmp.S (revision 4920) > >> @@ -0,0 +1,45 @@ > >> +.global __setjmp > >> +.global _setjmp > >> +.global setjmp > >> +.type __setjmp,%function > >> +.type _setjmp,%function > >> +.type setjmp,%function > >> +__setjmp: > >> +_setjmp: > >> +setjmp: > >> + mov ip,r0 > >> +#if defined(__thumb__) > >> + stmia ip!,{v1,v2,v3,v4,v5,v6,sl,fp,lr} > >> + str sp, [ip], #4 > >> +#else > >> + stmia ip!,{v1,v2,v3,v4,v5,v6,sl,fp,sp,lr} > >> +#endif > > > > My leaning is to just always use the version that works on thumb, > > Presumaby there's a longjmp.S bit too because: > > ldmia ip!, {v1,v2,v3,v4,v5,v6,sl,fp,sp,lr} > > Make sure you're aware of erratum 752419: > > https://sourceware.org/ml/binutils/2011-05/msg00087.html > > On a board with a lot of serial traffic, it takes about 3 minutes for > uClibc (yes, including the current -ng) to crash (by loading the link > register into the stack pointer and stomping code before the setjmp). > > The emcraft guys fixed this in their cortex-m uClibc fork way back when > (http://www.emcraft.com/som/building-uclibc), and of _course_ never > bothered to send it upstream. (Or put it on github with their kernel and > uboot.) But the source is in their distro tarball, and their patch to > uClibc's __longjmp.S is basically: > > --- uclibc-1.0.17/libc/sysdeps/linux/arm/__longjmp.S > +++ emcraft/libc/sysdeps/linux/arm/__longjmp.S > @@ -56,9 +60,15 @@ > #if defined(__thumb2__) > /* Thumb-2 does not allow loading sp with ldm. */ > ldmia ip!, {v1-v6, sl, fp} > +#if 0 > ldr sp, [ip], #4 > ldr lr, [ip], #4 > #else > + ldr sp, [ip, #0] > + ldr lr, [ip, #4] > + add ip, #8 > +#endif > +#else > ldmia ip!, {v1-v6, sl, fp, sp, lr} > #endif > > > (Dunno why the "add ip, #8" because it's a scratch register and this is > the last use in the function, but maybe they just made it do exactly > what the #else case does?) > > cc-ing buildroot because this is still broken in their november release. I am wondering why you don't cc me or any uclibc related list? You still believe uClibc projects should die? It is really cool and very nice that the communication between Rich and me is always fruitful. I always report any bugs I find while testing musl on new or old platforms (mostly via IRC channel) and I always take care that bugfixes for musl targets end up in buildroot. (mipsr6 support, binutils microblaze problems, ..) I really would like to see a similar open communication between the nommu community, you and me. Just ignoring uClibc-ng does not will make it die. As a good starting point, a nice bug report and/or test application which allows me to reproduce the problem would be really appreciated. best regards Waldemar
Sorry for the delay responding, lots of deadlines and travel recently... (Ok, for a definition of "recently" that goes back a decade: http://lists.busybox.net/pipermail/busybox/2005-June/048743.html) On 12/15/2016 12:51 PM, Waldemar Brodkorb wrote: > Hi, > Rob Landley wrote, > >> On 12/08/2016 03:11 PM, Rich Felker wrote: >>> On Tue, Dec 06, 2016 at 11:52:29PM -0600, Rob Landley wrote: >> Make sure you're aware of erratum 752419: >> >> https://sourceware.org/ml/binutils/2011-05/msg00087.html ... >> cc-ing buildroot because this is still broken in their november release. > > I am wondering why you don't cc me or any uclibc related list? I cc'd the buildroot list, which only has uClibc-based cortex-m support at the moment. Why do you suppose I did that? Did you want me to send it to the uclibc.org mailing list which hasn't had a single post this month except your announcement of your fork's release? The list where nobody's noticed the chrome browser can't access https://lists.uclibc.org (archives, subscription page, etc) for weeks now? And yes, I publicized that fact when I noticed it: https://twitter.com/landley/status/806202364822597632 Several people replied to that tweet, if nobody bothered to poke uclibc.org I'm clearly not the only one who thinks the project is dead. Your fork clearly hasn't fixed any of the structural issues uClibc developed over the years. We are discussing a patch to cortex-m that I found because code was crashing, and when gdb was deployed we found out that code in libpthread/libthreads/pthread.c was getting corrupted, and we worked out that the corruption ended right after the sigsetjmp() call in __pthread_timedsuspend_new() and that all the data before it looked like stack contents, stack grows down on every linux target except pa-risc so that's actually where the corruption started... and we went from there. This was debugging through the _old_ legacy pthreads implementation, which is the only option on cortex-m because even the one buildroot has today never got NPTL working there. Musl has nptl for every target it supports (including a nommu one), but the uClibc developers have spent TEN YEARS trying to make that work since the first out-of-tree NTPL version that sjhill promised to release after his OLS presentation, then after his customer paid him enough, then... If you don't understand WHY this is such a bad thing, then you clearly don't know the history of the project. You don't now what the problems that led to all the multi-year gaps between releases WERE. Has your fork solved the locales problem? http://lists.busybox.net/pipermail/uclibc/2015-June/049000.html Has your fork solved the nptl issue? http://lists.busybox.net/pipermail/uclibc/2008-September/020151.html http://lists.busybox.net/pipermail/uclibc/2008-September/020169.html http://lists.busybox.net/pipermail/uclibc/2008-September/020171.html http://lists.busybox.net/pipermail/uclibc/2008-October/041201.html Did you fix the fact that huge swaths of uClibc (especially headers) are just snapshots of old versions of glibc stuff advertising things uClibc doesn't actually implement and pretending to be glibc? http://lists.busybox.net/pipermail/uclibc/2006-March/014811.html Do you have a sane "make defconfig" that lets people build uClibc without learning what over a hundred individual config options do and making decisions about whether or not they need each one? This issue doesn't even come _up_ with musl, it fundamentally avoids most of the structural problems that strangled uClibc development, by design. > You still believe uClibc projects should die? No, I believe uClibc _already_ died. I believe this because I was there. I did my turn as Sisyphus on that project. I pushed the boulder uphill over an over for over a decade. And yes, I mean a full decade, starting in 2003: http://lists.busybox.net/pipermail/uclibc/2003-August/027643.html I used to prod the maintainer to get releases out by sending him birthday cakes when the existing release was a year old: http://lists.busybox.net/pipermail/uclibc/2005-January/010877.html http://lists.busybox.net/pipermail/uclibc/2006-March/014921.html http://lists.busybox.net/pipermail/uclibc/2006-March/014923.html http://lists.busybox.net/pipermail/uclibc/2006-December/037750.html http://lists.busybox.net/pipermail/uclibc/2006-December/037766.html I had to fight hard for the very IDEA of cutting releases: http://lists.busybox.net/pipermail/uclibc/2006-March/014722.html http://lists.busybox.net/pipermail/uclibc/2006-March/014881.html http://lists.busybox.net/pipermail/uclibc/2006-March/014885.html Here's a changelog I spent 3 days researching after a particularly long development cycle (because users had expressed reluctance to move to new versions when so _much_ had changed at once that the couldn't debug anything that broke): http://lists.busybox.net/pipermail/uclibc/2006-July/016032.html I used to prod other projects to support uClibc too. For example, it took me three years to finally convince qemu to stop being incompatible with uClibc static binaries on powerpc, but they finally made the change: https://lists.gnu.org/archive/html/qemu-devel/2010-02/msg00800.html Circa 2010 I was still trying to convince people uClibc would recover when everybody _else_ had declared it dead: http://lists.busybox.net/pipermail/uclibc/2010-April/043835.html Buildroot itself was finally ready to declare uClibc dead a couple years ago: http://lists.busybox.net/pipermail/buildroot/2014-February/089789.html Which is when you stepped in to continue beating the dead horse, so they didn't have to decide. It's nice that you're maintaining buildroot's uClibc so they don't have to maintain their own fork anymore (like emcraft still does, or https://github.com/mickael-guene/uclibc/tree/uClibc-0.9.33.2-fdpic-m). Your version is more interesting to me than random other attempt du jour like https://github.com/davidgfnet/uClibc-Os because buildroot uses your version. But cortex-m still only supports pthreads in 2016 and even that's buggy in ways that were fixed out of tree quite a while ago. The release I fished this bugfix out of is a year old. I don't have their source control to see how old the fix really is, but emcraft's "preferred" kernel (https://github.com/emcraftsystems/linux-emcraft) forked off from mainline 7 years ago, and cortex-m support for Linux is their core business, so there's a guess how long ago somebody actually _using_ this might have noticed it. In case you really _don't_ know the history, let me walk you through how the uClibc project died, going back through about ten years of accumulated scar tissue, and why three different maintainers before you failed to fix it. The most prominent reasons were a classic "project tumor", an absentee maintainer, development forking itself to death, and the code itself forking to death (most prominently over NPTL). --- Project Tumor The first problem is that buildroot forked off and sucked away all uClibc's developers for several years. This doesn't mean buildroot is bad, QEMU similarly forked off from tinycc and sucked away all that project's developers. QEMU started life with the purpose of running Wine on non-x86 platforms (https://www.winehq.org/pipermail/wine-devel/2003-March/015577.html). I usually summarize buildroot as "a test harness for uClibc that grew legs" but the actual history is more complciated, it seems to have been inspired by a build system for the "tuxscreen" project (http://lists.busybox.net/pipermail/uclibc/2002-February/002542.html) which Erik turned into a toolchain builder for uClibc (http://lists.busybox.net/pipermail/uclibc/2002-August/024891.html when he abandoned the old toolchain wrapper because of libgcc_s.so issues) then added the ability to do smoketest boots under User Mode Linux (which seems to have been brought up at http://lists.busybox.net/pipermail/busybox/2001-April/037259.html) and so on. For years its main purpose was people would go "How do I build X with uClibc" and he'd go http://lists.busybox.net/pipermail/uclibc/2002-August/004089.html The problem is, buildroot had no obvious project boundaries (you couldn't say "no, this is not part of buildroot") so it expanded and expanded, turning into a Linux distro which took up more and more of the uClibc developers' time. Erik publicly complained that this was negatively impacting uClibc development over a decade ago: http://lists.busybox.net/pipermail/uclibc/2003-August/027564.html I don't participate in buildroot development much (yes I still owe you guys a new toybox patch) but I was there from the beginning, as in I created the first buildroot config file (http://lists.busybox.net/pipermail/uclibc/2003-August/027559.html) and documentation (http://lists.busybox.net/pipermail/uclibc/2003-August/027531.html). It was also obvious to _me_ that buildroot traffic was drowning out uClibc development back at the start: http://lists.busybox.net/pipermail/uclibc/2003-November/028342.html But I didn't have the ability to fix it until Erik handed off busybox maintainership to me and thus gave me root access to the shared server, and I abused that access to create this buildroot list, the first post in the archive of which is: https://lists.busybox.net/pipermail/buildroot/2006-July/012219.html And then I tried to politely shunt the buildroot traffic off the uClibc list: https://lists.busybox.net/pipermail/uclibc/2006-July/015977.html But that was after buildroot traffic had drowned out uClibc development traffic for THREE YEARS. Needless to say, this hurt the project's development. Here are a few highlights from the uClibc thread that led _up_ to me creating the buildroot list: http://lists.busybox.net/pipermail/uclibc/2005-October/033703.html http://lists.busybox.net/pipermail/uclibc/2005-October/033709.html http://lists.busybox.net/pipermail/uclibc/2005-October/033712.html http://lists.busybox.net/pipermail/uclibc/2005-October/033720.html I.E. my motivation was that buildroot was smothering uClibc, but by the time I was in a position to separate the two uClibc development had been significantly eroded. --- Absentee maintainer The second problem uClibc died from was an absentee maintainer: Erik ran a struggling startup that took so much of his time it impacted his marriage. The prominent developers under him didn't fare much better, buildroot or no: Manuel Nova was never the same after his fiancee died (you may notice the various "donate money in memory of" headers in the code, he got really _bitter_ as time went on and eventually wandered off). Glenn McGrath burned out and left uClibc and BusyBox due to trying his hand at license enforcement and having a horrible experience with it (ala http://lists.busybox.net/pipermail/busybox/2005-April/048256.html which was one of the things I cited when starting my own license enforcement effort a few years alter, to take the burden off other people trying to do it themselves). SJ Hill got mercenary and refused to release his NPTL work until his employer paid outstanding invoices (which dragged on for _years_ but we'll get to that). I took busybox off of Erik's hands in part so he'd have more time to spend on uClibc (and also so busybox didn't go the _way_ of uClibc): http://lists.busybox.net/pipermail/busybox/2005-January/047510.html http://lists.busybox.net/pipermail/busybox/2005-January/047601.html http://lists.busybox.net/pipermail/busybox/2005-May/048481.html http://lists.busybox.net/pipermail/busybox/2005-May/048565.html And so on scaling up to me taking over busybox maintainership. (Alas the web archive has some holes in it, ala http://lists.busybox.net/pipermail/busybox/2005-June/048965.html). But to clarify: I took work off or Erik's plate when he _asked_ for help over in busybox-land: http://lists.busybox.net/pipermail/busybox/2005-April/048250.html http://lists.busybox.net/pipermail/busybox/2005-April/048255.html And it took up all my spare time. In May 2005 I posted to the busybox mailing list 104 times. http://lists.busybox.net/pipermail/busybox/2005-May/date.html I didn't have TIME to do the same for uClibc, but I still poked at various other issues such as the fact there was no official place to get kernel headers from when building a C library (the kernel guys said that coming up with sanitized kernel headers for use by userspace was a distro maintainers' problem, hence http://lists.busybox.net/pipermail/uclibc/2004-June/030119.html and http://lists.busybox.net/pipermail/uclibc/2006-February/035260.html and http://lists.pld-linux.org/mailman/pipermail/llh-discuss/2006-March/000016.html and https://lwn.net/Articles/244375/ and so on). Those sort of things reduced the number of people building their own uClibc systems, outside of something like buildroot that does it for you. Use of the project actually _declined_ as it got harder to make it work. (Side note: the eglibc project was created by developers commenting on how the death of uClibc made their project necessary. Wanna know how long ago that was?) Having chosen to get behind busybox and push, I didn't have a similar amount of spare bandwidth to directly help out uClibc with. Taking busybox off Erik's plate helped out uClibc a little bit, but it didn't last. Erik's participation continued to decline until he eventually left open source development entirely. --- Forked development Another reason the uClibc project died was it forked itself to death. Ten years ago a guy named sjhill did the first uClibc NPTL port: https://www.kernel.org/doc/ols/2006/ols2006v1-pages-409-420.pdf But he refused to release the code, first for various trivial reasons (ala http://lists.busybox.net/pipermail/uclibc/2006-March/035697.html) then he wanted to wait until after his OLS presentation, then he wanted to wait until whoever had sponsored the work (I forget) paid him more for it... His reasons for not merging it kept changing, and he kept dangling the idea of merging it but never doing so for years. Here's me declaring his work dead in an IRC conversation a couple years later after years of it never going in: http://infobot.rikers.org/%23uclibc/20080810.html.gz Other developers started maintaining out of tree forks as a matter of course, and complaining that merging code into mainline was inconveniencing their forks. Doing this eventually chased away the most active remaining developer who _was_ merging code into mainline, convincing him that _his_ fork should be out of tree too: http://lists.busybox.net/pipermail/uclibc/2006-March/015014.html In the ensuing discussion other prominent developers admitted that they valued their private forks more than mainline. Manuel Nova said, and I quote, "I can't ethicly (at least in my code of ethics) justify handing out bug fixes to my employer's competitors until necessary." http://lists.busybox.net/pipermail/uclibc/2006-March/015018.html http://lists.busybox.net/pipermail/uclibc/2006-April/015077.html http://lists.busybox.net/pipermail/uclibc/2006-April/015080.html Here's sjhill calling me a hypocrite for saying code _should_ go upstream: http://lists.busybox.net/pipermail/uclibc/2006-April/015103.html I participated fairly extensively in that thread: http://lists.busybox.net/pipermail/uclibc/2006-April/015083.html http://lists.busybox.net/pipermail/uclibc/2006-April/015097.html http://lists.busybox.net/pipermail/uclibc/2006-April/015127.html But the maintainer, Erik, didn't. Half his response in that thread was to propose removing architectures to reduce the maintenance burden. No really: http://lists.busybox.net/pipermail/uclibc/2006-April/015195.html Erik did not object to the project forking itself to death, specifically the "we must prevent code going upstream to not inconvenience these out of tree forks" thread topic was fine by him. I objected to his lack of objection: http://lists.busybox.net/pipermail/uclibc/2006-April/015085.html But it wasn't _my_ project, and I couldn't overrule its maintainer when he didn't care to prevent it from forking itself to death. --- Forked code Development wasn't the only thing that fragmented, the in-tree codebase did too. A persistent habit of uClibc's was cloning large chunks of glibc. Since the two projects were under the same license, the uClibc developers could copy a glibc subsystem (such as pthreads) more or less verbatim, and then try to strip it down to be smaller afterwards. Their snapshot would inevitably fall out of date with glibc's, and rather than try to update it themselves they'd eventually clone a less stale version and put it next to the other one in the tree with a menuconfig option to select the "old" or "new" version. On more than one occasion they had three snapshots of the same infrastructure in play at once. Here's an incomplete list of some of the duplicate infrastructure you had to select old or new versions of in menuconfig: http://lists.busybox.net/pipermail/uclibc/2008-December/041639.html Just educating yourself on what your OPTIONS were was a daunting task. And the menuconfig options and help text were often completely nonsensical. I used to go through and try to audit menuconfig: http://lists.busybox.net/pipermail/uclibc/2008-August/040766.html http://lists.busybox.net/pipermail/uclibc/2009-August/042851.html But tended to get pushback from people going "no, it makes sense to _me_ and I refuse to let it be changed": http://lists.busybox.net/pipermail/uclibc/2009-August/042868.html Needing menuconfig at all is another design problem: configuring uClibc is not a trivial exercise, and all sorts of configuration changes change the ABI and break binary compatibility. (And of course no uClibc version upgrade was ever binary compatible with any previous version.) You don't have this problem with musl-libc, not only _is_ there no menuconfig stage, but they tried for binary compatibility with glibc (as in trying to make it a drop-in replacement capable of running the flash plugin and such) for the longest time, and thus musl has a reasonably stable ABI. (It's not 100% glibc compatible but they know what their ABI _is_.) But back to duplicate code: different architectures even had their own versions of headers that _should_ have been mostly the same, but were full of version skew. Cleaning this up was a perennial problem, with occasional small victories: http://lists.busybox.net/pipermail/uclibc-cvs/2010-April/027938.html But mostly, wrestling with this burned out developers and sucked up huge amounts of effort while falling behind the rate of increase in clutter and scar tissue in the code (or else the code going stale and not having current features), while introducing churn and breakage into the development cycle that exacerbated the lack of releases (because if such a cleanup broke something nobody would notice for a year or more; the churn made releases dangerous to do, so the developers simply avoided having any). The worst instance of redundant code was uClibc's NPTL support, or lack thereof. In Linux 2.6 Rusty Russell invented Futexes and the "New Posix Threading Library" was invented on top of that circa 2004. http://www.drdobbs.com/open-source/nptl-the-new-implementation-of-threads-f/184406204 The first uClibc version of NPTL support was the aforementioned work by sjhill, who didn't merge it upstream. As a result, several _other_ NPTL implementations were done for other architectures, each starting from scratch and sharing zero code with any of the others: http://lists.busybox.net/pipermail/uclibc/2006-March/014793.html http://lists.busybox.net/pipermail/uclibc/2006-December/037765.html http://lists.busybox.net/pipermail/uclibc/2007-August/039206.html This combined with the earlier "don't destabilize my out of tree fork by merging anything upstream" nonsense to ensure that all the different NPTL implementations stayed as out-of-tree forks for years and years, slowly diverging, because there was no way to get them upstream. Merging any of them would inconvenience the others! No, they had to be unified _before_ they could be merged, which was a chicken and egg problem. This led to at least four complete and active NPTL ports done by four groups of people for four different architectures (arm, mips, powerpc, sh), which had no common code. Multiple summit-like meetings were held online to try to deal with this, but we still had multiple versions of the old pthreads code NPTL was supposed to replace, and couldn't even agree on unifying or removing _those_ duplicates: http://lists.busybox.net/pipermail/uclibc/2008-September/041039.html http://lists.busybox.net/pipermail/uclibc/2008-September/020198.html So the situation dragged on for years and years and the code grew more and more scar tissue as everybody's cleanup efforts focused on their own fork and not mainline. When releases _did_ happen, they were just a subset of the development, major features like NPTL were kicked down the road again and again, in various deveopment branches but not in any release version. --- _TWO_ new maintainers didn't help Circa 2007 uClibc was so clearly dying I stopped being polite about it. By this point the project was a giant mass of scar tissue from partial merges of various developers' out of tree forks leaving half-finished code all over the tree. Nobody could get major subsystems like locales to work at all, there were multiple regex implementations... Unifying that mess was a giant cleanup job that nobody had the bandwidth to do, and the longer it went on the bigger a job it became. Endless _partial_ cleanups just made the problem _worse_, the churn destabilizing the tree and making releases harder to do. Along the way Erik basically stopped participating at all. So a year or so after he handed BusyBox off to me, I poked him to hand off uClibc maintainership to somebody and just do Buildroot. (We were talking a lot behind the scenes because of the GPL enforcement suits, which seemed like a good idea at the time but wound up drawing Erik further away from day-to-day programming). But the new uClibc maintainer Erik chose was Mike Frysinger, who maintained a prominent out of tree fork for blackfin (proprietary, only customers could access it). And over the next year, Mike never put out a release. He allowed the gap since the last release to grow to its highest level ever, and then stopped posting to the uClibc list entirely to focus on his proprietary blackfin port (which was only available to customers). When it had been FOUR MONTHS since Mike last posted to the list, while everybody else was trying to get a release together (because hey, new maintainer would unblock the project, right?) So after the new maintainer didn't cut a release for a year and didn't post to the list for four motnhs straight, I got sick of him: http://lists.busybox.net/pipermail/uclibc/2008-September/019987.html And basically staged a coup and appointed a new maintainer who had the benefit of actually SHOWING UP: http://lists.busybox.net/pipermail/uclibc/2008-October/020339.html (Remember, root access to the server, and the previous maintainer answered my email.) I pitched in and started testing everything, and sending in patches to try to get a release out and get development unstuck under the new maintainer (Bernhard, who is still the current nominal maintainer of uClibc, I.E. the guy _you_ stepped in to replace): http://lists.busybox.net/pipermail/uclibc/2008-September/020033.html http://lists.busybox.net/pipermail/uclibc/2008-September/020178.html http://lists.busybox.net/pipermail/uclibc/2008-September/020182.html http://lists.busybox.net/pipermail/uclibc/2008-September/020180.html Of course bernhard's last release was in 2012, and he let things go over three years after that without a release before I basically gave up, stopped performing CPR, and called it. The uClibc project was dead. Presumably you know it from there, but you're now the FOURTH maintainer of the project _since_ it died. You came into a project I'd pushed for 10 years and started collecting trivial bugfixes without addressing any of the serious architectural issues (like needlessly duplicated per-architecture headers snapshot from ancient glibc versions, with a #define pretending to be glibc (musl doesn't) and declaring all sorts of stuff the library doesn't actually implement). And you're wondering why I have a more pessimistic outlook of your chances than you do? The big elephant in the development room was the NPTL code, and cortex-m is still using pthreads, not nptl, and WITHIN that pthreads mess is _old and _new versions of thread wait for supporting 2.2 kernels. (The menuconfig option has been replaced with _old and _new suffixes on the functions. Great.) That does not say to me that you've actually _fixed_ anything. Sure you can maintain your own version. Just like BeOS isn't really dead as long as the Haiku project persists, or osfree.org is keeping OS/2 alive, or reactos can keep windows 95 alive, the multiple open source AmigaOS clone attempts. But I'm not sure why I should CARE. You've said that your reason for supporting uclibc-ng was for two architectures, ARC and Xtensa: http://www.openwall.com/lists/musl/2015/05/30/1 That is why _you_ care. I personally think that adding support for those to musl-libc would be a far better use of your time, but it's not my job to tell you want to do. It's your hobby time. Do what you like with it. Just don't expect me to take you seriously after ten years of this. > It is really cool and very nice that the communication between Rich > and me is always fruitful. I always report any bugs I find while > testing musl on new or old platforms (mostly via IRC channel) and I > always take care that bugfixes for musl targets end up in buildroot. > (mipsr6 support, binutils microblaze problems, ..) > > I really would like to see a similar open communication between the > nommu community, you and me. > > Just ignoring uClibc-ng does not will make it die. I cc'd buildroot. That is a real project, which still uses uClibc for the targets Rich hasn't gotten around to porting musl to yet. I cc'd them so they would be aware of the issue. I could have just sent it to the musl list, but chose to inform buildroot as well. (Oddly enough, lists.buildroot.org doesn't have a web archive of the list. I have to go to lists.busybox.net to see the archive? I would have thought they'd moved it over by now. I can _email_ @buildroot...) > As a good starting point, a nice bug report and/or test application > which allows me to reproduce the problem would be really > appreciated. I spent several days trying to come up with a decent test case, it was Jonathan finding the eratta that says "this particular instruction has to get interrupted" that let me know what was actually going on (and that a reliable reproduction sequence can't be isolated, you need something like the serial port going to hammer the system with interrupts to make it happen in under an hour). That said, the board crashing under load is pretty easy to spot. Figuring out _why_ was the hard part. :) Of course this particular group of people using uClibc for a new cortex-m project were originally specced to use "uclinux". As in http://www.uclinux.org/ which had its a total disk failure take out its CVS repository a few years back, but one guy is still putting out releases every year by editing the previous year's release files, with no source control. Therefore the project still isn't dead despite the newest file in the "http download" link on the left being from 2003 and their "the developers" page having a... ahem, _slightly_younger_ picture of Jeff Dionne than https://lwn.net/Articles/647636/ (who moved to Japan in 2003 and hasn't particularly been involved with that project since then)... I pointed them at buildroot instead, and I informed buildroot of a bug using their stuff. If uclinux wants to complain I didn't cc: them on this bug either... > best regards > Waldemar Rob
Hello, On Tue, 20 Dec 2016 01:18:40 -0600, Rob Landley wrote: > I cc'd the buildroot list, which only has uClibc-based cortex-m support > at the moment. Why do you suppose I did that? > > Did you want me to send it to the uclibc.org mailing list which hasn't > had a single post this month except your announcement of your fork's > release? The list where nobody's noticed the chrome browser can't access > https://lists.uclibc.org (archives, subscription page, etc) for weeks > now? And yes, I publicized that fact when I noticed it: Do you realize that the uclibc-ng project has a mailing list? It is active, people post patches, and they get merged. I posted patches on this list, they were merged by Waldemar within the next day or two. > Your fork clearly hasn't fixed any of the structural issues uClibc > developed over the years. Waldemar has fixed the main problem of uClibc: the lack of regular releases, the lack of a responsive maintainer that merges patches. Then, from a technical point of view, Waldemar has added new features, dropped badly supported architectures, cleaned up a lot of things, improved the test suite, and more. There are probably a tons of other things to improve in uClibc-ng, but it's just a matter of receiving contributions: we can no longer blame the lack of maintainership. [... snip the rest of the text, TLDR ... ] Thomas
On 12/20/2016 02:26 AM, Thomas Petazzoni wrote: > Hello, > > On Tue, 20 Dec 2016 01:18:40 -0600, Rob Landley wrote: > >> I cc'd the buildroot list, which only has uClibc-based cortex-m support >> at the moment. Why do you suppose I did that? >> >> Did you want me to send it to the uclibc.org mailing list which hasn't >> had a single post this month except your announcement of your fork's >> release? The list where nobody's noticed the chrome browser can't access >> https://lists.uclibc.org (archives, subscription page, etc) for weeks >> now? And yes, I publicized that fact when I noticed it: > > Do you realize that the uclibc-ng project has a mailing list? No, I hadn't noticed. Good to know, I guess? >> Your fork clearly hasn't fixed any of the structural issues uClibc >> developed over the years. > > Waldemar has fixed the main problem of uClibc: the lack of regular > releases, the lack of a responsive maintainer that merges patches. Bernhard was pretty active in the years right after he became maintainer. In the long run 10 years of accumulated technical debt was too much for him, but maybe it'll be different this time. > Then, from a technical point of view, Waldemar has added new features, > dropped badly supported architectures, cleaned up a lot of things, > improved the test suite, and more. There are probably a tons of other > things to improve in uClibc-ng, but it's just a matter of receiving > contributions: we can no longer blame the lack of maintainership. Good luck to him. I have nothing against uclibc-ng the same way I have nothing against olibc. If I wasn't already regression testing against glibc, bionic, and musl, I might pay attention to this one. > [... snip the rest of the text, TLDR ... ] He asked. :) Rob
Hi Rob, First of all, thank you very much for this writeup on the history of uClibc and Buildroot. It was a really interesting read and filled in a lot of the blanks for me. Something like this should appear on lwn.net. On 20-12-16 08:18, Rob Landley wrote: [snip] > Did you want me to send it to the uclibc.org mailing list which hasn't > had a single post this month except your announcement of your fork's > release? Funny thing: that very post points to the (very active, as Thomas wrote) uclibc-ng mailing list... > Your fork clearly hasn't fixed any of the structural issues uClibc > developed over the years. It is pretty rare that a fork fixes the problems with the original project. eglibc is one of the few ones that come to mind, I expect uClibc-ng to replace uClibc completely, perhaps taking over the original name at some point. > We are discussing a patch to cortex-m that I > found because code was crashing, and when gdb was deployed we found out > that code in libpthread/libthreads/pthread.c was getting corrupted, and > we worked out that the corruption ended right after the sigsetjmp() call > in __pthread_timedsuspend_new() and that all the data before it looked > like stack contents, stack grows down on every linux target except > pa-risc so that's actually where the corruption started... and we went > from there. > > This was debugging through the _old_ legacy pthreads implementation, > which is the only option on cortex-m because even the one buildroot has > today never got NPTL working there. Much like the musl that buildroot has today doesn't have NPTL support for ARM NOMMU ;-) [snip] > I used to prod other projects to support uClibc too. For example, it > took me three years to finally convince qemu to stop being incompatible > with uClibc static binaries on powerpc, but they finally made the change: Then you have your work cut out with musl... In Buildroot, we currently have 48 packages that are excluded on musl, and 114 that need to be patched to be able to build with musl. And that's not taking into account the global hacks we introduced for musl: cdefs.h, queue.h, linux headers patching... The problem with musl is that the world is not posix :-/ Ignoring this fact moves the problems from libc to distros. [snip] > Buildroot itself was finally ready to declare uClibc dead a couple years > ago: > > http://lists.busybox.net/pipermail/buildroot/2014-February/089789.html > > Which is when you stepped in to continue beating the dead horse, so they > didn't have to decide. Note that when we finally switched to uClibc-ng as a default (in 2015), we already had musl support for more than a year. > It's nice that you're maintaining buildroot's uClibc so they don't have > to maintain their own fork anymore You seem to have things wrong here. We never maintained a buildroot uClibc, other than picking some of the uClibc commits and applying them on top of 0.9.33.2. When Waldemar's fork appeared, we started offering that fork as a choice, then after a few months we switched to uClibc-ng as the default version, and about a year later we dropped support for the original uClibc-ng. > (like emcraft still does, or > https://github.com/mickael-guene/uclibc/tree/uClibc-0.9.33.2-fdpic-m). > Your version is more interesting to me than random other attempt du jour > like https://github.com/davidgfnet/uClibc-Os because buildroot uses your > version. I think the main reason why uClibc-ng is more interesting is that it is the only fork that is actually active: it has a maintainer, a mailing list, regular releases, a bug tracker, ... . [snip] > The first problem is that buildroot forked off and sucked away all > uClibc's developers for several years. That's an interesting way of looking at it... But back in those days, Buildroot development suffered pretty much from the same problems as uClibc, no? [snip] > In the ensuing discussion other prominent developers admitted that they > valued their private forks more than mainline. Manuel Nova said, and I > quote, "I can't ethicly (at least in my code of ethics) justify handing > out bug fixes to my employer's competitors until necessary." Now that's a Quote of the Week :-) [snip] > Presumably you know it from there, but you're now the FOURTH maintainer > of the project _since_ it died. Fourth time is the charm :-) [snip] > I cc'd buildroot. That is a real project, which still uses uClibc for > the targets Rich hasn't gotten around to porting musl to yet. We do about as much uClibc maintainance as we do musl maintainance, so you should probably send all musl-related patches to the buildroot mailing list as well :-P > I cc'd > them so they would be aware of the issue. I could have just sent it to > the musl list, but chose to inform buildroot as well. > > (Oddly enough, lists.buildroot.org doesn't have a web archive of the > list. I have to go to lists.busybox.net to see the archive? I would have > thought they'd moved it over by now. I can _email_ @buildroot...) IIRC Peter looked at that but it turns out not to be so trivial to migrate a mailman setup. Regards, Arnout [snip]
On 12/20/2016 05:37 PM, Arnout Vandecappelle wrote: > Hi Rob, > > First of all, thank you very much for this writeup on the history of uClibc and > Buildroot. It was a really interesting read and filled in a lot of the blanks > for me. Something like this should appear on lwn.net. Computer history's a hobby of mine (http://landley.net/history/mirror). I occasionally try to document lived experience ala http://landley.net/aboriginal/history.html or http://landley.livejournal.com/41393.html or tell a larger story I pieced together in research like two part http://landley.net/notes-2010.html#17-07-2010 and http://landley.net/notes-2010.html#19-07-2010 . Sometimes my history is highly opinionated (https://landley.net/notes-2014.html#04-09-2014) Sometimes a random post I make goes viral ala http://www.osnews.com/story/25556/Understanding_the_bin_sbin_usr_bin_usr_sbin_Split but mostly not. If you think _this_ was a linkbomb, you haven't read the green bits and appendices of https://landley.net/writing/halloween9.html tl;dr asking me "why did X happen" is dangerous. :) > Funny thing: that very post points to the (very active, as Thomas wrote) > uclibc-ng mailing list... As I said, I hadn't clicked on the link labeled "release notes". If you don't know there's a community there, it doesn't necessarily indicate there's a community there. But good to know. >> Your fork clearly hasn't fixed any of the structural issues uClibc >> developed over the years. > > It is pretty rare that a fork fixes the problems with the original project. > eglibc is one of the few ones that come to mind, I expect uClibc-ng to replace > uClibc completely, perhaps taking over the original name at some point. Quite likely, given gcc->pcc->egcs->gcc last decade, x.org replacing xfree86, shortly after I wrote that "patch penguin" thing Alan Cox took a year off from Linux development to get an MBA so his tree _wouldn't_ displace Linus's tree... It happens. But I've got my plate full regression testing against glibc, bionic, and musl. My periodic attempts to add a fourth have mostly centered around BSD variants. https://landley.net/notes-2013.html#15-09-2013 https://landley.net/notes-2014.html#01-04-2014 https://landley.net/notes-2014.html#11-09-2014 All the new uClibc installs I've seen in the past ~2 years have been people who think "uclinux" is still synonymous with "nommu". (I've been working with Jeff Dionne to try to get http://nommu.org and such off the ground, but I'm juggling too many other balls to give this one the time it deserves.) >> This was debugging through the _old_ legacy pthreads implementation, >> which is the only option on cortex-m because even the one buildroot has >> today never got NPTL working there. > > Much like the musl that buildroot has today doesn't have NPTL support for ARM > NOMMU ;-) See thread starting at http://www.openwall.com/lists/musl/2016/12/07/2 and ongoing. I.E. I'm working on that. > [snip] >> I used to prod other projects to support uClibc too. For example, it >> took me three years to finally convince qemu to stop being incompatible >> with uClibc static binaries on powerpc, but they finally made the change: > > Then you have your work cut out with musl... In Buildroot, we currently have 48 > packages that are excluded on musl, and 114 that need to be patched to be able > to build with musl. And that's not taking into account the global hacks we > introduced for musl: cdefs.h, queue.h, linux headers patching... Musl is the default C library of ChromeOS, and one of the defaults on Docker. That gives it a largeish installed base. (And it's a "young" energetic project with the cleanest code out there, easy to add another architecture, responsive knowledgeable maintainer, etc.) Bionic is the default C library of Android. That gives it a ridiculously large installed base. Elliott Hughes (the bionic maintainer) is the second most active contributor to my toybox project, so they've added more than one feature for me already. Most recently, he's been trying to adapt the android NDK so I can build toybox a bionic toolchain I didn't have to build from soruce (see http://lists.landley.net/pipermail/toybox-landley.net/2016-December/008767.html ). Glibc is the default C library of all the other big iron "Cloud" linux distros as the PC gets kicked up into the server space like mainframes and minicomputers before it. (Boring but lucrative.) I have reasons to be interested in all three of these. > The problem with musl is that the world is not posix :-/ Ignoring this fact > moves the problems from libc to distros. The "Year of the Linux Desktop" you've heard so much about was 1998. (And yes, it lasted about a year.) Just as economies of scale shifting from desktop to laptop components gave us "blade servers", the economies of scale from a billion annual phone sales are going to switch everything over to that and price "big iron" PC components out of the market before too much longer. Faceboot already runs its servers on ARM. Google is playing its cards close to its chest but I doubt the powerpc side of http://www.eetimes.com/document.asp?doc_id=1329374 means much in the long run (ppc unit volume came from game consoles last generation, but I'm told xbone and ps4 are both 8-way AMD x86-64 chips?) Sure you can do conventional linux on arm, and big iron tends to be conservative (ala sticking with cobol as long as it could). but it also never invents anything new because those aren't the machines anybody actually sits in front of. The machines people sit in front of are phones, and tablets (big phones). There's always the issue of "what do people write code on" (see Sun's perennial attacks on Solaris-x86 eliminating its entire developer base) If android development becomes self-hosting (which I'm _working_ on, ala http://landley.net/aboriginal/about.html#selfhost and https://www.youtube.com/watch?v=SGmtP5Lg_t0 and https://lwn.net/Articles/629362/ ) then you stop cross compiling from x86 (mostly windows machines outside of Google, according to Elliott's interview at http://androidbackstage.blogspot.com/2016/07/episode-53-adb-on-adb.html) and start compiling natively on Android machines. And that means bionic. *shrug* I could be wrong. The future's always hard to predict. >> It's nice that you're maintaining buildroot's uClibc so they don't have >> to maintain their own fork anymore > > You seem to have things wrong here. We never maintained a buildroot uClibc, > other than picking some of the uClibc commits and applying them on top of > 0.9.33.2. Maybe not officially, but de-facto buildroot did have its own uClibc. Then again I had over a dozen uClibc patches at https://github.com/landley/aboriginal/tree/master/sources/patches before http://lists.landley.net/pipermail/aboriginal-landley.net/2016-May/002567.html and http://lists.landley.net/pipermail/aboriginal-landley.net/2016-November/002591.html . > When Waldemar's fork appeared, we started offering that fork as a > choice, then after a few months we switched to uClibc-ng as the default version, > and about a year later we dropped support for the original uClibc-ng. Somebody once said that in a vacuum, all something has to do is exist. (Woody Allen also said "80% of life is showing up", but the quote I can't quite remember was better.) >> (like emcraft still does, or >> https://github.com/mickael-guene/uclibc/tree/uClibc-0.9.33.2-fdpic-m). >> Your version is more interesting to me than random other attempt du jour >> like https://github.com/davidgfnet/uClibc-Os because buildroot uses your >> version. > > I think the main reason why uClibc-ng is more interesting is that it is the > only fork that is actually active: it has a maintainer, a mailing list, regular > releases, a bug tracker, ... . And buildroot uses it. > [snip] >> The first problem is that buildroot forked off and sucked away all >> uClibc's developers for several years. > > That's an interesting way of looking at it... But back in those days, Buildroot > development suffered pretty much from the same problems as uClibc, no? I didn't give the project its own list to _hurt_ buildroot. :) (I could go into more extended analysis here, but it would be more tl;dr. I'm glad Erik handed over to Thomas, it was very good for the project.) > [snip] >> Presumably you know it from there, but you're now the FOURTH maintainer >> of the project _since_ it died. > > Fourth time is the charm :-) This implies Bernhard burned down, fell over, and then sank into the swamp. Seems a bit harsh. (I have searched the internet. There is no supercut of the Swamp Castle construction speech and Sinclair explaining to Delenn why they built Babylon 5 after the first 4 were destroyed. I has a disappoint.) (Yes, I am aware of http://koti.kapsi.fi/sph/b5/montypython.html .) > [snip] >> I cc'd buildroot. That is a real project, which still uses uClibc for >> the targets Rich hasn't gotten around to porting musl to yet. > > We do about as much uClibc maintainance as we do musl maintainance, so you > should probably send all musl-related patches to the buildroot mailing list as > well :-P I build my own musl systems and toolchains. The project I was debugging used buildroot, so I sent you the fix to help them. >> I cc'd >> them so they would be aware of the issue. I could have just sent it to >> the musl list, but chose to inform buildroot as well. >> >> (Oddly enough, lists.buildroot.org doesn't have a web archive of the >> list. I have to go to lists.busybox.net to see the archive? I would have >> thought they'd moved it over by now. I can _email_ @buildroot...) > > IIRC Peter looked at that but it turns out not to be so trivial to migrate a > mailman setup. Ugh, tell me about it. People got interested in a post I made (https://utcc.utoronto.ca/~cks/space/blog/unix/GccEarlyEnthusiasm) and then dreamhost deleted a couple months of the web archive for the _third_ time and that was one of the ones that went. (And then gmane went down and _their_ copy went too. Oh well.) *shrug*. Dreamhost is cheap. Rob
Hi Rob, Rob Landley wrote, > > I am wondering why you don't cc me or any uclibc related list? > > I cc'd the buildroot list, which only has uClibc-based cortex-m support > at the moment. Why do you suppose I did that? That is not completely true, OpenADK has support for it, too. But I am not sure you declare OpenADK a real project. It seems to me that in your definition of a open source project, the people behind must either get money for their work or the project needs millions of installations. > Did you want me to send it to the uclibc.org mailing list which hasn't > had a single post this month except your announcement of your fork's > release? No, I would like to see your suggestions and patches on the uClibc-ng mailing list: http://mailman.uclibc-ng.org/cgi-bin/mailman/listinfo/devel/ > Has your fork solved the locales problem? > > http://lists.busybox.net/pipermail/uclibc/2015-June/049000.html I don't support the binary locale package. I am not very interested in locale support. > Has your fork solved the nptl issue? > > http://lists.busybox.net/pipermail/uclibc/2008-September/020151.html > http://lists.busybox.net/pipermail/uclibc/2008-September/020169.html > http://lists.busybox.net/pipermail/uclibc/2008-September/020171.html > http://lists.busybox.net/pipermail/uclibc/2008-October/041201.html Not sure. But in uClibc-ng there is Linuxthreads support for every supported architecture (excluding METAG) and NPTL for the global players. I added NPTL support from GNU libc recently, but Microblaze seems indeed bitrotted inside GNU libc as nobody is doing any test suite runs. https://sourceware.org/ml/libc-alpha/2016-07/msg00640.html There is even missing DWARF2 support in GCC upstream, so I think it is even not compile time tested. > http://lists.busybox.net/pipermail/uclibc/2006-March/014811.html > > Do you have a sane "make defconfig" that lets people build uClibc > without learning what over a hundred individual config options do and > making decisions about whether or not they need each one? This issue > doesn't even come _up_ with musl, it fundamentally avoids most of the > structural problems that strangled uClibc development, by design. I regulary cleanup needless options and keep good defaults. The number of options get lower. I still like the idea of being able to build a complete toolchain without thread support or other stuff disabled. I will try to remove more options in the future. > > You still believe uClibc projects should die? > > No, I believe uClibc _already_ died. I believe this because I was there. Yes, uClibc is dead. uClibc-ng is alive! > But cortex-m still only supports pthreads in 2016 and even that's buggy > in ways that were fixed out of tree quite a while ago. The release I > fished this bugfix out of is a year old. I don't have their source > control to see how old the fix really is, but emcraft's "preferred" > kernel (https://github.com/emcraftsystems/linux-emcraft) forked off from > mainline 7 years ago, and cortex-m support for Linux is their core > business, so there's a guess how long ago somebody actually _using_ this > might have noticed it. So, may be you can prepare a patch for the uClibc-ng list? Or do you want to see the bug open for another year? > In case you really _don't_ know the history, let me walk you through how > the uClibc project died, going back through about ten years of > accumulated scar tissue, and why three different maintainers before you > failed to fix it. Thanks for the history walkthrough. Do you know why Bernhard is so unresponsive and more about the time shortly after his last release? > You came into a project I'd pushed for 10 years and started collecting > trivial bugfixes without addressing any of the serious architectural > issues (like needlessly duplicated per-architecture headers snapshot > from ancient glibc versions, with a #define pretending to be glibc (musl > doesn't) and declaring all sorts of stuff the library doesn't actually > implement). And you're wondering why I have a more pessimistic outlook > of your chances than you do? You have to start somewhere, otherwise nothing happens. > The big elephant in the development room was the NPTL code, and cortex-m > is still using pthreads, not nptl, and WITHIN that pthreads mess is _old > and _new versions of thread wait for supporting 2.2 kernels. (The > menuconfig option has been replaced with _old and _new suffixes on the > functions. Great.) Linuxthreads.new is gone. uClibc-ng only support Linuxthreads and NPTL. > You've said that your reason for supporting uclibc-ng was for two > architectures, ARC and Xtensa: > > http://www.openwall.com/lists/musl/2015/05/30/1 > That is why _you_ care. I personally think that adding support for those > to musl-libc would be a far better use of your time, but it's not my job > to tell you want to do. It's your hobby time. Do what you like with it. > Just don't expect me to take you seriously after ten years of this. That is not entirely right. I care for every supported uClibc-ng architecture. uClibc-ng might be used for different use cases: - old non mainstream architectures like lm32, avr32, cris, fr-v, .. - noMMU systems like arm, coldfire, bfin, c6x, .. - architectures not supported by any other C library like arc, xtensa, nds32, h8300, .. - vintage unix hardware like SGI O2, Sparcstations, ... See my table which architectures are not supported by musl/glibc: http://uclibc-ng.org/wiki/matrix Indeed Max has started a musl implementaton for Xtensa: https://github.com/jcmvbkbc/musl-xtensa The Synopsys people trying to add support for GNU libc. But even then uClibc-ng is always a good choice for regression testing and testing between the different C libraries. > I cc'd buildroot. That is a real project, which still uses uClibc for > the targets Rich hasn't gotten around to porting musl to yet. I cc'd > them so they would be aware of the issue. I could have just sent it to > the musl list, but chose to inform buildroot as well. Other real projects use uClibc-ng, too. So may be next time you sent it to our list directly ;) best regards Waldemar
Hi Arnout, On Wed, Dec 21, 2016 at 12:37:47AM +0100, Arnout Vandecappelle wrote: > > I used to prod other projects to support uClibc too. For example, it > > took me three years to finally convince qemu to stop being incompatible > > with uClibc static binaries on powerpc, but they finally made the change: > > Then you have your work cut out with musl... In Buildroot, we currently have 48 > packages that are excluded on musl, and 114 that need to be patched to be able > to build with musl. And that's not taking into account the global hacks we > introduced for musl: cdefs.h, queue.h, linux headers patching... Would you suggest musl to define __GLIBC__[1] and __USE_MISC[2]? > The problem with musl is that the world is not posix :-/ Ignoring this fact > moves the problems from libc to distros. Part of the problem is that the kernel sees the world as glibc only. [1] https://git.busybox.net/buildroot/commit/?id=196932cd91ef5d58bd3142bbea939fb55636a540 [2] https://git.busybox.net/buildroot/commit/?id=4470dd9b1bdbb9b39e9fb13f27bcaa044719de6d baruch
On 12/21/2016 12:18 AM, Waldemar Brodkorb wrote: > Hi Rob, > Rob Landley wrote, I missed this the first time because gmail is crap. (I wash email through it for the spam filtering, but there's no way to get it to STOP duplicate killing, so a reply like this gets randomly sorted into the buildroot folder, the musl folder, or my inbox depending on which copy arrvied first. Makes following threads impossible, let alone archiving them. Yes I have complained to Google multiple times over the past few years, they're too big to hear me.) So I just noticed it in the musl web archive, but I think it's off topic here. I assume your list will bounce posts from non-subscribers, so lemme send my reply to the old uClibc mailing list instead. (*shrug* Still subscribed there, and you're the only one who posts there, so...) Rob
--- uclibc-1.0.17/libc/sysdeps/linux/arm/__longjmp.S +++ emcraft/libc/sysdeps/linux/arm/__longjmp.S @@ -56,9 +60,15 @@ #if defined(__thumb2__) /* Thumb-2 does not allow loading sp with ldm. */ ldmia ip!, {v1-v6, sl, fp} +#if 0 ldr sp, [ip], #4 ldr lr, [ip], #4 #else + ldr sp, [ip, #0] + ldr lr, [ip, #4] + add ip, #8 +#endif +#else ldmia ip!, {v1-v6, sl, fp, sp, lr} #endif