[OpenWrt-Devel,v3] gemini: Support sysupgrade on DIR-685

This makes sysupgrade work on the D-Link DIR-685 after
initial factory install.

We create the platform.sh script to support sysupgrade
on more targets as we move on with sysupgrade support.

Cc: Petr Štetiar <ynezz@true.cz>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
---
ChangeLog v2->v3:
- Drop the WRGG magic check: after thinking about it this
  check only MD5-sums the kernel on the sysupgrade images
  so it is not helpful for checking the stuff we actually
  write to flash, including the rootfs. So skip it on
  sysupgrade.
ChangeLog v1->v2:
- Append metadata to sysupgrade image
- Require metadata in platform.sh
- Strip comments
---
 .../gemini/base-files/lib/upgrade/platform.sh | 40 +++++++++++++++++++
 target/linux/gemini/image/Makefile            |  5 ++-
 2 files changed, 44 insertions(+), 1 deletion(-)
 create mode 100644 target/linux/gemini/base-files/lib/upgrade/platform.sh

Linus Walleij <linus.walleij@linaro.org> [2019-05-20 22:25:21]:

Hi,

I don't want to drag out more of your time so,

> +platform_find_part_size() {
> +	local first dev size erasesize name
> +	while read dev size erasesize name; do
> +		name=${name#'"'}; name=${name%'"'}
> +		[ "$name" = "$1" ] && {
> +			echo "$size"
> +			break
> +		}
> +	done < /proc/mtd
> +}

I've removed this (as we've agreed in v2)

> +platform_do_upgrade() {
> +	local board=$(board_name)
> +
> +	v "board=$board"

Fixed this leftover.

> +	case "$board" in
> +	dlink,dir-685 )
> +		PART_NAME=firmware
> +		default_do_upgrade "$ARGV"
> +		;;
> +	*)
> +
> +		;;
> +	esac

Cleaned up this.

> @@ -143,9 +144,11 @@ define Device/dlink_dir-685
> +	SUPPORTED_DEVICES += dlink,dir-685

Removed this.

>  endef
>  TARGET_DEVICES += dlink_dir-685

And merged[1] it into my staging tree, so please check it and let me know if
it's ok with you or if you prefer to go with v4, thanks.

1. https://git.openwrt.org/5f456d7acc

-- ynezz

On Mon, May 20, 2019 at 10:56 PM Petr Štetiar <ynezz@true.cz> wrote:

> And merged[1] it into my staging tree, so please check it and let me know if
> it's ok with you or if you prefer to go with v4, thanks.

Thanks man, sorry if I'm a bit confused around how you want the
sysupgrade to work.

Maybe we could open a documentation page for new devices, like
here is how we want you to implement sysupgrade, I had a hard time
to figure out that all was circuling around the platform.sh script and
how that was called from the core sysupgrade.

I can start while I still have it in fresh memory and you can chime
in on how you want people to do generic checksum etc.

Yours,
Linus Walleij

Linus Walleij <linus.walleij@linaro.org> [2019-05-20 23:48:41]:

[adding Alberto, Thomas and Javier to the Cc: loop]

> how you want the sysupgrade to work.

it's just preferred to share as much common code as possible.

> Maybe we could open a documentation page for new devices, like here is how
> we want you to implement sysupgrade, I had a hard time to figure out that
> all was circuling around the platform.sh script and how that was called from
> the core sysupgrade.

indeed, some documentation would be certainly nice and welcome, but I think,
that it's clear, that having the documentation decoupled from the code (wiki
today) leads often to outdated (incomplete, even factually wrong or misleadin)
documentation which might be even more confusing for the starters then current
"read the code Luke" approach.

I mean, maybe this is the right time to rethink[1,2], so we could have
docs/Documentation folder in the source tree, like it's done in the kernel and
other projects.

1. https://lede.readthedocs.io/en/latest/
2. https://lists.openwrt.org/pipermail/openwrt-devel/2017-November/009888.html

-- ynezz

On 21/05/19 11:13, Petr Štetiar wrote:
> Linus Walleij <linus.walleij@linaro.org> [2019-05-20 23:48:41]:
>
> [adding Alberto, Thomas and Javier to the Cc: loop]
>
>> how you want the sysupgrade to work.
> it's just preferred to share as much common code as possible.
>
>> Maybe we could open a documentation page for new devices, like here is how
>> we want you to implement sysupgrade, I had a hard time to figure out that
>> all was circuling around the platform.sh script and how that was called from
>> the core sysupgrade.
> indeed, some documentation would be certainly nice and welcome, but I think,
> that it's clear, that having the documentation decoupled from the code (wiki
> today) leads often to outdated (incomplete, even factually wrong or misleadin)
> documentation which might be even more confusing for the starters then current
> "read the code Luke" approach.
>
> I mean, maybe this is the right time to rethink[1,2], so we could have
> docs/Documentation folder in the source tree, like it's done in the kernel and
> other projects.
>
> 1. https://lede.readthedocs.io/en/latest/
> 2. https://lists.openwrt.org/pipermail/openwrt-devel/2017-November/009888.html
>
> -- ynezz


The only clear thing is that documentation does not write itself and that

you can't rely on users to write developer documentation.

Where it is kept is irrelevant, what is relevant is that people doing 
commits

(members and contributors) actually write it and keep it updated 
wherever it is kept.


I'd like to point out that it was yours truly that deleted the 
documentation

folder in the current OpenWrt tree

with commit 882f4d2d63272abce8c1966983aa10178e2e971f

as it was horribly outdated and completely useless

(and we got people rightfully complaining about it in the forums).


-Alberto.

On 20/05/19 23:48, Linus Walleij wrote:
> On Mon, May 20, 2019 at 10:56 PM Petr Štetiar <ynezz@true.cz> wrote:
>
>> And merged[1] it into my staging tree, so please check it and let me know if
>> it's ok with you or if you prefer to go with v4, thanks.
> Thanks man, sorry if I'm a bit confused around how you want the
> sysupgrade to work.
>
> Maybe we could open a documentation page for new devices, like
> here is how we want you to implement sysupgrade, I had a hard time
> to figure out that all was circuling around the platform.sh script and
> how that was called from the core sysupgrade.
>
> I can start while I still have it in fresh memory and you can chime
> in on how you want people to do generic checksum etc.
>
> Yours,
> Linus Walleij
>
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel


To avoid losing information, I'd suggest to write it down in a wiki page 
in the

developer section https://openwrt.org/docs/guide-developer/start

open this (currently empty) page and write it here

https://openwrt.org/docs/guide-developer/sysupgrade-process

You don't need to worry too much about formatting and making it look cool,

I will do that when you're done.

I'm afraid that waiting for people to decide where they want

to bikeshed the developer docs will end nowhere, just like it did in the 
past.


-Alberto

Alberto Bursi <bobafetthotmail@gmail.com> [2019-05-21 11:37:31]:

> The only clear thing is that documentation does not write itself and that
> you can't rely on users to write developer documentation.

You actually can enforce it somehow, as you either include/update
documentation to reflect your changes or your contribution won't be considered
for merge/review.

It's not a panacea, but it works relatively well in the kernel (DT docs for
example).

> with commit 882f4d2d63272abce8c1966983aa10178e2e971f
> as it was horribly outdated and completely useless

Maybe, that at that time GitHub considered Tex binary format and thus wouldn't
allow web based PRs, thus limiting potential contributions? :-)

I think, that you either have approachable documentation format(Markdown/ReST)
and tooling around, which would allow easy contributions or you end up with
the outdated documentation.

-- ynezz

On 21/05/19 12:11, Petr Štetiar wrote:
> Alberto Bursi <bobafetthotmail@gmail.com> [2019-05-21 11:37:31]:
>
>> The only clear thing is that documentation does not write itself and that
>> you can't rely on users to write developer documentation.
> You actually can enforce it somehow, as you either include/update
> documentation to reflect your changes or your contribution won't be considered
> for merge/review.
>
> It's not a panacea, but it works relatively well in the kernel (DT docs for
> example).
>

And this can't be done with the docs in a wiki because?

That's just an implementation detail.

What matters is that someone is enforcing the rule to write and update docs.

This is what isn't happening.


>> with commit 882f4d2d63272abce8c1966983aa10178e2e971f
>> as it was horribly outdated and completely useless
> Maybe, that at that time GitHub considered Tex binary format and thus wouldn't
> allow web based PRs, thus limiting potential contributions? :-)

Yeah, because a lot of people contributed to the LEDE site when it was 
source in Github.

Github web interface is trash, and it's not user-friendly even for code 
contributions.

It's completely alien for people that want to contribute docs.



> I think, that you either have approachable documentation format(Markdown/ReST)
> and tooling around, which would allow easy contributions or you end up with
> the outdated documentation.
>
> -- ynezz


The wiki is working fine for user documentation, we have people that 
reworked a lot of the docs,

like for example the firewall section was completely rewritten to update 
it,

and there is random people that add stuff every now and then.

There are a couple semi-official maintainers for VPN and other hot topics.

(and I know, I'm notified of EVERY edit ANYONE does in the wiki)


The only thing that is lacking is the developer docs, which as I said 
cannot

be written by users, no amount of additional user-friendliness will help.

It has to be written by project members or contributors that actually 
worked on the code.


Since this is a project-wide thing you need to decide it among 
yourselves, and then start writing it

or enforce the rules on contributors.


-Alberto

Alberto Bursi <bobafetthotmail@gmail.com> [2019-05-21 11:44:37]:

> On 20/05/19 23:48, Linus Walleij wrote:
> > On Mon, May 20, 2019 at 10:56 PM Petr Štetiar <ynezz@true.cz> wrote:
> > 
> > I can start while I still have it in fresh memory and you can chime
> > in on how you want people to do generic checksum etc.
>
> You don't need to worry too much about formatting and making it look cool,

It's not about the formatting, it's about the content. How could I (or possibly
someone else) chime in and improve the docs (easily)?

You mean, that Linus starts adding a new wiki content, then he sends me the
link to the wiki via email, I'm going to log into that wiki (learn just
another wiki markup syntax meanwhile the page loads in order to forget all of
this once I close that tab), then I'll send a reply to Linus (and all other
interested) over the email, that I've added some changes and so on? 

From a developer's point of view it seems like a waste of time and it will
inevitably lead to just another wiki page with the outdated content anyway.

BTW are we going to add links to the wiki into the code? So in case if someone
touches the code one day, then (s)he don't forget to update the wiki page
(sometimes more then one) with the documentation as well? Or are we going to
do the opposite, adding links to the code on the Wiki so in case I'm going to
change something, I'll first check wiki if there's any reference to that code?

Maybe I'm exaggerating, but what is faster then `git grep` ? How could you
potentionaly automate code/documentation checks if you've documentation
somewhere else?

> I'm afraid that waiting for people to decide where they want to bikeshed the
> developer docs will end nowhere, just like it did in the past.

Well, the difference is, that I'm willing to help make it happen this time, if
there's still some interest around this topic. It simply makes sense to me.

Actually I'm not saying, that I'm going to prepare patches which would add
this feature to the tree, but I'm willing help with testing and giving it my
+1 in a potential vote.

I've abused this thread in order to spark exactly this discussion and to see
if there's some interest in potentialy moving forward with this. It will serve
also as a small reminder to me, that this might be next topic I should bring
to to the discussion on our meeting in June.

Don't take me wrong, you're all doing amazing work on the wiki, I'm sorry that
I'm not contributing there as well, but Wiki simply doesn't fits well into my
mutt/vim/git daily workflow.

In the end, we could simply just delete that docs folder again in 2022 if we
find out, that it didn't worked as expected :-)

-- ynezz

On 21/05/19 13:10, Petr Štetiar wrote:
> Alberto Bursi <bobafetthotmail@gmail.com> [2019-05-21 11:44:37]:
>
>> On 20/05/19 23:48, Linus Walleij wrote:
>>> On Mon, May 20, 2019 at 10:56 PM Petr Štetiar <ynezz@true.cz> wrote:
>>>
>>> I can start while I still have it in fresh memory and you can chime
>>> in on how you want people to do generic checksum etc.
>> You don't need to worry too much about formatting and making it look cool,
> It's not about the formatting, it's about the content. How could I (or possibly
> someone else) chime in and improve the docs (easily)?


As I said in the mail, he should write it in the wiki as a stopgap measure,

it's better to have it in the wiki with the limitations we all know,

instead of waiting for a perfect solution that may

or may not happen later.


As for the permanent place for development/technical documentation,

it's up to you members and core contributors to decide

where and how to write it down.

The only thing I care about is that someone must be enforcing

to write technical/development docs somewhere.


The only thing that we MUST use a wiki for is for tutorials

(installation and configuration), device data, and other

user-contributed documentation, as users don't

know nor need to learn git or Github just to

write down a tutorial and upload a few

screenshots of a board.


> BTW are we going to add links to the wiki into the code?

Would you add paths to the internal docs in the code?


I personally don't see why you want it. Finding the document about

a specific OpenWrt component isn't terribly hard even without a direct link.


But still, I'm neutral on this too. You choose what you think is best.

I'm not a dev, I don't know what is most suited for your workflow.


>> I'm afraid that waiting for people to decide where they want to bikeshed the
>> developer docs will end nowhere, just like it did in the past.
> Well, the difference is, that I'm willing to help make it happen this time, if
> there's still some interest around this topic. It simply makes sense to me.
>
> Actually I'm not saying, that I'm going to prepare patches which would add
> this feature to the tree, but I'm willing help with testing and giving it my
> +1 in a potential vote.


This is why I'm afraid this isn't going to end anywhere.

Actually writing documentation requires time and effort,

and I fully understand that developers don't like to be using

their time to write docs. Especially so in their free time.


Just ACKing and reviewing isn't enough to make this happen.


Maybe have the leg work of this documentation work done by an intern

in a Google Summer of Code or something.


>
> I've abused this thread in order to spark exactly this discussion and to see
> if there's some interest in potentialy moving forward with this. It will serve
> also as a small reminder to me, that this might be next topic I should bring
> to to the discussion on our meeting in June.


Please talk with the other members about this point, as it's important

and not addressed properly by the current wiki setup.


As I also said above, I'm available to rework or add formatting or do other

editorial work to make it look good and have a coherent structure,

wherever you decide to place it.

You just need someone to write correct raw information.


-Alberto