| Message ID | 96e98b7d-ee04-4581-a55c-a4a99dd3c8af@googlegroups.com |
|---|---|
| State | Rejected |
| Headers | show |
| Series | Update fails when ubifs is compressed - fixed and patch attached | expand |
Hi Matt, On 10/05/2018 18:34, mattmatician@outlook.com wrote: > I found when trying to use an ubifs.gz file and adding compressed = true; to the sw-description resulted in a failure to perform the swupdate. > This is a known issue, already raised by seeting encrypted = true, see commit 72c2a0c5eeac382916bb26b274d64b28bd1d6be0. However, there is not an easy solution because the MTD library requires to know the size of the volume before starting. > I found this was a problem with the following lines in > - handlers/ubivol_handler.c > > ------------ > > err = ubi_update_start(libubi, fdout, bytes); > if (err) { > ERROR("cannot start volume \"%s\" update", node); > return -1; > } > Yes. ub_update_start() requires the toral numbér of bytes before starting to write. > --------- > > The ubi_update_start command 'enables' writing to the number of bytes specified by the variable bytes. > > However, the variable 'bytes' contains the size of the compressed ubifs.gz file, and not the size of the decompressed ubifs file, as required. > True. > By obtaining the decompressed filesize from the gzipped file using this logic: > > https://stackoverflow.com/questions/9715046/find-the-size-of-the-file-inside-a-gzip-file/9727599#9727599 > > I have created a patch that solves this issue, but note (as described in the link above) this limits the original ubifs file size to 4 GiB. > > Patch is below: > ######################################### > > diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c > index 0c6fcbf..6a88487 100644 > --- a/handlers/ubivol_handler.c > +++ b/handlers/ubivol_handler.c > @@ -42,6 +42,7 @@ static int update_volume(libubi_t libubi, struct img_type *img, > char node[64]; > int err; > char sbuf[128]; > + uint32_t decompressed_bytes; > > bytes = img->size; > if (img->is_encrypted) { > @@ -91,7 +92,17 @@ static int update_volume(libubi_t libubi, struct img_type *img, > ERROR("cannot open UBI volume \"%s\"", node); > return -1; > } > - err = ubi_update_start(libubi, fdout, bytes); > + > + if (img->compressed) { > + lseek(img->fdin, img->size-4, SEEK_CUR); > + read(img->fdin, &decompressed_bytes, 4); > + lseek(img->fdin, -img->size, SEEK_CUR); NAK. You forget that SWUpdate works in most cases with streams, not with files. You cannot seek in a stream - this works just in case of a local file. Try to update from network with install-directly activated, this does not work. > + } > + else { > + decompressed_bytes = bytes; > + } > + > + err = ubi_update_start(libubi, fdout, decompressed_bytes); > if (err) { > ERROR("cannot start volume \"%s\" update", node); > return -1; > Best regards, Stefano Babic
On Thursday, 10 May 2018 17:34:43 UTC+1, mattma...@outlook.com wrote: > I found when trying to use an ubifs.gz file and adding compressed = true; to the sw-description resulted in a failure to perform the swupdate. > > I found this was a problem with the following lines in > - handlers/ubivol_handler.c > > ------------ > > err = ubi_update_start(libubi, fdout, bytes); > if (err) { > ERROR("cannot start volume \"%s\" update", node); > return -1; > } > > --------- > > The ubi_update_start command 'enables' writing to the number of bytes specified by the variable bytes. > > However, the variable 'bytes' contains the size of the compressed ubifs.gz file, and not the size of the decompressed ubifs file, as required. > > By obtaining the decompressed filesize from the gzipped file using this logic: > > https://stackoverflow.com/questions/9715046/find-the-size-of-the-file-inside-a-gzip-file/9727599#9727599 > > I have created a patch that solves this issue, but note (as described in the link above) this limits the original ubifs file size to 4 GiB. > > Patch is below: > ######################################### > > diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c > index 0c6fcbf..6a88487 100644 > --- a/handlers/ubivol_handler.c > +++ b/handlers/ubivol_handler.c > @@ -42,6 +42,7 @@ static int update_volume(libubi_t libubi, struct img_type *img, > char node[64]; > int err; > char sbuf[128]; > + uint32_t decompressed_bytes; > > bytes = img->size; > if (img->is_encrypted) { > @@ -91,7 +92,17 @@ static int update_volume(libubi_t libubi, struct img_type *img, > ERROR("cannot open UBI volume \"%s\"", node); > return -1; > } > - err = ubi_update_start(libubi, fdout, bytes); > + > + if (img->compressed) { > + lseek(img->fdin, img->size-4, SEEK_CUR); > + read(img->fdin, &decompressed_bytes, 4); > + lseek(img->fdin, -img->size, SEEK_CUR); > + } > + else { > + decompressed_bytes = bytes; > + } > + > + err = ubi_update_start(libubi, fdout, decompressed_bytes); > if (err) { > ERROR("cannot start volume \"%s\" update", node); > return -1; Ah okay, silly me. Originally, I had passed the decompressed size to the handler with the "data" attribute. This would work with stream installed-directly aswell, no...? How about adding a new attribute to the ubivol type in sw_description. A new 'decompressed_size' field could become mandatory if compressed is true...?
Hi Matt, On 10/05/2018 22:16, mattmatician@outlook.com wrote: > On Thursday, 10 May 2018 17:34:43 UTC+1, mattma...@outlook.com wrote: >> I found when trying to use an ubifs.gz file and adding compressed = true; to the sw-description resulted in a failure to perform the swupdate. >> >> I found this was a problem with the following lines in >> - handlers/ubivol_handler.c >> >> ------------ >> >> err = ubi_update_start(libubi, fdout, bytes); >> if (err) { >> ERROR("cannot start volume \"%s\" update", node); >> return -1; >> } >> >> --------- >> >> The ubi_update_start command 'enables' writing to the number of bytes specified by the variable bytes. >> >> However, the variable 'bytes' contains the size of the compressed ubifs.gz file, and not the size of the decompressed ubifs file, as required. >> >> By obtaining the decompressed filesize from the gzipped file using this logic: >> >> https://stackoverflow.com/questions/9715046/find-the-size-of-the-file-inside-a-gzip-file/9727599#9727599 >> >> I have created a patch that solves this issue, but note (as described in the link above) this limits the original ubifs file size to 4 GiB. >> >> Patch is below: >> ######################################### >> >> diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c >> index 0c6fcbf..6a88487 100644 >> --- a/handlers/ubivol_handler.c >> +++ b/handlers/ubivol_handler.c >> @@ -42,6 +42,7 @@ static int update_volume(libubi_t libubi, struct img_type *img, >> char node[64]; >> int err; >> char sbuf[128]; >> + uint32_t decompressed_bytes; >> >> bytes = img->size; >> if (img->is_encrypted) { >> @@ -91,7 +92,17 @@ static int update_volume(libubi_t libubi, struct img_type *img, >> ERROR("cannot open UBI volume \"%s\"", node); >> return -1; >> } >> - err = ubi_update_start(libubi, fdout, bytes); >> + >> + if (img->compressed) { >> + lseek(img->fdin, img->size-4, SEEK_CUR); >> + read(img->fdin, &decompressed_bytes, 4); >> + lseek(img->fdin, -img->size, SEEK_CUR); >> + } >> + else { >> + decompressed_bytes = bytes; >> + } >> + >> + err = ubi_update_start(libubi, fdout, decompressed_bytes); >> if (err) { >> ERROR("cannot start volume \"%s\" update", node); >> return -1; > > Ah okay, silly me. > > Originally, I had passed the decompressed size to the handler with the "data" attribute. This would work with stream installed-directly aswell, no...? Yes, this works. > > How about adding a new attribute to the ubivol type in sw_description. A new 'decompressed_size' field could become mandatory if compressed is true...? We have now "properties" for such cases - see as reference the swuforwarder handler. Anyway, I am ask myself why the UBI volume should be compressed when UBI runs the decompression on-the-fly. IMHO it is much more efficient to not compress the volume, but deliver a already compressed UBIFS and the target decompress it on demand. Feature is in mainline kernel since a very long time. Best regards, Stefano Babic
On Thursday, May 10, 2018 at 9:40:00 PM UTC+1, Stefano Babic wrote: > Hi Matt, > > On 10/05/2018 22:16, matt wrote: > > On Thursday, 10 May 2018 17:34:43 UTC+1, mattma...@outlook.com wrote: > >> I found when trying to use an ubifs.gz file and adding compressed = true; to the sw-description resulted in a failure to perform the swupdate. > >> > >> I found this was a problem with the following lines in > >> - handlers/ubivol_handler.c > >> > >> ------------ > >> > >> err = ubi_update_start(libubi, fdout, bytes); > >> if (err) { > >> ERROR("cannot start volume \"%s\" update", node); > >> return -1; > >> } > >> > >> --------- > >> > >> The ubi_update_start command 'enables' writing to the number of bytes specified by the variable bytes. > >> > >> However, the variable 'bytes' contains the size of the compressed ubifs.gz file, and not the size of the decompressed ubifs file, as required. > >> > >> By obtaining the decompressed filesize from the gzipped file using this logic: > >> > >> https://stackoverflow.com/questions/9715046/find-the-size-of-the-file-inside-a-gzip-file/9727599#9727599 > >> > >> I have created a patch that solves this issue, but note (as described in the link above) this limits the original ubifs file size to 4 GiB. > >> > >> Patch is below: > >> ######################################### > >> > >> diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c > >> index 0c6fcbf..6a88487 100644 > >> --- a/handlers/ubivol_handler.c > >> +++ b/handlers/ubivol_handler.c > >> @@ -42,6 +42,7 @@ static int update_volume(libubi_t libubi, struct img_type *img, > >> char node[64]; > >> int err; > >> char sbuf[128]; > >> + uint32_t decompressed_bytes; > >> > >> bytes = img->size; > >> if (img->is_encrypted) { > >> @@ -91,7 +92,17 @@ static int update_volume(libubi_t libubi, struct img_type *img, > >> ERROR("cannot open UBI volume \"%s\"", node); > >> return -1; > >> } > >> - err = ubi_update_start(libubi, fdout, bytes); > >> + > >> + if (img->compressed) { > >> + lseek(img->fdin, img->size-4, SEEK_CUR); > >> + read(img->fdin, &decompressed_bytes, 4); > >> + lseek(img->fdin, -img->size, SEEK_CUR); > >> + } > >> + else { > >> + decompressed_bytes = bytes; > >> + } > >> + > >> + err = ubi_update_start(libubi, fdout, decompressed_bytes); > >> if (err) { > >> ERROR("cannot start volume \"%s\" update", node); > >> return -1; > > > > Ah okay, silly me. > > > > Originally, I had passed the decompressed size to the handler with the "data" attribute. This would work with stream installed-directly aswell, no...? > > Yes, this works. > > > > > How about adding a new attribute to the ubivol type in sw_description. A new 'decompressed_size' field could become mandatory if compressed is true...? > > We have now "properties" for such cases - see as reference the > swuforwarder handler. > > Anyway, I am ask myself why the UBI volume should be compressed when UBI > runs the decompression on-the-fly. IMHO it is much more efficient to not > compress the volume, but deliver a already compressed UBIFS and the > target decompress it on demand. Feature is in mainline kernel since a > very long time. > > Best regards, > Stefano Babic > > > -- > ===================================================================== > DENX Software Engineering GmbH, Managing Director: Wolfgang Denk > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany > Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de > ===================================================================== Hi Stefano, Thanks, I will explore the "properties". And yes, I have seen you suggest using runtime compression in the ubifs in previous compression questions. Firstly, my target hardware may have low bandwidth but will have high storage. I don't want to inhibit my performance at runtime to decrease the swu size. Secondly, I generated differentl compressed ubifs to gather the following sizes: Uncompressed ubifs: 73MB -> Gzipped 27MB Zlibbed ubifs: 39MB Notably, the Zlibbed ubifs is 39MB vs the Uncompressed, and then gzipped 27MB. Increasing my ubifs my ~50% is not ideal, and has the added problem of slowing my read/write times. Regards, Matt
Hi Matt, On 11/05/2018 10:45, mattbeaumontgts@gmail.com wrote: > On Thursday, May 10, 2018 at 9:40:00 PM UTC+1, Stefano Babic wrote: >> Hi Matt, >> >> On 10/05/2018 22:16, matt wrote: >>> On Thursday, 10 May 2018 17:34:43 UTC+1, mattma...@outlook.com wrote: >>>> I found when trying to use an ubifs.gz file and adding compressed = true; to the sw-description resulted in a failure to perform the swupdate. >>>> >>>> I found this was a problem with the following lines in >>>> - handlers/ubivol_handler.c >>>> >>>> ------------ >>>> >>>> err = ubi_update_start(libubi, fdout, bytes); >>>> if (err) { >>>> ERROR("cannot start volume \"%s\" update", node); >>>> return -1; >>>> } >>>> >>>> --------- >>>> >>>> The ubi_update_start command 'enables' writing to the number of bytes specified by the variable bytes. >>>> >>>> However, the variable 'bytes' contains the size of the compressed ubifs.gz file, and not the size of the decompressed ubifs file, as required. >>>> >>>> By obtaining the decompressed filesize from the gzipped file using this logic: >>>> >>>> https://stackoverflow.com/questions/9715046/find-the-size-of-the-file-inside-a-gzip-file/9727599#9727599 >>>> >>>> I have created a patch that solves this issue, but note (as described in the link above) this limits the original ubifs file size to 4 GiB. >>>> >>>> Patch is below: >>>> ######################################### >>>> >>>> diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c >>>> index 0c6fcbf..6a88487 100644 >>>> --- a/handlers/ubivol_handler.c >>>> +++ b/handlers/ubivol_handler.c >>>> @@ -42,6 +42,7 @@ static int update_volume(libubi_t libubi, struct img_type *img, >>>> char node[64]; >>>> int err; >>>> char sbuf[128]; >>>> + uint32_t decompressed_bytes; >>>> >>>> bytes = img->size; >>>> if (img->is_encrypted) { >>>> @@ -91,7 +92,17 @@ static int update_volume(libubi_t libubi, struct img_type *img, >>>> ERROR("cannot open UBI volume \"%s\"", node); >>>> return -1; >>>> } >>>> - err = ubi_update_start(libubi, fdout, bytes); >>>> + >>>> + if (img->compressed) { >>>> + lseek(img->fdin, img->size-4, SEEK_CUR); >>>> + read(img->fdin, &decompressed_bytes, 4); >>>> + lseek(img->fdin, -img->size, SEEK_CUR); >>>> + } >>>> + else { >>>> + decompressed_bytes = bytes; >>>> + } >>>> + >>>> + err = ubi_update_start(libubi, fdout, decompressed_bytes); >>>> if (err) { >>>> ERROR("cannot start volume \"%s\" update", node); >>>> return -1; >>> >>> Ah okay, silly me. >>> >>> Originally, I had passed the decompressed size to the handler with the "data" attribute. This would work with stream installed-directly aswell, no...? >> >> Yes, this works. >> >>> >>> How about adding a new attribute to the ubivol type in sw_description. A new 'decompressed_size' field could become mandatory if compressed is true...? >> >> We have now "properties" for such cases - see as reference the >> swuforwarder handler. >> >> Anyway, I am ask myself why the UBI volume should be compressed when UBI >> runs the decompression on-the-fly. IMHO it is much more efficient to not >> compress the volume, but deliver a already compressed UBIFS and the >> target decompress it on demand. Feature is in mainline kernel since a >> very long time. >> >> Best regards, >> Stefano Babic >> >> >> -- >> ===================================================================== >> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk >> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany >> Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de >> ===================================================================== > > Hi Stefano, > > Thanks, I will explore the "properties". > > And yes, I have seen you suggest using runtime compression in the ubifs in previous compression questions. > > Firstly, my target hardware may have low bandwidth but will have high storage. I don't want to inhibit my performance at runtime to decrease the swu size. ok > > Secondly, I generated differentl compressed ubifs to gather the following sizes: > > Uncompressed ubifs: 73MB -> Gzipped 27MB > Zlibbed ubifs: 39MB > > Notably, the Zlibbed ubifs is 39MB vs the Uncompressed, and then gzipped 27MB. By the way, UBI supports LZO, too, and the compression rate is higher as with zlib. > > Increasing my ubifs my ~50% is not ideal, and has the added problem of slowing my read/write times. Ok - regarding the decompression problem, we are building a work-around, but IMHO the correct solution should be done in kernel. Currently, drivers/mtd/ubi/upd.c checks for the receivd bytes to set automatically the update marker, see: 362 ubi_assert(vol->upd_received <= vol->upd_bytes); 363 if (vol->upd_received == vol->upd_bytes) { 364 err = ubi_wl_flush(ubi, UBI_ALL, UBI_ALL); 365 if (err) 366 return err; 367 /* The update is finished, clear the update marker */ 368 err = clear_update_marker(ubi, vol, vol->upd_bytes); However, this forbids streaming if we do not know, as for compressed images, the whole size of the volume. I guess this question could be post to MTD ML, asking if the en-of-file can be determined in a different was, for example with an (rather new) ioctl. Best regards, Stefano Babic
Hi Stefano,
Thanks for your time Stefano, I'm new to this.
I've written some code to take the decompressed size from the properties dict, I've added the field to sw-description respectively.
However, I suspect I will take your advise and use runtime-compression only.
Also, does the following link not suggest that Zlib compresses better then Lzo?
http://www.linux-mtd.infradead.org/misc.html
###############################
diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c
index 0c6fcbf..d906f1f 100644
--- a/handlers/ubivol_handler.c
+++ b/handlers/ubivol_handler.c
@@ -37,7 +37,7 @@ static struct ubi_part *search_volume(const char *str, struct ubilist *list)
static int update_volume(libubi_t libubi, struct img_type *img,
struct ubi_vol_info *vol)
{
- long long bytes;
+ long long bytes, decompressed_bytes;
int fdout;
char node[64];
int err;
@@ -91,7 +91,19 @@ static int update_volume(libubi_t libubi, struct img_type *img,
ERROR("cannot open UBI volume \"%s\"", node);
return -1;
}
- err = ubi_update_start(libubi, fdout, bytes);
+
+ if (img->compressed) {
+ decompressed_bytes = strtoll(dict_get_value(&img->properties, "decompressed_size"), NULL, 10);
+ if (!decompressed_bytes) {
+ ERROR("UBIFS to be decompressed, but decompressed_size not found");
+ return -EINVAL;
+ }
+ }
+ else {
+ decompressed_bytes = bytes;
+ }
+
+ err = ubi_update_start(libubi, fdout, decompressed_bytes);
if (err) {
ERROR("cannot start volume \"%s\" update", node);
return -1;
######################################
Thanks,
Matt
On Friday, May 11, 2018 at 10:39:38 AM UTC+1, Stefano Babic wrote:
> Hi Matt,
>
> On 11/05/2018 10:45, mattbeaumontgts@gmail.com wrote:
> > On Thursday, May 10, 2018 at 9:40:00 PM UTC+1, Stefano Babic wrote:
> >> Hi Matt,
> >>
> >> On 10/05/2018 22:16, matt wrote:
> >>> On Thursday, 10 May 2018 17:34:43 UTC+1, mattma...@outlook.com wrote:
> >>>> I found when trying to use an ubifs.gz file and adding compressed = true; to the sw-description resulted in a failure to perform the swupdate.
> >>>>
> >>>> I found this was a problem with the following lines in
> >>>> - handlers/ubivol_handler.c
> >>>>
> >>>> ------------
> >>>>
> >>>> err = ubi_update_start(libubi, fdout, bytes);
> >>>> if (err) {
> >>>> ERROR("cannot start volume \"%s\" update", node);
> >>>> return -1;
> >>>> }
> >>>>
> >>>> ---------
> >>>>
> >>>> The ubi_update_start command 'enables' writing to the number of bytes specified by the variable bytes.
> >>>>
> >>>> However, the variable 'bytes' contains the size of the compressed ubifs.gz file, and not the size of the decompressed ubifs file, as required.
> >>>>
> >>>> By obtaining the decompressed filesize from the gzipped file using this logic:
> >>>>
> >>>> https://stackoverflow.com/questions/9715046/find-the-size-of-the-file-inside-a-gzip-file/9727599#9727599
> >>>>
> >>>> I have created a patch that solves this issue, but note (as described in the link above) this limits the original ubifs file size to 4 GiB.
> >>>>
> >>>> Patch is below:
> >>>> #########################################
> >>>>
> >>>> diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c
> >>>> index 0c6fcbf..6a88487 100644
> >>>> --- a/handlers/ubivol_handler.c
> >>>> +++ b/handlers/ubivol_handler.c
> >>>> @@ -42,6 +42,7 @@ static int update_volume(libubi_t libubi, struct img_type *img,
> >>>> char node[64];
> >>>> int err;
> >>>> char sbuf[128];
> >>>> + uint32_t decompressed_bytes;
> >>>>
> >>>> bytes = img->size;
> >>>> if (img->is_encrypted) {
> >>>> @@ -91,7 +92,17 @@ static int update_volume(libubi_t libubi, struct img_type *img,
> >>>> ERROR("cannot open UBI volume \"%s\"", node);
> >>>> return -1;
> >>>> }
> >>>> - err = ubi_update_start(libubi, fdout, bytes);
> >>>> +
> >>>> + if (img->compressed) {
> >>>> + lseek(img->fdin, img->size-4, SEEK_CUR);
> >>>> + read(img->fdin, &decompressed_bytes, 4);
> >>>> + lseek(img->fdin, -img->size, SEEK_CUR);
> >>>> + }
> >>>> + else {
> >>>> + decompressed_bytes = bytes;
> >>>> + }
> >>>> +
> >>>> + err = ubi_update_start(libubi, fdout, decompressed_bytes);
> >>>> if (err) {
> >>>> ERROR("cannot start volume \"%s\" update", node);
> >>>> return -1;
> >>>
> >>> Ah okay, silly me.
> >>>
> >>> Originally, I had passed the decompressed size to the handler with the "data" attribute. This would work with stream installed-directly aswell, no...?
> >>
> >> Yes, this works.
> >>
> >>>
> >>> How about adding a new attribute to the ubivol type in sw_description. A new 'decompressed_size' field could become mandatory if compressed is true...?
> >>
> >> We have now "properties" for such cases - see as reference the
> >> swuforwarder handler.
> >>
> >> Anyway, I am ask myself why the UBI volume should be compressed when UBI
> >> runs the decompression on-the-fly. IMHO it is much more efficient to not
> >> compress the volume, but deliver a already compressed UBIFS and the
> >> target decompress it on demand. Feature is in mainline kernel since a
> >> very long time.
> >>
> >> Best regards,
> >> Stefano Babic
> >>
> >>
> >> --
> >> =====================================================================
> >> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> >> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> >> Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
> >> =====================================================================
> >
> > Hi Stefano,
> >
> > Thanks, I will explore the "properties".
> >
> > And yes, I have seen you suggest using runtime compression in the ubifs in previous compression questions.
> >
> > Firstly, my target hardware may have low bandwidth but will have high storage. I don't want to inhibit my performance at runtime to decrease the swu size.
>
> ok
>
> >
> > Secondly, I generated differentl compressed ubifs to gather the following sizes:
> >
> > Uncompressed ubifs: 73MB -> Gzipped 27MB
> > Zlibbed ubifs: 39MB
> >
> > Notably, the Zlibbed ubifs is 39MB vs the Uncompressed, and then gzipped 27MB.
>
> By the way, UBI supports LZO, too, and the compression rate is higher as
> with zlib.
>
> >
> > Increasing my ubifs my ~50% is not ideal, and has the added problem of slowing my read/write times.
>
> Ok - regarding the decompression problem, we are building a work-around,
> but IMHO the correct solution should be done in kernel. Currently,
> drivers/mtd/ubi/upd.c checks for the receivd bytes to set automatically
> the update marker, see:
>
> 362 ubi_assert(vol->upd_received <= vol->upd_bytes);
> 363 if (vol->upd_received == vol->upd_bytes) {
> 364 err = ubi_wl_flush(ubi, UBI_ALL, UBI_ALL);
> 365 if (err)
> 366 return err;
> 367 /* The update is finished, clear the update marker */
> 368 err = clear_update_marker(ubi, vol, vol->upd_bytes);
>
>
> However, this forbids streaming if we do not know, as for compressed
> images, the whole size of the volume. I guess this question could be
> post to MTD ML, asking if the en-of-file can be determined in a
> different was, for example with an (rather new) ioctl.
>
> Best regards,
> Stefano Babic
>
>
> --
> =====================================================================
> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
> =====================================================================
Hi Matt, On 11/05/2018 11:55, mattbeaumontgts@gmail.com wrote: > Hi Stefano, > > Thanks for your time Stefano, I'm new to this. > > I've written some code to take the decompressed size from the properties dict, I've added the field to sw-description respectively. > > However, I suspect I will take your advise and use runtime-compression only. > > Also, does the following link not suggest that Zlib compresses better then Lzo? > http://www.linux-mtd.infradead.org/misc.html > My fault, yes. LZO is faster, but zlib has better compression. > ############################### > > diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c > index 0c6fcbf..d906f1f 100644 > --- a/handlers/ubivol_handler.c > +++ b/handlers/ubivol_handler.c > @@ -37,7 +37,7 @@ static struct ubi_part *search_volume(const char *str, struct ubilist *list) > static int update_volume(libubi_t libubi, struct img_type *img, > struct ubi_vol_info *vol) > { > - long long bytes; > + long long bytes, decompressed_bytes; > int fdout; > char node[64]; > int err; > @@ -91,7 +91,19 @@ static int update_volume(libubi_t libubi, struct img_type *img, > ERROR("cannot open UBI volume \"%s\"", node); > return -1; > } > - err = ubi_update_start(libubi, fdout, bytes); > + > + if (img->compressed) { > + decompressed_bytes = strtoll(dict_get_value(&img->properties, "decompressed_size"), NULL, 10); > + if (!decompressed_bytes) { > + ERROR("UBIFS to be decompressed, but decompressed_size not found"); > + return -EINVAL; > + } > + } > + else { > + decompressed_bytes = bytes; > + } > + > + err = ubi_update_start(libubi, fdout, decompressed_bytes); > if (err) { > ERROR("cannot start volume \"%s\" update", node); > return -1; > > ###################################### > > Thanks, > Best regards, Stefano Babic
diff --git a/handlers/ubivol_handler.c b/handlers/ubivol_handler.c index 0c6fcbf..6a88487 100644 --- a/handlers/ubivol_handler.c +++ b/handlers/ubivol_handler.c @@ -42,6 +42,7 @@ static int update_volume(libubi_t libubi, struct img_type *img, char node[64]; int err; char sbuf[128]; + uint32_t decompressed_bytes; bytes = img->size; if (img->is_encrypted) { @@ -91,7 +92,17 @@ static int update_volume(libubi_t libubi, struct img_type *img, ERROR("cannot open UBI volume \"%s\"", node); return -1; } - err = ubi_update_start(libubi, fdout, bytes); + + if (img->compressed) { + lseek(img->fdin, img->size-4, SEEK_CUR); + read(img->fdin, &decompressed_bytes, 4); + lseek(img->fdin, -img->size, SEEK_CUR); + } + else { + decompressed_bytes = bytes; + } + + err = ubi_update_start(libubi, fdout, decompressed_bytes); if (err) { ERROR("cannot start volume \"%s\" update", node); return -1;
I found when trying to use an ubifs.gz file and adding compressed = true; to the sw-description resulted in a failure to perform the swupdate. I found this was a problem with the following lines in - handlers/ubivol_handler.c ------------ err = ubi_update_start(libubi, fdout, bytes); if (err) { ERROR("cannot start volume \"%s\" update", node); return -1; } --------- The ubi_update_start command 'enables' writing to the number of bytes specified by the variable bytes. However, the variable 'bytes' contains the size of the compressed ubifs.gz file, and not the size of the decompressed ubifs file, as required. By obtaining the decompressed filesize from the gzipped file using this logic: https://stackoverflow.com/questions/9715046/find-the-size-of-the-file-inside-a-gzip-file/9727599#9727599 I have created a patch that solves this issue, but note (as described in the link above) this limits the original ubifs file size to 4 GiB. Patch is below: #########################################