Message ID | 53BA4B98.1080000@huawei.com |
---|---|
State | RFC |
Headers | show |
On Mon, 2014-07-07 at 15:26 +0800, hujianyang wrote: > +"-h, --help print help message\n" > +"-n, --lnum logic eraseblock num to dump\n" I envision that in the future people may want to specify PEB number instead of LEB number, and '-n' becomes a confusing name for the option. Could you please only support '--lnum', and drop '-n' altogether. Then in the future someone may add '--pnum' support. > +"-i, --info show explicit information about NODEs\n" I do not understand what this means from the description. Would you please try to describe it better in the help text? > +"-H, --header show only header information\n" > +"-N, --node show only NODEs information\n" Same for these. Thanks!
> I envision that in the future people may want to specify PEB number > instead of LEB number, and '-n' becomes a confusing name for the option. > Could you please only support '--lnum', and drop '-n' altogether. Then > in the future someone may add '--pnum' support. Yes, you are right. I'll change it in next version. > >> +"-i, --info show explicit information about NODEs\n" > > I do not understand what this means from the description. Would you > please try to describe it better in the help text? > >> +"-H, --header show only header information\n" >> +"-N, --node show only NODEs information\n" > > Same for these. > > Thanks! > -H for UBI-level info and -N for UBIFS-level info. -i means dump UBIFS Nodes by function ubifs_dump_node, not just scan them. I will re-describe them. Thanks.
On Wed, 2014-07-16 at 16:53 +0800, hujianyang wrote: > > I envision that in the future people may want to specify PEB number > > instead of LEB number, and '-n' becomes a confusing name for the option. > > Could you please only support '--lnum', and drop '-n' altogether. Then > > in the future someone may add '--pnum' support. > > Yes, you are right. I'll change it in next version. > > > > >> +"-i, --info show explicit information about NODEs\n" > > > > I do not understand what this means from the description. Would you > > please try to describe it better in the help text? > > > >> +"-H, --header show only header information\n" > >> +"-N, --node show only NODEs information\n" > > > > Same for these. > > > > Thanks! > > > > -H for UBI-level info and -N for UBIFS-level info. > -i means dump UBIFS Nodes by function ubifs_dump_node, not just > scan them. > > I will re-describe them. Things like UBI volume table, UBI fast-map stuff are not "headers", so I am not sure if using word "headers" a good idea. Probably we want options like --ubifs and --ubi to denote ubi and ubifs-level stuff. The default would be "everything". But again, if you start smaller, and upstream a good tool for UBIFS-level stuff, it will be easier to add UBI stuff separately. Besides, I have some additional vision, which you do not have to implement, but which should be taken into account. E.g., ubidump which does not need UBI/UBIFS drivers, ubidump which can deal with an image generated with nanddump without "mounting" it, etc. So I was thinking doing small steps at a time would make it easier for me and for you to make a tool which has limited functionality today, but which can be later extended to support more functionality.
> > Things like UBI volume table, UBI fast-map stuff are not "headers", so I > am not sure if using word "headers" a good idea. Probably we want > options like --ubifs and --ubi to denote ubi and ubifs-level stuff. The > default would be "everything". > > But again, if you start smaller, and upstream a good tool for > UBIFS-level stuff, it will be easier to add UBI stuff separately. > > Besides, I have some additional vision, which you do not have to > implement, but which should be taken into account. E.g., ubidump which > does not need UBI/UBIFS drivers, ubidump which can deal with an image > generated with nanddump without "mounting" it, etc. So I was thinking > doing small steps at a time would make it easier for me and for you to > make a tool which has limited functionality today, but which can be > later extended to support more functionality. > It seems you what more than me. I think you are right. I need to reconsider how to realize these above, not hole of them, but a good architecture that can be extended easily. So I think getting data from mtd driver is a good choice and then run it with an image file. Current UBI functionality has lots of limit and it is basing on UBI driver. But we need to do more work in user space (rebuilding volume table and so on) in this way. I think it's worth. Give me some time to re-create this utility. I will send it to you if I finished it. If you get some new ideas, please tell me as soon as possible. Thanks~! Hu
On Wed, 2014-07-16 at 19:27 +0800, hujianyang wrote: > > > > Things like UBI volume table, UBI fast-map stuff are not "headers", so I > > am not sure if using word "headers" a good idea. Probably we want > > options like --ubifs and --ubi to denote ubi and ubifs-level stuff. The > > default would be "everything". > > > > But again, if you start smaller, and upstream a good tool for > > UBIFS-level stuff, it will be easier to add UBI stuff separately. > > > > Besides, I have some additional vision, which you do not have to > > implement, but which should be taken into account. E.g., ubidump which > > does not need UBI/UBIFS drivers, ubidump which can deal with an image > > generated with nanddump without "mounting" it, etc. So I was thinking > > doing small steps at a time would make it easier for me and for you to > > make a tool which has limited functionality today, but which can be > > later extended to support more functionality. > > > > It seems you what more than me. I think you are right. I need to > reconsider how to realize these above, not hole of them, but a > good architecture that can be extended easily. > > So I think getting data from mtd driver is a good choice and then > run it with an image file. Current UBI functionality has lots of > limit and it is basing on UBI driver. But we need to do more work > in user space (rebuilding volume table and so on) in this way. I > think it's worth. > > Give me some time to re-create this utility. I will send it to you > if I finished it. If you get some new ideas, please tell me as soon > as possible. I am not trying to make you do a lot more than you need. If you are analyzing an MTD image without "mounting" it, the only way to find out the LEB->PEB mapping is to do full scan, and basically copy the UBI driver code. This is a lot of work. In your case you have the driver, and it is fine to add an ioctl which provides LEB->PEB mapping. This is a lot simpler. So I envision that the tool would work like this. $ ubidump my.img --lnum 5 - dump LEB 5, will need to scan the entire image. $ ubidump /dev/ubiX_Y --lnum 5 - will just ask the UBI driver to give the PEB number for LEB 5, then find out the MTD device for this volume (should be possible by checking the sysfs files), and then reads the MTD device, and gets the UBI-level information from there. Something like this, just quick thoughts.
On Wed, 2014-07-16 at 19:27 +0800, hujianyang wrote: > If you get some new ideas, please tell me as soon > as possible. The problem is that I do not know if I have more ideas. You may spend a lot of time on this stuff and then I send one of my ideas may render your work useless. This is why I suggest to start small, do just the UBIFS nodes dump, nothing else. Just forget about UBI. Then let's review that and make part of mtd-utils. This will clarify the vision, and we may add UBI support then separately.
> > This is why I suggest to start small, do just the UBIFS nodes dump, > nothing else. Just forget about UBI. Then let's review that and make > part of mtd-utils. This will clarify the vision, and we may add UBI > support then separately. > So just use simple way to do UBIFS dump and then step by step make it a strong utility. It's not too hard. I will first realize an UBIFS dump as you showed by mtd driver.
On 16 Jul 2014, hujianyang at huawei.com wrote: >> But again, if you start smaller, and upstream a good tool for >> UBIFS-level stuff, it will be easier to add UBI stuff separately. >> Besides, I have some additional vision, which you do not have to >> implement, but which should be taken into account. E.g., ubidump >> which does not need UBI/UBIFS drivers, ubidump which can deal with an >> image generated with nanddump without "mounting" it, etc. So I was >> thinking doing small steps at a time would make it easier for me and >> for you to make a tool which has limited functionality today, but >> which can be later extended to support more functionality. > It seems you what more than me. I think you are right. I need to > reconsider how to realize these above, not hole of them, but a > good architecture that can be extended easily. > So I think getting data from mtd driver is a good choice and then > run it with an image file. Current UBI functionality has lots of > limit and it is basing on UBI driver. But we need to do more work > in user space (rebuilding volume table and so on) in this way. I > think it's worth. > Give me some time to re-create this utility. I will send it to you > if I finished it. If you get some new ideas, please tell me as soon > as possible. I started some thing like this. The source is attached. As really you only want 'read only' access, there is a minimal portion of the UBI/UbiFs code that is needed. I think that only the 'ubi-media.h' is needed. The Linux is high performance and handles read/write with different fault conditions. If you write code for read-only/single thread I think it will be much more simple than the active UBI/UbiFS code in the Linux kernel. I did not look at the UbiFS layer as much. It is far more complex that UBI imho; of course, I just saw a little of the internal volumes and the fast map features. However, fast map itself should not be needed for this utility. For your use case of analysis in a running system, I think that reads of '/dev/mtd0ro', etc can be used. This way recovery features can also be developed if we identified some inconsistency; Ie, it is not required to 'mount' the volume before analysis. The attached source just scans an file image and create an leb/peb mapping. I only used the 'ubi-media.h' as documentation. Fwiw, Bill Pringlemeir.
> > I started some thing like this. The source is attached. As really you > only want 'read only' access, there is a minimal portion of the > UBI/UbiFs code that is needed. I think that only the 'ubi-media.h' is > needed. The Linux is high performance and handles read/write with > different fault conditions. > > If you write code for read-only/single thread I think it will be much > more simple than the active UBI/UbiFS code in the Linux kernel. I did > not look at the UbiFS layer as much. It is far more complex that UBI > imho; of course, I just saw a little of the internal volumes and the > fast map features. However, fast map itself should not be needed for > this utility. > > For your use case of analysis in a running system, I think that reads of > '/dev/mtd0ro', etc can be used. This way recovery features can also be > developed if we identified some inconsistency; Ie, it is not required to > 'mount' the volume before analysis. > > The attached source just scans an file image and create an leb/peb > mapping. I only used the 'ubi-media.h' as documentation. > > Fwiw, > Bill Pringlemeir. > Hi Bill, I've read your code. This code shows me a better way to get EC header and VID header than my 'ioctl' design. Thanks~! I think my former work started in a wrong way. Using MTD functionality as yours seems better to develop a new utility we want. So in the next step, I would like to resend a new patch set which just read fs data by mtd_read(). I think a new ioctl is needed to translate specified eraseblock num from lnum to pnum. I will not care about volume table or fastmap this time. This patch set will not support dumping image file(something like -f) either. As Artem said, we should do this step by step. Thanks for your advice. I will resend this utility soon and Cc to you. Hu
On 22 Jul 2014, hujianyang@huawei.com wrote: > I've read your code. This code shows me a better way to get EC header > and VID header than my 'ioctl' design. Thanks~! > I think my former work started in a wrong way. Using MTD functionality > as yours seems better to develop a new utility we want. > So in the next step, I would like to resend a new patch set which just > read fs data by mtd_read(). I think a new ioctl is needed to translate > specified eraseblock num from lnum to pnum. The code I sent is not very clear on this. However, the variables 'eba_map' and 'pba_map' are static arrays that do this mapping. Basically, the algorithm is, for each erase block, read EC header. if(EC good) read VID header if(VID good) place_in_map(). The structure of the base UBI is very simple. An issue the code I sent doesn't handle is when two headers map to the same 'logical block'. The 'sqnum' should resolve this. place_in_map() if(LEB empty) map PEB. else if(LEB sqnum < new sqnum) /* care of wrapping or large change? */ map new PEB to LEB. For this utility, we might wish to keep the older PEB around in case the current version is corrupt at a higher (UbiFS) layer or maybe not. However, writing our own layer gives this flexibility and we don't have to add code to the Linux MTD layers. More importantly, the utility will not need (a compatible) Linux MTD layer on the host machine to check the UBI/UbiFs dump, if we can do this without an 'ioctl()'. Thanks for attempting this. I think many people realize the value of such a tool. Regards, Bill Pringlemeir.
On Tue, 2014-07-22 at 16:15 +0800, hujianyang wrote: > I've read your code. This code shows me a better way to get EC header > and VID header than my 'ioctl' design. Thanks~! > > I think my former work started in a wrong way. Using MTD functionality > as yours seems better to develop a new utility we want. > > So in the next step, I would like to resend a new patch set which just > read fs data by mtd_read(). I think a new ioctl is needed to translate > specified eraseblock num from lnum to pnum. I will not care about volume > table or fastmap this time. This patch set will not support dumping > image file(something like -f) either. As Artem said, we should do this > step by step. > > Thanks for your advice. I will resend this utility soon and Cc to you. If possible, please, try to implement few things at a time. Try to make your initial submission smaller, and then add more stuff on top. This way it is going to be easier to merge the work. Thanks!
> > If possible, please, try to implement few things at a time. Try to make > your initial submission smaller, and then add more stuff on top. This > way it is going to be easier to merge the work. Thanks! > I've resend the v2 patches. I spend a short vacation last week and was busy with other stuff when I came back to work. I think we can just apply binary data dump, ignore ubif/ubifs format data at first to make initial submission smaller. How do you feel about this? just like: ubidump /dev/ubiX_Y --lnum n output binary/hex data.
diff --git a/ubi-utils/ubidump.c b/ubi-utils/ubidump.c new file mode 100644 index 0000000..70808a5 --- /dev/null +++ b/ubi-utils/ubidump.c @@ -0,0 +1,287 @@ +/* + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See + * the GNU General Public License for more details. + */ + +/* + * An utility to dump UBI/UBIFS format data in eraseblock + * + * Author: Hu Jianyang <hujianyang@huawei.com> + */ + +#define PROGRAM_NAME "ubidump" + +#include <stdlib.h> +#include <getopt.h> + +#include <libubi.h> +#include <libubifs.h> +#include <libmtd.h> +#include <mtd_swab.h> +#include "common.h" +#include "ubiutils-common.h" + +/* The variables below are set by command line arguments */ +struct args { + const char *vol; + int lnum; + int info:1; + int header:1; + int node:1; +}; + +static struct args args = +{ + .vol = NULL, + .lnum = -1, + .info = 0, + .header = 0, + .node = 0, +}; + +static const char doc[] = PROGRAM_NAME " version " VERSION + " - an utility to dump UBI/UBIFS format data in eraseblock"; + +static const char optionsstr[] = +"-h, --help print help message\n" +"-n, --lnum logic eraseblock num to dump\n" +"-i, --info show explicit information about NODEs\n" +"-H, --header show only header information\n" +"-N, --node show only NODEs information\n" +"-V, --version print program version"; + +static const char usage[] = +"Usage: " PROGRAM_NAME " <UBI volume node file name> [-l <lnum>] [-i]\n" +"\t\t\t[--help] [--version]\n\n" +"Example 1: " PROGRAM_NAME " /dev/ubi0_1 -n 2 - dump leb 2 in volume 1\n" +"Example 2: " PROGRAM_NAME " /dev/ubi0_0 -n 1234 -i - dump leb 1234 with explicit info\n"; + +static const struct option long_options[] = { + { .name = "help", .has_arg = 0, .flag = NULL, .val = 'h' }, + { .name = "lnum", .has_arg = 1, .flag = NULL, .val = 'n' }, + { .name = "info", .has_arg = 0, .flag = NULL, .val = 'i' }, + { .name = "header", .has_arg = 0, .flag = NULL, .val = 'H' }, + { .name = "node", .has_arg = 0, .flag = NULL, .val = 'N' }, + { .name = "version", .has_arg = 0, .flag = NULL, .val = 's' }, + { NULL, 0, NULL, 0} +}; + +static int parse_opt(int argc, char * const argv[]) +{ + while (1) { + int key, error = 0; + + key = getopt_long(argc, argv, "h?in:HN", long_options, NULL); + if (key == -1) + break; + + switch (key) { + case 'n': + args.lnum = simple_strtoul(optarg, &error); + if (error || args.lnum < 0) + return errmsg("bad lnum: \"%s\"", optarg); + break; + + case 'i': + args.info = 1; + break; + + case 'H': + args.header = 1; + break; + + case 'N': + args.node = 1; + break; + + case 'h': + case '?': + printf("%s\n\n", doc); + printf("%s\n\n", usage); + printf("%s\n", optionsstr); + exit(EXIT_SUCCESS); + + case 'V': + common_print_version(); + exit(EXIT_SUCCESS); + + case ':': + return errmsg("parameter is missing"); + + default: + fprintf(stderr, "Use -h for help\n"); + return -1; + } + } + if (optind == argc) + return errmsg("UBI device name was not specified (use -h for help)"); + else if (optind != argc - 1) + return errmsg("more then one UBI device specified (use -h for help)"); + + args.vol = argv[optind]; + + if (args.lnum < 0) + return errmsg("lnum was not specified (use -h for help)"); + if (args.header && (args.info || args.node)) + return errmsg("cannot specify -H with -i or -N"); + + return 0; +} + +static void dump_header(struct ubi_ebdump_req *header) +{ + struct ubi_ec_hdr *ec_hdr = (struct ubi_ec_hdr *)header->ec_hdr; + struct ubi_vid_hdr *vid_hdr = (struct ubi_vid_hdr *)header->vid_hdr; + + printf("lnum %d is mapped to pnum %d\n", header->lnum, header->pnum); + + printf("Erase counter header dump:\n"); + printf("\tmagic %#08x\n", be32_to_cpu(ec_hdr->magic)); + printf("\tversion %d\n", (int)ec_hdr->version); + printf("\tec %llu\n", (long long)be64_to_cpu(ec_hdr->ec)); + printf("\tvid_hdr_offset %d\n", be32_to_cpu(ec_hdr->vid_hdr_offset)); + printf("\tdata_offset %d\n", be32_to_cpu(ec_hdr->data_offset)); + printf("\timage_seq %d\n", be32_to_cpu(ec_hdr->image_seq)); + printf("\thdr_crc %#08x\n", be32_to_cpu(ec_hdr->hdr_crc)); + + printf("Volume identifier header dump:\n"); + printf("\tmagic %08x\n", be32_to_cpu(vid_hdr->magic)); + printf("\tversion %d\n", (int)vid_hdr->version); + printf("\tvol_type %d\n", (int)vid_hdr->vol_type); + printf("\tcopy_flag %d\n", (int)vid_hdr->copy_flag); + printf("\tcompat %d\n", (int)vid_hdr->compat); + printf("\tvol_id %d\n", be32_to_cpu(vid_hdr->vol_id)); + printf("\tlnum %d\n", be32_to_cpu(vid_hdr->lnum)); + printf("\tdata_size %d\n", be32_to_cpu(vid_hdr->data_size)); + printf("\tused_ebs %d\n", be32_to_cpu(vid_hdr->used_ebs)); + printf("\tdata_pad %d\n", be32_to_cpu(vid_hdr->data_pad)); + printf("\tsqnum %llu\n", + (unsigned long long)be64_to_cpu(vid_hdr->sqnum)); + printf("\thdr_crc %08x\n", be32_to_cpu(vid_hdr->hdr_crc)); +} + +static int dump_leb(libubi_t libubi, int fd, struct ubi_vol_info *vol_info) +{ + int ret, leb_size, lnum; + char *buf; + + leb_size = vol_info->leb_size; + lnum = args.lnum; + + ret = ubi_is_mapped(fd, lnum); + if (ret == 0) { + errmsg("lnum %d is not mapped", lnum); + goto out; + } else if (ret < 0) { + sys_errmsg("ubi_is_mapped() failed"); + goto out; + } + + /* dump peb header */ + if (!args.node) { + struct ubi_ebdump_req request; + + memset(&request, 0, sizeof(struct ubi_ebdump_req)); + request.lnum = lnum; + request.pnum = -1; + ret = ubi_leb_dump_header(libubi, fd, &request); + if (ret < 0) { + sys_errmsg("cannot dump header of lnum %d", lnum); + goto out; + } else if (request.pnum == -1) { + errmsg("lnum %d is not mapped", lnum); + goto out; + } + + dump_header(&request); + } + + /* dump NODEs */ + if (!args.header) { + off_t offs = lnum * leb_size; + + buf = malloc(leb_size); + if (!buf) + return errmsg("cannot allocate %d bytes of memory", + leb_size); + + ret = pread(fd, buf, leb_size, offs); + if (ret != leb_size) { + errmsg("cannot read %d bytes at lnum %d, read %d", + leb_size, lnum, ret); + goto out_free; + } + ubifs_scan(lnum, buf, leb_size, 0, args.info); + } + +out_free: + free(buf); +out: + return ret; +} + +int main(int argc, char * const argv[]) +{ + int err, fd; + libubi_t libubi; + struct ubi_vol_info vol_info; + + err = parse_opt(argc, argv); + if (err) + return -1; + + libubi = libubi_open(); + if (!libubi) { + if (errno == 0) + errmsg("UBI is not present in the system"); + else + sys_errmsg("cannot open libubi"); + goto out_libubi; + } + + err = ubi_probe_node(libubi, args.vol); + if (err == 1) { + errmsg("\"%s\" is an UBI device node, not an UBI volume node", + args.vol); + goto out_libubi; + } else if (err < 0) { + if (errno == ENODEV) + errmsg("\"%s\" is not an UBI volume node", args.vol); + else + sys_errmsg("error while probing \"%s\"", args.vol); + goto out_libubi; + } + + err = ubi_get_vol_info(libubi, args.vol, &vol_info); + if (err) { + sys_errmsg("cannot get information about UBI volume \"%s\"", + args.vol); + goto out_libubi; + } + + fd = open(args.vol, O_RDONLY); + if (fd == -1) { + sys_errmsg("cannot open UBI volume \"%s\"", args.vol); + goto out_libubi; + } + err = dump_leb(libubi, fd, &vol_info); + if (err) + goto out_fd; + + close(fd); + libubi_close(libubi); + return 0; + +out_fd: + close(fd); +out_libubi: + libubi_close(libubi); + return -1; +}
This utility is referred to ubiupdatevol. I think I should move 'dump_header()' in this file to other place but I don't know where. This utility don't support dump binary date because I didn't find a useful helper function like 'print_dump_hex()' in kernel and kernel may dump the data which is corrupted so I think no need to dump them here. Signed-off-by: hujianyang <hujianyang@huawei.com> --- ubi-utils/ubidump.c | 287 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 ubi-utils/ubidump.c