| Message ID | 20190628155640.24103-1-metepolat2000@gmail.com |
|---|---|
| Headers | show |
| Series | Update REST API: Add 'project patches as mbox' field | expand |
On 28-06-2019 17:56, Mete Polat wrote: > Patchwork already has the ability to export patches, series, covers and bundles > as an mbox file. This patch extends that ability to projects as well. Therefore > a new url and api field has been introduced. > > Updated REST API version to 1.2 > > New REST API url: api/1.2/ > New internal url: project/<project_id>/list/mbox/ > New project rest api field: patches_mbox > > Like patchwork.example.com/project/<project_id>/list/ the final project.mbox > only includes patches. Careful naming has been made in case the api should > support export of a whole project in the future as well. > > Changes since v1: > - Add missing url for api version 1.2 > > Mete Polat (5): > Add option to get all project patches in one mbox > Add urls to get all project patches in one mbox > Add api endpoint for project patches as mbox > Update api documentation for v1.2 > Add release notes: project patches as mbox > > docs/api/rest/index.rst | 50 +- > docs/api/rest/schemas/v1.1.rst | 4 +- > docs/api/rest/schemas/v1.2.rst | 5 + > docs/api/schemas/generate_schema.py | 4 +- > docs/api/schemas/latest/patchwork.yaml | 7 +- > docs/api/schemas/patchwork.j2 | 7 + > docs/api/schemas/v1.2/patchwork.yaml | 2319 +++++++++++++++++ Probably too large to get unmoderated through the mailinglist. Anyway: Patch 4/5 is missing. Even at https://lists.ozlabs.org/pipermail/patchwork/2019-June/thread.html How does get patch four of the five to the mailinglist(archive)? > patchwork/api/project.py | 11 +- > patchwork/models.py | 25 +- > patchwork/urls.py | 6 +- > patchwork/views/patch.py | 12 + > patchwork/views/utils.py | 13 + > ...project-patches-mbox-623f8c9d4cf6a952.yaml | 6 + > 13 files changed, 2428 insertions(+), 41 deletions(-) > create mode 100644 docs/api/rest/schemas/v1.2.rst > create mode 100644 docs/api/schemas/v1.2/patchwork.yaml > create mode 100644 releasenotes/notes/project-patches-mbox-623f8c9d4cf6a952.yaml > Cheers Geert Stappers DevOps Engineer @ Hendrikx ITC
Hi Mete, > Patchwork already has the ability to export patches, series, covers and bundles > as an mbox file. This patch extends that ability to projects as well. Therefore > a new url and api field has been introduced. > Thanks for your contribution! I checked https://lore.kernel.org/patchwork/project/lkml/list/?archive=both&state=* at the moment it has 1012231 patches. I'm a bit worried about how long that would take to export and the load it would put on an server to serve a request for all the patches. I think some of the lists on ozlabs will also have hundred of megabytes of patches. What do you see as the use case for this? Regards, Daniel > Updated REST API version to 1.2 > > New REST API url: api/1.2/ > New internal url: project/<project_id>/list/mbox/ > New project rest api field: patches_mbox > > Like patchwork.example.com/project/<project_id>/list/ the final project.mbox > only includes patches. Careful naming has been made in case the api should > support export of a whole project in the future as well. > > Changes since v1: > - Add missing url for api version 1.2 > > Mete Polat (5): > Add option to get all project patches in one mbox > Add urls to get all project patches in one mbox > Add api endpoint for project patches as mbox > Update api documentation for v1.2 > Add release notes: project patches as mbox > > docs/api/rest/index.rst | 50 +- > docs/api/rest/schemas/v1.1.rst | 4 +- > docs/api/rest/schemas/v1.2.rst | 5 + > docs/api/schemas/generate_schema.py | 4 +- > docs/api/schemas/latest/patchwork.yaml | 7 +- > docs/api/schemas/patchwork.j2 | 7 + > docs/api/schemas/v1.2/patchwork.yaml | 2319 +++++++++++++++++ > patchwork/api/project.py | 11 +- > patchwork/models.py | 25 +- > patchwork/urls.py | 6 +- > patchwork/views/patch.py | 12 + > patchwork/views/utils.py | 13 + > ...project-patches-mbox-623f8c9d4cf6a952.yaml | 6 + > 13 files changed, 2428 insertions(+), 41 deletions(-) > create mode 100644 docs/api/rest/schemas/v1.2.rst > create mode 100644 docs/api/schemas/v1.2/patchwork.yaml > create mode 100644 releasenotes/notes/project-patches-mbox-623f8c9d4cf6a952.yaml > > -- > 2.22.0 > > _______________________________________________ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork
> Hi Mete, > >> Patchwork already has the ability to export patches, series, covers and bundles >> as an mbox file. This patch extends that ability to projects as well. Therefore >> a new url and api field has been introduced. >> > Thanks for your contribution! > > I checked > https://lore.kernel.org/patchwork/project/lkml/list/?archive=both&state=* > at the moment it has 1012231 patches. I'm a bit worried about how long > that would take to export and the load it would put on an server to > serve a request for all the patches. I think some of the lists on ozlabs > will also have hundred of megabytes of patches. > > What do you see as the use case for this? > > Regards, > Daniel Hello Daniel, this patch is our effort in combining Patchwork with PaStA (Patch Stack Analysis). We described our intention in detail in our mail 'Patch stack analysis'. We have to download all patches in a project in order to analyze and compare different ones. This is a one-time job. Later on, incoming patches are downloaded individually (similar to what snowpatch does). This approach allows us to keep the data between patchwork and PaStA consistent. We have understand the problem of server load. Do you have any suggestion for a solution? Regards, Mete >> Updated REST API version to 1.2 >> >> New REST API url: api/1.2/ >> New internal url: project/<project_id>/list/mbox/ >> New project rest api field: patches_mbox >> >> Like patchwork.example.com/project/<project_id>/list/ the final project.mbox >> only includes patches. Careful naming has been made in case the api should >> support export of a whole project in the future as well. >> >> Changes since v1: >> - Add missing url for api version 1.2 >> >> Mete Polat (5): >> Add option to get all project patches in one mbox >> Add urls to get all project patches in one mbox >> Add api endpoint for project patches as mbox >> Update api documentation for v1.2 >> Add release notes: project patches as mbox >> >> docs/api/rest/index.rst | 50 +- >> docs/api/rest/schemas/v1.1.rst | 4 +- >> docs/api/rest/schemas/v1.2.rst | 5 + >> docs/api/schemas/generate_schema.py | 4 +- >> docs/api/schemas/latest/patchwork.yaml | 7 +- >> docs/api/schemas/patchwork.j2 | 7 + >> docs/api/schemas/v1.2/patchwork.yaml | 2319 +++++++++++++++++ >> patchwork/api/project.py | 11 +- >> patchwork/models.py | 25 +- >> patchwork/urls.py | 6 +- >> patchwork/views/patch.py | 12 + >> patchwork/views/utils.py | 13 + >> ...project-patches-mbox-623f8c9d4cf6a952.yaml | 6 + >> 13 files changed, 2428 insertions(+), 41 deletions(-) >> create mode 100644 docs/api/rest/schemas/v1.2.rst >> create mode 100644 docs/api/schemas/v1.2/patchwork.yaml >> create mode 100644 releasenotes/notes/project-patches-mbox-623f8c9d4cf6a952.yaml >> >> -- >> 2.22.0 >> >> _______________________________________________ >> Patchwork mailing list >> Patchwork@lists.ozlabs.org >> https://lists.ozlabs.org/listinfo/patchwork
> On 28-06-2019 17:56, Mete Polat wrote: > >> Patchwork already has the ability to export patches, series, covers and bundles >> as an mbox file. This patch extends that ability to projects as well. Therefore >> a new url and api field has been introduced. >> >> Updated REST API version to 1.2 >> >> New REST API url: api/1.2/ >> New internal url: project/<project_id>/list/mbox/ >> New project rest api field: patches_mbox >> >> Like patchwork.example.com/project/<project_id>/list/ the final project.mbox >> only includes patches. Careful naming has been made in case the api should >> support export of a whole project in the future as well. >> >> Changes since v1: >> - Add missing url for api version 1.2 >> >> Mete Polat (5): >> Add option to get all project patches in one mbox >> Add urls to get all project patches in one mbox >> Add api endpoint for project patches as mbox >> Update api documentation for v1.2 >> Add release notes: project patches as mbox >> >> docs/api/rest/index.rst | 50 +- >> docs/api/rest/schemas/v1.1.rst | 4 +- >> docs/api/rest/schemas/v1.2.rst | 5 + >> docs/api/schemas/generate_schema.py | 4 +- >> docs/api/schemas/latest/patchwork.yaml | 7 +- >> docs/api/schemas/patchwork.j2 | 7 + >> docs/api/schemas/v1.2/patchwork.yaml | 2319 +++++++++++++++++ > Probably too large to get unmoderated through the mailinglist. I agree. The patch is 72KB big. > Anyway: Patch 4/5 is missing. Even at > https://lists.ozlabs.org/pipermail/patchwork/2019-June/thread.html > > > How does get patch four of the five to the mailinglist(archive)? > > > >> patchwork/api/project.py | 11 +- >> patchwork/models.py | 25 +- >> patchwork/urls.py | 6 +- >> patchwork/views/patch.py | 12 + >> patchwork/views/utils.py | 13 + >> ...project-patches-mbox-623f8c9d4cf6a952.yaml | 6 + >> 13 files changed, 2428 insertions(+), 41 deletions(-) >> create mode 100644 docs/api/rest/schemas/v1.2.rst >> create mode 100644 docs/api/schemas/v1.2/patchwork.yaml >> create mode 100644 releasenotes/notes/project-patches-mbox-623f8c9d4cf6a952.yaml >> > > Cheers > > Geert Stappers > > DevOps Engineer @ Hendrikx ITC > > > _______________________________________________ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork
Hi Mete, > this patch is our effort in combining Patchwork with PaStA (Patch Stack > Analysis). We described our intention in detail in our mail 'Patch > stack analysis'. Ah right, sorry, I lose track of who is doing what! > We have to download all patches in a project in order to analyze and > compare different ones. This is a one-time job. Later on, incoming > patches are downloaded individually (similar to what snowpatch does). > This approach allows us to keep the data between patchwork and PaStA > consistent. > We have understand the problem of server load. Do you have any > suggestion for a solution? So there are two possible complimenatry approaches I can think of: - gather the data from a download of the mailing list that patchwork injests. For LKML you can get this from https://www.kernel.org/lore.html, for example. You could then pass this through a local patchwork instance. (Let me know if you want my scripts for importing a public-archive git repo into patchwork.) - add a management command to export a project as an mbox and then coordinate with patchwork admins at the instance you're interested in to run the export at a time that suits them and provide you with a heavily compressed copy of the output. Regards, Daniel > > Regards, > > Mete >>> Updated REST API version to 1.2 >>> >>> New REST API url: api/1.2/ >>> New internal url: project/<project_id>/list/mbox/ >>> New project rest api field: patches_mbox >>> >>> Like patchwork.example.com/project/<project_id>/list/ the final project.mbox >>> only includes patches. Careful naming has been made in case the api should >>> support export of a whole project in the future as well. >>> >>> Changes since v1: >>> - Add missing url for api version 1.2 >>> >>> Mete Polat (5): >>> Add option to get all project patches in one mbox >>> Add urls to get all project patches in one mbox >>> Add api endpoint for project patches as mbox >>> Update api documentation for v1.2 >>> Add release notes: project patches as mbox >>> >>> docs/api/rest/index.rst | 50 +- >>> docs/api/rest/schemas/v1.1.rst | 4 +- >>> docs/api/rest/schemas/v1.2.rst | 5 + >>> docs/api/schemas/generate_schema.py | 4 +- >>> docs/api/schemas/latest/patchwork.yaml | 7 +- >>> docs/api/schemas/patchwork.j2 | 7 + >>> docs/api/schemas/v1.2/patchwork.yaml | 2319 +++++++++++++++++ >>> patchwork/api/project.py | 11 +- >>> patchwork/models.py | 25 +- >>> patchwork/urls.py | 6 +- >>> patchwork/views/patch.py | 12 + >>> patchwork/views/utils.py | 13 + >>> ...project-patches-mbox-623f8c9d4cf6a952.yaml | 6 + >>> 13 files changed, 2428 insertions(+), 41 deletions(-) >>> create mode 100644 docs/api/rest/schemas/v1.2.rst >>> create mode 100644 docs/api/schemas/v1.2/patchwork.yaml >>> create mode 100644 releasenotes/notes/project-patches-mbox-623f8c9d4cf6a952.yaml >>> >>> -- >>> 2.22.0 >>> >>> _______________________________________________ >>> Patchwork mailing list >>> Patchwork@lists.ozlabs.org >>> https://lists.ozlabs.org/listinfo/patchwork
On 2/7/19 10:26 am, Daniel Axtens wrote: > So there are two possible complimenatry approaches I can think of: > > - gather the data from a download of the mailing list that patchwork > injests. For LKML you can get this from > https://www.kernel.org/lore.html, for example. > You could then pass this through a local patchwork instance. (Let me > know if you want my scripts for importing a public-archive git repo > into patchwork.) > > - add a management command to export a project as an mbox and then > coordinate with patchwork admins at the instance you're interested in > to run the export at a time that suits them and provide you with a > heavily compressed copy of the output. > If we were to add an API for this kind of bulk mbox export, I think it would need to export a fixed number of emails at a time (100 or 250 or something like that). For patchwork setups where the raw mailing list data isn't easily retrievable, a management command to export the whole project as mbox could be extended fairly easily to "export the past N days of the project as mbox". Then just put that in your crontab and have it export a new archive every so often, which you compress and then serve up statically. > Regards, > Daniel > > >> >> Regards, >> >> Mete >>>> Updated REST API version to 1.2 >>>> >>>> New REST API url: api/1.2/ >>>> New internal url: project/<project_id>/list/mbox/ >>>> New project rest api field: patches_mbox >>>> >>>> Like patchwork.example.com/project/<project_id>/list/ the final project.mbox >>>> only includes patches. Careful naming has been made in case the api should >>>> support export of a whole project in the future as well. >>>> >>>> Changes since v1: >>>> - Add missing url for api version 1.2 >>>> >>>> Mete Polat (5): >>>> Add option to get all project patches in one mbox >>>> Add urls to get all project patches in one mbox >>>> Add api endpoint for project patches as mbox >>>> Update api documentation for v1.2 >>>> Add release notes: project patches as mbox >>>> >>>> docs/api/rest/index.rst | 50 +- >>>> docs/api/rest/schemas/v1.1.rst | 4 +- >>>> docs/api/rest/schemas/v1.2.rst | 5 + >>>> docs/api/schemas/generate_schema.py | 4 +- >>>> docs/api/schemas/latest/patchwork.yaml | 7 +- >>>> docs/api/schemas/patchwork.j2 | 7 + >>>> docs/api/schemas/v1.2/patchwork.yaml | 2319 +++++++++++++++++ >>>> patchwork/api/project.py | 11 +- >>>> patchwork/models.py | 25 +- >>>> patchwork/urls.py | 6 +- >>>> patchwork/views/patch.py | 12 + >>>> patchwork/views/utils.py | 13 + >>>> ...project-patches-mbox-623f8c9d4cf6a952.yaml | 6 + >>>> 13 files changed, 2428 insertions(+), 41 deletions(-) >>>> create mode 100644 docs/api/rest/schemas/v1.2.rst >>>> create mode 100644 docs/api/schemas/v1.2/patchwork.yaml >>>> create mode 100644 releasenotes/notes/project-patches-mbox-623f8c9d4cf6a952.yaml >>>> >>>> -- >>>> 2.22.0 >>>> >>>> _______________________________________________ >>>> Patchwork mailing list >>>> Patchwork@lists.ozlabs.org >>>> https://lists.ozlabs.org/listinfo/patchwork > _______________________________________________ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork >
Hi Daniel, hi Andrew, (I am mentoring Mete) On Tue, Jul 2, 2019 at 6:33 AM Andrew Donnellan <ajd@linux.ibm.com> wrote: > > On 2/7/19 10:26 am, Daniel Axtens wrote: > > So there are two possible complimenatry approaches I can think of: > > > > - gather the data from a download of the mailing list that patchwork > > injests. For LKML you can get this from > > https://www.kernel.org/lore.html, for example. > > You could then pass this through a local patchwork instance. (Let me > > know if you want my scripts for importing a public-archive git repo > > into patchwork.) > > In fact, Pasta can already take public inbox git repositories as input, and from the point of reproducibility of results, this is our preference for analysis with pasta. During the design discussion, I was considering that Mete could simply extend patchwork to also take public inboxes as input. We made this second priority, as: 1. we expected that to be a bit more complex and driving our focus a bit away of original goal combining patchwork and pasta towards a secondary goal of making public inbox and patchwork work nicely together. 2. we expected that our first alpha users might not be mailing lists administrators, but users that just run on their personal inbox and would be interested to try out pasta's capabilities, but do not have a public inbox setup for their personal inbox. The public inbox to patchwork integration would certainly be nice and helpful for using it on mailing lists that are already set up to be archived with public inbox, and it would make it easier to keep the state of pasta and patchwork consistent. Please let us know where to find those scripts. We can then try them out and see if the setup would principally work for us and which extensions and changes we would need. > > - add a management command to export a project as an mbox and then > > coordinate with patchwork admins at the instance you're interested in > > to run the export at a time that suits them and provide you with a > > heavily compressed copy of the output. > > > If we were to add an API for this kind of bulk mbox export, I think it > would need to export a fixed number of emails at a time (100 or 250 or > something like that). > > For patchwork setups where the raw mailing list data isn't easily > retrievable, a management command to export the whole project as mbox > could be extended fairly easily to "export the past N days of the > project as mbox". Then just put that in your crontab and have it export > a new archive every so often, which you compress and then serve up > statically. > It sounds reasonable to make the functionality we want a management command rather than using the REST API. We will continue to dig into that and let you know if we hit any unexpected issues. Best regards, Lukas
Hi all, > The public inbox to patchwork integration would certainly be nice and > helpful for using it on mailing lists that are already set up to be > archived with public inbox, and it would make it easier to keep the > state of pasta and patchwork consistent. > > Please let us know where to find those scripts. We can then try them > out and see if the setup would principally work for us and which > extensions and changes we would need. Check out https://github.com/daxtens/patchwork/commit/6d522cf75f909dc26dc608c5771d210f6d1649c7 It converts the public-inbox git repo into mbox files and uses parsearchive to import them. I will warn you in advance that the script is not at all sophisticated and so for a list like LKML the process is still excruciatingly slow. > It sounds reasonable to make the functionality we want a management > command rather than using the REST API. We will continue to dig into > that and let you know if we hit any unexpected issues. Sounds good. Regards, Daniel > > > Best regards, > > Lukas
On 01-07-2019 17:31, Mete Polat wrote: >> On 28-06-2019 17:56, Mete Polat wrote: >> >>> Patchwork already has the ability to export patches, series, covers and bundles >>> as an mbox file. This patch extends that ability to projects as well. Therefore >>> a new url and api field has been introduced. >>> >>> Updated REST API version to 1.2 .... >>> Mete Polat (5): >>> Add option to get all project patches in one mbox >>> Add urls to get all project patches in one mbox >>> Add api endpoint for project patches as mbox >>> Update api documentation for v1.2 >>> Add release notes: project patches as mbox >>> >>> docs/api/rest/index.rst | 50 +- >>> docs/api/rest/schemas/v1.1.rst | 4 +- >>> docs/api/rest/schemas/v1.2.rst | 5 + >>> docs/api/schemas/generate_schema.py | 4 +- >>> docs/api/schemas/latest/patchwork.yaml | 7 +- >>> docs/api/schemas/patchwork.j2 | 7 + >>> docs/api/schemas/v1.2/patchwork.yaml | 2319 +++++++++++++++++ >> Probably too large to get unmoderated through the mailinglist. > I agree. The patch is 72KB big. >> Anyway: Patch 4/5 is missing. Even at >> https://lists.ozlabs.org/pipermail/patchwork/2019-June/thread.html >> >> >> How does get patch four of the five to the mailinglist(archive)? >> Reminder. I hope it helps this cool project. Geert Stappers
>>> Probably too large to get unmoderated through the mailinglist. >> I agree. The patch is 72KB big. >>> Anyway: Patch 4/5 is missing. Even at >>> https://lists.ozlabs.org/pipermail/patchwork/2019-June/thread.html >>> >>> >>> How does get patch four of the five to the mailinglist(archive)? >>> > Reminder. I hope it helps this cool project. For what it's worth, what to do in future if you have a large patch is: 1) send the patches that you can to the list as regular mail 2) include in your cover letter a link to a signed tag that we can pull to get the patches that do not make it to the list. Please make sure that patches that cannot go to the list are as simple to review as possible (e.g. automatically generated) because they miss out on review on the list. In this case I don't think you'll end up needing to resend that patch because I think we landed on a management command rather than changing the API, but maybe it will help in future. Regards, Daniel > > > Geert Stappers > > > _______________________________________________ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork