[RFC,1/5] Documentation: dt-bindings: add documentation on new DT binding format

Documentation explaining the syntax and format of the YAML-based DT binding
documentation.

Signed-off-by: Matt Porter <mporter@konsulko.com>
---
 .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt

On 8/27/2015 10:23 PM, Matt Porter wrote:
> Documentation explaining the syntax and format of the YAML-based DT binding
> documentation.
> 
> Signed-off-by: Matt Porter <mporter@konsulko.com>
> ---
>  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
>  1 file changed, 106 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> 
> diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> new file mode 100644
> index 0000000..f9acc22
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> @@ -0,0 +1,106 @@
> +--------------------------
> +Device Tree Binding Format
> +--------------------------
> +
> +Background
> +----------
> +
> +DT bindings historically were written as text in prose format which
> +led to issues in usability of that source documentation. Some of
> +these issues include the need to programmatically process binding
> +source documentation to do DTS validation, perform mass updates to
> +format/style, and to generate publishable documentation in HTML or
> +PDF form.
> +
> +Overview
> +--------
> +
> +The DT binding format is based on the YAML text markup language.
> +Although there are many text markup options available, YAML
> +fulfills all requirements considered for a DT binding source format
> +which include:
> +
> +1) Must be human readable
> +2) Must be easily translated to other data formats (XML, JSON, etc).
> +3) Must have sufficient tools and libraries to enable developers to
> +   build new tools for DT binding processing
> +4) Must have a complete spec to refer to syntax
> +
> +YAML is documentated in the specification found at
> +http://www.yaml.org/spec/1.2/spec.html
> +
> +The required YAML DT binding tag format and syntax are defined in
> +the following sections.
> +
> +YAML DT Binding Syntax
> +----------------------
> +
> +* Lines starting with "#" are comments and not part of the binding itself
> +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> +* "---" starts a binding document
> +* "..." ends a binding document
> +* Multiple binding documents may exist in a single file
> +* Tabs are not permitted
> +* Scope is denoted by indentation of two spaces
> +* Key value pairs are denoted by "key: value"
> +* Sequences are denoted by "- "
> +* Scalar values may convert newlines to spaces and preserve blank
> +  lines for long description formatting using ">"
> +* Scalar values may escape all reserved characters and preserve
> +  newlines by using "|" to denote literal style
> +
> +For additional information on YAML syntax, refer to the specification
> +at http://www.yaml.org/spec/1.2/spec.html
> +
> +YAML DT Binding Format
> +----------------------
> +
> +The following YAML types are supported in the DT binding format:
> +
> +* [R] id: unique identifier in property form (e.g. skel-device)
> +
> +* [R] title: title of the binding
> +
> +* [O] maintainer: sequence of maintainers
> +      [R] name: name and email of maintainer or mailing list in RFC822
> +                form.
> +
> +* [O] description: full description of the binding
> +
> +* [O] compatible: sequence of valid compatible descriptors
> +      [R] name: the compatible string surrounded in double quotes
> +      [O] deprecated: a deprecated compatible string surrounded in
> +                      double quotes
> +      [O] description: description of the compatible string
> +
> +* [O] required: sequence of required properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] optional: sequence of optional properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] deprecated: sequence of deprecated properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id

I commented in reply to patch 0 that we should think through how
this structure will support adding new features as the next step
after converting existing bindings.

The specific case I started thinking about was the distinction between
required, optional, required-if, and optional-if.  A property might
be required in all cases, optional in all cases, required in some
specified cases, and/or optional in some specified cases.  So a
property could be both "required-if" and "optional-if".  Or it
might just be one of the two.

Here is where my YAML ignorance jumps out.  I really do not know the
best way to approach it.  And I do not know if the syntax I am using
is even remotely correct - I am trying to extrapolate from the examples
in the spec, without having read past the examples yet.  My naive knee
jerk alternatives where (by example, because I can explain it more
concisely that way instead of the way it should be described in this file):

---- (1)

required:
  - &P1
    name: "prop1"
    required-if: < some condition >

optional:
  - *P1
    optional-if: < some other condition>
  - name: "prop2"
    required: true
  - name: "prop3"
    optional: true

---- (2)

property:
  - name: "prop1"
    required-if: < some condition >
    optional-if: < some other condition >
  - name: "prop2"
    required: true
  - name: "prop3"
    optional: true


In a related question, I commented in my reply to patch 0 that some of
the information about the compatible got lost in the conversion from
the existing binding to the YAML format.  That led me to wonder why
the compatible property should be at the YAML top level (same as
"optional" and "required").  My (again naive, YAML amateur) thought
is that the form should be something like:

[O] property:
    [R] optional: | required: | required-if: | optional-if: | required-if: optional-if:
    [O] deprecated: a deprecated compatible string surrounded in
                    double quotes | ???
    [R] name: name of the property surrounded in double quotes
    [R] description: description of the property
    [O] reference: optional reference to a binding id

Apologies for bizarre format for "optional:", but for the purposes of this
comment, hopefully my intent will be clear.

Then "compatible" becomes just another "property:", just as it is in the
.dts, but any processing or analysis tool knows that the name "compatible"
is used in a very specific way and has magical meaning for .dts.

Then if compatible is just another "property:" then the information that
lost was in the conversion process would not be lost, it would appear
in "description:".


> +
> +* [R] example: sequence of examples:
> +      [R] dts: DT source of example usage. The example text must use
> +               literal style ("|") so that it retains indentation and
> +               newlines.
> +      [O] description: description of the example
> +
> +Note: [R] and [O] denote required and optional fields, respectively.
> +
> +Skeleton Binding
> +----------------
> +
> +The skeleton.yaml binding found in the top of the DT binding tree
> +is the canonical example of syntax and format to use when writing
> +a DT binding document. It is maintained with the latest formatting
> +conventions, making it the best starting point when writing a new DT
> +binding.

It would be good to have a comment of YAML style.  Looking at the spec,
there appear to be multiple ways to format content that is functionally
equivalent.  For example:

   # a flow sequence
   - [name        , hr, avg  ]
   - [Mark McGwire, 65, 0.278]
   - [Sammy Sosa  , 63, 0.288]

vs.

   # a flow mapping
   Mark McGwire: {hr: 65, avg: 0.278}
   Sammy Sosa: {
       hr: 63,
       avg: 0.288
     }

This is my first quick reading of the YAML spec, so maybe a flow sequence and
a flow mapping are not equivalent, but that was my naive impression.

I think that in the interest of human readability (requirement 1), that a
specific formatting style should be specified.  That will make life easier
for people who do not know YAML, but are (1) merely copying from existing binding
docs to create new ones, or (2) trying to understand how different bindings
differ.

-Frank
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter@konsulko.com> wrote:
> Documentation explaining the syntax and format of the YAML-based DT binding
> documentation.
>
> Signed-off-by: Matt Porter <mporter@konsulko.com>
> ---
>  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
>  1 file changed, 106 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
>
> diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> new file mode 100644
> index 0000000..f9acc22
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> @@ -0,0 +1,106 @@
> +--------------------------
> +Device Tree Binding Format
> +--------------------------
> +
> +Background
> +----------
> +
> +DT bindings historically were written as text in prose format which
> +led to issues in usability of that source documentation. Some of
> +these issues include the need to programmatically process binding
> +source documentation to do DTS validation, perform mass updates to
> +format/style, and to generate publishable documentation in HTML or
> +PDF form.
> +
> +Overview
> +--------
> +
> +The DT binding format is based on the YAML text markup language.
> +Although there are many text markup options available, YAML
> +fulfills all requirements considered for a DT binding source format
> +which include:
> +
> +1) Must be human readable
> +2) Must be easily translated to other data formats (XML, JSON, etc).
> +3) Must have sufficient tools and libraries to enable developers to
> +   build new tools for DT binding processing
> +4) Must have a complete spec to refer to syntax
> +
> +YAML is documentated in the specification found at
> +http://www.yaml.org/spec/1.2/spec.html
> +
> +The required YAML DT binding tag format and syntax are defined in
> +the following sections.
> +
> +YAML DT Binding Syntax
> +----------------------
> +
> +* Lines starting with "#" are comments and not part of the binding itself
> +* "%YAML 1.2" starts a file, indicating the version of YAML in use

I think it would be good to specify a DT Binding Format version as well.
I would expect we may expand the syntax, or we may alter it, so it
would be good to have a number for our specific syntax.  This would
help in some circumstances, if we make multiple passes over the
documents to achieve our conversion.  Maybe this should be added
below.

> +* "---" starts a binding document
> +* "..." ends a binding document
> +* Multiple binding documents may exist in a single file
> +* Tabs are not permitted
> +* Scope is denoted by indentation of two spaces
> +* Key value pairs are denoted by "key: value"
> +* Sequences are denoted by "- "
> +* Scalar values may convert newlines to spaces and preserve blank
> +  lines for long description formatting using ">"
> +* Scalar values may escape all reserved characters and preserve
> +  newlines by using "|" to denote literal style
> +
> +For additional information on YAML syntax, refer to the specification
> +at http://www.yaml.org/spec/1.2/spec.html
> +
> +YAML DT Binding Format
> +----------------------
> +
> +The following YAML types are supported in the DT binding format:
> +

Based on my comment above, I recommend:
[R] binding-format: 1

> +* [R] id: unique identifier in property form (e.g. skel-device)
> +
> +* [R] title: title of the binding
> +
> +* [O] maintainer: sequence of maintainers
> +      [R] name: name and email of maintainer or mailing list in RFC822
> +                form.
> +
> +* [O] description: full description of the binding
> +
> +* [O] compatible: sequence of valid compatible descriptors
> +      [R] name: the compatible string surrounded in double quotes
> +      [O] deprecated: a deprecated compatible string surrounded in
> +                      double quotes
> +      [O] description: description of the compatible string

compatible is a property like all the others, and I think it's confusing
to have it in a separate section.  For example, when listing other
properties, the "name:" field specifies the property name, but for
the compatible section the "name:" field specifies the possible values
for the compatible property.  This is (at least to me) quite confusing.

Can we make this a regular property, with an option 'deprecated' attribute.
I guess we may need to distinguish between deprecated properties and
deprecated vales of properties.
The above 'deprecated' specifies deprecated values for the compatible
property, while
the section below specifies deprecated properties.  I don't see a way
to indicate a deprecated
value for a still-existing property, but that may be too much detail for now.


> +* [O] required: sequence of required properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id

I would recommend we add types and value constraints to the
binding schema:

[O] type: type of value for this property, and if specified must be
one of the following (without quotes):
"string", "number", "array", "phandle"
[O] value constraint: specifies a limit on the value for a property.
This can be specified as English text, or be a range or choice constraint.
The latter two should be expressed with the following exact english
phrases (without quotes):
"must be in the range x to y, inclusive"
where x and y are numbers
"must be one of <choice 1>, <choice 2>, ..."
where <choice x> is a string, which should be quoted
if it includes a comma.

Any other constraint is expressed in free-form English text

I would recommend adding these to this binding schema doc, but
not requiring them on the first conversion pass of the docs.

Basically my idea is that you want the each pass over the binding
documents to be as painless as possible.  Every time a user encounters
something they're not sure about, it will slow them down.  There will
be lots of ambiguous situations, and we want the process to be as
lossless as possible, so I would recommend a strategy of doing as
much as people can, and if in doubt just put any outstanding text
in the descriptions.  The descriptions will always be free-form text,
and we can come along later and convert these to types, constraints,
as we see fit.

> +* [O] optional: sequence of optional properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] deprecated: sequence of deprecated properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [R] example: sequence of examples:
> +      [R] dts: DT source of example usage. The example text must use
> +               literal style ("|") so that it retains indentation and
> +               newlines.
> +      [O] description: description of the example
> +

I'd recommend having a top-level notes section:
* [O] note: any other notes about this node that are not covered by
preceding property descriptions.

> +Note: [R] and [O] denote required and optional fields, respectively.

I'd put this note before the format description.

 -- Tim

> +
> +Skeleton Binding
> +----------------
> +
> +The skeleton.yaml binding found in the top of the DT binding tree
> +is the canonical example of syntax and format to use when writing
> +a DT binding document. It is maintained with the latest formatting
> +conventions, making it the best starting point when writing a new DT
> +binding.
> --
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Fri, Aug 28, 2015 at 01:23:49AM -0400, Matt Porter wrote:
> Documentation explaining the syntax and format of the YAML-based DT binding
> documentation.

Sorry I've taken so long to comment; I've had sickness and a backlog
of day-job work to deal with.

> Signed-off-by: Matt Porter <mporter@konsulko.com>
> ---
>  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
>  1 file changed, 106 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> 
> diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> new file mode 100644
> index 0000000..f9acc22
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> @@ -0,0 +1,106 @@
> +--------------------------
> +Device Tree Binding Format
> +--------------------------
> +
> +Background
> +----------
> +
> +DT bindings historically were written as text in prose format which
> +led to issues in usability of that source documentation. Some of
> +these issues include the need to programmatically process binding
> +source documentation to do DTS validation, perform mass updates to
> +format/style, and to generate publishable documentation in HTML or
> +PDF form.
> +
> +Overview
> +--------
> +
> +The DT binding format is based on the YAML text markup language.
> +Although there are many text markup options available, YAML
> +fulfills all requirements considered for a DT binding source format
> +which include:
> +
> +1) Must be human readable
> +2) Must be easily translated to other data formats (XML, JSON, etc).
> +3) Must have sufficient tools and libraries to enable developers to
> +   build new tools for DT binding processing
> +4) Must have a complete spec to refer to syntax
> +
> +YAML is documentated in the specification found at
> +http://www.yaml.org/spec/1.2/spec.html

I've read through this.  At least tentatively, YAML seems like a
pretty good choice to me.  It seems to have a pretty good balance
between human readability and machine parsability.


> +The required YAML DT binding tag format and syntax are defined in
> +the following sections.
> +
> +YAML DT Binding Syntax
> +----------------------
> +
> +* Lines starting with "#" are comments and not part of the binding itself
> +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> +* "---" starts a binding document
> +* "..." ends a binding document
> +* Multiple binding documents may exist in a single file
> +* Tabs are not permitted
> +* Scope is denoted by indentation of two spaces

I think the recommended indentation should be increased.  At just 2
spaces it's easy to lose track of indentation level if you scroll down
a page.

8 would match the kernel code style, but might get cumbersome since I
think our YAML structures will probably be more deeply nested in
practice than code blocks generally should be.

So, I'd suggest 4.

> +* Key value pairs are denoted by "key: value"
> +* Sequences are denoted by "- "
> +* Scalar values may convert newlines to spaces and preserve blank
> +  lines for long description formatting using ">"
> +* Scalar values may escape all reserved characters and preserve
> +  newlines by using "|" to denote literal style
> +
> +For additional information on YAML syntax, refer to the specification
> +at http://www.yaml.org/spec/1.2/spec.html
> +
> +YAML DT Binding Format
> +----------------------
> +
> +The following YAML types are supported in the DT binding format:

I find the wording above really confusing.  Mentioning "types" makes
me thinnk you're talking about the actual YAML tags, which usually
have typing information, but actually you seem to be talking about the
expected members of the top level YAML mapping.

Speaking of YAML tags, we really should specify the YAML schema we're
using here.  From the rest of the stuff, I'm assuming it's the "Core"
schema described in section 10.3 of the YAML spec.  That seems a
reasonable first choice to me, although eventually we may want to
define our own YAML schema.

> +* [R] id: unique identifier in property form (e.g. skel-device)
> +
> +* [R] title: title of the binding
> +
> +* [O] maintainer: sequence of maintainers
> +      [R] name: name and email of maintainer or mailing list in RFC822
> +                form.
> +
> +* [O] description: full description of the binding
> +
> +* [O] compatible: sequence of valid compatible descriptors
> +      [R] name: the compatible string surrounded in double quotes
> +      [O] deprecated: a deprecated compatible string surrounded in
> +                      double quotes
> +      [O] description: description of the compatible string

So, when thinking about DT validation, I like to make a distinction
between the binding's "key" and its requirements.  By the "key"
(because I haven't thought of a better term) I mean the thing that
tells you this binding is in play.  From a validator's point of view,
given a DT node and a library of bindings, it needs to check that if
the node matches the key, then it must match all the binding's requirements.

For most bindings that will be a particular value listed in
'compatible', but there are other cases.  I'm fine if 'compatible' is
the only style of "key" we support initially, but I'd like to see a
higher level structural break between the "key" part of the binding
and the requirements.

> +* [O] required: sequence of required properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] optional: sequence of optional properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] deprecated: sequence of deprecated properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id

I'm not sure if it will work better to put require / optional
etc. properties into different sections as you have here, or if it
would be better to tag each property with "required", "optional",
"required-when" and so forth.

> +* [R] example: sequence of examples:
> +      [R] dts: DT source of example usage. The example text must use
> +               literal style ("|") so that it retains indentation and
> +               newlines.
> +      [O] description: description of the example
> +
> +Note: [R] and [O] denote required and optional fields, respectively.
> +
> +Skeleton Binding
> +----------------
> +
> +The skeleton.yaml binding found in the top of the DT binding tree
> +is the canonical example of syntax and format to use when writing
> +a DT binding document. It is maintained with the latest formatting
> +conventions, making it the best starting point when writing a new DT
> +binding.

So the other thing I'd like to see in our very first implementation is
at least minimal checking of the binding itself.  Initially, just
checking that it is valid YAML would be enough, I think.  But the
quoting and nesting rules in YAML are pretty subtle, so I could easily
see someone writing something based on existing examples that looks ok
to a human, but doesn't quite meet the YAML spec rules.

On Tue, Sep 01, 2015 at 09:59:14AM -0700, Tim Bird wrote:
> On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter@konsulko.com> wrote:
> > Documentation explaining the syntax and format of the YAML-based DT binding
> > documentation.
> >
> > Signed-off-by: Matt Porter <mporter@konsulko.com>
> > ---
> >  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
> >  1 file changed, 106 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> >
> > diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> > new file mode 100644
> > index 0000000..f9acc22
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> > @@ -0,0 +1,106 @@
> > +--------------------------
> > +Device Tree Binding Format
> > +--------------------------
> > +
> > +Background
> > +----------
> > +
> > +DT bindings historically were written as text in prose format which
> > +led to issues in usability of that source documentation. Some of
> > +these issues include the need to programmatically process binding
> > +source documentation to do DTS validation, perform mass updates to
> > +format/style, and to generate publishable documentation in HTML or
> > +PDF form.
> > +
> > +Overview
> > +--------
> > +
> > +The DT binding format is based on the YAML text markup language.
> > +Although there are many text markup options available, YAML
> > +fulfills all requirements considered for a DT binding source format
> > +which include:
> > +
> > +1) Must be human readable
> > +2) Must be easily translated to other data formats (XML, JSON, etc).
> > +3) Must have sufficient tools and libraries to enable developers to
> > +   build new tools for DT binding processing
> > +4) Must have a complete spec to refer to syntax
> > +
> > +YAML is documentated in the specification found at
> > +http://www.yaml.org/spec/1.2/spec.html
> > +
> > +The required YAML DT binding tag format and syntax are defined in
> > +the following sections.
> > +
> > +YAML DT Binding Syntax
> > +----------------------
> > +
> > +* Lines starting with "#" are comments and not part of the binding itself
> > +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> 
> I think it would be good to specify a DT Binding Format version as well.
> I would expect we may expand the syntax, or we may alter it, so it
> would be good to have a number for our specific syntax.  This would
> help in some circumstances, if we make multiple passes over the
> documents to achieve our conversion.  Maybe this should be added
> below.

Good point.  In fact, I think it's absolutely essential that we
include that.  This is a complicated business and we're bound to make
mistakes in the binding definition format - a version number gives us
a way to fix them without requiring changing all the existing bindings
at once.

[snip]
> > +* [O] compatible: sequence of valid compatible descriptors
> > +      [R] name: the compatible string surrounded in double quotes
> > +      [O] deprecated: a deprecated compatible string surrounded in
> > +                      double quotes
> > +      [O] description: description of the compatible string
> 
> compatible is a property like all the others, and I think it's confusing
> to have it in a separate section.  For example, when listing other
> properties, the "name:" field specifies the property name, but for
> the compatible section the "name:" field specifies the possible values
> for the compatible property.  This is (at least to me) quite confusing.
> 
> Can we make this a regular property, with an option 'deprecated' attribute.
> I guess we may need to distinguish between deprecated properties and
> deprecated vales of properties.
> The above 'deprecated' specifies deprecated values for the compatible
> property, while
> the section below specifies deprecated properties.  I don't see a way
> to indicate a deprecated
> value for a still-existing property, but that may be too much detail for now.

So, I think folding compatible in with the rest is a mistake.  There
are two important distinctions between compatible and the other
property descriptions:

1) (See my other post for context) The most important is that
compatible is the "key" for the binding.  It's what lets you know the
binding is in play, rather than just saying what the requirements are
once you know it's a relevant binding.

2) The condition we're typically listing for compatible is that it
must contain the given compatible string - it's valid and common to
also have additional compatible strings (e.g. for the future,
backwards-compatible revision of this device).  For most other
properties we're describing the *whole* value it should have.

Eventually I think we'll want a general way of expressing such
"equals" versus "contains" conditions, but that's a pretty hard
problem, so for the first cut I think it's reasonable to special case
'compatible'.


Incidentally, note that it would also be quite common for 'compatible'
to appear in *both* the "key" and the requirements.  e.g. the binding
for "foo,bardev-revision-b" could require that compatible also list
"foo,bardev-revision-a" if it's backwards compatible.

> > +* [O] required: sequence of required properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> 
> I would recommend we add types and value constraints to the
> binding schema:
> 
> [O] type: type of value for this property, and if specified must be
> one of the following (without quotes):
> "string", "number", "array", "phandle"

It might be worth re-using the type names used in old OF documents
here.  Although "prop-encoded-array" tends to cover an awful lot there.

> [O] value constraint: specifies a limit on the value for a property.
> This can be specified as English text, or be a range or choice constraint.
> The latter two should be expressed with the following exact english
> phrases (without quotes):
> "must be in the range x to y, inclusive"
> where x and y are numbers
> "must be one of <choice 1>, <choice 2>, ..."
> where <choice x> is a string, which should be quoted
> if it includes a comma.
> 
> Any other constraint is expressed in free-form English text

Requiring exact english phrases when we're already have a flexible
encoding scheme with YAML is pretty horrid.

I'd suggest instead that English constraints just go in the
description.  We can add new fields for specific types of machine
verifiable constraints as we implement them (e.g. "max", "min",
"allowed-values").

> I would recommend adding these to this binding schema doc, but
> not requiring them on the first conversion pass of the docs.
> 
> Basically my idea is that you want the each pass over the binding
> documents to be as painless as possible.  Every time a user encounters
> something they're not sure about, it will slow them down.  There will
> be lots of ambiguous situations, and we want the process to be as
> lossless as possible, so I would recommend a strategy of doing as
> much as people can, and if in doubt just put any outstanding text
> in the descriptions.  The descriptions will always be free-form text,
> and we can come along later and convert these to types, constraints,
> as we see fit.
> 
> > +* [O] optional: sequence of optional properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] deprecated: sequence of deprecated properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [R] example: sequence of examples:
> > +      [R] dts: DT source of example usage. The example text must use
> > +               literal style ("|") so that it retains indentation and
> > +               newlines.
> > +      [O] description: description of the example
> > +
> 
> I'd recommend having a top-level notes section:
> * [O] note: any other notes about this node that are not covered by
> preceding property descriptions.

Good call.  Although I'd suggest "description" to match the name used
elsewhere.

> > +Note: [R] and [O] denote required and optional fields, respectively.
> 
> I'd put this note before the format description.
> 
>  -- Tim
> 
> > +
> > +Skeleton Binding
> > +----------------
> > +
> > +The skeleton.yaml binding found in the top of the DT binding tree
> > +is the canonical example of syntax and format to use when writing
> > +a DT binding document. It is maintained with the latest formatting
> > +conventions, making it the best starting point when writing a new DT
> > +binding.
> > --

On Sun, Aug 30, 2015 at 03:04:33PM -0700, Frank Rowand wrote:
> On 8/27/2015 10:23 PM, Matt Porter wrote:
> > Documentation explaining the syntax and format of the YAML-based DT binding
> > documentation.
> > 
> > Signed-off-by: Matt Porter <mporter@konsulko.com>
> > ---
> >  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
> >  1 file changed, 106 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> > new file mode 100644
> > index 0000000..f9acc22
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> > @@ -0,0 +1,106 @@
> > +--------------------------
> > +Device Tree Binding Format
> > +--------------------------
> > +
> > +Background
> > +----------
> > +
> > +DT bindings historically were written as text in prose format which
> > +led to issues in usability of that source documentation. Some of
> > +these issues include the need to programmatically process binding
> > +source documentation to do DTS validation, perform mass updates to
> > +format/style, and to generate publishable documentation in HTML or
> > +PDF form.
> > +
> > +Overview
> > +--------
> > +
> > +The DT binding format is based on the YAML text markup language.
> > +Although there are many text markup options available, YAML
> > +fulfills all requirements considered for a DT binding source format
> > +which include:
> > +
> > +1) Must be human readable
> > +2) Must be easily translated to other data formats (XML, JSON, etc).
> > +3) Must have sufficient tools and libraries to enable developers to
> > +   build new tools for DT binding processing
> > +4) Must have a complete spec to refer to syntax
> > +
> > +YAML is documentated in the specification found at
> > +http://www.yaml.org/spec/1.2/spec.html
> > +
> > +The required YAML DT binding tag format and syntax are defined in
> > +the following sections.
> > +
> > +YAML DT Binding Syntax
> > +----------------------
> > +
> > +* Lines starting with "#" are comments and not part of the binding itself
> > +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> > +* "---" starts a binding document
> > +* "..." ends a binding document
> > +* Multiple binding documents may exist in a single file
> > +* Tabs are not permitted
> > +* Scope is denoted by indentation of two spaces
> > +* Key value pairs are denoted by "key: value"
> > +* Sequences are denoted by "- "
> > +* Scalar values may convert newlines to spaces and preserve blank
> > +  lines for long description formatting using ">"
> > +* Scalar values may escape all reserved characters and preserve
> > +  newlines by using "|" to denote literal style
> > +
> > +For additional information on YAML syntax, refer to the specification
> > +at http://www.yaml.org/spec/1.2/spec.html
> > +
> > +YAML DT Binding Format
> > +----------------------
> > +
> > +The following YAML types are supported in the DT binding format:
> > +
> > +* [R] id: unique identifier in property form (e.g. skel-device)
> > +
> > +* [R] title: title of the binding
> > +
> > +* [O] maintainer: sequence of maintainers
> > +      [R] name: name and email of maintainer or mailing list in RFC822
> > +                form.
> > +
> > +* [O] description: full description of the binding
> > +
> > +* [O] compatible: sequence of valid compatible descriptors
> > +      [R] name: the compatible string surrounded in double quotes
> > +      [O] deprecated: a deprecated compatible string surrounded in
> > +                      double quotes
> > +      [O] description: description of the compatible string
> > +
> > +* [O] required: sequence of required properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] optional: sequence of optional properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] deprecated: sequence of deprecated properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> 
> I commented in reply to patch 0 that we should think through how
> this structure will support adding new features as the next step
> after converting existing bindings.
> 
> The specific case I started thinking about was the distinction between
> required, optional, required-if, and optional-if.  A property might
> be required in all cases, optional in all cases, required in some
> specified cases, and/or optional in some specified cases.  So a
> property could be both "required-if" and "optional-if".  Or it
> might just be one of the two.
 
Ok, so this is similar to the issue Rob points out about AND and
ORs of compatible strings.

> Here is where my YAML ignorance jumps out.  I really do not know the
> best way to approach it.  And I do not know if the syntax I am using
> is even remotely correct - I am trying to extrapolate from the examples
> in the spec, without having read past the examples yet.  My naive knee
> jerk alternatives where (by example, because I can explain it more
> concisely that way instead of the way it should be described in this file):

I will confess that I'm not an expert either so we can learn together.
:)

> 
> ---- (1)
> 
> required:
>   - &P1
>     name: "prop1"
>     required-if: < some condition >
> 
> optional:
>   - *P1
>     optional-if: < some other condition>
>   - name: "prop2"
>     required: true
>   - name: "prop3"
>     optional: true
> 

Couple concerns here. First, we have to consider properties being
inherited from a generic binding rather than repeating them in
a peripheral binding. The conditions in which a generic property
may be required can be different in a child peripheral binding.
So you may have a property inhereit that is optional in the generic
binding, but is required in a particular child peripheral binding.

Now, having said that, if we make use of the anchor/alias syntax
as above, now we have to have unique anchor names across all generic
bindings. Further, we have this issue of yaml parsers not having
the ability now to directly include other files. Certainly we could
make a wrapper to pull in each referenced generic binding into a
single yaml document to be run through the parser.

The second thing is that in the example above we still have
the required-if tag populated within the optional section which
is not ideal.

> ---- (2)
> 
> property:
>   - name: "prop1"
>     required-if: < some condition >
>     optional-if: < some other condition >
>   - name: "prop2"
>     required: true
>   - name: "prop3"
>     optional: true

This looks better to me. If coupled with the C like constraint
expressions it should work well for most cases.

I'm thinking of the resource-names.txt common case such as:

property:
  - name: "reg"
    required-if: iface(mmio)
  - name "reg-names"
    required-if: prop(reg)

Keep in mind I'm not sure yet what we'll allow in these conditional
statements. I just believe that we'll want the same syntax for
constraints and property conditionals

> 
> 
> In a related question, I commented in my reply to patch 0 that some of
> the information about the compatible got lost in the conversion from
> the existing binding to the YAML format.  That led me to wonder why
> the compatible property should be at the YAML top level (same as
> "optional" and "required").  My (again naive, YAML amateur) thought
> is that the form should be something like:
> 
> [O] property:
>     [R] optional: | required: | required-if: | optional-if: | required-if: optional-if:
>     [O] deprecated: a deprecated compatible string surrounded in
>                     double quotes | ???
>     [R] name: name of the property surrounded in double quotes
>     [R] description: description of the property
>     [O] reference: optional reference to a binding id

TBH, compatible started getting treated separately from other properties
simply because of my initial take on readability in the yaml source.
There's technically no reason it can't be folded back in as yet another
property as above. However, we need a way to show ands and ors for
compatibles in a sane manner. Consider a device where the generic
compatible string from the original part in the family is gendev1001
and subsequent family members requiring tweaks exist.

property:
  - name: "foo,gendev1001"
    required: true
  - name: "foo,gendev1005"
    required-or: true
  - name: "foo,gendev1006"
    required-or: true
  - name: "foo,gendev1007"
    required-or: true

Not sure here, but that's one way to do it with the keys driving it
versus a conditional expression.

or

property:
  - name: "foo,gendev1001"
    required: true
  - name: "foo,gendev1005" || "foo,gendev1006" || "foo,gendev1007"
    required-if: !"foo,gendev1001"

which seems to look a lot better to me. A
not-yet-existing-smart-validator would check that a dts had:

compatible = "foo,gendev100[5|6|7]", "foo,gendev1001"

or simply

compatible = "foo,gendev1001"

> Apologies for bizarre format for "optional:", but for the purposes of this
> comment, hopefully my intent will be clear.
> 
> Then "compatible" becomes just another "property:", just as it is in the
> .dts, but any processing or analysis tool knows that the name "compatible"
> is used in a very specific way and has magical meaning for .dts.
> 
> Then if compatible is just another "property:" then the information that
> lost was in the conversion process would not be lost, it would appear
> in "description:".

Sounds good.

> 
> 
> > +
> > +* [R] example: sequence of examples:
> > +      [R] dts: DT source of example usage. The example text must use
> > +               literal style ("|") so that it retains indentation and
> > +               newlines.
> > +      [O] description: description of the example
> > +
> > +Note: [R] and [O] denote required and optional fields, respectively.
> > +
> > +Skeleton Binding
> > +----------------
> > +
> > +The skeleton.yaml binding found in the top of the DT binding tree
> > +is the canonical example of syntax and format to use when writing
> > +a DT binding document. It is maintained with the latest formatting
> > +conventions, making it the best starting point when writing a new DT
> > +binding.
> 
> It would be good to have a comment of YAML style.  Looking at the spec,
> there appear to be multiple ways to format content that is functionally
> equivalent.  For example:
> 
>    # a flow sequence
>    - [name        , hr, avg  ]
>    - [Mark McGwire, 65, 0.278]
>    - [Sammy Sosa  , 63, 0.288]
> 
> vs.
> 
>    # a flow mapping
>    Mark McGwire: {hr: 65, avg: 0.278}
>    Sammy Sosa: {
>        hr: 63,
>        avg: 0.288
>      }
> 
> This is my first quick reading of the YAML spec, so maybe a flow sequence and
> a flow mapping are not equivalent, but that was my naive impression.

The examples section in the intro is incomplete and glosses over some
stuff such that it's expected to get confused by this part. I got lost
there at first as well.

The quick summary is that YAML allows either Flow Style or Block Style
of describing nodes. The two example above use flow style as described
in Chapter 7 to show a sequence construct and separately a mapping
construct. I chose to use block style through the proposed YAML dt
binding format as it's my personal preference for human readability.
Compare, for example, the format of doing sequences in flow style
(section 7.4.1) versus block style (section 8.2.1) and you'll see that
I'm making heavy use of block style sequences throughout this format.

> I think that in the interest of human readability (requirement 1), that a
> specific formatting style should be specified.  That will make life easier
> for people who do not know YAML, but are (1) merely copying from existing binding
> docs to create new ones, or (2) trying to understand how different bindings
> differ.

Agreed. I'll explicitly mention that we use block style formatting now.
Also, David asked that we explicitly mention the YAML schema used (which
is the default core schema that also parsers have to implement).

-Matt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Sep 01, 2015 at 09:59:14AM -0700, Tim Bird wrote:
> On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter@konsulko.com> wrote:
> > Documentation explaining the syntax and format of the YAML-based DT binding
> > documentation.
> >
> > Signed-off-by: Matt Porter <mporter@konsulko.com>
> > ---
> >  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
> >  1 file changed, 106 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> >
> > diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> > new file mode 100644
> > index 0000000..f9acc22
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> > @@ -0,0 +1,106 @@
> > +--------------------------
> > +Device Tree Binding Format
> > +--------------------------
> > +
> > +Background
> > +----------
> > +
> > +DT bindings historically were written as text in prose format which
> > +led to issues in usability of that source documentation. Some of
> > +these issues include the need to programmatically process binding
> > +source documentation to do DTS validation, perform mass updates to
> > +format/style, and to generate publishable documentation in HTML or
> > +PDF form.
> > +
> > +Overview
> > +--------
> > +
> > +The DT binding format is based on the YAML text markup language.
> > +Although there are many text markup options available, YAML
> > +fulfills all requirements considered for a DT binding source format
> > +which include:
> > +
> > +1) Must be human readable
> > +2) Must be easily translated to other data formats (XML, JSON, etc).
> > +3) Must have sufficient tools and libraries to enable developers to
> > +   build new tools for DT binding processing
> > +4) Must have a complete spec to refer to syntax
> > +
> > +YAML is documentated in the specification found at
> > +http://www.yaml.org/spec/1.2/spec.html
> > +
> > +The required YAML DT binding tag format and syntax are defined in
> > +the following sections.
> > +
> > +YAML DT Binding Syntax
> > +----------------------
> > +
> > +* Lines starting with "#" are comments and not part of the binding itself
> > +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> 
> I think it would be good to specify a DT Binding Format version as well.
> I would expect we may expand the syntax, or we may alter it, so it
> would be good to have a number for our specific syntax.  This would
> help in some circumstances, if we make multiple passes over the
> documents to achieve our conversion.  Maybe this should be added
> below.

Added this in v2, thanks.

> 
> > +* "---" starts a binding document
> > +* "..." ends a binding document
> > +* Multiple binding documents may exist in a single file
> > +* Tabs are not permitted
> > +* Scope is denoted by indentation of two spaces
> > +* Key value pairs are denoted by "key: value"
> > +* Sequences are denoted by "- "
> > +* Scalar values may convert newlines to spaces and preserve blank
> > +  lines for long description formatting using ">"
> > +* Scalar values may escape all reserved characters and preserve
> > +  newlines by using "|" to denote literal style
> > +
> > +For additional information on YAML syntax, refer to the specification
> > +at http://www.yaml.org/spec/1.2/spec.html
> > +
> > +YAML DT Binding Format
> > +----------------------
> > +
> > +The following YAML types are supported in the DT binding format:
> > +
> 
> Based on my comment above, I recommend:
> [R] binding-format: 1
> 
> > +* [R] id: unique identifier in property form (e.g. skel-device)
> > +
> > +* [R] title: title of the binding
> > +
> > +* [O] maintainer: sequence of maintainers
> > +      [R] name: name and email of maintainer or mailing list in RFC822
> > +                form.
> > +
> > +* [O] description: full description of the binding
> > +
> > +* [O] compatible: sequence of valid compatible descriptors
> > +      [R] name: the compatible string surrounded in double quotes
> > +      [O] deprecated: a deprecated compatible string surrounded in
> > +                      double quotes
> > +      [O] description: description of the compatible string
> 
> compatible is a property like all the others, and I think it's confusing
> to have it in a separate section.  For example, when listing other
> properties, the "name:" field specifies the property name, but for
> the compatible section the "name:" field specifies the possible values
> for the compatible property.  This is (at least to me) quite confusing.
> 
> Can we make this a regular property, with an option 'deprecated' attribute.
> I guess we may need to distinguish between deprecated properties and
> deprecated vales of properties.
> The above 'deprecated' specifies deprecated values for the compatible
> property, while
> the section below specifies deprecated properties.  I don't see a way
> to indicate a deprecated
> value for a still-existing property, but that may be too much detail for now.

I've made an attempt at this in v2. There's some ugly fall-out I haven't
decided the best way to handle. That is, I can show a allowable
compatible strings including a specific, generic type syntax using
C-like expressions. However, showing deprecated compatible requires
a special key. I'll be interested if anybody can suggestion something
cleaner. There's definitely a trade-off in folding compatibles into
properties

> 
> 
> > +* [O] required: sequence of required properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> 
> I would recommend we add types and value constraints to the
> binding schema:
> 
> [O] type: type of value for this property, and if specified must be
> one of the following (without quotes):
> "string", "number", "array", "phandle"

I modified this a bit to be "string", "int", "array", "phandle", and
"empty". I find "empty" to be useful for those properties and a
validator will find that useful to validate those boolean type
property switches it encounters i.e. skip constraint checking

> [O] value constraint: specifies a limit on the value for a property.
> This can be specified as English text, or be a range or choice constraint.
> The latter two should be expressed with the following exact english
> phrases (without quotes):
> "must be in the range x to y, inclusive"
> where x and y are numbers
> "must be one of <choice 1>, <choice 2>, ..."
> where <choice x> is a string, which should be quoted
> if it includes a comma.

Got them, but Pantelis' C syntax suggestion for constraints seemed
popular and flexible so I went with that initially, please let me
know what you think of that.

> 
> Any other constraint is expressed in free-form English text

Yes, I believe the description field can hold this information
until bindings are fully converted with constraint info in the 
C-syntax (if we stick with that).

> I would recommend adding these to this binding schema doc, but
> not requiring them on the first conversion pass of the docs.

Sounds good.

> Basically my idea is that you want the each pass over the binding
> documents to be as painless as possible.  Every time a user encounters
> something they're not sure about, it will slow them down.  There will
> be lots of ambiguous situations, and we want the process to be as
> lossless as possible, so I would recommend a strategy of doing as
> much as people can, and if in doubt just put any outstanding text
> in the descriptions.  The descriptions will always be free-form text,
> and we can come along later and convert these to types, constraints,
> as we see fit.

Agreed!

> > +* [O] optional: sequence of optional properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] deprecated: sequence of deprecated properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [R] example: sequence of examples:
> > +      [R] dts: DT source of example usage. The example text must use
> > +               literal style ("|") so that it retains indentation and
> > +               newlines.
> > +      [O] description: description of the example
> > +
> 
> I'd recommend having a top-level notes section:
> * [O] note: any other notes about this node that are not covered by
> preceding property descriptions.

Added in v2, thanks.

> 
> > +Note: [R] and [O] denote required and optional fields, respectively.
> 
> I'd put this note before the format description.

Done, thanks.

> 
>  -- Tim
> 
> > +
> > +Skeleton Binding
> > +----------------
> > +
> > +The skeleton.yaml binding found in the top of the DT binding tree
> > +is the canonical example of syntax and format to use when writing
> > +a DT binding document. It is maintained with the latest formatting
> > +conventions, making it the best starting point when writing a new DT
> > +binding.
> > --
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html