[v4,1/8] dmaengine: of_dma: Support for DMA routers

DMA routers are transparent devices used to mux DMA requests from
peripherals to DMA controllers. They are used when the SoC integrates more
devices with DMA requests then their controller can handle.
DRA7x is one example of such SoC, where the sDMA can hanlde 128 DMA request
lines, but in SoC level it has 205 DMA requests.

The of_dma_router will be registered as of_dma_controller with special
xlate function and additional parameters. The driver for the router is
responsible to craft the dma_spec (in the of_dma_route_allocate callback)
which can be used to requests a DMA channel from the real DMA controller.
This way the router can be transparent for the system while remaining generic
enough to be used in different environments.

Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
---
 Documentation/devicetree/bindings/dma/dma.txt | 28 +++++++++
 drivers/dma/dmaengine.c                       |  7 +++
 drivers/dma/of-dma.c                          | 86 +++++++++++++++++++++++++++
 include/linux/dmaengine.h                     | 17 ++++++
 include/linux/of_dma.h                        | 21 +++++++
 5 files changed, 159 insertions(+)

Hi,

On Wed, Apr 08, 2015 at 04:14:45PM +0300, Peter Ujfalusi wrote:
> DMA routers are transparent devices used to mux DMA requests from
> peripherals to DMA controllers. They are used when the SoC integrates more
> devices with DMA requests then their controller can handle.
> DRA7x is one example of such SoC, where the sDMA can hanlde 128 DMA request
> lines, but in SoC level it has 205 DMA requests.
> 
> The of_dma_router will be registered as of_dma_controller with special
> xlate function and additional parameters. The driver for the router is
> responsible to craft the dma_spec (in the of_dma_route_allocate callback)
> which can be used to requests a DMA channel from the real DMA controller.
> This way the router can be transparent for the system while remaining generic
> enough to be used in different environments.
> 
> Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
> ---
>  Documentation/devicetree/bindings/dma/dma.txt | 28 +++++++++
>  drivers/dma/dmaengine.c                       |  7 +++
>  drivers/dma/of-dma.c                          | 86 +++++++++++++++++++++++++++
>  include/linux/dmaengine.h                     | 17 ++++++
>  include/linux/of_dma.h                        | 21 +++++++
>  5 files changed, 159 insertions(+)

Can that be moved to a header / C file of its own?

There's a lot of various code already in dmaengine.h and dmaengine.c,
it would be really great to avoid adding more random stuff in there.

> 
> diff --git a/Documentation/devicetree/bindings/dma/dma.txt b/Documentation/devicetree/bindings/dma/dma.txt
> index 82104271e754..6312fb00ce8d 100644
> --- a/Documentation/devicetree/bindings/dma/dma.txt
> +++ b/Documentation/devicetree/bindings/dma/dma.txt
> @@ -31,6 +31,34 @@ Example:
>  		dma-requests = <127>;
>  	};
>  
> +* DMA router
> +
> +DMA routers are transparent IP blocks used to route DMA request lines from
> +devices to the DMA controller. Some SoCs (like TI DRA7x) have more peripherals
> +integrated with DMA requests than what the DMA controller can handle directly.
> +
> +Required property:
> +- dma-masters:		phandle of the DMA controller or list of phandles for
> +			the DMA controllers the router can direct the signal to.
> +- #dma-cells: 		Must be at least 1. Used to provide DMA router specific
> +			information. See DMA client binding below for more
> +			details.
> +
> +Optional properties:
> +- dma-requests: 	Number of incoming request lines the router can handle.
> +- In the node pointed by the dma-masters:
> +	- dma-requests:	The router driver might need to look for this in order
> +			to configure the routing.
> +
> +Example:
> +	sdma_xbar: dma-router@4a002b78 {
> +		compatible = "ti,dra7-dma-crossbar";
> +		reg = <0x4a002b78 0xfc>;
> +		#dma-cells = <1>;
> +		dma-requests = <205>;
> +		ti,dma-safe-map = <0>;
> +		dma-masters = <&sdma>;
> +	};
>  
>  * DMA client
>  
> diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
> index 24967c89f5d4..c7aa232b4a59 100644
> --- a/drivers/dma/dmaengine.c
> +++ b/drivers/dma/dmaengine.c
> @@ -267,6 +267,13 @@ static void dma_chan_put(struct dma_chan *chan)
>  	/* This channel is not in use anymore, free it */
>  	if (!chan->client_count && chan->device->device_free_chan_resources)
>  		chan->device->device_free_chan_resources(chan);
> +
> +	/* If the channel is used via a DMA request router, free the mapping */
> +	if (chan->router && chan->router->route_free) {
> +		chan->router->route_free(chan->router->dev, chan->route_data);
> +		chan->router = NULL;
> +		chan->route_data = NULL;
> +	}
>  }
>  
>  enum dma_status dma_sync_wait(struct dma_chan *chan, dma_cookie_t cookie)
> diff --git a/drivers/dma/of-dma.c b/drivers/dma/of-dma.c
> index cbd4a8aff120..98b3b61624f0 100644
> --- a/drivers/dma/of-dma.c
> +++ b/drivers/dma/of-dma.c
> @@ -45,6 +45,47 @@ static struct of_dma *of_dma_find_controller(struct of_phandle_args *dma_spec)
>  }
>  
>  /**
> + * of_dma_router_xlate - translation function for router devices
> + * @dma_spec:	pointer to DMA specifier as found in the device tree
> + * @of_dma:	pointer to DMA controller data (router information)
> + *
> + * The function creates new dma_spec to be passed to the router driver's
> + * of_dma_route_allocate() function to prepare a dma_spec which will be used
> + * to request channel from the real DMA controller.
> + */
> +static struct dma_chan *of_dma_router_xlate(struct of_phandle_args *dma_spec,
> +					    struct of_dma *ofdma)
> +{
> +	struct dma_chan		*chan;
> +	struct of_dma		*ofdma_target;
> +	struct of_phandle_args	dma_spec_target;
> +	void			*route_data;
> +
> +	/* translate the request for the real DMA controller */
> +	memcpy(&dma_spec_target, dma_spec, sizeof(dma_spec_target));
> +	route_data = ofdma->of_dma_route_allocate(&dma_spec_target, ofdma);
> +
> +	ofdma_target = of_dma_find_controller(&dma_spec_target);
> +	if (!ofdma_target)
> +		return NULL;
> +
> +	chan = ofdma_target->of_dma_xlate(&dma_spec_target, ofdma_target);
> +	if (chan) {
> +		chan->router = ofdma->dma_router;
> +		chan->route_data = route_data;
> +	} else {
> +		ofdma->dma_router->route_free(ofdma->dma_router->dev, route_data);

This line is over 80 characters.

> +	}
> +
> +	/*
> +	 * Need to put the node back since the ofdma->of_dma_route_allocate
> +	 * has taken it for generating the new, translated dma_spec
> +	 */
> +	of_node_put(dma_spec_target.np);
> +	return chan;
> +}
> +
> +/**
>   * of_dma_controller_register - Register a DMA controller to DT DMA helpers
>   * @np:			device node of DMA controller
>   * @of_dma_xlate:	translation function which converts a phandle
> @@ -110,6 +151,51 @@ void of_dma_controller_free(struct device_node *np)
>  EXPORT_SYMBOL_GPL(of_dma_controller_free);
>  
>  /**
> + * of_dma_router_register - Register a DMA router to DT DMA helpers as a
> + *			    controller
> + * @np:				device node of DMA router
> + * @of_dma_route_allocate:	setup function for the router which need to
> + *				modify the dma_spec for the DMA controller to
> + *				use and to set up the requested route.
> + * @dma_router:			pointer to dma_router structure to be used when
> + *				the route need to be free up.
> + *
> + * Returns 0 on success or appropriate errno value on error.
> + *
> + * Allocated memory should be freed with appropriate of_dma_controller_free()
> + * call.
> + */
> +int of_dma_router_register(struct device_node *np,
> +			   void *(*of_dma_route_allocate)
> +			   (struct of_phandle_args *, struct of_dma *),
> +			   struct dma_router *dma_router)
> +{
> +	struct of_dma	*ofdma;
> +
> +	if (!np || !of_dma_route_allocate || !dma_router) {
> +		pr_err("%s: not enough information provided\n", __func__);
> +		return -EINVAL;
> +	}
> +
> +	ofdma = kzalloc(sizeof(*ofdma), GFP_KERNEL);
> +	if (!ofdma)
> +		return -ENOMEM;

Is that expected that you allocate through kzalloc, but never have a
matching free function implemented?

> +	ofdma->of_node = np;
> +	ofdma->of_dma_xlate = of_dma_router_xlate;
> +	ofdma->of_dma_route_allocate = of_dma_route_allocate;
> +	ofdma->dma_router = dma_router;
> +
> +	/* Now queue of_dma controller structure in list */
> +	mutex_lock(&of_dma_lock);
> +	list_add_tail(&ofdma->of_dma_controllers, &of_dma_list);
> +	mutex_unlock(&of_dma_lock);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(of_dma_router_register);
> +
> +/**
>   * of_dma_match_channel - Check if a DMA specifier matches name
>   * @np:		device node to look for DMA channels
>   * @name:	channel name to be matched
> diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
> index ad419757241f..abf63ceabef9 100644
> --- a/include/linux/dmaengine.h
> +++ b/include/linux/dmaengine.h
> @@ -222,6 +222,16 @@ struct dma_chan_percpu {
>  };
>  
>  /**
> + * struct dma_router - DMA router structure
> + * @dev: pointer to the DMA router device
> + * @route_free: function to be called when the route can be disconnected
> + */
> +struct dma_router {
> +	struct device *dev;
> +	void (*route_free)(struct device *dev, void *route_data);
> +};
> +
> +/**
>   * struct dma_chan - devices supply DMA channels, clients use them
>   * @device: ptr to the dma device who supplies this channel, always !%NULL
>   * @cookie: last cookie value returned to client
> @@ -232,6 +242,8 @@ struct dma_chan_percpu {
>   * @local: per-cpu pointer to a struct dma_chan_percpu
>   * @client_count: how many clients are using this channel
>   * @table_count: number of appearances in the mem-to-mem allocation table
> + * @router: pointer to the DMA router structure
> + * @route_data: channel specific data for the router
>   * @private: private data for certain client-channel associations
>   */
>  struct dma_chan {
> @@ -247,6 +259,11 @@ struct dma_chan {
>  	struct dma_chan_percpu __percpu *local;
>  	int client_count;
>  	int table_count;
> +
> +	/* DMA router */
> +	struct dma_router *router;
> +	void *route_data;
> +
>  	void *private;
>  };
>  
> diff --git a/include/linux/of_dma.h b/include/linux/of_dma.h
> index 56bc026c143f..734e449f87c1 100644
> --- a/include/linux/of_dma.h
> +++ b/include/linux/of_dma.h
> @@ -23,6 +23,9 @@ struct of_dma {
>  	struct device_node	*of_node;
>  	struct dma_chan		*(*of_dma_xlate)
>  				(struct of_phandle_args *, struct of_dma *);
> +	void			*(*of_dma_route_allocate)
> +				(struct of_phandle_args *, struct of_dma *);
> +	struct dma_router	*dma_router;

I don't really see why this is really tied to the device tree.

Couldn't we use the device_alloc_chan_resources to do that?

Thanks,
Maxime

On 04/08/2015 06:42 PM, Maxime Ripard wrote:
>> ---
>>  Documentation/devicetree/bindings/dma/dma.txt | 28 +++++++++
>>  drivers/dma/dmaengine.c                       |  7 +++
>>  drivers/dma/of-dma.c                          | 86 +++++++++++++++++++++++++++
>>  include/linux/dmaengine.h                     | 17 ++++++
>>  include/linux/of_dma.h                        | 21 +++++++
>>  5 files changed, 159 insertions(+)
> 
> Can that be moved to a header / C file of its own?
> 
> There's a lot of various code already in dmaengine.h and dmaengine.c,
> it would be really great to avoid adding more random stuff in there.

This patch adds the core support for DMA signal routers. It adds fairly small
amount of generic code to the core to achieve this. I don't think it would be
better to create let's say of-dma-router.c and .h just for this and export
functions from of-dma.c to be used outside of the file.

>> +	chan = ofdma_target->of_dma_xlate(&dma_spec_target, ofdma_target);
>> +	if (chan) {
>> +		chan->router = ofdma->dma_router;
>> +		chan->route_data = route_data;
>> +	} else {
>> +		ofdma->dma_router->route_free(ofdma->dma_router->dev, route_data);
> 
> This line is over 80 characters.

Oh, true.

>> +int of_dma_router_register(struct device_node *np,
>> +			   void *(*of_dma_route_allocate)
>> +			   (struct of_phandle_args *, struct of_dma *),
>> +			   struct dma_router *dma_router)
>> +{
>> +	struct of_dma	*ofdma;
>> +
>> +	if (!np || !of_dma_route_allocate || !dma_router) {
>> +		pr_err("%s: not enough information provided\n", __func__);
>> +		return -EINVAL;
>> +	}
>> +
>> +	ofdma = kzalloc(sizeof(*ofdma), GFP_KERNEL);
>> +	if (!ofdma)
>> +		return -ENOMEM;
> 
> Is that expected that you allocate through kzalloc, but never have a
> matching free function implemented?

The free is via the of_dma_router_free() in case the router is removed
runtime, which is unlikely to happen since it will cause all DMA request to fail.

>> diff --git a/include/linux/of_dma.h b/include/linux/of_dma.h
>> index 56bc026c143f..734e449f87c1 100644
>> --- a/include/linux/of_dma.h
>> +++ b/include/linux/of_dma.h
>> @@ -23,6 +23,9 @@ struct of_dma {
>>  	struct device_node	*of_node;
>>  	struct dma_chan		*(*of_dma_xlate)
>>  				(struct of_phandle_args *, struct of_dma *);
>> +	void			*(*of_dma_route_allocate)
>> +				(struct of_phandle_args *, struct of_dma *);
>> +	struct dma_router	*dma_router;
> 
> I don't really see why this is really tied to the device tree.

The signal router is not a DMA device, it is represented in the device tree
and the code will do the needed translation, which is transparent for the DMA
clients and also for the DMA controllers. Neither should know about the signal
router.
Similar translation can be done for ACPI.

> Couldn't we use the device_alloc_chan_resources to do that?

Not really. The router itself is not a DMA controller. The routing need to be
configured before the device_alloc_chan_resources can be called for the real
DMA controller. The signal router (core part + the HW driver) need to prepare
the route and do the translation so the filter function of the DMA driver can
validate the translated request.

On Thu, Apr 09, 2015 at 11:24:58AM +0300, Peter Ujfalusi wrote:
> On 04/08/2015 06:42 PM, Maxime Ripard wrote:
> >> ---
> >>  Documentation/devicetree/bindings/dma/dma.txt | 28 +++++++++
> >>  drivers/dma/dmaengine.c                       |  7 +++
> >>  drivers/dma/of-dma.c                          | 86 +++++++++++++++++++++++++++
> >>  include/linux/dmaengine.h                     | 17 ++++++
> >>  include/linux/of_dma.h                        | 21 +++++++
> >>  5 files changed, 159 insertions(+)
> > 
> > Can that be moved to a header / C file of its own?
> > 
> > There's a lot of various code already in dmaengine.h and dmaengine.c,
> > it would be really great to avoid adding more random stuff in there.
> 
> This patch adds the core support for DMA signal routers. It adds
> fairly small amount of generic code to the core to achieve this. I
> don't think it would be better to create let's say of-dma-router.c
> and .h just for this and export functions from of-dma.c to be used
> outside of the file.

A lot of "a fairly small amount of generic code" has been added over
time, and we ended up in the current situation.

It's a bit sad if we just end up moving this just after it got merged,
especially if it doesn't have any kind of dependency on any of the
core function.

> >> +int of_dma_router_register(struct device_node *np,
> >> +			   void *(*of_dma_route_allocate)
> >> +			   (struct of_phandle_args *, struct of_dma *),
> >> +			   struct dma_router *dma_router)
> >> +{
> >> +	struct of_dma	*ofdma;
> >> +
> >> +	if (!np || !of_dma_route_allocate || !dma_router) {
> >> +		pr_err("%s: not enough information provided\n", __func__);
> >> +		return -EINVAL;
> >> +	}
> >> +
> >> +	ofdma = kzalloc(sizeof(*ofdma), GFP_KERNEL);
> >> +	if (!ofdma)
> >> +		return -ENOMEM;
> > 
> > Is that expected that you allocate through kzalloc, but never have a
> > matching free function implemented?
> 
> The free is via the of_dma_router_free() in case the router is removed
> runtime, which is unlikely to happen since it will cause all DMA request to fail.

Ok.

> >> diff --git a/include/linux/of_dma.h b/include/linux/of_dma.h
> >> index 56bc026c143f..734e449f87c1 100644
> >> --- a/include/linux/of_dma.h
> >> +++ b/include/linux/of_dma.h
> >> @@ -23,6 +23,9 @@ struct of_dma {
> >>  	struct device_node	*of_node;
> >>  	struct dma_chan		*(*of_dma_xlate)
> >>  				(struct of_phandle_args *, struct of_dma *);
> >> +	void			*(*of_dma_route_allocate)
> >> +				(struct of_phandle_args *, struct of_dma *);
> >> +	struct dma_router	*dma_router;
> > 
> > I don't really see why this is really tied to the device tree.
> 
> The signal router is not a DMA device, it is represented in the device tree
> and the code will do the needed translation, which is transparent for the DMA
> clients and also for the DMA controllers. Neither should know about the signal
> router.

Yeah, I got that part, and we do agree on that.

> Similar translation can be done for ACPI.

But this argument is exactly why it shouldn't be tied to the device
tree. We wouldn't like to re-do all this all over again for ACPI,
while your code seems to just handle that very well, wouldn't we?

> > Couldn't we use the device_alloc_chan_resources to do that?
> 
> Not really. The router itself is not a DMA controller. The routing
> need to be configured before the device_alloc_chan_resources can be
> called for the real DMA controller. The signal router (core part +
> the HW driver) need to prepare the route and do the translation so
> the filter function of the DMA driver can validate the translated
> request.

Ah, yes, hence why you need a custom xlate function. Got it, thanks!

Maxime

On 04/10/2015 10:40 AM, Maxime Ripard wrote:
> On Thu, Apr 09, 2015 at 11:24:58AM +0300, Peter Ujfalusi wrote:
>> On 04/08/2015 06:42 PM, Maxime Ripard wrote:
>>>> ---
>>>>  Documentation/devicetree/bindings/dma/dma.txt | 28 +++++++++
>>>>  drivers/dma/dmaengine.c                       |  7 +++
>>>>  drivers/dma/of-dma.c                          | 86 +++++++++++++++++++++++++++
>>>>  include/linux/dmaengine.h                     | 17 ++++++
>>>>  include/linux/of_dma.h                        | 21 +++++++
>>>>  5 files changed, 159 insertions(+)
>>>
>>> Can that be moved to a header / C file of its own?
>>>
>>> There's a lot of various code already in dmaengine.h and dmaengine.c,
>>> it would be really great to avoid adding more random stuff in there.
>>
>> This patch adds the core support for DMA signal routers. It adds
>> fairly small amount of generic code to the core to achieve this. I
>> don't think it would be better to create let's say of-dma-router.c
>> and .h just for this and export functions from of-dma.c to be used
>> outside of the file.
> 
> A lot of "a fairly small amount of generic code" has been added over
> time, and we ended up in the current situation.

So that we have fairly good dmaengine core? If people using the code, they
will eventually fix things, add new features. When dmaengine came to existence
we did not had DMA routers, now that the SoCs are getting even more
complicated they are coming (in case of DRA7x, it is already here).

> It's a bit sad if we just end up moving this just after it got merged,
> especially if it doesn't have any kind of dependency on any of the
> core function.

This provides core functionality to handle DMA signal routers via DT. It does
depend on the DT DMA parsing core code as it integrates with it.

>> The signal router is not a DMA device, it is represented in the device tree
>> and the code will do the needed translation, which is transparent for the DMA
>> clients and also for the DMA controllers. Neither should know about the signal
>> router.
> 
> Yeah, I got that part, and we do agree on that.
> 
>> Similar translation can be done for ACPI.
> 
> But this argument is exactly why it shouldn't be tied to the device
> tree. We wouldn't like to re-do all this all over again for ACPI,
> while your code seems to just handle that very well, wouldn't we?

The data structures used by the DT and ACPI layers are different. Underneath
DT and ACPI we do not have enough information on how to handle the DMA signal
router. If you happen to have multiple DMA controllers in the system for
example how would you tell how the routers are connected?
In DRA7x we have sDMA and eDMA controllers. Both have similar DMA crossbar. It
is only possible to know the hierarchy of the components via DT. I'm sure this
is the same for ACPI. You take away the ACPI layer and you loose the topology
of the system.
Probably with huge architectural change in the dmaengine core (and thus with
all the drivers) this might be possible, but I don't think that is a valid
route to take.