Architecture RFC: Field token architecture

Problem/Motivation

We need field tokens for Drupal 7. This capability used to be provided by CCK, but CCK was moved into core without this functionality, and Drupal 7 shipped without it in order to meet its release date.

Token integration for D6 was fairly simple. For the most part, fields only defined small bits of information. In Drupal 6, a node reference field token might be:

[field_node_reference-nid]

This would only provide the first value of the field; multivalue fields were not supported.

In Drupal 7, however, tokens can be chained in the following way:

[node:field_node_reference:author:mail]

This allows you to get to the email address of the author of the node that is referenced by the nodereference field. Whew!

However, there are complications:

1. Fields can contain multiple values, and this is especially relevant for taxonomy fields. So you need to reference the delta:
   

   [node:field_some_node_reference:0:author:mail]
   

2. In addition to field values, fields also contain metadata about the field itself (such as help text or description) that users might want to reference as a token. So you want to separate those, like:
   

   [node:field_some_node_reference:values:0:author:mail]
   [node:field_some_node_reference:field:machine_name]
   

3. Users are going to also want "simple" versions of fields, similar to what they had in Drupal 6, which output in a standard way without dealing with deltas.

   For example:

   [node:field_some_node_reference] == the output of the field run through field_view_field()

4. Dynamic tokens need good enough context in help. For multivalue fields, this is challenging because the Token help can only list things it knows about ahead of time, and everything else must be manually specified by the user.

   For example, looking at the UI in admin/help/token, you'll see the following for "Nodes > Date created > Custom format:"

   [node:created:custom:?]
   

   We can't go any further than this, because the elements after :custom: might be anything (some custom combination of date parts).

   We run into this with multivalue fields as well; the best we could do is:

   [node:field_name:values:?]
   

   But the actual token value might be:

   [node:field_name:values:0:node:author:created]
   

   We need to preserve a "drill down" ability for chained tokens even when the delta is unknown.

Proposed resolution

One of the first blocking issues is what to output from the token at the "top level" of the field. We suggest the following approach:

> node
  > field_attendees (depends on field name) <-- Field output from field_view_field()
     > values <-- everything under here is raw values from the database.
       > 0
         > uid = 345
         > name = User 345's name
       > 1
         > uid = 453
         > name = User 453's name
     > field <-- everything under here is metadata about the field itself.
        > field-name
        > help-text

So in Drupal 6, a Pathauto definition for this field might've been:

user/[field_attendees-uid]

In Drupal 7, it will be this instead:

user/[node:field_attendees:values:0:uid]

While this requires more typing, it is also *way* more precise, and allows you to get to *exactly* the data you want, without undue asumptions.

To handle the dynamic help bits, Dave has proposed using a base token type of "Array" (see #1195874: Need to figure out how to create nested tokens from the array token type) which could then be "extended" by field tokens. This would allow us to give a token type definition to a 'first' and/or 'last' keyword value, from which we could also provide chained token help for a field's value.

Remaining tasks

The proposed architecture in the issue summary needs to be vetted by folks with field API experience. Constructive comments welcome! :)

Solving #1195874: Need to figure out how to create nested tokens from the array token type is the biggest blocker, as we can then re-use this same mechanism for fields. This currently works for simple key => value arrays, but needs to handle a situation with chained objects. See the issue for more details.

Once that's finished, then we can adjust the current field token code to conform to the spec in this issue, utilizing the array pattern.

Also note that #1058912: Prevent recursive tokens is something that's getting encountered as well while building up token lists.

User interface changes

TODO: Fill in later.

API changes

TODO: Fill in later.