Multiple updates for doc parser

[toby.collett@…]

The attached patch is made up of the following parts:

1) fixes to the tree walker to allow the parser to run on non dojo trees (also simplifies it)

2) Adds 'tags' as a special comment field

3) Allows filtering of what is included in the output files based on the tags, private members and can strip source.

4) Allows configuration of which paths, namespace prefixes and filters are run from the command line

Tags work as follows

Idea:
        Add tags to doc parser. These could be useful for segmenting generated 
        documentation for different audiences. (eg. internal team vs clients, 
        contribs vs users)
        
        When the docparser runs it would produce a set of tags for each 
        object/function/var/...
        
        1. They could be used to filter the parser output before it was used in 
        the ApiRef so there is no overhead of downloading stuff that is 
        never/should never be displayed.
        
        2. They could be used in the display of documentation in any form, or as 
        it is converted to another form (XSD).
        
        In source, tags are specified whereever you can have a summary. Comma or 
        whitespace separated. If we can get them added to other places easily 
        then cool.
        function foo() {
          //tags: internal, special_tag
          //summary: foo function
          ...
        }
        
        In XML tags are added under different items as:
          ...  <tag>internal</tag><tag>special-tag</tag> ...
        This makes it fairly straightforward to use them in XSD or when parsing 
        with code.
        
        In JSON tags are presented as an array:
        { ... , tags: ['internal', 'special-tag'], ... }
        
        Use Case 1:
        In our map product we have a series of markermanagers that control how 
        markers are drawn. One thing they handle is whether and how markers are 
        clustered together.
        
        As a client, when you initialise the map you can specify a manager to 
        use. But apart from specifying it,
        the manager is internal and there is no client API for it. But because 
        there are a number of managers they have a documented API. Just not for 
        clients to use. So this could be tagged 'internal'.
        
        Use Case 2:
        Say use of an object or method is deprecated after V1. It should be 
        marked as such in the V2 docs and hidden by default. This item could be 
        tagged 'deprecated'.

[robert.coup@…]

Since some people have asked, I've added some brief docs on how to use the changes here. These should be added to ​http://dojo.jot.com/WikiHome/DocParserInstructions if/when the patch is accepted. Comments and suggestions are welcome!

• Parsing another namespace or set of code besides dojo. This will find all JS files under the specified path, and use the specified namespace. The default is to parse the dojo code. Once you've done this you can view the results immediately by opening /dojo/api/index.html.

  php5 -d memory_limit=100M -d error_reporting=2039 docparser.php --path=/path/to/mynamespace --namespace=mynamespace
  

• Passing --filter_source will exclude any source snippets from the JSON and XML output.
• Passing --filter_private will exclude any variables or methods named with an '_' prefix from the JSON and XML output.

Tags

Tags you can add to your code like this:

function(){
  // tags: mycustomtag, deprecated, internal_only
  // summary: Soon we will have enough treasure to rule all of New Jersey.
  // description: Or we could just get a new roomate.
  //    Look, you go find him.  He don't yell at you.
  //    All I ever try to do is make him smile and sing around
  //    him and dance around him and he just lays into me.
  //    He told me to get in the freezer 'cause there was a carnival in there.
  // returns:  Look, a Bananarama tape!
}

You can add them anywhere you can put a summary note currently (where that is we'll leave as an exercise for the reader). They can be letters, numbers and underscores and are case sensitive. These get added to a tags block in the JSON and XML output, and are currently unused by anything.

However, you can filter the docparser output using tags. For example:

• to exclude all the items tagged 'deprecated': --filter_exclude_tags=deprecated
• to include only the items tagged 'new': --filter_include_only_tags=new
• specify more than one tag for these options separated by commas.
• you can combine filter_exclude_tags and filter_include_only_tags.

[robert.coup@…]

Note that this patch applies to HEAD after [7844] which fixes some issue with parsing dojo.declare() statements.

[potatosculptor@…]

Apologies if I'm being dense, but your patch cites (revision 656). That's obviously not a dojo SVN revision #. If I switch that to 7844, tortoiseSVN informs me the "The patch seems outdated" .. the file line and patch line do not match. I've got todays HEAD, but I've also tried against 0.4.2

[toby.collett@…]

Possibly the patch was from our vendor branch, I did check it applied against head but who knows what happened, I have attached three new patches which separately add: tags, filters and namespace setting. These are definitely against head as of 3 mins ago [7906].

[toby.collett@…]

Patch to add filter options to the parser

[toby.collett@…]

adds support for user definable namespaces/paths

[toby.collett@…]

adds custom tags for filtering documentation

[dante]

did this ever get considered? merged? deprecated?

[dylan]

Neil, please decide what to do with this prior to 1.1 final.

[bill]

defer dormant enhancements to "future" (they don't look like they're getting fixed soon so might as well mark as such)

[Neil Roberts]

New parser works differently than this patch. Will add stripping features if requested.