Opened 14 years ago

Closed 14 years ago

Last modified 12 years ago

#379 closed enhancement (fixed)

[patch] Alternative io transport and script loading allowing cross-domain use of Dojo package

Reported by: Krzysztof.Finowicki@… Owned by: James Burke
Priority: high Milestone:
Component: Core Version: 0.2
Keywords: cross domain transport script loading Cc: alexey@…;
Blocked By: Blocking:

Description

Alternative "io" transport, based on well known dynamic script loading, would allow cross-domain use of Dojo package, giving e.g. possibility to publish widgets insertable on other sites. This should be accompanied by alternative implementation of Dojo script packaging by native <script> tag loading, and reimplementation of dojo.hostenv.getText (with protocol for server support) to enable resolving templatePath while loading widgets from cross-domain defined templates. See http://dojotoolkit.org/pipermail/dojo-interest/2005-September/000979.html, http://dojotoolkit.org/pipermail/dojo-interest/2006-January/003399.html for votes of interest and http://dojotoolkit.org/pipermail/dojo-interest/2006-January/003470.html for more detailed suggestions, references and discussion.

Attachments (5)

scriptsrcioAndBackforward1.patch.txt (85.0 KB) - added by jrburke@… 14 years ago.
Patch file that also contains the patch for ticket #439
scriptsrcio2.patch.txt (50.3 KB) - added by jrburke@… 14 years ago.
Patch file with style guide fixes
scriptsrcTest1.patch.txt (7.1 KB) - added by jrburke@… 14 years ago.
Remove duplicate code in test files.
xdomain.patch.txt (36.7 KB) - added by James Burke 14 years ago.
Cross domain package loading patch.
xdomain.build.xml.patch.txt (3.7 KB) - added by James Burke 14 years ago.
Patch file for build.xml. Forgot to include it in the original patch.

Download all attachments as: .zip

Change History (21)

comment:1 Changed 14 years ago by tagneto@…

I maintain Tagneto, and an implementation of a dynamic script loading mechanism, DSR (http://tagneto.org/how/reference/js/DynamicScriptRequest.html). I was planning to study the Dojo code to see if it would be possible to integrate that mechanism with the dojo.io package. I also wanted to try some other Dojo integrations (http://tagneto.blogspot.com/2006/01/clarifying-tagnetos-mission.html).

However, if there is a Dojo committer that would rather do the work, and would like to use the DSR API and/or implementation, please feel free to do so (just let me know so I don't duplicate any effort). If for some reason licensing is a concern, I'm open to getting it resolved. I don't have any strong licensing desires, I just want the source open.

comment:2 Changed 14 years ago by Josh Reed (obitus9982@…

Here is the library that I put together for doing this. The library is pretty simple and is inspired by Dojo (the package concept):

http://services.chronos.org/xqe/syndicated/cswl.js

There is some automatically generated documentation for the 'widgets' that I've put together:

http://services.chronos.org/xqe/syndicated/docs/

We've just started including these widgets in our main site and with our partner sites and it has gone well. In general, I'd like to see Dojo support something along these lines so that I could take full advantage of all of the goodies it offers (like the event system and widgets).

comment:3 Changed 14 years ago by jrburke@…

I looked at the code mentioned in the previous post by Josh Reed. It contains the same core piece of code as the tagneto implementation (create a script dom element, set the source, add it to head element). I'll still use the tagneto code as the base for an implementation since it also has support for timeouts/error checking, assured callback syntax, and supporting segmenting large requests into a set of smaller GET requests. My CLA is on file with the Dojo Foundation. Here are some high-level design notes that I will be using to create a patch to address this enhancement request:

1) Implement a src/io/ScriptSrcIO.js file that defines a dojo.io.ScriptSrcTransport? class. This class may be a subclass of dojo.io.XMLHTTPTransport, since ScriptSrcTransport? can support the caching and inflight/interval check mechanisms. I was hoping to use the implementation of some of XMLHTTPTransport's methods, but if it doesn't work out, then I would just have ScriptSrcTransport? implement the transport interface.

2) ScriptSrcTransport?.canHandle() would only return true if the following are all true:

  • mime type is "text/javascript" or "text/json",
  • it is a GET request
  • kwArgsfile? and kwArgsmultipart? are not defined.
  • it is an asynchronous request

I want to make sure ScriptSrcTransport? get registers itself as a transport provider after XMLHTTPTransport so that XMLHTTPTransport still gets first shot at the bind request. For a cross-domain bind(), do we need some auto-detection that requires the use of ScriptSrcTransport?, or just have the user explicitly set the transport when calling dojo.io.bind()? I'm fine with just using the explicit transport for now.

3) ScriptSrcTransport?.bind() would support kwArgsformNode? and back/forward. Very long URLs would use the multiple request feature of DSR:

http://tagneto.org/how/reference/js/DynamicScriptRequest.html#Multipart_Requests

4) Bob Ippolito's jsonp methodology (http://bob.pythonmac.org/archives/2005/12/05/remote-json-jsonp/) could be supported in ScriptSrcTransport?. Perhaps if ScriptSrcTransport? sees that one of the URL parameters is "jsonp", then it would use the jsonp string as the id of the request, replace the jsonp string with "onscriptload", and the window.onscriptload would be smart enough to use the original jsonp string for the application callback.

Instead of proxying the jsonp requests through onscriptload, perhaps something more sophisticated could be done using dojo.event.connect(), but I'm not sure that will work nice with the cache/inflight checks (timeout support?), and it would be nice to avoid having dojo.io depend on dojo.event.

5) Add timeout callback support at least for XMLHTTPTransport and ScriptSrcTransport?. It is almost there for XMLHTTPTransport, just missing a definition of a timeout callback and a timeout wait time on dojo.io.Request. What about adding "timeout" as the timeout function property on dojo.io.Request and "timeoutInterval" as the property that specifies how many seconds to wait before calling the timeout function callback?

6) Make sure the tests in tests/io to pass with the changes. Add tests for the ScriptSrcTransport?. I have a test page that uses another JavaScript? file to simulate simple and multipart responses (so that a server is not required for the tests):

http://www.tagneto.org/test/test/jstest/srvrscripttest/Test.html

I would adapt that page to create tests that fit into the Dojo testing framework.

Any changes to the above are welcome.

James Burke

comment:4 Changed 14 years ago by jrburke@…

Summary: Alternative io transport and script loading allowing cross-domain use of Dojo package[patch] Alternative io transport and script loading allowing cross-domain use of Dojo package

I have a patch that implements a ScriptSrcTransport?. It supports the following types of script src requests:

It has been tested in Windows Firefox 1.5, MSIE 6.0, Opera 8.51 and OSX Safari 2.0.3.

I will upload the Tortoise SVN patch file (that includes a test page) to this ticket. The patch also includes the back/forward changes done for ticket #439. The patch is also available here:

http://tagneto.org/dojo/patches/scriptsrcioAndBackforward1.patch.txt

You can try the test page (which has notes about how to use the new transport) here:

http://tagneto.org/dojo/snapshot/tests/io/test_ScriptSrcIO.html

Please let me know if you would like any changes. I'm happy to make them no matter how trivial you might think them to be. My CLA is on file. Or, if you just don't want this patch, that is OK too.

If all of this looks good and can get integrated with the SVN source, I would be happy to look at how to allow loading dojo packages across domains using this transport. It has a few dependencies on dojo.io and dojo.io.BrowserIO, but just for dealing with forms and converting JS objects to query params. There is also a dojo.undo.browser dependency, but that is a convenience dependency.

James Burke

Changed 14 years ago by jrburke@…

Patch file that also contains the patch for ticket #439

comment:5 Changed 14 years ago by alex

Milestone: 0.3release
Owner: changed from anonymous to alex
Status: newassigned

James, this is great stuff. Will merge once I get a chance to review.

Thanks so much for your substantial and high-quality contributions.

comment:6 Changed 14 years ago by alex

Hey James,

I'm merging the back/forward patches independently. Is there a reason that they are provided together in this patch?

Additionally, there are serious style issues with portions of the patch. Would it be too much trouble for you to work up a revised version that uses tabs and adheres to the Dojo style guide?

Regards

comment:7 Changed 14 years ago by jrburke@…

I only included the back/forward patch since ScriptSrcIO.js referred to the dojo.undo.browser package, and I wasn't sure when that one was going to be merged (trying to save you some time with one patch if you hadn't gotten to the back/forward yet).

I apologize on the style issues. I've been trying to follow the style guide, but I don't think I have my editor tuned quite yet to catch all the tab issues, and I'm still learning to internalize the style. Also, I just realized I forgot to properly style the test pages. I was too focused on getting the source looking good.

I will scrub the patch for the following issues and resubmit. Please let me know if I goofed up anything else in the style guide:

  • Use tabs instead of spaces
  • Indentation of split lines
  • Brace and parenthenses positioning
  • Executable statements in conditionals MUST be avoided.
  • The use of magic numbers in the code SHOULD be avoided; they SHOULD be declared using named "constants" instead.
  • Iterator called index instead of i

Also, I will remove the back/forward patch. So the patch for this ticket (379) will assume a dojo.undo.browser.addToHistory exists.

Thank you, and sorry for the style mistakes.

comment:8 Changed 14 years ago by jrburke@…

The following patch file hopefully addresses the style goofs (I will also attach to this ticket). It does not include the patch for ticket 439/back-forward refactoring:

http://tagneto.org/dojo/patches/scriptsrcio2.patch.txt

Snapshot test page is here:

http://tagneto.org/dojo/snapshot/tests/io/test_ScriptSrcIO.html

Changed 14 years ago by jrburke@…

Attachment: scriptsrcio2.patch.txt added

Patch file with style guide fixes

comment:9 Changed 14 years ago by alex

Resolution: fixed
Status: assignedclosed

great patch James. Merged.

comment:10 Changed 14 years ago by jrburke@…

Alex, Thank you for merging in the patch. Unfortunately, there was a problem with the patch file that I generated with TortoiseSVN: it included the test files in tests/io/scriptsrc/ twice (Test.js, TestJsonp?.js, TestMultipart?.js). I think it is an issue if I add a directory that is not in the repository. This caused those files to have their content repeated inside the file. The other files in the patch are fine.

End result, the test page outputs success messages twice. I created a patch file that removes the duplicate code in the test files, based on the current SVN HEAD:

http://tagneto.org/dojo/patches/scriptsrcTest1.patch.txt

I will also upload it to this ticket. Thank you.

Changed 14 years ago by jrburke@…

Attachment: scriptsrcTest1.patch.txt added

Remove duplicate code in test files.

comment:11 Changed 14 years ago by Krzysztof.Finowicki@…

Cc: alex@… jrburke@… alexey@… added
Resolution: fixed
Status: closedreopened

Hey to all,

(The same content I've put on dojo-interest).

Great, great job done (regarding ScriptSrcIO.js). I'm impressed by efficiency of handling tickets in DOJO. However, I have some remarks and comments.

  1. As explained in http://dojotoolkit.org/pipermail/dojo-interest/2006-January/003470.html (at 2. Cross-domain script packaging and loading), this is a half of the game in a scenario when DOJO-based components implemented in one domain (A) are to be made insertable on an "external" Web page that is installed on another domain (B). The page from domain (B) does not want to host and load DOJO library explicitly (e.g. choice of DOJO or another library is responsibility of (A), and (B) does not want to be changed in spite of that). Instead, (B) loads only some initializing script from (A) and marks placeholders for specific widgets to be loaded and filled from (A). In this scenario, various specific widgets are to be loaded from various server-delivered templates from (A), and accompanied by additional specific Javascript modules (packages) loaded from (A). The set and range of specific components delivered from (A) is not known beforehand, so they cannot be loaded in a preprocessed package. Thus the problem of cross-domain script (and widget) packaging and loading still needs resolving.

I do not want to be regarded as somebody ordering requests, but I just want DOJO to reach closed functionality in that matter.

Additional possibilities implemented in ScriptSrcIO.js allow only for data transport, not for packages/widgets transport. The latter one requires modification of dojo.hostenv.getText which is bound to XmlHttpRequest?.responseText instead of using dynamically added <script> tags. The 'same domain' condition verified in dojo.hostenv.loadPath should be also released; reimplementing dojo.hostenv.loadUri so as to add and load <script> tag (and not dj_eval it, because it will be eval'ed by browser) would be also necessary.

I would like to stress out that this is not the same situation like http://blog.dojotoolkit.org/2006/01/30/dojo-iamalpha-and-cdns, but rather opposite one, so setting baseScriptUri will not help.

  1. I have some comment on so called JSONP convention used in parameters of dojo.io.bind function. In my opinion, using literal "jsonp=callback" in GET parameters may be too restrictive in various contexts. Some servers can deliver JSON data string and prepend it with desired callback name retrieved from "jsonp" parameter. Another server solution may want to deliver e.g. HTML chunks (e.g. to be inserted as innerHTML). HTML has nothing to do with JSON, so fixing "jsonp" parameter name is some kind of limitation (why one cannot name it "htmlp" for example, or use other general name, e.g. just "callback").

So I propose more general solution in parameters to dojo.io.bind function (see http://tagneto.org/dojo/snapshot/tests/io/test_ScriptSrcIO.html#JSONP):

	callbackpar: "myCallbackLoadParam",
//This means a callback name should be pointed by param myCallbackLoadParam;
	callbackfun: "myCallbackLoadFunction",
//This means a callback will be myCallbackLoadFunction(data);

In this way, passing parameters:

	callbackpar: "jsonp", 
	callbackfun: "myJsonpLoadFunction", 

will result in placing "jsonp=myJsonpLoadFunction" in GET request. Of course, "jsonp" may be default value for callbackpar argument.

By the way, may be I did not quite catch implementation of

342         this._finish = function(state, callback, event){ 
343                 if(callback == "jsonp"){ 
344                         var jsonpData = event; 
345                         eval(state.jsonp + "(jsonpData);"); 
346                         state.isDone = true; //...

in http://trac.dojotoolkit.org/browser/trunk/src/io/ScriptSrcIO.js - is completing the string with the 'jsonp' call implemented on the server or on the client?

Similarly, I don't understand where the window.onscriptload function has to be called in 'jsonp' and 'not-jsonp' requests.

  1. The last comment regards implementation of multipart DSR (see http://tagneto.org/how/reference/js/DynamicScriptRequest.html#mozTocId97657. I understand that implementing this functionality results from GET URL length restrictions, which differs between browsers, browser versions and web servers. Should be the same generalization and improvement implemented in ALL OTHER 'GET' requests (or not implemented at all - why ScriptSrcIO has to be better than other?). By the way, AjaxExtended? by Alex Serebryakov http://ajaxextended.com/ uses similar technique.

Regards

Krzysztof

comment:12 Changed 14 years ago by jrburke@…

Cc: alex@… jrburke@… removed

Krzysztof,

(I removed Alex (the dojo Alex) and me from the CC list since we get any updates to this ticket via the dojo-checkins listserv):

  1. I agree that the ScriptSrcIO.js does not satisfy the cross-domain (xdomain for the rest of this message) loading of dojo packages. I was hoping to use parts of ScriptSrcIO.js for the xdomain package loading. However, now that I have spent more time with the dojo code, I'm not sure if there will be much reuse.

I am looking into the xdomain package loading, but I'm still mapping out the changes that need to be made. I was looking at some of the things that you mentioned (dojo.hostenv.getText and loadPath). My first pass at a design was this:

  • Modifying the loadModule function (I think that is the name, don't have the source with me) to use a document.write() to write out the script tag for the package if it is a xdomain package, keep an inflight array/object of the script src URLs.
  • Modify dojo.provide (or the backing function) to provide the notification that the script has loaded (marks the request done in the inflight object).
  • Once all inflight requests are done, trigger the dojo.loaded notification.

I want it to play nice with the existing getText() approach -- only use the script src for xdomain requests.

There are some issues that are a bit tricky:

  • We can't just override getText() because I believe that is a synchronous call, but the script src thing is async by nature.
  • I'm concerned about all the packages doing document.writes() at different stages. Are there any weirdness when the 3rd level package that gets included does a document.write()? What if this happens after the browser thinks the page is loaded, but before we get notification of the load? I want to use appendChild if the HEAD is created and document.write otherwise, but I'll have to experiment.
  • Is there a race condition with dojo.provide() saying the loading is done, but the rest of the script is still loading (does the browser wait until the whole file is downloaded before executing script? I think so, but I can't say for sure).
  • Loading HTML for widgets might be a problem, since only JavaScript? can be in the response for a script src URL. Perhaps we provide a tool to allow widget authors to convert the HTML into a javascript-safe string that can be loaded by this technique. Have them save it in a location like MyWidgetHtml?.html.js. Need to think more about this though.

That is where I am so far. Corrections and feedback is welcome.

2) Concerning JSONP: I added it because one of the other dojo contributors proposed that methodology and I have heard of some services using it:

http://bob.pythonmac.org/archives/2005/12/05/remote-json-jsonp/

Since script src requests are used, a service should not return HTML since only script is allowed for a script src request. I'm not sure about allowing a parameter that specifies the callback name. I feel it would be good to try to standardize on a limited set (ideally one -- my preference is window.onscriptload) to make things easier. Unfortunately, I think this is a new area of discussion, but one that needs to happen.

All that said, you could still use the existing ScriptSrcIO with your own callback. Just use the Simple or Polling examples. You will just have to feed the callback string as part of the "content" object parameter for dojo.io.bind.

You question on the jsonp implementation in ScriptSrcIO.js: What it does is actually replace the jsonp parameter with something that points to dojo.io.ScriptSrcTransport? so that we can properly handle timeouts. In the this._finish method you quoted, that is ScriptSrcTransport? getting the jsonp callback from the server, and it is now forwarding the response back to the original jsonp target.

window.onscriptload is only used for DSR requests (bind() requests that specify an apiId or say useRequestId). It is not used for jsonp requests.

3) Concerning multipart GET requests in DSR: The main goal of DSR was to provide a robust cross-domain alternative to XMLHTTPRequest (XHR). Since XHR can do POSTs, I wanted a mechanism to handle large requests. I think for the 80% (or even 90%) case, most script src requests won't need multipart support. But there are cases (I've had them in my own development) when this is needed. It does require a special server implementation to assemble the parts, but it does provide a solution. However, if enough people feel like this is too much bloat for such a small use case, I can look at extracting it to an optional package that can be loaded for that case.

I did look at AjaxExtended? http://ajaxextended.com/. Alex Serebryakov did a nice job of simulating the XMLHTTPRequest API, and he certainly grasped the issue about needing a request ID to properly match up the completed responses, and a multipart need. I used the DSR implementation in ScriptSrcIO.js mostly because:

  • I'm more familiar with it, and I have signed a CLA to contribute it to dojo.
  • The simulation of XHR's API was not needed since all of that is hidden behind dojo.io.bind()'s interface.
  • I prefer to use normal name=value pairs for CGI parameters. AjaxExtended? used a $ syntax. However, I can appreciate that this allowed for simulating sending up HTTP headers (that the custom server implementation uses), and it is a more compact syntax.
  • It didn't support request timeouts natively. Since SCRIPT SRC requests are a bit brittle (no callbacks if the server is unreachable, 404s, etc...), I felt having timeout support was important.

So, it is a nice library, just some different choices done for good reasons.

I was also interested in pushing DSR as a general spec for these types of requests. I think it is important to have a known, well-defined global callback (like window.onscriptload) so that it is easy for services that don't want to run server processes to create custom responses for each request (like the JavaScript?-friendly RSS feeds described in the Possible Applications section):

http://tagneto.org/how/reference/js/DynamicScriptRequest.html#Possible_Applications

I actually prefer DSR to jsonp for this reason, and because DSR specified a syntax for sending the status of the request as well as the data, instead of relying on a custom syntax for each server API. But it is important to get more feedback to find the best solution.

I hope that answers your questions. If it didn't please let me know. I think this is a very interesting area of development.

James

comment:13 Changed 14 years ago by alex

Owner: changed from alex to James Burke
Status: reopenednew

passing off to James as he's now a committer and is implementing the solution

comment:14 Changed 14 years ago by James Burke

I have a patch for the cross domain loading. I will upload it this ticket. See the following wiki page for some info:

http://dojo.jot.com/WikiHome/CrossDomainPackageLoading

If the patch is good, I'll merge it with the trunk, and fill out the wiki page with more info. You can try the tests here:

http://dojotoolkit.org/~jburke/djxd/tests/xdomain/

Changed 14 years ago by James Burke

Attachment: xdomain.patch.txt added

Cross domain package loading patch.

Changed 14 years ago by James Burke

Attachment: xdomain.build.xml.patch.txt added

Patch file for build.xml. Forgot to include it in the original patch.

comment:15 Changed 14 years ago by James Burke

Resolution: fixed
Status: newclosed

OK, got the basics working. There are some subtleties with widgets, but it is good to go now. If any bugs come up on it, file them as separate, specific bugs. See the wiki page for info on how to use it:

http://dojo.jot.com/WikiHome/CrossDomainPackageLoading

comment:16 Changed 12 years ago by (none)

Milestone: 0.3release

Milestone 0.3release deleted

Note: See TracTickets for help on using tickets.