Opened 5 years ago

Last modified 2 years ago

#18424 new defect

dojo/request/node doesn't support proxies

Reported by: lee Owned by: Bryan Forbes
Priority: undecided Milestone: 1.14
Component: IO Version: 1.10.3
Keywords: Cc:
Blocked By: Blocking:

Description

I'll find it difficult to provide a test case for this but after speaking with neonstalwart on irc's intern channel it made sense to report this here first.

I'm working in a corporate environment behind a proxy & not been able to get intern tests running against browserstack.

I also wanted to raise it incase others had seen this. From what I've seen so far, sniffing http etc, I think the problem is request/node doesn't support requests (running in node.js) behind a proxy correctly. I have to try and replicate this locally from home and will then provide tests. Initial findings are the path header isn't being set to the full URL and the host (name) and port aren't being set to the proxy host and port.

Change History (7)

comment:1 Changed 5 years ago by bill

Component: CoreIO
Owner: set to Bryan Forbes

comment:2 Changed 5 years ago by lee

So I've setup privoxy from home and so far this seems to be replicating the same issue. I've gotten no way forward yet determining where proxy configuration isn't set correctly in dojo, but have been able to successfully route dojo/request through my local proxy by bootstrapping https://github.com/goinstant/global-tunnel.

e.g.:

var globalTunnel = require('global-tunnel');

globalTunnel.initialize({
	host: 'localhost',
	port: 8118
});

dojoConfig = {
....
    deps : ["test1.js"]
};

// Now load the Dojo loader
require("./dojo/dojo.js");

comment:3 Changed 5 years ago by lee

So, I believe that request/node doesn't meet the requirements for http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html (sec 5.1.2) i.e. "The absoluteURI form is REQUIRED when the request is being made to a proxy" (as well as the host and port headers set to the proxy host and port). This doesn't take into account more advanced proxying i.e. tunnelling, but not important at the moment.

dojo/request/node uses node.http directly so I'll investigate setting those options

comment:4 Changed 5 years ago by lee

This is a difficult issue to resolve on my own without any feedback; I have made progress but it's not entirely clear this is the right way to go because proxy servers appear to have differing requirements.

Replicating locally on my own machine I saw I could successfully proxy a request (via my local privoxy) to apache on my own machine, however when I tried to request a resource from the DTK site (via my local privoxy) I saw different behaviour (a 404).

When trying to request a resource from the DTK site, it was confusing comparing the proxied requests I made via the changes to dojo/request/node to the requests when boostrapping https://github.com/goinstant/global-tunnel. The changes I made to https://github.com/dojo/dojo/blob/master/request/node.js#L50 were:

	var reqOptions = response.requestOptions = {
		//hostname: url.hostname, //lbod commented out
		port: url.port,
		socketPath: options.socketPath,
		method: options.method,
		headers: options.headers,
		agent: options.agent,
		pfx: options.pfx,
		key: options.key,
		passphrase: options.passphrase,
		cert: options.cert,
		ca: options.ca,
		ciphers: options.ciphers,
		rejectUnauthorized: options.rejectUnauthorized === false ? false : true
	};
	if (useproxy) { // just a hardcoded var for the moment
		reqOptions.host = '127.0.0.1'; // privoxy running locally
		reqOptions.port = 8118; // privoxy port
		reqOptions.path = url.protocol + '//' + url.host + ':80' + url.path;
	} else {
		reqOptions.hostname = url.hostname;
	}			

Comparing the requests look like this http://pastebin.com/xWDR2xaX

It seems from the pastebin that I'm likely missing or misconfiguring headers. However it does also raise the question why are we trying to reinvent the wheel when there are already node.js request libraries to deal with this e.g. https://github.com/request/request supports setting http_proxy/https_proxy env variables. Some further reading can be found here http://stackoverflow.com/questions/3862813/how-can-i-use-an-http-proxy

Last edited 5 years ago by lee (previous) (diff)

comment:5 Changed 5 years ago by lee

I think there are 2 downstream issues now depending on this; https://github.com/theintern/digdug/issues/16 & https://github.com/theintern/digdug/issues/17

Last edited 5 years ago by lee (previous) (diff)

comment:6 Changed 4 years ago by dylan

Milestone: tbd1.12

comment:7 Changed 2 years ago by dylan

Milestone: 1.131.14
Note: See TracTickets for help on using tickets.