Implement FTP resuming

See: https://www.mozdev.org/bugs/show_bug.cgi?id=15152

*** Ticket #354 marked duplicate of this one ***

Has this been reported to Firefox devs as "feature request: give built-in FTP resuming"? I think it would be good for the future to avoid having other download extensions reinvent the wheel (although who would want any download extension other than DTA!).

There actually is some resuming functionality within Firefox (nsIResumableChannel), but it doesn't fit our needs.

In particular you can only resume downloads that you already know (via the computed entity). This makes chunked downloading much harder than it has to be and it is impossible to download from multiple ftp mirrors at once.
Futhermore I'm not sure how (well) supported nsIResumableChannel is in all our supported applications...

Therefore rolling out our own ftp implementation is not reinventing the wheel.

Replying to MaierMan:

Unluckily mozilla does not provide us with appropriate tools for doing so, so it looks like we have to roll out an ftp channel implementation ourselves. fireftp has it's own implementation too, in js, maybe we can borrow from it if license allows.

Firefox(xulrunner) has implemented a resumable ftp now. We can follow the way below to make ftp download resumable. It only affects chrome/content/dta/manager.js

At about line number 1693, there is a try catch and catch contains nothing. We insert the following lines to this catch.

try {
    let rc = this._chan.QueryInterface(Ci.nsIResumableChannel);
    rc.resumeAt(c.start+c.written, null);
} catch(ex) {
}

At about line number 2038, there is a single statement. We replace it with another try catch as below:

      try {
            let rc = this._chan.QueryInterface(Ci.nsIResumableChannel);
            d.resumable = true;
        } catch(ex) {
            d.resumable = false;
        }

These works because nsIFtpChannel implements nsIResumableChannel now.

I have tested on my machine, ftp downloading can be paused and resumed now.

Replying to ZhouWeikuan:

Replying to MaierMan:

Unluckily mozilla does not provide us with appropriate tools for doing so, so it looks like we have to roll out an ftp channel implementation ourselves. fireftp has it's own implementation too, in js, maybe we can borrow from it if license allows.

Firefox(xulrunner) has implemented a resumable ftp now. We can follow the way below to make ftp download resumable. It only affects chrome/content/dta/manager.js ...

I already explored the possiblity of using nsIResumableChannel: #186

Problem is, that this won't (or at least shouldn't) work, as:

• entityID might not be null:

  mResumeRequested = (mStartPos || !mEntityID.IsEmpty());
  

• The connection must not be open:

  NS_ENSURE_TRUE(!IsPending(), NS_ERROR_IN_PROGRESS);
  

(Both taken from the implementation of nsIFTPChannel in nsFTPChannel.cpp).

One might however get the entity id from the initial channel.

However this again wouldn't work when getting files from different mirrors in parallel, as we don't have an "initial" channel for most of those where we can lookup the entity ID.

And the generator of those entityIDs for http channels is "flawed" anyway, as it expects servers to behave correctly. But in fact many load-balanced server clusters are misconfigured so that you will get a different ETag because the request doesn't go to the same machine and the server in question uses for example inode in ETag (Apache, lighttpd) or a machine id (IIS). Examples that come to mind are kernel.org or microsoft.com.
This might as well be a problem for some load balanced ftp server clusters.

Oh, and having a real patch would be nice. ;)

Replying to MaierMan:

Replying to ZhouWeikuan:

Firefox(xulrunner) has implemented a resumable ftp now. We can follow the way below to make ftp download resumable. It only affects chrome/content/dta/manager.js ...

I already explored the possiblity of using nsIResumableChannel: #186 Problem is, that this won't (or at least shouldn't) work, as: * entityID might not be null: {{{

mResumeRequested = (mStartPos !mEntityID.IsEmpty?()); }}}

If my understanding is right, this statement means if mStartPos is not zero, or mEntityID is not empty, there is a request to resume ftp channel. So if entityID is null, it doesn't matter here, for we have specified the start pos.

* The connection must not be open: {{{ NS_ENSURE_TRUE(IsPending(), NS_ERROR_IN_PROGRESS); }}}

The channel is not open when we call resumeAt. It is before the asyncOpen statement. Actually, FTP and Http can share the same way to resume at some position.

(Both taken from the implementation of nsIFTPChannel in nsFTPChannel.cpp). One might however get the entity id from the initial channel. However this again wouldn't work when getting files from different mirrors in parallel, as we don't have an "initial" channel for most of those where we can lookup the entity ID. And the generator of those entityIDs for http channels is "flawed" anyway, as it expects servers to behave correctly. But in fact many load-balanced server clusters are misconfigured so that you will get a different ETag because the request doesn't go to the same machine and the server in question uses for example inode in ETag (Apache, lighttpd) or a machine id (IIS). Examples that come to mind are kernel.org or microsoft.com.

Yes, entityID is flawed. If we ignore entityID and set it null, we can resume ftp downloading at least. We have resolved this bug:) This will corrupt the file if several mirrors do not have the same file. It appears in rare cases, and it is an another bug.

This might as well be a problem for some load balanced ftp server clusters. Oh, and having a real patch would be nice. ;)

Yes, entityID is not trustable. So set it null?

The post below may help some on download resuming. http://developer.mozilla.org/en/docs/Implementing_Download_Resuming

Btw: after I modify the original manager.js of dTa, the file downloading is resumable and it may be multithreading for I have seen several chunks start to run. But there is just one chunk running after several seconds, I guess it is because the ftp server limit one thread for one IP, or it is because I provide a null entity id.

Indeed... You may resume ftp using nsIResumableChannel. Not sure if I simply erred when I first evaluated that interface/method or if it wasn't supported back then and afterwards changed...

However, as the mozilla ftp implementation is quirky at best, there was some more work required besides resumeAt/resumable-flag:

• if the server returns an error on REST or RETR (e.g. not supported or too many connections) then nsFTPChannel will assume the file in question was actually an directory, and then will try to CWD to that non-existing directory. Which then of course gives and 5xx error which is helpfully displayed as a modal warning dialog :p
• although nsFTPChannel queries MDTM it does not make that value accessible in any way (like nsIPropertyBag2)
• 5xx errors are always displayed via a modal alert dialog

Anyway, implemented the bits in [1037], [1038]

Testing welcome.

Seems to work. Closing as fixed.