forum.bittorrent.org

The 8472

Azureus Developer

Being more ISP-friendly

Well, the issue of bittorrent not being very ISP-friendly comes up repeatedly, either in the form of people like the P4P consortium claiming that using local peers would result in speedups or - on the other end of the spectrum - traffic shaping ala sandvine.

I think this will be an important topic for the future of bittorrent in the long term but corporate interests, PR and politics usually surrounding these issues don't help the process. This is an engineering issue and should be treated as such. I would prefer if we could get a few ISP engineers (or anyone working on the internet infrastructure) to join, but as long as that's not the case we still can bounce around a few ideas.

The main goal is to maintain performance for the enduser at least or even improve it while easing the load on various parts of the internet infrastructure. Simply easing the load regardless of user-side performance is not realistic, as users would not like it, they'd stick to the faster clients.

I have already listed a few ideas here: http://www.azureuswiki.com/index.php/Us … /P2PvsISPs

So, if anyone has better/other ideas, approaches to simplify or optimize the ones listed or reasons why they're not practicable... speak.

Consider this thread as brainstorming.

Last edited by The 8472 (2008-03-23 13:15:28)

---

Az dev

dave

Editor

Re: Being more ISP-friendly

I suggest all client developers read the web page referenced and written by "The 8472".

To this I would add two items:

- Careful allocation of upload slots.  The number of upload slots should be based on long-run estimated access capacity, not on the number of active torrents and should not be arbitrarily large.  There is no reason to squeeze upload slots to window sizes well below 1 packet per round-trip time since TCP behaves poorly in this regime.

- Local Service Discovery: uses administrative-local multicast to locate peers sharing an infohash.   This only helps with finding peers very close on the network (usu the same LAN).  This is already implemented in uTorrent and mainline but there currently is no BEP for it.

Azureus has or had support for caching.  BitTorrent tried to work with CacheLogic to achieve the same.  For reasons which I am not thoroughly acquainted, the relationship between BitTorrent and CacheLogic fell apart.  However, I believe the argument behind caching is sound. 

I call for draft standards-track BEPs for the following topics:

- P2P Caching Support
- Local Service Discovery

I also call for informational BEPs that draw on content developer experience regarding the effective use of the BitTorrent protocols beyond the protocol semantics described in Bram's specification.  Much wisdom has undoubtedly accumulated over the years that is documented nowhere.

The 8472

Azureus Developer

Re: Being more ISP-friendly

uses administrative-local multicast to locate peers sharing an infohash.

Btw, azureus has that too. But this generally only helps in some special cases, maybe college networks or when you're only using LAN-internal transfers anyway. I don't think this does affect the bulk of bittorrent traffic in any way.

About peer caching, we had it. It was scrapped due to the lack of ISP-side support, i think i only saw 2 ISPs using it, and those were rather small ones.

Last edited by The 8472 (2008-03-24 04:15:20)

---

Az dev

The 8472

Azureus Developer

Re: Being more ISP-friendly

Since version 3.0.4.0 Azureus uses request limiting instead of read limiting by default, this allows it to minimize the number of peers used iff the global download limit is reached, and thus indirectly reduces the upload slots that are used by the other side. Request limiting also has the advantage that it keeps the network buffers empty / tcp windows open. The downside is that it can only be applied if a download speed limit is set manually.

Last edited by The 8472 (2008-03-24 05:34:25)

---

Az dev

dave

Editor

Re: Being more ISP-friendly

Even the most recent version of uTorrent has too many upload slots.  It allocates 4 per torrent when the number should be global across all torrents.  This is being corrected.

uTorrent will increase the number if the upload limit is not fully utilized.  However this assumes the upload limit is set or estimated correctly.  To avoid large numbers of slots, the number should be upper bounded by some function of the bandwidth-delay product.  Because delay will vary between links and the purpose of an upper bound is to prevent the system from becoming unreasonable, we can pick a canonical value for Internet RTT, e.g., 200 ms. 

This will still tend to give you small number of slots with home DSL or Cable modem connections so you might want to include an additional safety factor.  For most home users in the US, there doesn't need to be more than 8 global upload slots.  For modem users, even 1 MTU in flight per RTT is likely to saturate the link so it makes sense to impose a lower bound based on the unchoke algorithm, e.g., 4 upload slots which is the current canonical value used by uTorrent.

I have no experience with request limiting (as opposed to rate limiting).

dave

Editor

Re: Being more ISP-friendly

The downside is that it can only be applied if a download speed limit is set manually.

Why doesn't Azureus estimate uplink capacity?

The 8472

Azureus Developer

Re: Being more ISP-friendly

errm... request limiting depends on the download speed, not upload speed. And yes, we do estimate both, but automatically limiting the download speed would lead to lots of user-side complaints if the estimate goes wrong.

Request limiting works by limiting the # of outstanding BT_REQUEST messages so that the download speed is capped to the set value by not requesting faster than speed / 16KiB/s chunks instead of simply limiting the socket reads. This strategy allows one to focus the requests on as few peers as necessary.

If you want to see that in action simply set a download speed limit and download a heavily overseeded torrent with Azureus, you'll see that (let's say) 50 peers unchoke you but Azureus only downloads from the fastest 3-4 peers to saturate the cap.

Also, tweaking too many variables automatically at once (# of upload slots, # of active torrents, upload speed, download speed) can have a negative impact on the accuracy of the estimates.

Last edited by The 8472 (2008-03-24 15:03:37)

---

Az dev

dave

Editor

Re: Being more ISP-friendly

errm... request limiting depends on the download speed, not upload speed.

Uhhh.... why only download speed?  I mean the local uplink is more likely congested.  Not that downlink shouldn't be considered-- as you allude -- any well-seeded swarm might saturate end-user downlinks.

Also downloading chunks from many sources rather than a few limits information between TCP control loops.  Doesn't this lead to more aggressive aggregate TCP behavior since exponential is no different from linear increase when around a window size of 1, and even more so when operating in the timeout regime since a single packet arrival resets the exponential backoff back to zero.  Wouldn't it be better to try to maintain 1 packet in flight at all times for each control loop?  For small bitrates this might be impossible but that can be handled by imposing a minimum bound on the number of slots so that other BitTorrent mechanisms can operate (e.g., optimistic unchoke).

And yes, we do estimate both,

So does uTorrent/mainline/DNA.

Request limiting works by limiting the # of outstanding BT_REQUEST messages so that the download speed is capped to the set value by not requesting faster than speed / 16KiB/s chunks instead of simply limiting the socket reads. This strategy allows one to focus the requests on as few peers as necessary.

Also tends to solve another problem that sometimes leads to wastage: downloading from so many peers that many do not finish transferring a single chunk within an unchoke interval.

The 8472

Azureus Developer

Re: Being more ISP-friendly

Uhhh.... why only download speed?

Because only the downloading side sends requests Of course, an uploader could refuse to respond to more than N requests at any time, but that would make piece picking and request scheduling into a true nightmare.

Bittorrent is a pull-mechanism. So the uploader controls the number of his upload slots (unchokes) and his overall send rate limit, the downloader controls the requests. Hence request limiting is a measure at the downloader side. Properly distributing the upload slots and choosing the upload limit properly is a measure at the uploader side.

Also downloading chunks from many sources rather than a few limits information between TCP control loops.

yes, that's exactly where request limiting improves the situation

Also tends to solve another problem that sometimes leads to wastage: downloading from so many peers that many do not finish transferring a single chunk within an unchoke interval.

This problem is only a real problem on slow torrents or torrents where all peers have a restricted set of pieces they can offer. Request limiting can only be performed when you hit the set download speed limit... which rarely happens under such conditions. So sadly, it won't improve that condition... at least not directly. It indirectly improves it by leaving upload slots of some users on some torrents unused, thus resulting in a better upload speed for the remaining slots.

Last edited by The 8472 (2008-03-28 11:54:45)

---

Az dev

dave

Editor

Re: Being more ISP-friendly

Uhhh.... why only download speed?

Because only the downloading side sends requests

Ahhh... of course.

Also downloading chunks from many sources rather than a few limits information between TCP control loops.

yes, that's exactly where request limiting improves the situation

Sort of.  Do you focus the request limiting to a handful of seeds?  If you are downloading from 100 seeds then request limiting makes sure that requests are completed in a timely manner but it might switch between control loops at the end of each request thereby losing congestion state information.

Also tends to solve another problem that sometimes leads to wastage: downloading from so many peers that many do not finish transferring a single chunk within an unchoke interval.

This problem is only a real problem on slow torrents or torrents where all peers have a restricted set of pieces they can offer.

The more likely problem is failure to complete a "piece" in a single unchoke interval, which occurs quite easily with a large file if the client doesn't download different chunks from the same piece from different sources.   uTorrent does this, but the logic is quite complicated.

Having a large number of seeds also seems to happen more often with DNA than with BitTorrent in general because peers tend to hang around and seed for a long time.

dave

Editor

Re: Being more ISP-friendly

Also downloading chunks from many sources rather than a few limits information between TCP control loops.

yes, that's exactly where request limiting improves the situation

Of course! I think I understand your point.  If you rate limit the number of requests but you greedily allocate them to the fastest peer below its pending limit then the mechanism will tend to focus on a small number of peers.  Is this how the mechanism works? 

It seems like that would be close (but not exactly) the minimum number of connections necessary to fully utilize the down rate limit.

That might perform similarly to setting the number of down slots but without requiring an explicit mechanism to adjust the number of down slots.

The 8472

Azureus Developer

Re: Being more ISP-friendly

Is this how the mechanism works?

correct. and you can do lots of smart things with the "greedily allocate them to the fastest peer"-part.

Oh, and i just remembered this also has a slight preference for fast peers with a lower latency as they'll be able to reach the same speed with less requests, which might ultimately prefer closer peers iff they're equally fast (or faster) as a further-away peer. But this probably only happens under ideal conditions.

----------------------------------------------

Edit:

Another thing we can consider: The optimistic unchoking behavior while seeding. Assuming that currently seeding simply rotates peers through optimistic unchoking and otherwise uploads to everyone at an equal speed (if possible, otherwise dropping the slowest one for the optimistics) then randomized locality-preference could be implemented in the while-seeding-optimistics. This would not hurt the user's experience and the impact on the swarm would depend on how aggressive this biasing would be.

But this would still require some locality-information.

Last edited by The 8472 (2008-04-02 10:27:48)

---

Az dev

dave

Editor

Re: Being more ISP-friendly

Another thing we can consider: The optimistic unchoking behavior while seeding. Assuming that currently seeding simply rotates peers through optimistic unchoking and otherwise uploads to everyone at an equal speed (if possible, otherwise dropping the slowest one for the optimistics) then randomized locality-preference could be implemented in the while-seeding-optimistics. This would not hurt the user's experience and the impact on the swarm would depend on how aggressive this biasing would be.

But this would still require some locality-information.

We bounced this same idea around.  We are building a simulator to test this out...along with initial connect order.   With PEX you could relatively quickly (within minutes) learn every peer in the swarm even for swarms with 1000's of peers.  We do not open connections to all of them due to protocol overhead and Win XP's half-open connection limit.  For large swarms and especially for shorter files, each peer would never sample all other peers.  In the absence of samples, start with peers that are closer.

ghazel

BitTorrent Developer

Re: Being more ISP-friendly

But this would still require some locality-information.

I've had two ideas for gathering locality information. I'm assuming ISPs are motivated to provide it, and BitTorrent will generally shift traffic around if it makes sense.

First, and perhaps easiest for the ISP, using reverse DNS of the peer's IP to search for a service:
Say it's 11-12-13-14.vilage1.city2.dictrict3.isp.lt then it should test:
p2p-local-zones.vilage1.city2.dictrict3.isp.lt
p2p-local-zones.city2.dictrict3.isp.lt
p2p-local-zones.disctrict3.isp.lt
p2p-local-zones.isp.lt

The first one to respond with a list of CIDR ranges terminates the lookup. IPs in those ranges are preferred, for example attempting to fill say 90% of the peer list with them.

Second, and far less latency prone, using anycast. The ISP routes a given IP to their local p2p-local-zone box, and the client only has to send a simple UDP query to that IP, and if it gets a response, obey it the same way.

The 8472

Azureus Developer

Re: Being more ISP-friendly

The problem  is that this does not work well with wholesale ISPs and small ones or hosts which do not have a reverse host entry which they could use to build the lookup. Well, i guess it's a matter of maintenance since every ISP, especially smaller ones, would have to maintain lists of near-by ISPs too, as it's unlikely to find someone using the same torrent within a small ISP.

I agree that anycast would be great as it would even work if the local ISP doesn't support it but its upstream partner does for example - just like 6to4 or the decentralized DNS root servers. The problem here is that there is no anycast block, thus some AS would have to offer an address (and thus a whole /24 block since that's the smallest block most internet routers will accept) for public use by ISPs.

It would be great if one could obtain better topology information, maybe by probing along the (trace)route to various targets.

Take australia for example, they don't have a problem with last-mile bandwidth but with international traffic, so it would be important to prefer not only peers of the local ISP but also of near-by ISPs.

---

Az dev

Himaatin

Member

Re: Being more ISP-friendly

With IPv6 this should be a lot easier.
You can generally assume, that a /64 is always in the same local net. A /48 belongs to the same organisation, and a /32 to the same provider. (Of course there are always exception to it)

The 8472

Azureus Developer

Re: Being more ISP-friendly

Heh, i just realized that one point i listed here: http://forum.bittorrent.org/viewtopic.php?id=26 might also improve any measure that relies on locality. Namely the coalescing of swarms with identical content, this would make the long tail a bit less long and improve the chance that you'll find a near-by peer that's useful.

---

Az dev