You are not logged in.
The Sandvine Attack
About 6 months ago BitTorrent (BT) users (particularly on Comcast) started noticing their upload ratios plummet. Torrents which the trackers indicated should have a dozen peers and seeds were showing very little activity. The culprit: forged TCP packets with the RESET flag set (TCP_RST, or RST). The perpetrator: a Sandvine box.
The Sandvine attack works by eavesdropping on communications between two BT clients, and inserting forged TCP packets to disrupt those communications. In implementation, Comcast is not using them to completely block BT, but is using them to disrupt BT upload traffic as an out-of-network rate limiting device. The more BT upload traffic there is on the local Comcast network, the more often BT streams get hit with a TCP_RST. According to Comcast, this limiting is done on a total upload volume basis for the local network segment.
http://arstechnica.com/news.ars/post/20 … tices.html
Sandvine has 3 modes of operation, TCP_RST is the most well known, because it is the easiest to recognize. The Sandvine box may also attack a TCP connection by sending fake acknowledgments (TCP_ACK, or ACK). In TCP, there is no "retransmit" control. Instead, a missing ACK is an indication that the segment failed to make it to the receiver. Fake ACKs will cause the sender to accelerate the disbursement of packets onto the network, even though the receiver may actually have missed a segment of the stream. The result is that the sender "thinks" it has successfully sent its data, while the receiver is waiting for the re-transmit of the missing segment. This attack puts the TCP stream into an unknown state, because TCP has been designed to handle missing or duplicate packets, not forged ones. Sending forged TCP_RST packets to both ends of the stream is an effective means of terminating a communication, because if either end obeys the reset, the connection will be half-open, and eventually closed with a real TCP_RST from the other end.
Simply dropping TCP_RST packets is possible on POSIX systems (Linux & Mac PC's, and Linux-based firewalls that allow telnet access) using iptables or ipfw. Windows based users are S.O.L. on this one. However, simply dropping TCP_RST can produce serious problems, especially when the RSTs are legit. In the case of an ACK attack (or the not-yet implemented TCP window attack), legit RSTs will be especially important.
Defeating It
Protocol Obfuscation
The key is to prevent the Sandvine box from identifying BT traffic. A full encryption of all the BT communications using SSL is the most "obvious" and "final" of all solutions; however, SSL uses a lot of bandwidth and processor overhead, especially for streams which transmit a small amount of data, such as tracker traffic. This is not really a concern for client traffic, but for large trackers (or moderate trackers on slow connections) the use of a full SSL solution would considerably increase their bandwidth and processor demands.
The best solution is one that defeats the Sandvine box, but does not increase bandwidth requirements or significantly increase processor requirements. Fortunately, it is not possible for a Sandvine box to monitor ALL communications, so it needs to pick-and-choose which connections it listens to. This is apparently done by intercepting the tracker responses and extracting the list of peer IPs. All current BT implementations are vulnerable to this, even the so-called "trackerless" clients (which still contact the tracker). The Sandvine box can then make a match table to inspect all connections between the target IP and the list of peers. This presents an "easy" solution where we encrypt the list of peers, using the info hash as a shared secret. It is outlined in BEP 0008:
http://www.bittorrent.org/beps/bep_0008.html
Datagram Protocol
Another option has also been suggested which uses transmission control implemented atop of a datagram method (UDP), called UTP. Datagrams do not have a "state" which can be attacked. They are sent, and either make it or don't. Essentially, they are as close to the IP layer as one can get without actually bypassing the IP stack. While using UTP would defeat the Sandvine box for now, it still maintains a "state" implemented in the clients. A Sandvine v2 box could still attack the client-based state by sending forged datagrams. Even a full datagram based protocol could be abused and attacked by manipulating client-based behaviors. This returns us to protocol obfuscation/encryption.
Voluntary Rate Limiting Work-around
Because Comcast has set the Sandvine boxes to kick-in at a certain total upload congestion, the easiest solution is for the individual users to throttle their upload speeds. Comcast has not stated what this total rate is, and it's problematic to determine because cable modem networks are a shared circuit. Generally, about 450 users share 40 mbps down, and 10 mbps up. If everyone is talking, that's 90 kbps down, and 24 kbps up. As a general rule, not everyone will be talking at the same time, so a proper download:upload throttle on a cable modem connection would be about 21:6 KBps for all torrent traffic. This would achieve a 1:1 sharing rate on 300 MB of data in about 15 hours. Comcast only blocks unilateral uploads, so setting your download and upload throttles to 10 KBps total should get you a 1:1 ratio without ever seeing that TCP_RST. If you want to help seed a file, while downloading it, then set an upload throttle higher than your download (e.g. 6:12), but knock it down once the download is complete.
Unfortunately, many people run their torrents at the maximum upload rate of 345 kbps (384*0.9 for overhead -> 42 KBps), which will swamp the local nodes with only 25 peers (6 %) uploading. Of course, other people are using upstream bandwidth too. Keep in mind that VOIP uses about 64 kbps upstream, so a few phone calls takes out 1 BT uploader.
George Ou presents a projected breakdown of Comcast's network traffic here (note, this is George's own work, not real Comcast data):
http://blogs.zdnet.com/Ou/?p=1031
Last edited by aciani1 (2008-02-29 14:55:12)
Offline
Because Comcast has set the Sandvine boxes to kick-in at a certain total upload congestion
No, that's what they claim, but not what is actually happening. Various user reports and long-term sampling by Robert M. Topolski (the initial reporter of the issue, see http://fjallfoss.fcc.gov/prod/ecfs/retr … 6519843340 ) show that the RST-injecting happens 24/7.
Most public comcast statements have been identified full of... crafty wordings or even blantant lies, so i would not base countermeasures on their claims.
Custom UDP-transports do work btw, as you can also encrypt the flow control header fields, this way they cannot inject termination-packets, they'd be forced to do "normal" traffic shaping by droping packets occasionally.
When i investigated their shaping it appeared that any connection that was identified as bittorrent by their heuristics gets severed when it achieves a 10:1 upload:download speed ratio for, let's say 50 seconds.
See http://www.azureuswiki.com/index.php/Us … 72/Comcast for my results. Note that they were done with a user whom i wouldn't describe as a networking expert, so they have to be taken with a grain of salt, i'm especially not sure about the tracker part, as users have reported that HTTPS-trackers do not make a difference.
A full encryption of all the BT communications using SSL is the most "obvious" and "final" of all solutions
Far from it, bittorrent still has many characteristics that stand out, such as DHT, Tracker announces a high number of connections and packetsize/timing patterns.
To be truly stealthy we'd have to be more creative. encrypted TCP-transports for control-data bounced over only few repeater nodes to coalesce control flows (as long as no data flows occur) would reduce the number of connections, data flows could be established only when need arises and be done over fully encrypted UDP transports.
Then there also is the nuclear option: Carefully design everything to look like VoIP when it comes to headers, packet sizes and timing... they probably wouldn't dare to touch that.
Last edited by The 8472 (2008-03-02 08:12:17)
Az dev
Offline
UDP-based transports will work against RST-based attacks, but it requires upgrading both ends of every connection to support it. I am not saying that it cannot be done (or that it hasn't already been done). :-)
But we are ALSO looking for a solution that solves the problem for TCP and does not require dipping below the application layer.
--Dave
Offline
Hmm... noone has commented on this in 2 years, and it is the same everywhere else I look... hardly anyone is still talking about how to thwart Sandvine even now that it is being used at it's worst.
I just recently became victim of Comcasts return to throttling and felt a wave of deja vu come over me... Not Again!!!
Sadly, it is true and two nights of countless efforts to find a work around have lead me to one conclusion... Sandvine must die and we must kill it at it's source... the networks it is designed to forge packets for.
Here is the concept:
Sandvine uses Deep Packet Inspection to scan data flowing over the network and determine if the connecting IPs and their bandwidth consumption require throttling. The offending IPs get RST or ACK packets sent and the connection get left half open. What most places don't tell you is that the IP is TIED to the port the transmission is traveling on when Sandvine inspects it and that is how it sends the forged packets.
Some sites have posted mediocre work arounds involving setting the port to be random upon client startup and scheduled tasks to stop and restart the client.This is faulty as each restart requires the DHT buckets to refill and the Trackers to update wasting precious time. Others mention ways to insert scripting to randomize the ports via WebUIs... but no solid process has been found from what I have read and the whole idea seems faulty and not easy for the masses to use.
My recent test show that for 120 to 240 seconds after the port is randomized and applied in uTorrent, speeds for both upload and download reach peak efficiencies while using the recommended settings for your connection speed via the uTorrent best settings thread.
Once Sandvine catches on though, I get throttled again until I randomize the port.
Now I know what everyone is thinking right off the bat... What about port forwarding? Grow up... we live in a time where DHT, UPnP and other port mapping technologies exist and I have better results not using a forwarded port than i do with one. That said, my tests did not require any port mappings to be set to see speed benefits.
So, running with this premise, I have hypothesized that implementing a randomizing layer for each and every transmission of packets would be the best solution.
The idea is that using the recommended settings, the average user should never have more than 2000 concurring connections up and downstream. Using that data based on your settings, a set number of random, not known assigned, ports could be generated and cycled through randomly as you connect to peers to download and upload the packets and pieces. By doing this we actually achieve two key goals in this battle... one being an automatic at the IP level fix for bandwidth shaping technologies, and the other being the Sandvine killer itself.
How do you ask... some of you may see it already... by randomizing the port attached to each piece being downloaded, Sandvine will be expecting packets and making decisions on what port to block on your IP while you have already finished that stream and begun many more on alternate ports... the idea being each stream/port pairing finishes downloading before Sandvine can understand that the port is no longer transmitting. The only way to deal with this would be for Sandvine to inspect every packet completely as well as log and store for database reference each port/IP pairing as well as start sending RST and ACK packets across multiple IP/port pairings. This means EXPONENTIAL increases in resource usage, bandwidth consumption by the ISP itself and over all a nsty down turn in the ROI of bandwidth shaping.
In short, this implementation would be like opening the flood gates onto the Sandvine camp and showing them just how weak there flawed bandwidth shaping policies are.
So if you made it this far without hating me for rambling, please tell me what you think? I did not know where I should post this but I guess the source was a good first choice.
Thank you for reading and seeding
DaemonFox
Last edited by daemonfox (2010-04-21 17:18:56)
Offline
Then there also is the nuclear option: Carefully design everything to look like VoIP when it comes to headers, packet sizes and timing... they probably wouldn't dare to touch that.
That's a bad idea.
Many broadband providers and network owners also provide telephony (and sometimes VoIP) services to clients in their areas. That means that they won't be too happy about "foreign" VoIP traffic and might throttle or delay it.
Offline
I think I'm affected by some kind of throttling since days there is no download or upload (or just marginal) happening. What can I do.
Greetings Aron
Offline
ok thanks for that info
greets
Offline
I fully support moving to SSL because it is a permanent solution to traffic shaping. Further more the MSE protocol has problems (http://forum.bittorrent.org/viewtopic.php?id=396).
CPU overhead of SSL is now quite small. Use the "openssl speed" command to get some raw figures; on my PC (a 2.4 GHz Core2 from two years ago), RC4 appears to be about twice faster than AES, but AES is already at 160 MBytes/s, i.e. 16 times faster than 100baseT Ethernet can transmit. The SSL cipher suite "SSL_RSA_WITH_RC4_128_MD5" is very light weight and there for would work well for BitTorrent.
Network overhead is about packet size. The initial handshake for SSL implies some asymmetric cryptography; the DHE cipher suites (when the server certificates is used for digital signatures only) imply a ServerKeyExchange message which will need a few hundred extra bytes compared with a RSA key exchange. This is a one-time cost, and clients will reuse sessions (continuing a previous SSL session with a symmetric-only shortened key exchange). This process of resuming SSL sessions is key for BitTorrent where many small chunks will be distributed among a subset of nodes.
Offline
If MSE is not effective then SSL would not be effective for the very same reasons.
Az dev
Offline
If someone implanted traffic shaping for BitTorrent over SSL then all protocols that use SSL (HTTPS,POPS...) would also be affected. Using a SSL based VPN is a common and well known solution to the bloking or rate limiting of MSE/BitTorrent. My proposal is make BitTorrent more secure by utilizing this proven defensive measure and I'm happy to see that others agree with me.
Last edited by TheRook (2011-05-08 14:15:32)
Offline
That is not correct. More sophisticated traffic shaping solutions do not break the encryption, they use timing/packet size patterns/side-channel information to identify streams. SSL wouldn't change that.
VPNs only work because they aggregate all bittorrent links into a single connection. Bittorrent over SSL on the other hand would establish 1 SSL connection to each bittorrent client that you're connecting to.
Those are two vastly different things. I suggest you refrain from making such suggestions until you have acquired some actual knowledge in that area.
Az dev
Offline
Reinventing the wheal is one of the worst mistakes you can make when attempting to build a secure system. Thus, MSE should have never been built and it should be deprecated. Writing your own protocol can be fun but others shouldn't be subjected its shortcomings.
Last edited by TheRook (2011-05-13 15:01:07)
Offline
I think that you hand wave away the performance problems of openssl a bit too lightly.
In my experience, the diffie hellman handshake is currently my far the most expensive operations in bittorrent. Even more expensive than sha-1 and rc4. Obviously it depends on peer churn. Using ssl would be pouring gas onto the fire.
Another possible conclusion from the fact that people can identify and throttle MSE is to stop using it all together.
Offline
@Arvid SSL does only what it has to in order to insure the best possible secuirty. This is not something you can just "cut out" of a protocol. Further more, you can use a ssl cipher suite with a DH Key exchange, with SHA1 and RC4. But I'm not sure why you would want to because SHA1 and RC4 are both broken primitives.
Last edited by TheRook (2011-08-17 11:30:47)
Offline