Transport-level protocols in user space
| Did you know...? LWN.net is a subscriber-supported publication; we rely on subscribers to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the net. |
The Linux networking developers have long held a strong opinion about user-space protocol implementations: they should be avoided in favor of making the in-kernel implementation better. So it might be surprising to see a veteran networking developer post a patch set aimed at making user-space implementations easier. A look at this patch and its motivations shines an interesting light on changes that are taking place in the networking world.
The developer in question is Tom Herbert; the patch set is titled "transports over UDP", or "TOU". This patch set allows any transport-layer protocol (such as TCP or SCTP) to be packaged up and carried within a stream of UDP packets. The mechanism by which this is done is described in this draft RFC (also written by Tom); it uses the draft generic UDP encapsulation (GUE) specification for the actual encapsulation. The patch set adds basic (still incomplete) TOU support to the kernel, with the ability to encapsulate TCP connections in particular.
Any destination that is reachable via UDP should also, in principle, be reachable directly using a transport-layer protocol, so one might well wonder why it is worth the trouble to hide that protocol inside UDP. The fact that this encapsulation comes with (according to Tom's benchmark numbers) a 10-15% performance loss also serves to increase eyebrow elevation. It turns out, though, that there are a few advantages to encapsulating a protocol like TCP in this way.
The first of those is that, as mentioned above, it facilitates running the transport-layer protocol stack in user space. The usual reason for bypassing the kernel in favor of user-space networking is to maximize performance for specific types of communications, but that is not the intent here. Tom (and his employer Facebook) envision putting the transport-layer protocol code into desktop and mobile client applications instead, and the main reason to do that is to speed progress on the development and adoption of newer protocol versions. Tom described the purpose this way:
This is, in other words, an attempt to address the protocol ossification problem by taking more direct control of the protocol code on both sides of the connection. That will, it is thought, allow network protocols to start evolving at a reasonable pace again — when users are running the right user-space implementation, at least.
The other benefit to TOU is related to the protocol ossification problem, but also to the privacy and security concerns that are prevalent on the net. Once the transport-layer headers (and payload) are encapsulated within a UDP packet, they no longer need to be sent in the clear. The TOU draft specifies that DTLS (datagram transport layer security) can be used to encrypt the payload of the UDP packets, hiding the transport-layer information entirely from any system other than the endpoints. It thus brings back the idea of end-to-end networking, where the endpoints know about a specific communication and all other machines in the middle exist only to serve as dumb transport. Eavesdropping on communications — or even collecting metadata — would be much harder in a world where TOU was widely deployed and used.
TOU can thus provide privacy, but there is another advantage: routers along the path ("middleboxes") will be unable to inspect packets and filter out specific protocols. This filtering makes the addition of any new protocol difficult at best. Adding a simple flag for explicit congestion notification (ECN) made much of the net unreachable; the developers of multipath TCP have had to go to considerable lengths to disguise the new protocol as boring vanilla TCP to get it past middleboxes. But once a middlebox loses visibility into the protocol that is actually being used, it also loses the ability to filter it. That too, Tom hopes, will fight ossification and speed the adoption of new protocols.
It is also worth noting that encapsulating a TCP connection in this way separates it from the underlying network, making the endpoints of TCP connections mobile. That, too, could be useful in a world where computers move rapidly from one network to the next.
Before all this can happen, though, the TOU patches need to find their way into the kernel. Networking maintainer David Miller did not reject them, but he did express some strong misgivings about the whole thing, starting with the additional maintenance burden that it implies:
He suggested that, in the future, developers would treat TOU-related bug reports the way reports involving proprietary modules are treated now — by ignoring them. He also worried that the middlebox problem might not be quite so easily solved:
Tom responded that the bulk of the net allows UDP traffic now, and that a number of higher-level protocols, including QUIC, have been successfully deployed. The nascent path layer UDP substrate effort is working to improve flow control for UDP-based transport protocols. Overall, that seems to be the direction the net is taking, and TOU should fit into it well, so it may not be unreasonable to believe that it will get through middleboxes in the future.
It would seem that the way to save the Internet is to take the
higher-level protocols out of the kernel's hands and bury them, instead, in
the applications. There is one potential problem here, though, that has
not yet really been discussed on the list. TOU could be used to achieve
rapid deployment of new Internet standards. Or it could be used to deploy
a whole new set of proprietary transport-level protocols that are not
easily accessible outside of closed-source applications. It could be that
TOU-like technologies (including the already-deployed QUIC) will, instead,
retard the adoption of open Internet standards by rendering them
irrelevant. Tom acknowledged that the
whole idea
"
In summary, enabling this mode might lead to a world where the
network evolves more quickly and is more resistant to surveillance and
eavesdropping. On the other hand, it could lead to a fragmented net where
each large company has its own transport-layer protocols baked into its own
proprietary applications. Guarantees that things would go the right way
would appear to be in short supply; this area is going to be interesting to
watch.
conflicts with our idea of what the architecture of
the Internet should be
", but also asserted that the development
community cannot ignore it.
Index entries for this article Kernel Networking
(Log in to post comments)
Transport-level protocols in user space
Posted Jun 21, 2016 6:23 UTC (Tue) by xman (guest, #46972) [Link]
On the down side, the whole point of ECN is that any hop along the way can potentially signal congestion by flipping the bits. You lose that if they aren't manipulating the embedded protocol.
Transport-level protocols in user space
Posted Jun 21, 2016 7:44 UTC (Tue) by kugel (subscriber, #70540) [Link]
I guess together with KCM it can even be relatively performant.
Transport-level protocols in user space
Posted Jun 21, 2016 8:41 UTC (Tue) by iarenaza (subscriber, #4812) [Link]
Transport-level protocols in user space
Posted Jun 21, 2016 9:22 UTC (Tue) by kugel (subscriber, #70540) [Link]
Transport-level protocols in user space
Posted Jun 21, 2016 13:19 UTC (Tue) by NAR (subscriber, #1313) [Link]
Transport-level protocols in user space
Posted Jun 21, 2016 16:32 UTC (Tue) by zlynx (guest, #2285) [Link]
Or did you mean that the downside is that with these customized protocols embedded in UDP, there is no longer any way for routers to do things like flipping ECN bits and they now have no choice but to drop packets.
Transport-level protocols in user space
Posted Jun 21, 2016 8:16 UTC (Tue) by marcH (subscriber, #57642) [Link]
Exerting control on applications put aside, performance looks like the only real reason for TCP to be in the kernel. User-space libraries are obviously enough for code re-use.
I bet the above just summarized what has already been explained in a gazillion of research papers (of which there shouldn't be a gazillion in the first place; but I digress). Ossification, research: pick one.
> "We also now have to debug against every single userland TCP implementation someone can come up with, the barrier to entry is insanely low with TOU."
Wow, "barriers to entry" presented as a good thing: that's serious "ossification" indeed. Looks like some would like a GPL'ed version of the Internet: one you cannot fork freely. By the way: are other kernels besides Linux allowed to network in such a world? Only if they can demonstrate a slow and conservative enough development pace?
OSI and telcos are back?
> Rather, I think people are going to start adding rules to block TOU tunnels entirely because they cannot inspect nor conditionally filter/rewrite the contents. This is even more likely if Joe Random and so easily can do their own userland TCP stack over TOU.
Whoever filters Facebook today may switch to new ways in the future. Whoever doesn't now, is very unlikely to filter TOU in the future. So this will make little difference if any.
Whatever happens here, the most powerful "barrier to entry" will always remain: Metcalfe's law. It will be more than enough to clamp the number of Joe Random implementations. Metcalfe's law is cruel for "small"; but it's especially crueler for "small + buggy".
> I suspect a lot of bug reports will go straight to /dev/null once it is clear that it is a TOU TCP connection.
Bug reports from Joe Random getting ignored: what's new? TOU-related bug reports from Facebook, Amazon or Google coming with tentative fix attached? I bet they will be looked at.
Transport-level protocols in user space
Posted Jun 21, 2016 11:14 UTC (Tue) by ballombe (subscriber, #9523) [Link]
No, the real end is the user.
The kernel has always been a great way for the user to keep control on the applications.
Transport-level protocols in user space
Posted Jun 21, 2016 15:51 UTC (Tue) by marcH (subscriber, #57642) [Link]
The closer to the user, the closer to the principle. From that perspective TOU is a massive step forward. Whether the e2e principle is good or bad, when and how is a much bigger question.
> The kernel has always been a great way for the user to keep control on the applications.
The kernel has always been a great way to keep control on the applications.
Kernels were and are still designed to keep applications and users from interfering with each other (and with the network). It's only recent that so many people can afford exclusive ownership and control of a pocket computer[*]. It took ages to get to this https://developer.android.com/training/permissions/reques... and despite all Google's efforts I bet most users still don't take advantage of it: too complicated. Most users are not admins of their own devices because there's no market interest: it's way too technical. (Un)installing/disabling/upgrading applications is control that is simple enough for all users to actually understand and perform. At the risk of hurting the feelings of this kernel-centric community: the vast majority of users have no more interest in controlling the kernel running in their pocket than for controlling the setup box in their home. Too far away from them. Just make sure my apps work and get out of the way. Security? Yes please, make it secure for me as long as I don't need to know about it. Privacy? What's that again?
[*] this evolution is obvious when you see that each Android application is assigned a separate user ID.
Transport-level protocols in user space
Posted Jun 25, 2016 19:01 UTC (Sat) by robert_s (subscriber, #42402) [Link]
I disagree that the application is closer to the user than the OS. The application is closer to the corporatized (and often rather greedy) vision the application developer has. The application likes to think it exists in a vacuum on your system without having to interoperate with or work alongside things it doesn't want to. One of the OS's important jobs is to police that on behalf of the user.
Transport-level protocols in user space
Posted Jun 26, 2016 10:37 UTC (Sun) by Fowl (subscriber, #65667) [Link]
Transport-level protocols in user space
Posted Jun 26, 2016 18:38 UTC (Sun) by marcH (subscriber, #57642) [Link]
A) run-time control, e.g.: root permission, cgroups, firewalls, etc.
B) "build/install control", sorry for the lack of a better term. e.g.: choosing which application to trust and install, a.k.a: "is there an app for that?"
> One of the OS's important jobs is to police that...
Yet is is, and this can be type of control/freedom A.
> ... on behalf of the user.
... on behalf of the *administrator*.
Freedom A obviously did not exist on multi-user systems. Now that everyone has a single-user computer in their pocket you'd think people have it. Except not because it's way too technical and complicated and scary, so all users besides geeks still want someone else to manage this on their behalf: see the massive success of secure/verified boot, DRMs, managed set-up boxes, etc.[*]. So nice theory but was never really practised on a wide scale. In practice the OS is owned just like the rest of the network is.
This leaves only freedom B as a (admittedly limited) type of control for most users.
[*] now add Linux TCP/IP maintainers to that list?
Transport-level protocols in user space
Posted Jul 1, 2016 21:21 UTC (Fri) by Wol (subscriber, #4433) [Link]
I think it is also confusing two types of user. The user/owner, and the (ab)user/nonowner.
Just because I'm the user doesn't mean I have any special rights. The owner may be someone completely different who wants to use the kernel to secure HIS rights.
Cheers,
Wol
Transport-level protocols in user space
Posted Jun 21, 2016 12:06 UTC (Tue) by k3ninho (subscriber, #50375) [Link]
Interfaces are hard, so we went sho- ...and built AWS and the cloud instead.
I've got a grumble with the idea that we build things with a fixed set of data fields but simly don't make use of the metadata available at creation (authorship, compilation, deployment) to indicate interoperability and control compatbility. We're missing the opportunity that the metadata about the version of the program provides to show the intent of interoperability so we can easily know whether programs are compatible.
Separation of interface and implementation is a helpful distinction to make and is widely adopted everywhere but Linux kernel, the distributions' library space and (consequently) the conatainerized tools for distributed applications in the cloud. Stuff like Telegram has a magic number which can be queried to say which versions of its messaging are supported and the magic number is appended to your calls. With this additional layer of abstraction (ha!) between the supported interfaces and the could improve the research side of computer science while respecting the ossification that's needed for engineering robust systems.
K3n. (Yes, I hear you say 'show us the code' so I'll slink away quietly.)
Transport-level protocols in user space
Posted Jun 21, 2016 15:28 UTC (Tue) by gdt (subscriber, #6284) [Link]
Exerting control on applications put aside…
Except that this is key. Networking authors fear that moving congestion control to userspace will lead to parties adopting congestion control algorithms which work well for the individual but which lead to congestion collapse for the network or lead to extreme unfairness between differing classes of connections. A high barrier to entry for the creation and wide dissemination of such algorithms is seen as desirable to prevent that Internet-breaking outcome.
Similar concerns have been expressed by researchers (eg, the last decade of Floyd's work before she retired), so the picture you paint of “ossification, research: pick one” is far too simplistic.
Concern about the fairness of connections between newer congestion control algorithms and established congestion control algorithms isn't limited to Linux. It can be found in the design of Microsoft's Compound TCP. CTCP is so focussed on avoiding extreme unfairness to connections from Windows Xp computers that it models the congestion window of that older congestion control algorithm and only exceeds that algorithm's datarate when unfairness will not be a result. Interestingly, many of the congestion control parameters are locked down on consumer versions of Windows to prevent misuse.
There are ways forward to userspace congestion control, but the proposed API is simply ‘trust userspace’ which is probably too simple and too trusting. Of course there is a performance penalty for being less simple and less trusting. Meaning that there's no immediately-apparent choice of API, but that a skilled design of careful choices is needed.
OSI and telcos are back? Oh please, that's rhetoric from ancient history. The situation in the IETF has changed so very much in thirty years that you can now often see telcos/ISPs being on the side of the angels and opposing equipment vendors wishing to cut corners for sooner ‘time to market’.
Transport-level protocols in user space
Posted Jun 21, 2016 16:03 UTC (Tue) by marcH (subscriber, #57642) [Link]
You never had someone in your household using Bittorrent. Skype and others already use their own congestion control; wake up and smell the coffee.
The correct location to enforce traffic control and contain congestion is subscriber/bill paying/peering interface. Then please leave end users the freedom to flood their own network or not. They decide which network applications they use or not.
Transport-level protocols in user space
Posted Jun 21, 2016 20:17 UTC (Tue) by nim-nim (subscriber, #34454) [Link]
Want to watch video? Ni problem, youtube, akamai, whatever will saturate your link.
Why is only one person in the building getting the video? Well the CDN saturated the link of course. No provision for a second watcher. All the javascript knows is push as many bits as possible till nothing else fits
(at which point your home becomes a shared network? I now have 7 streaming-capable network nodes at home, most of them appliances in different rooms. Does it qualifies?)
Transport-level protocols in user space
Posted Jun 22, 2016 4:37 UTC (Wed) by marcH (subscriber, #57642) [Link]
As long as its only your link: not a problem since you know the person who is saturating it and can do something about it.
> The result of such thinking is hilarious on any shared network.
The result is hilarious... today.
> (at which point your home becomes a shared network? I now have 7 streaming-capable network nodes at home, most of them appliances in different rooms. Does it qualifies?)
So you are new to all this. Welcome.
Transport-level protocols in user space
Posted Jun 23, 2016 17:27 UTC (Thu) by cwillu (guest, #67268) [Link]
>As long as its only your link: not a problem since you know the person who is saturating it and can do something about it.
Ahhh... the myopic view of the bachelor.
"Honey, I'm trying to play counterstrike, can you stop watching netflix?" is not a good answer.
Transport-level protocols in user space
Posted Jun 23, 2016 19:42 UTC (Thu) by nybble41 (subscriber, #55106) [Link]
We're long past the point where it is reasonable to simply assume that every device and application will employ fair congestion algorithms. Effective traffic shaping policies at the border are a necessity, and render tight control over the endpoints' congestion algorithms unnecessary. In any case, it doesn't make sense to allow applications to use UDP (which has no built-in congestion control) and yet forbid them from taking charge of the TCP congestion algorithm.
Transport-level protocols in user space
Posted Jun 27, 2016 4:29 UTC (Mon) by gdt (subscriber, #6284) [Link]
To date most UDP algorithms have had flow-control. DNS, for example, has one packet in flight per query. Where UDP algorithms don't do congestion control then we already see congestion collapse, such as with high-rate videoconferencing systems taking out campus traffic.
Rate-limiting of client-side interfaces isn't enough. To suggest otherwise is to think only of the case of congestion is where the bottleneck link is the link to the customer. But there are two client-side interfaces on any congested path, so the the suggestion of using ingress rate-limiting fails on its own terms, as it fails to address the issue of congestion on the *far* client-side link.
In any case rate-limiting is too large a stick to be deployable. The customer has paid for 10Gbps. Rate limiting says they can't transmit at 10Gbps, even if that causes no congestion.
What you are handwavingly asking for is that the client-side interface ask "is this connection's sending rate reasonable given the apparent congestion control parameters?" That is, the router needs to reverse-engineer the TCP algorithm's variables. Easy enough to do for a TCP stream, but to do it for UDP streams then the only reasonable way to do that is for the router to tunnel them in a TCP stream to the far client-side egress interface. It also adds a new performance parameter to interfaces: the number of concurrent client-side connections they can support; and it's not difficult to see how that can become an anchor for an effective denial of service (and yeah, you could soft-fail and not monitor connections, in which case the library mentioned in another comment starts by opening 100,001 connections).
Now maybe networks will have to do that. But let's not pretend it will be cheap, either finanically, in terms of robustness, or in terms of latency (a three-way TCP handshake before that UDP packet progresses into the network).
The whole point of the TCP/IP protocols is that they found a better-performing and more robust design than "enforce congestion control in the network". Moving away from that design necessarily means less performance and less robustness. Maybe a failure of the commons will force us to that situation. But let's not pretend it is optimal.
Transport-level protocols in user space
Posted Jun 27, 2016 6:43 UTC (Mon) by marcH (subscriber, #57642) [Link]
I don't think anyone suggested otherwise yet.
> it fails to address the issue of congestion on the *far* client-side link.
When you receive too much it's because you asked for it. For instance, this guy currently saturating your link with his brain-dead video conference application is almost always someone you know and someone you're trying to communicate with using the same brain-dead application. I bet you may both switch to a smarter and gentler application next time.
> What you are handwavingly asking for is that the client-side interface ask "is this connection's sending rate reasonable given the apparent congestion control parameters?" That is, the router needs to reverse-engineer the TCP algorithm's variables.
What's done in the network doesn't have to be that real-time, that fine-grained and that close to TCP: leave that to the applications.
> The whole point of the TCP/IP protocols is that they found a better-performing and more robust design than "enforce congestion control in the network".
What you and TCP are "handwavingly" asking is for everything to be nice and behave on the Internet. TCP's congestion control was a very smart afterthought, however gentlemen agreements can only last for so long:
http://www.stevesouders.com/blog/2008/03/20/roundup-on-pa...
https://trac.filezilla-project.org/ticket/2309
The per-connection "fairness" never made sense anyway: what's fair about someone or something multitasking getting more bandwidth?
Transport-level protocols in user space
Posted Jun 27, 2016 17:34 UTC (Mon) by raven667 (subscriber, #5198) [Link]
I'm not sure that's really true, so much in society depends on people doing the right thing even when an authority isn't watching, civilization seems to still exist. The network is just another facet of human interaction.
Transport-level protocols in user space
Posted Jun 30, 2016 3:51 UTC (Thu) by marcH (subscriber, #57642) [Link]
Whatever "real-world" agency is actually policing the internet has much bigger fish to fry than congestion control. Among many others, ransomware looks higher priority to me. Good luck with that one already.
I don't think anyone expects the internet to look anything like a civilization any time soon. I hope the Linux network stack maintainers are not under such an illusion.
Transport-level protocols in user space
Posted Jul 2, 2016 13:16 UTC (Sat) by Wol (subscriber, #4433) [Link]
> I'm not sure that's really true, so much in society depends on people doing the right thing even when an authority isn't watching, civilization seems to still exist. The network is just another facet of human interaction.
The problem here is that the people feeling the pain are not the people inflicting it. Even within a household. And as soon as the choke-point moves "out there", all the evidence says that people are indifferent (at best!) to the pain they inflict on other people - if it's some impersonal "them", it's not seen as important.
I think the fairest thing to do is choke traffic by IP address. And if it's IPv6, you choke based on the network portion (iiuc, v6 comes as a network half and device half, and it's allocated by the network half). That way, cheating by opening multiple connections won't work :-)
Cheers,
Wol
Transport-level protocols in user space
Posted Jun 27, 2016 17:16 UTC (Mon) by nybble41 (subscriber, #55106) [Link]
At the application level, which was the whole point. The proposal which has gathered so much opposition was merely to give applications control over the congestion algorithm for TCP, when they already have full control over the congestion algorithms (if they exist at all) for UDP-based protocols.
This is not an argument that congestion algorithms are useless, but rather that mandatory QoS is needed at the routers regardless of any congestion algorithms (well-behaved or otherwise) at the endpoints. You can't simply rely on the endpoints to do the right thing for the health of the entire network.
> In any case rate-limiting is too large a stick to be deployable. The customer has paid for 10Gbps. Rate limiting says they can't transmit at 10Gbps, even if that causes no congestion.
You don't set a fixed rate, you limit the amount of data in the buffer, prioritizing packets according some "fairness" policy and dropping anything that can't be transmitted in a reasonable time. This is exactly what adaptive AQM (e.g. CoDel) was designed for, and it works quite well. If there is no congestion then you can transmit at line rate; if congestion appears then you start dropping packets selectively to maintain the configured QoS policy.
Of course, you only get to set the QoS policy on your own routers. The endpoints are generally the lowest-bandwidth and thus most congested points, but if congestion does appear elsewhere in the network then the administrators of those routers will determine the policy. In my opinion the best default policy would be a fair division of available bandwidth according to source addresses for IPv4 or source /64 prefixes for IPv6—by user, in other words, to the extent that one can be readily identified.
Transport-level protocols in user space
Posted Jun 29, 2016 1:48 UTC (Wed) by gdt (subscriber, #6284) [Link]
The endpoints are generally the lowest-bandwidth and thus most congested points
I see that assumption a lot in this discussion. For many nations the undersea capacity is the bottleneck; for Australia's NBN it is the links to the ISP connection point (the 'CVC'); for a campus network it might be the campus link, shared between thousands of downstream users; for a metro ethernet network it might be the link from the basement switch of the building, shared between all occupants of that multistory residential block.
In short, congestion immediately north of the endpoint -- although common for US consumers -- isn't universal enough to anchor a design for congestion control enforcement .
Transport-level protocols in user space
Posted Jun 29, 2016 7:59 UTC (Wed) by micka (subscriber, #38720) [Link]
north of ? I'm confused... Does it say something about the orientation of landline or cable installations ?
Transport-level protocols in user space
Posted Jul 2, 2016 13:18 UTC (Sat) by Wol (subscriber, #4433) [Link]
Cheers,
Wol
Transport-level protocols in user space
Posted Jun 29, 2016 17:56 UTC (Wed) by nybble41 (subscriber, #55106) [Link]
Short of the adoption of a protocol like Backward ECN (https://en.wikipedia.org/wiki/Network_congestion#BECN) there isn't much that *can* be done at the ISP or national level beyond attempting to ensure fair access for each end-user.
Transport-level protocols in user space
Posted Jul 1, 2016 21:25 UTC (Fri) by Wol (subscriber, #4433) [Link]
I think you're assuming knowledge and ability way beyond what is actually available in most homes. My router is supplied by the telco, and I have no idea how to configure it - I have actually actively wanted to NOT get involved in messing with it.
Cheers,
Wol
Transport-level protocols in user space
Posted Jun 25, 2016 7:06 UTC (Sat) by marcH (subscriber, #57642) [Link]
Far from ideal, however and unless you're a (myopic?) geek it's pretty much the only practical answer available for any network performance issue. Random example among many, many others: http://www.agnilam.com/league-of-ledgends-ping-lag-reduce...
For the special case of professional bachel..^H gamers who can't stand the latency of a single extra packet queued in their way at any point in time, this will stay the answer for the foreseeable future. There are just too many buffers all over the place since most *other* consumers want throughput (or at least they're made to believe they do) and typically don't have a clue what latency is. So good luck policing ALL these buffers.
I remember the days before mobile phones when telcos were trying to sell you additional landlines (on the same wires). I wonder if some tried that with gamers now?
> I think the point was that you could do your own traffic shaping on your home router to enforce fairness between devices (or whatever other policy you want), not that you would manually shut off devices which are using excessive bandwidth.
It was both. The former for the few dozens of LWN readers, and the latter for everyone else.
> We're long past the point where it is reasonable to simply assume that every device and application will employ fair congestion algorithms. Effective traffic shaping policies at the border are a necessity, and render tight control over the endpoints' congestion algorithms unnecessary. In any case, it doesn't make sense to allow applications to use UDP (which has no built-in congestion control) and yet forbid them from taking charge of the TCP congestion algorithm.
Quoting this entirely cause it's what I wish I could have written.
Transport-level protocols in user space
Posted Jun 21, 2016 17:06 UTC (Tue) by mtaht (guest, #11087) [Link]
It's not just "bufferbloat"!. App writers will open dozens of tcp connections, middlebox writers and nat will make deploying new message based transports like sctp and dccp impossible, and nobody will care until sneakernet becomes the best way to share maps and cat videos again.
Videoconferencing apps like jitsy and freeswitch do no congestion control already, and things that do do it, have been slow to take off or be incorporated in videoconferencing apps.
The latest wave of the "distributed web" folk are leveraging a library that allows an application to create connections over every possible transport (udp,tcp,utp,sctp,etc), "any way you want", and "as many as you want"... which applications like ipfs, actually do use any way they want, with as many as they possibly can, simultaneously... I've been working really hard on just making rational, limited uses on multiple tcp flows work halfway decently for years now... and I stared blankly at the presenter so proud of this work in this lib, and wondered if I should grab the mic, or merely go drinking. (I went drinking).
Nagle's basic premise - "every app is entitled to *one* packet in the network", is dead. So I give up. Bring it on. One benefit to collapsing networks on a more regular basis would lead to the development of more internal, rather than cloud applications, in particular.
Some other steps forward would be to, flippantly:
I would support a development API that had a question+response segment you had to answer correctly in order to write the next line of code.
#include <TOU.h>
Compiler: Have you read and understood http://ee.lbl.gov/papers/congavoid.pdf ?
Compiler: Have you read and understood rfc5348?
compiler: "OK, go ahead, and should your application become popular and you be wrong, my god have mercy on your soul, and you fix before the Internet collapses. There will be a quiz, later."
...
Bram Cohen has some lovely stories... one, about bittorrent taking out all of south korea at one point, and he was trying to build a low priority application in the first place. There was a very good idea in early versions of bittorrent - reducing the MSS size on congested links so a connection could still be transferring some data with minimal impact - he had to rip it out because hardware deployed in the field overloaded on such small packets in such volume.
(I think it's a deployable idea today). There's also been some work on making tcp's use sub-packet windows, and certainly I regard the IW10 default of linux as horrifically stupid from outside of the datacenter, looking in (e.g from the home or business).
...
So, if we end up writing applications in javascript with udp... which seem inevitable at this point, do I think app writers will be suitably cautious? No.
Which is where fq, enforced on the routers, comes into play, the only application you can hurt (mostly) is yourself. It has the benefit of having not been tried to any large extent, is much last nasty than what a typical middlebox would attempt to impose, and merely requires updating a few billion routers to use it before things get much worse.
aqm, enforced on the routers, also, applies saner limits to application crazyness.
I would, of course, love it, if a facebook tried to help out on getting stuff like fq_codel out to the edges before inflicting more hurt on the internet in this way.
...
I could see adding some sort of per-application limits on accessing stuff through this new API, also, like, no more than 6, and some application of fq techniques to multiplex app traffic through it, so no one application, accessing this api, could dominate the local machine.
...
Another thing that would help is to try and make sure the ipv6 deployment does have heavy users of newer protocols so that they are not firewalled or middleboxed away. dctp, sctp, udp-lite, etc. I would love it if a facebook or google were to make a service available over any of these native protocols over ipv6.
Transport-level protocols in user space
Posted Jun 21, 2016 21:18 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]
If applications can't deal with it - then it's a problem for individual end-user nodes.
Transport-level protocols in user space
Posted Jun 30, 2016 5:37 UTC (Thu) by marcH (subscriber, #57642) [Link]
> Bug reports from Joe Random getting ignored: what's new?
Example: one of the many, spot reports of bufferbloat. A very simple one. Reported in 2004, more than 5 years before Jim Gettys thoroughly studied bufferbloat and gave it a catchy name. According to https://lwn.net/Articles/617070/, this had been ignored until 2014.
I find nothing wrong with ignoring bug reports from nobodies; the reputation system serves a useful filtering purpose and this is the way it's supposed to work. Everything is fine... as long as no one pretends that all reports get looked at.
> TOU-related bug reports from Facebook, Amazon or Google coming with tentative fix attached? I bet they will be looked at.
Transport-level protocols in user space
Posted Jun 30, 2016 13:33 UTC (Thu) by paulj (subscriber, #341) [Link]
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
Fedora 24. qdisc + 1k qlen.
:( :( :(
(And yeah, I've raised bugs in the past, given suggested scripts for NM to set lower qlen - rejected).
Transport-level protocols in user space
Posted Jul 5, 2016 9:39 UTC (Tue) by Lennie (subscriber, #49641) [Link]
"Going back to what Marc said in an earlier e-mail about having txqueuelen in the unit of bytes rather than packets to provide a fixed queueing delay in ms rather than packets. Maintaining txqueuelen in ms would be an ideal solution, but probably hard to achieve in practice."
That is actually what CoDel does, drop some packets when the length of the queue measured _in time_ becomes to long (simplification).
Transport-level protocols in user space
Posted Jul 5, 2016 9:48 UTC (Tue) by paulj (subscriber, #341) [Link]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
But wlan still seems stuck on qdisc and a massive qlen. :(
Transport-level protocols in user space
Posted Jul 17, 2016 11:45 UTC (Sun) by mtaht (guest, #11087) [Link]
but we are making serious progress, now on two chipsets.
http://blog.cerowrt.org/post/fq_codel_on_ath10k/
https://blog.tohojo.dk/2016/06/fixing-the-wifi-performanc...
Coming soon to a kernel tree near you, I hope.
Transport-level protocols in user space
Posted Jul 17, 2016 14:57 UTC (Sun) by Lennie (subscriber, #49641) [Link]
Transport-level protocols in user space
Posted Jul 17, 2016 21:32 UTC (Sun) by flussence (subscriber, #85566) [Link]
Transport-level protocols in user space
Posted Jun 27, 2016 12:09 UTC (Mon) by smitty_one_each (subscriber, #28989) [Link]
There is always some minimal amount of metadata that needs to be managed across the application. Moving that management out of TCP might (or might not) bring joy.
Transport-level protocols in user space
Posted Jun 28, 2016 18:05 UTC (Tue) by flussence (subscriber, #85566) [Link]
NoSQL in this analogy would be something along the lines of SCTP+IPv6; that'd be a real improvement (and naturally, still wouldn't make everyone happy), but we don't have the luxury of choosing an appropriate tool for the job.
Transport-level protocols in user space
Posted Jul 18, 2016 16:16 UTC (Mon) by Hugne (guest, #82663) [Link]
TCP doesn't really like when the datapath changes to worse conditions.. New CCA's needed?