I dont see a way to characterize this one as simple ignorance. Matt Stoller falsely repeated the claim that Authentium had craigslist on a blacklist five days after Authentium posted the explanation as a comment on his own blog. Timothy Karr has pushed this on this blog, the Free Press Action HQ blog, and his own Media Citizen blog, also with the blacklist wording and also well after having the correct explanation in hand.
This is dishonest, plain and simple. If Karr and Stoller have any integrity, they will retract and apologize for lying.
PBCliberal
· 3 years ago
Craig Newmarks site is screwed up and hes blaming Cox for it - and seeking a new law. Thats taking Internet retardation to a whole new level.
From the summary:
Summary: craigslist told Cox to please speak to it very slowly. Cox did, but for longer than craigslist explicitly requested. Fixing this for craigslist could break other sites, so some caution in shipping a fix is justified.
I'd say both sites are screwed up, and "what we have here is a failure to communicate," and real live human administrators at both ends of the connection need to look at their configuration and work together.
Let's look at this in context. Its not the "superhighway" that is at issue here, its the "offramps," at which a free marketplace doesn't or barely exists. ("...only 2 percent of Americans get high-speed Internet access from someone other than their local phone company or cable provider--Findings section of S2360).
These offramps are controlled (at the telcos) by organizations who have decimated their staffs, often retiring or firing their most comptent and experienced employees because this represents the highest cost savings.
So no matter whom is at fault here, it is important that there be some way to compel these monopolies to respond when their systems are broken or even appear to be. Now that net neutrality agreements are expiring at the new mega-telcos and forced competition has been discarded by the FCC, the easy answer for the understaffed telcos who block access to sites (even inadvertantly), is: "Maybe we are, so what?"
The new megatelcos were quite happy with net neutrality when the carrot they were offered in exchange for it was the ability to eat other telcos. There's no reason they can't swallow it now through direct legislation, even though they're not being given a prize for it.
Jon Garfunkel
· 3 years ago
A few curious twists here:
One, I followed up with Tom Foremski, who was the prime amplifier of the story up to Stoller. I felt that as a blogger riding on his decades of his journalistic experience, he ought to get around to doing a follow-up. He told me a week ago that he'd like to, but was focusing on the Scoble announcement. So I would press Foremski for a follow-up.
Two, MyDD is running BlogAds for "What is the Future of the Internet" -- aka the non-network-neutrality camp. I wonder if anyone's noticed.
Richard Bennett
· 3 years ago
In the first place, the reclassification of DSL as an Information Service doesn't affect Cox, they're a cable provider, not a telco or a mega-telco.
In the second place, the problem is strictly at the Craig's List side and Craig's List has the ability to fix it anytime they want with no help from anybody.
Forcing the "monopolies" to respond in one way or another to Craig's incompetence isn't going to happen, no matter how many "neutrality" laws we pass.
PBCliberal
· 3 years ago
So, Richard, let me make sure I understand this. You say:
Forcing the monopolies to respond in one way or another to Craigs incompetence isnt going to happen, no matter how many neutrality laws we pass.
Alexa today rates Craig's List #27 of around 96,000,000 urls. The reason Craig's list is so highly rated is that most of the Internet is able to contact his list, despite his misconfiguration. Yet somehow that misconfiguration creates a three-month hiatus at a cable company that just happens to have a competing service.
The way a free market would solve this, is that Cox subscribers would have a littany of alternatives to choose from so there would be real business pressure on it to work with Craig to solve this. By claiming it is Craig's problem alone, you're contravening the very "authoritative" summary you posted, and raising the thorny question of why enough other subnets have so little problem taking Craig's packets that he's #27, yet cox.net is #1,182 and coc.com is #603.
I'll take it as agreement with me when you say the monopolies aren't going to respond to this issue no matter how many neutrality laws we pass. They'll find some way to circumvent the issue (how many years was AT&T able to perpetuate the myth that hooking an additional telephone to your home service would cause the immediate collapse of the public switched network)?
Therefore, net neutrality is only a first step until we can get some real competition at the last mile level.
Jim Lippard
· 3 years ago
PBCLiberal: "The way a free market would solve this, is that Cox subscribers would have a littany of alternatives to choose from so there would be real business pressure on it to work with Craig to solve this. By claiming it is Craigs problem alone, youre contravening the very authoritative summary you posted, and raising the thorny question of why enough other subnets have so little problem taking Craigs packets that hes #27, yet cox.net is #1,182 and coc.com is #603."
1. There already are solutions for Cox subscribers: (a) deinstall the software firewall (and as a security professional I'd recommend turning on an alternative rather than going without one) or (b) install the free beta that addresses the problem. The first solution has always been available and the second has been available since a couple of weeks after the problem was first reported. I'm a Cox customer, but never experienced this issue because I've never used the Cox-provided Authentium software.
2. There already are alternative providers for many, if not most Cox subscribers. You're right that last-mile competition is a concern, but it's not as big a concern in many metropolitan areas as some seem to think.
3. Comparing the popularity of the craigslist.org website to the popularity of the cox.net website isn't a good measure of the number of respective users--there are a lot more Cox eyeball customers using the Internet than there are users of Cox's web site.
Jon Garfunkel: "I followed up with Tom Foremski, who was the prime amplifier of the story up to Stoller. I felt that as a blogger riding on his decades of his journalistic experience, he ought to get around to doing a follow-up. He told me a week ago that hed like to, but was focusing on the Scoble announcement. So I would press Foremski for a follow-up."
Given that the complaint against Cox/Authentium is for taking months to fix the problem, it seems like there's a valid complaint against Foremski for taking months to correct the record. It's easier to post an update to a blog than it is to fix, test, and release a piece of widely-used consumer software for production.
Jim Lippard
· 3 years ago
I wrote "it seems like theres a valid complaint against Foremski for taking months to correct the record." Sorry, had the February date of the first report of the problem in mind. Foremski first posted about this on June 6, 12 days ago, and followed up on June 8 with some reader comments that included the correct explanation amongst them.
So there's no valid complaint against Foremski for "taking months to correct the record." Let my record stand corrected...
PBCliberal
· 3 years ago
3. Comparing the popularity of the craigslist.org website to the popularity of the cox.net website isnt a good measure of the number of respective usersthere are a lot more Cox eyeball customers using the Internet than there are users of Coxs web site.
Absolutely. This speaks to the level of customer service, which comes back to why we need free market pressures to movtiate these megaproviders to act responsibly. Since Cox and BellSouth's coverage area coincide, I know that a lot of the "alternatives" to BellSouth service are nothing but BellSouth dressed us another way. For instance, I am on a network that is not BellSouth, where BellSouth leases my provider far more than just access to the phone lines. I suspect this will markedly change now that they need not even project the pretense of providing competition. Of course, there's satellite, and even (now don't go a callin' me socialist) the spectre of municipal wifi.
As a matter of good customer service,Cox should be highly motivated to solve problems that affect popular websites. The contrast in the popularity of the website was included to show how popular Craigslist is compared with Cox, which offers a competing service. This is the classic case of content providers vs. content providers who also own the edge routers.
As a security professional, I assume you're used to the blank look on the faces of users to whom you say the word "Firewall," and the head shaking that usually comes tossing out: "Network Address Translation." The only people who should ever disable their firewalls are those experienced enough to raise their systems from the ashes if they are successfully attacked. I think most of today's internet users aren't in that category.
Richard Bennett
· 3 years ago
Nonetheless, the bug is Craig's, and he could have fixed it months ago. But instead of doing that, he helped organize a poltical movement to shackle ISPs. He's running around the world whining about "level playing fields" and still hasn't fixed his bug. He's a regular Mary Mapes.
Who's responsible for Craig Newmark's tech support?
max
· 3 years ago
"As a matter of good customer service,Cox should be highly motivated to solve problems that affect popular websites...."
Since when does Craig's list get to offload it's technical support duties for it's users onto third party telcos or ISPs?
PBCliberal
· 3 years ago
Since when does Craigs list get to offload its technical support duties for its users onto third party telcos or ISPs?
One of the claims being made by Lippard in this thread, is that one of the solutions offered to Cox users was to
install the free beta that addresses the problem.
So I'd say they've done that already. The duopoly boosters are talking out of both sides of their mouthes. They are simultaneously arguing that the blame lays squarely with Craigslist and that Cox is fixing it.
Actually, it was probably the users who wanted to get to Craiglist who offloaded their concerns onto their telcos and ISPs regarding a third-party site they believed they were paying to access.
Be careful with this line of argument, because it undercuts the "power of the marketplace" line being peddled as a reason we net neutrality supporters are trying to fix a problem that doesn't exist.
If the #27 website can't get Cox to act when its undoubtely being on its best behavior even though it is not compelled, one can only imagine the range of possibilities when the restrictions expire everywhere.
max
· 3 years ago
install the free beta that addresses the problem.
So Id say theyve done that already. The duopoly boosters are talking out of both sides of their mouthes. They are simultaneously arguing that the blame lays squarely with Craigslist and that Cox is fixing it.
Not exactly. Offering customers advice on installing a beta doesn't mean that the beta is fully supported by Cox, what Cox is offering is a "hack" or "work around" to a problem outside of their control. It's a common problem, but I don't go screaming to my congressman about it.
From a technical perspective, the blame and *correct* solution to the *problem* (Technically defined as Hosts violating RFCs) is to have Craigslist fix their content director/loadbalancer or server to be less RFC ignorant.
If the #27 website cant get Cox to act when its undoubtely being on its best behavior even though it is not compelled, one can only imagine the range of possibilities when the restrictions expire everywhere.
BTW: the popularity of a website is no excuse for having it violate basic protocol behaviour... that arguement sounds suspiciously Microsoft-esque.
PBCliberal
· 3 years ago
BTW: the popularity of a website is no excuse for having it violate basic protocol behaviour that arguement sounds suspiciously Microsoft-esque.
There is no question that everyone should follow the RFCs as closely as possible. Have you read RFC793? Authentium has, and their response was:
Our firewall driver responds by sending data only one byte at a time, even after the server increases the TCP window size. This is the glitch we have fixed and are QA testing.
I've read RFC793, and I'm not sure Craigslist is technically outside the specification, its just bad practice, unless its being used as a "cue" to send a 1-octet packet, which is what Authentium assumes it is. So, strictly speaking, Authentium is broken in that it responds with a single octet when none was allowed.
From what I've read, Authentium's Achillies Heel is that it doesn't respond correctly on subsequent packets when it is asked increase the window size. How this has been characterized as Craigslist's sole problem speaks more to the desire to deflect this issue than to comment on good protocol practice.
A lot of these arguments are taking the form that Craigslist is a fault because "they were the last one who could avoid the accident."
Sigivald
· 3 years ago
How can Cox "fix" the problem, anyway? Cox can't force its customers to upgrade their free firewall. Cox was already offering the upgraded beta version no?
And why, PBC, do you keep pushing at Cox, when the easy solution to the entire problem is for the Craigslist people to fix their broken window size?
You argument seems to take the form that Cox is at fault because they're not Craigslist. (And what "competing service" do they have? I mean, seriously? Nobody, figuratively speaking, has ever even heard of this service, which means it's not competing at all, as the only strength of something like Craigslist is that many people use it.
Hell, their website doesn't even mention such a service. The idea that they're somehow deliberately preventing a fix from going out to bolster their own competitor to Craigslist is... I don't even have words for what that is.)
Richard Bennett
· 3 years ago
Like a lot of networking problems that I see in my day job, this one involves the interaction of two bugs, one on Craig's List and the other in the Authentium firewall. Authentium didn't test their software with all possible buggy servers, and Craig's List hadn't encountered a firewall that didn't update the window size the second time it was sent. It's probably a "multiply by zero" issue.
So the bottom line is this: Authentium has fixed its bug, but Craig hasn't fixed his. Craig is still complaining about "discrimination" and Authentium is being gracious and taking full responsibility for the whole issue, even Craig's part.
And meanwhile, the Save The Internet/Kosola Krowd are still saying this is proof that we need harsh regulations against ISPs.
Cox Cable delivers Craig's List's packets to its customers computers just fine, so the problem can't remotely be attributed to malice on the part of Cox.
Craig Newmark has no credibility.
max
· 3 years ago
Ive read RFC793, and Im not sure Craigslist is technically outside the specification, its just bad practice, unless its being used as a cue to send a 1-octet packet, which is what Authentium assumes it is.
A zero window simply means that the client will need to recieve an ACK from $server after every attempt to communicate with $server. The result is an incredibly slow TCP conversation that requires additional overhead (chatty) communication that may exceed timeout thresholds of higher level protocols (E.g. HTTP) There is no rule against the client continuing the conversation with a server advertising a 0 Window. It just needs more ACKnowledgements from the server before requesting additional data.
From what Ive read, Authentiums Achillies Heel is that it doesnt respond correctly on subsequent packets when it is asked increase the window size.
This is only a problem when dealing with systems that don't properly negotiate window sizes in the first place, and it's not just an Authentium problem.. Many stateful firewalls (especially hostbased ones) only use the window sizes negotiated during the 3 way handshake, especially for stateless protocols like HTTP.
Craiglist's webserver appears to only boost the window after the 3 way handshake occurs. I've just confirmed that myself via tcpdump.
A lot of these arguments are taking the form that Craigslist is a fault because they were the last one who could avoid the accident.
I disagree. Craigslist is the only people in charge of what Windows their hardware/servers are advertising, and since running a 24/7 web infrastructure requires technically more clue than installing and operating a host based firewall, it seems that Craig's list could easily resolve all of the various problems they have with *MANY* firewalls (See their system status page for details) by fixing things on their end.
max
· 3 years ago
Richard said:
UPDATE 5: Craig Newmark still refuses to acknowledge his bug. All he has to do is correct his TCP settings and the whole problem goes away. Why wont he?
I'm not sure... but reading some of the issues Craigslist is having related to it's own firewall (see their system status page) I think they may be waiting on a vendor fix as well ;-)
All Comments
I dont see a way to characterize this one as simple ignorance. Matt Stoller falsely repeated the claim that Authentium had craigslist on a blacklist five days after Authentium posted the explanation as a comment on his own blog. Timothy Karr has pushed this on this blog, the Free Press Action HQ blog, and his own Media Citizen blog, also with the blacklist wording and also well after having the correct explanation in hand.
This is dishonest, plain and simple. If Karr and Stoller have any integrity, they will retract and apologize for lying.
From the summary:
I'd say both sites are screwed up, and "what we have here is a failure to communicate," and real live human administrators at both ends of the connection need to look at their configuration and work together.
Let's look at this in context. Its not the "superhighway" that is at issue here, its the "offramps," at which a free marketplace doesn't or barely exists. ("...only 2 percent of Americans get high-speed Internet access from someone other than their local phone company or cable provider--Findings section of S2360).
These offramps are controlled (at the telcos) by organizations who have decimated their staffs, often retiring or firing their most comptent and experienced employees because this represents the highest cost savings.
So no matter whom is at fault here, it is important that there be some way to compel these monopolies to respond when their systems are broken or even appear to be. Now that net neutrality agreements are expiring at the new mega-telcos and forced competition has been discarded by the FCC, the easy answer for the understaffed telcos who block access to sites (even inadvertantly), is: "Maybe we are, so what?"
The new megatelcos were quite happy with net neutrality when the carrot they were offered in exchange for it was the ability to eat other telcos. There's no reason they can't swallow it now through direct legislation, even though they're not being given a prize for it.
One, I followed up with Tom Foremski, who was the prime amplifier of the story up to Stoller. I felt that as a blogger riding on his decades of his journalistic experience, he ought to get around to doing a follow-up. He told me a week ago that he'd like to, but was focusing on the Scoble announcement. So I would press Foremski for a follow-up.
Two, MyDD is running BlogAds for "What is the Future of the Internet" -- aka the non-network-neutrality camp. I wonder if anyone's noticed.
In the second place, the problem is strictly at the Craig's List side and Craig's List has the ability to fix it anytime they want with no help from anybody.
Forcing the "monopolies" to respond in one way or another to Craig's incompetence isn't going to happen, no matter how many "neutrality" laws we pass.
Alexa today rates Craig's List #27 of around 96,000,000 urls. The reason Craig's list is so highly rated is that most of the Internet is able to contact his list, despite his misconfiguration. Yet somehow that misconfiguration creates a three-month hiatus at a cable company that just happens to have a competing service.
The way a free market would solve this, is that Cox subscribers would have a littany of alternatives to choose from so there would be real business pressure on it to work with Craig to solve this. By claiming it is Craig's problem alone, you're contravening the very "authoritative" summary you posted, and raising the thorny question of why enough other subnets have so little problem taking Craig's packets that he's #27, yet cox.net is #1,182 and coc.com is #603.
I'll take it as agreement with me when you say the monopolies aren't going to respond to this issue no matter how many neutrality laws we pass. They'll find some way to circumvent the issue (how many years was AT&T able to perpetuate the myth that hooking an additional telephone to your home service would cause the immediate collapse of the public switched network)?
Therefore, net neutrality is only a first step until we can get some real competition at the last mile level.
"The way a free market would solve this, is that Cox subscribers would have a littany of alternatives to choose from so there would be real business pressure on it to work with Craig to solve this. By claiming it is Craigs problem alone, youre contravening the very authoritative summary you posted, and raising the thorny question of why enough other subnets have so little problem taking Craigs packets that hes #27, yet cox.net is #1,182 and coc.com is #603."
1. There already are solutions for Cox subscribers: (a) deinstall the software firewall (and as a security professional I'd recommend turning on an alternative rather than going without one) or (b) install the free beta that addresses the problem. The first solution has always been available and the second has been available since a couple of weeks after the problem was first reported. I'm a Cox customer, but never experienced this issue because I've never used the Cox-provided Authentium software.
2. There already are alternative providers for many, if not most Cox subscribers. You're right that last-mile competition is a concern, but it's not as big a concern in many metropolitan areas as some seem to think.
3. Comparing the popularity of the craigslist.org website to the popularity of the cox.net website isn't a good measure of the number of respective users--there are a lot more Cox eyeball customers using the Internet than there are users of Cox's web site.
Jon Garfunkel: "I followed up with Tom Foremski, who was the prime amplifier of the story up to Stoller. I felt that as a blogger riding on his decades of his journalistic experience, he ought to get around to doing a follow-up. He told me a week ago that hed like to, but was focusing on the Scoble announcement. So I would press Foremski for a follow-up."
Given that the complaint against Cox/Authentium is for taking months to fix the problem, it seems like there's a valid complaint against Foremski for taking months to correct the record. It's easier to post an update to a blog than it is to fix, test, and release a piece of widely-used consumer software for production.
So there's no valid complaint against Foremski for "taking months to correct the record." Let my record stand corrected...
Absolutely. This speaks to the level of customer service, which comes back to why we need free market pressures to movtiate these megaproviders to act responsibly. Since Cox and BellSouth's coverage area coincide, I know that a lot of the "alternatives" to BellSouth service are nothing but BellSouth dressed us another way. For instance, I am on a network that is not BellSouth, where BellSouth leases my provider far more than just access to the phone lines. I suspect this will markedly change now that they need not even project the pretense of providing competition. Of course, there's satellite, and even (now don't go a callin' me socialist) the spectre of municipal wifi.
As a matter of good customer service,Cox should be highly motivated to solve problems that affect popular websites. The contrast in the popularity of the website was included to show how popular Craigslist is compared with Cox, which offers a competing service. This is the classic case of content providers vs. content providers who also own the edge routers.
As a security professional, I assume you're used to the blank look on the faces of users to whom you say the word "Firewall," and the head shaking that usually comes tossing out: "Network Address Translation." The only people who should ever disable their firewalls are those experienced enough to raise their systems from the ashes if they are successfully attacked. I think most of today's internet users aren't in that category.
Who's responsible for Craig Newmark's tech support?
Since when does Craig's list get to offload it's technical support duties for it's users onto third party telcos or ISPs?
One of the claims being made by Lippard in this thread, is that one of the solutions offered to Cox users was to
So I'd say they've done that already. The duopoly boosters are talking out of both sides of their mouthes. They are simultaneously arguing that the blame lays squarely with Craigslist and that Cox is fixing it.
Actually, it was probably the users who wanted to get to Craiglist who offloaded their concerns onto their telcos and ISPs regarding a third-party site they believed they were paying to access.
Be careful with this line of argument, because it undercuts the "power of the marketplace" line being peddled as a reason we net neutrality supporters are trying to fix a problem that doesn't exist.
If the #27 website can't get Cox to act when its undoubtely being on its best behavior even though it is not compelled, one can only imagine the range of possibilities when the restrictions expire everywhere.
So Id say theyve done that already. The duopoly boosters are talking out of both sides of their mouthes. They are simultaneously arguing that the blame lays squarely with Craigslist and that Cox is fixing it.
Not exactly. Offering customers advice on installing a beta doesn't mean that the beta is fully supported by Cox, what Cox is offering is a "hack" or "work around" to a problem outside of their control. It's a common problem, but I don't go screaming to my congressman about it.
From a technical perspective, the blame and *correct* solution to the *problem* (Technically defined as Hosts violating RFCs) is to have Craigslist fix their content director/loadbalancer or server to be less RFC ignorant.
If the #27 website cant get Cox to act when its undoubtely being on its best behavior even though it is not compelled, one can only imagine the range of possibilities when the restrictions expire everywhere.
BTW: the popularity of a website is no excuse for having it violate basic protocol behaviour... that arguement sounds suspiciously Microsoft-esque.
There is no question that everyone should follow the RFCs as closely as possible. Have you read RFC793? Authentium has, and their response was:
I've read RFC793, and I'm not sure Craigslist is technically outside the specification, its just bad practice, unless its being used as a "cue" to send a 1-octet packet, which is what Authentium assumes it is. So, strictly speaking, Authentium is broken in that it responds with a single octet when none was allowed.
From what I've read, Authentium's Achillies Heel is that it doesn't respond correctly on subsequent packets when it is asked increase the window size. How this has been characterized as Craigslist's sole problem speaks more to the desire to deflect this issue than to comment on good protocol practice.
A lot of these arguments are taking the form that Craigslist is a fault because "they were the last one who could avoid the accident."
And why, PBC, do you keep pushing at Cox, when the easy solution to the entire problem is for the Craigslist people to fix their broken window size?
You argument seems to take the form that Cox is at fault because they're not Craigslist. (And what "competing service" do they have? I mean, seriously? Nobody, figuratively speaking, has ever even heard of this service, which means it's not competing at all, as the only strength of something like Craigslist is that many people use it.
Hell, their website doesn't even mention such a service. The idea that they're somehow deliberately preventing a fix from going out to bolster their own competitor to Craigslist is... I don't even have words for what that is.)
So the bottom line is this: Authentium has fixed its bug, but Craig hasn't fixed his. Craig is still complaining about "discrimination" and Authentium is being gracious and taking full responsibility for the whole issue, even Craig's part.
And meanwhile, the Save The Internet/Kosola Krowd are still saying this is proof that we need harsh regulations against ISPs.
Cox Cable delivers Craig's List's packets to its customers computers just fine, so the problem can't remotely be attributed to malice on the part of Cox.
Craig Newmark has no credibility.
A zero window simply means that the client will need to recieve an ACK from $server after every attempt to communicate with $server. The result is an incredibly slow TCP conversation that requires additional overhead (chatty) communication that may exceed timeout thresholds of higher level protocols (E.g. HTTP) There is no rule against the client continuing the conversation with a server advertising a 0 Window. It just needs more ACKnowledgements from the server before requesting additional data.
From what Ive read, Authentiums Achillies Heel is that it doesnt respond correctly on subsequent packets when it is asked increase the window size.
This is only a problem when dealing with systems that don't properly negotiate window sizes in the first place, and it's not just an Authentium problem.. Many stateful firewalls (especially hostbased ones) only use the window sizes negotiated during the 3 way handshake, especially for stateless protocols like HTTP.
Craiglist's webserver appears to only boost the window after the 3 way handshake occurs. I've just confirmed that myself via tcpdump.
A lot of these arguments are taking the form that Craigslist is a fault because they were the last one who could avoid the accident.
I disagree. Craigslist is the only people in charge of what Windows their hardware/servers are advertising, and since running a 24/7 web infrastructure requires technically more clue than installing and operating a host based firewall, it seems that Craig's list could easily resolve all of the various problems they have with *MANY* firewalls (See their system status page for details) by fixing things on their end.
UPDATE 5: Craig Newmark still refuses to acknowledge his bug. All he has to do is correct his TCP settings and the whole problem goes away. Why wont he?
I'm not sure... but reading some of the issues Craigslist is having related to it's own firewall (see their system status page) I think they may be waiting on a vendor fix as well ;-)