Professor Jed Crandall Measures Global Internet Censorship and Surveillance

October 20, 2015

Over the next three years, UNM Associate Professor of Computer Science Jed Crandall and co-PIs Prof. Stephanie Forrest (UNM) and Prof. Michalis Faloutsos (University of California) will be utilizing CARC resources to measure and log global instances of censorship and surveillance of the internet. This effort is funded by a new NSF grant that is part of the Networking Technology and Systems (NeTS) Program, and will involve three parts: modeling high traffic internet choke points; modeling censorship on social media sites such as Facebook and Twitter, including studying not just fully censored content; and measuring the internet.

“Normally if you want to, you can only measure what happens between two points—you and somebody else,” Crandall explains. “From here in New Mexico we could send packets—messages with data and destination addresses—to anyone in the world and look at the responses. But we don’t learn anything about the paths between, for example, other countries. The technique then is to spoof the IP address of one foreign machine, and send a message to another. Intuitively that should be useless, because we would never see the response. But if you look at the way the buffers and resources are being managed, you can tell what’s happening between the two machines.”

“As an analogy,” Crandall continues, “say you have a pen pal in Norway, and he can only respond to one letter a day. But you want to find out about letters that are being delivered—or being censored—between Norway and a hypothetical person in Turkey. So you send a letter to your pen pal in Norway using your own address. And you also spoof a letter, pretending to be your friend in Norway, and send it to this person in Turkey. Now the person in Turkey replies to Norway, and Norway has to write back to him. But remember he can only respond to one message a day, which means it now takes him twice as long to respond to you. So you know he also got the message from Turkey.”

CARC will host the data collection servers and 500 TB mass storage system for the project. Professor Crandall will be storing the results of his three-year long experiment on an expansion of the existing Research Storage Consortium (RSC) petabyte storage system. CARC and UNM IT are collaborating to provide the high-bandwidth, 10 Gbps+ data connection to the outside world that is essential for the internet measurements. The machines designated to collect the data will be isolated from the rest of the CARC and UNM networks, thus allowing Crandall to perform in-depth analyses on raw information collected from the internet while maintaining UNM net ID security. The project is a new and unique use case for the RSC, in terms of both the technical implementation and the extreme storage requirements of the project. The project will generate ~17 TB in the form of millions of files, necessitating a complete write-out 0f 200 TB to tape every 120 days.

“Basically, we want to measure everything we can about the internet every day. We’re partnering with CARC because we need a lot of bandwith. When we scale up our measurements, it could be that we’re averaging more internet traffic than the rest of UNM while we’re doing the measurements,” says Crandall.

“What I think we’ll find is that the amount of censorship that’s going on is a lot more vast than people realize. The internet used to be a big global thing. Anyone could talk to anyone, across borders. But now nations, states and companies all want to control their own little corners of the internet. And so the Web is breaking up into factions. Every group has its own little internet, and they only talk to one antoher in proscribed ways that whoever controls each group allows. You see a lot of reports—this country is blocking this website, that country won’t allow such-and-such content, etc. But nobody has actually measured the totality of it. That’s what we hope to do.”

In an article by the Daily Lobo, Prof. Crandall also discusses ways that members of the UNM community can protect and learn about their own digital freedoms.

Center for Advanced Research Computing

MSC01 1190
1601 Central Ave. NE
Albuquerque, NM 87106

p: 505.277.8249
f:  505.277.8235
e: