Beth Noveck: think about how to open up the API of government

Former White House deputy CTO for open government Beth Noveck recently gave a TED Talk on a “more open source government. For more perspective in the vein of “open,” read CNN’s TED summary, “What if you could make anything you wanted?” Noveck’s talk is embedded below:

“…start by teaching young people that we live, not in a passive society, a read-only society, but in a writable society, where we have the power to change our communities, to change our institutions, that’s when we begin to really put ourselves on the pathway towards this open government innovation”

White House announces 200m in funding for big data research and development, hosts forum at AAAS

In 2012, making sense of big data through narrative and context, particularly unstructured data, is now a strategic imperative for leaders around the world, whether they serve in Washington, run media companies or trading floors in New York City or guide tech titans in Silicon Valley.

While big data carries the baggage of huge hype, the institutions of federal government are getting serious about its genuine promise. On Thursday morning, the Obama Administration announced a “Big Data Research and Development Initiative,” with more than $200 million in new commitments. (See fact sheet provided by the White House Office of Science and technology policy at the bottom of this post.)

“In the same way that past Federal investments in information-technology R&D led to dramatic advances in supercomputing and the creation of the Internet, the initiative we are launching today promises to transform our ability to use Big Data for scientific discovery, environmental and biomedical research, education, and national security,” said Dr. John P. Holdren, Assistant to the President and Director of the White House Office of Science and Technology Policy, in a prepared statement.

The research and development effort will focus on advancing “state-of-the-art core technologies” need for big data, harnessing said technologies “to accelerate the pace of discovery in science and engineering, strengthen our national security, and transform teaching and learning,” and “expand the workforce needed to develop and use Big Data technologies.”

In other words, the nation’s major research institutions will focus on improving available technology to collect and use big data, apply them to science and national security, and look for ways to train more data scientists.

“IBM views Big Data as organizations’ most valuable natural resource, and the ability to use technology to understand it holds enormous promise for society at large,” said David McQueeney, vice president of software, IBM Research, in a statement. “The Administration’s work to advance research and funding of big data projects, in partnership with the private sector, will help federal agencies accelerate innovations in science, engineering, education, business and government.”

While $200 million dollars is a relatively small amount of funding, particularly in the context of the federal budget or as compared to investments that are (probably) being made by Google or other major tech players, specific support for training and subsequent application of big data within federal government is important and sorely needed. The job market for data scientists in the private sector is so hot that government may well need to build up its own internal expertise, much in the same way Living Social is training coders at the Hungry Academy.

Big data is a big deal,” blogged Tom Kalil, deputy director for policy at White House OSTP, at the White House blog this morning.

We also want to challenge industry, research universities, and non-profits to join with the Administration to make the most of the opportunities created by Big Data. Clearly, the government can’t do this on its own. We need what the President calls an “all hands on deck” effort.

Some companies are already sponsoring Big Data-related competitions, and providing funding for university research. Universities are beginning to create new courses—and entire courses of study—to prepare the next generation of “data scientists.” Organizations like Data Without Borders are helping non-profits by providing pro bono data collection, analysis, and visualization. OSTP would be very interested in supporting the creation of a forum to highlight new public-private partnerships related to Big Data.

The White House is hosting a forum today in Washington to explore the challenges and opportunities of big data and discuss the investment. The event will be streamed online in live webcast from the headquarters of the AAAS in Washington, DC. I’ll be in attendance and sharing what I learn.

“Researchers in a growing number of fields are generating extremely large and complicated data sets, commonly referred to as ‘big data,'” reads the invitation to the event from the White House Office of Science and Technology Policy. “A wealth of information may be found within these sets, with enormous potential to shed light on some of the toughest and most pressing challenges facing the nation. To capitalize on this unprecedented opportunity — to extract insights, discover new patterns and make new connections across disciplines — we need better tools to access, store, search, visualize, and analyze these data.”

Speakers:

  • John Holdren, Assistant to the President and Director, White House Office of Science and Technology Policy
  • Subra Suresh, Director, National Science Foundation
  • Francis Collins, Director, National Institutes of Health
  • William Brinkman, Director, Department of Energy Office of Science

Panel discussion:

  • Moderator: Steve Lohr, New York Times, author of “Big Data’s Impact in the World
  • Alex Szalay, Johns Hopkins University
  • Lucila Ohno-Machado, UC San Diego
  • Daphne Koller, Stanford
  • James Manyika, McKinsey

What is big data?

Anyone planning for big data to use data for public good — or profit — through applied data science must know first understand what big data is.

On that count, turn to my colleague Edd Dumbill, who posted a useful definition last year on the O’Reilly Radar in his introduction to the big data landscape:

Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.

The hot IT buzzword of 2012, big data has become viable as cost-effective approaches have emerged to tame the volume, velocity and variability of massive data. Within this data lie valuable patterns and information, previously hidden because of the amount of work required to extract them. To leading corporations, such as Walmart or Google, this power has been in reach for some time, but at fantastic cost. Today’s commodity hardware, cloud architectures and open source software bring big data processing into the reach of the less well-resourced. Big data processing is eminently feasible for even the small garage startups, who can cheaply rent server time in the cloud.

Teams of data scientists are increasingly leveraging a powerful, growing set of common tools, whether they’re employed by government technologists opening cities, developers driving a revolution in healthcare or hacks and hackers defining the practice of data journalism.

To learn more about the growing ecosystem of big data tools, watch my interview with Cloudera architect Doug Cutting, embedded below. @Cutting created Lucerne and led the Hadoop project at Yahoo before he joined Cloudera. Apache Hadoop is an open source framework that allows distributed applications based upon the MapReduce paradigm to run on immense clusters of commodity hardware, which in turn enables the processing of massive amounts of big data.

Details on the administration’s big data investments

A fact sheet released by the White House OSTP follows, verbatim:

National Science Foundation and the National Institutes of Health – Core Techniques and Technologies for Advancing Big Data Science & Engineering

“Big Data” is a new joint solicitation supported by the National Science Foundation (NSF) and the National Institutes of Health (NIH) that will advance the core scientific and technological means of managing, analyzing, visualizing, and extracting useful information from large and diverse data sets. This will accelerate scientific discovery and lead to new fields of inquiry that would otherwise not be possible. NIH is particularly interested in imaging, molecular, cellular, electrophysiological, chemical, behavioral, epidemiological, clinical, and other data sets related to health and disease.

National Science Foundation: In addition to funding the Big Data solicitation, and keeping with its focus on basic research, NSF is implementing a comprehensive, long-term strategy that includes new methods to derive knowledge from data; infrastructure to manage, curate, and serve data to communities; and new approaches to education and workforce development. Specifically, NSF is:

· Encouraging research universities to develop interdisciplinary graduate programs to prepare the next generation of data scientists and engineers;
· Funding a $10 million Expeditions in Computing project based at the University of California, Berkeley, that will integrate three powerful approaches for turning data into information – machine learning, cloud computing, and crowd sourcing;
· Providing the first round of grants to support “EarthCube” – a system that will allow geoscientists to access, analyze and share information about our planet;
Issuing a $2 million award for a research training group to support training for undergraduates to use graphical and visualization techniques for complex data.
Providing $1.4 million in support for a focused research group of statisticians and biologists to determine protein structures and biological pathways.
· Convening researchers across disciplines to determine how Big Data can transform teaching and learning.

Department of Defense – Data to Decisions: The Department of Defense (DoD) is “placing a big bet on big data” investing approximately $250 million annually (with $60 million available for new research projects) across the Military Departments in a series of programs that will:

*Harness and utilize massive data in new ways and bring together sensing, perception and decision support to make truly autonomous systems that can maneuver and make decisions on their own.
*Improve situational awareness to help warfighters and analysts and provide increased support to operations. The Department is seeking a 100-fold increase in the ability of analysts to extract information from texts in any language, and a similar increase in the number of objects, activities, and events that an analyst can observe.

To accelerate innovation in Big Data that meets these and other requirements, DoD will announce a series of open prize competitions over the next several months.

In addition, the Defense Advanced Research Projects Agency (DARPA) is beginning the XDATA program, which intends to invest approximately $25 million annually for four years to develop computational techniques and software tools for analyzing large volumes of data, both semi-structured (e.g., tabular, relational, categorical, meta-data) and unstructured (e.g., text documents, message traffic). Central challenges to be addressed include:

· Developing scalable algorithms for processing imperfect data in distributed data stores; and
· Creating effective human-computer interaction tools for facilitating rapidly customizable visual reasoning for diverse missions.

The XDATA program will support open source software toolkits to enable flexible software development for users to process large volumes of data in timelines commensurate with mission workflows of targeted defense applications.

National Institutes of Health – 1000 Genomes Project Data Available on Cloud: The National Institutes of Health is announcing that the world’s largest set of data on human genetic variation – produced by the international 1000 Genomes Project – is now freely available on the Amazon Web Services (AWS) cloud. At 200 terabytes – the equivalent of 16 million file cabinets filled with text, or more than 30,000 standard DVDs – the current 1000 Genomes Project data set is a prime example of big data, where data sets become so massive that few researchers have the computing power to make best use of them. AWS is storing the 1000 Genomes Project as a publically available data set for free and researchers only will pay for the computing services that they use.

Department of Energy – Scientific Discovery Through Advanced Computing: The Department of Energy will provide $25 million in funding to establish the Scalable Data Management, Analysis and Visualization (SDAV) Institute. Led by the Energy Department’s Lawrence Berkeley National Laboratory, the SDAV Institute will bring together the expertise of six national laboratories and seven universities to develop new tools to help scientists manage and visualize data on the Department’s supercomputers, which will further streamline the processes that lead to discoveries made by scientists using the Department’s research facilities. The need for these new tools has grown as the simulations running on the Department’s supercomputers have increased in size and complexity.

US Geological Survey – Big Data for Earth System Science: USGS is announcing the latest awardees for grants it issues through its John Wesley Powell Center for Analysis and Synthesis. The Center catalyzes innovative thinking in Earth system science by providing scientists a place and time for in-depth analysis, state-of-the-art computing capabilities, and collaborative tools invaluable for making sense of huge data sets. These Big Data projects will improve our understanding of issues such as species response to climate change, earthquake recurrence rates, and the next generation of ecological indicators.”

Further details about each department’s or agency’s commitments can be found at the following websites by 2 pm today:

NSF: http://www.nsf.gov/news/news_summ.jsp?cntn_id=123607
HHS/NIH: http://www.nih.gov/news/health/mar2012/nhgri-29.htm
DOE: http://science.energy.gov/news/
DOD: www.DefenseInnovationMarketplace.mil
DARPA: http://www.darpa.mil/NewsEvents/Releases/2012/03/29.aspx
USGS: http://powellcenter.usgs.gov

IBM infographic on big data

Big Data: The New Natural Resource

This post and headline have been updated as more information on the big data R&D initiative became available.

White House releases Consumer Privacy Bill of Rights

After years of wrangling about online privacy in Washington, the White House has unveiled a Consumer Privacy Bill of Rights. A coalition of Internet giants, including Google, Yahoo, Microsoft, and AOL, have committed to adopt “Do Not Track technology” in most Web browsers by the end of 2012.

These companies, which deliver almost 90 percent of online behavioral advertisements, have agreed not to track consumers if these choose to opt out of online tracking using the Do Not Track mechanism, which will likely manifest as a button or browser plug-in. All companies that have made this commitment will be subject to FTC enforcement.

“American consumers can’t wait any longer for clear rules of the road that ensure their personal information is safe online,” said President Obama in a prepared statement. “As the Internet evolves, consumer trust is essential for the continued growth of the digital economy. That’s why an online privacy Bill of Rights is so important. For businesses to succeed online, consumers must feel secure. By following this blueprint, companies, consumer advocates and policymakers can help protect consumers and ensure the Internet remains a platform for innovation and economic growth.”

The announcement coincided with the release of a long awaited white paper: Consumer Data Privacy in a Networked World: A Framework for Protecting Privacy and Promoting Innovation in the Global Digital Economy. (Embedded below.)

The Center for Democracy & Technology (CDT) welcomed the Administration’s unveiling of this “Consumer Privacy Bill of Rights,” calling the industry announcement by industry to respect “Do Not Track” settings in Web browsers is “a positive step for consumer privacy.”

“The Administration’s call for a comprehensive privacy bill of rights comes at a pivotal time when there is a tremendous concern among consumers about their personal information,” said CDT President Leslie Harris in a prepared statement. “While we believe legislation will likely be necessary to achieve these protections, we support the White Paper’s call for the development of consensus rules on emerging privacy issues to be worked out by industry, civil society, and regulators.”

“For five years CDT has pushed for the development of a reliable ‘Do Not Track’ mechanism; today’s Digital Advertising Alliance announcement is an important step toward making ‘Do Not Track’ a reality for consumers,” said CDT’s Director of Consumer Privacy Justin Brookman in a prepared statement. “The industry deserves credit for this commitment, though the details of exactly what ‘Do Not Track’ means still need to be worked out,” Brookman said. “CDT will continue to work through the W3C standards setting process to develop strong and workable ‘Do Not Track’ guidelines.”

As Edward Wyatt reported at the New York Times, however, implementation of these online privacy guidelines won’t be just a matter of adding some lines of code:

Much remains to be done before consumers can click on a button in their Web browser to set their privacy standards. Congress will probably have to write legislation governing the collection and use of personal data, officials said, something that is unlikely to occur this year. And the companies that make browsers — Google, Microsoft, Apple and others — will have to agree to the new standards.

There will be a press conference tomorrow, streamed live from the White House. (Much more to come on this story tomorrow, though given that I’ll be traveling, you’ll be reading it elsewhere.)

A Consumer Privacy Bill of Rights

· Individual Control: Consumers have a right to exercise control over what personal data organizations collect from them and how they use it.

· Transparency: Consumers have a right to easily understandable information about privacy and security practices.

· Respect for Context: Consumers have a right to expect that organizations will collect, use, and disclose personal data in ways that are consistent with the context in which consumers provide the data.

· Security: Consumers have a right to secure and responsible handling of personal data.

· Access and Accuracy: Consumers have a right to access and correct personal data in usable formats, in a manner that is appropriate to the sensitivity of the data and the risk of adverse consequences to consumers if the data are inaccurate.

· Focused Collection: Consumers have a right to reasonable limits on the personal data that companies collect and retain.

· Accountability: Consumers have a right to have personal data handled by companies with appropriate measures in place to assure they adhere to the Consumer Privacy Bill of Rights.

White House Privacy White Paper

This story has been updated as more statements and news stories came online.

Celebrating science with the Geek in Chief at the White House Science Fair

Today in Washington, President Obama hosted the second annual White House Science Fair. Video of his comments is embedded below, along with a storify of exhibits and students from the day.

“The young people I met today, the young people behind me — you guys inspire me. It’s young people like you that make me so confident that America’s best days are still to come. When you work and study and excel at what you’re doing in math and science, when you compete in something like this, you’re not just trying to win a prize today. You’re getting America in shape to win the future. You’re making sure we have the best, smartest, most skilled workers in the world, so that the jobs and industries of tomorrow take root right here. You’re making sure we’ll always be home to the most creative entrepreneurs, the most advanced science labs and universities. You’re making sure America will win the race to the future.

So as an American, I’m proud of you. As your President, I think we need to make sure your success stories are happening all across our country.

And that’s why when I took office, I called for an all-hands-on-deck approach to science, math, technology and engineering. Let’s train more teachers. Let’s get more kids studying these subjects. Let’s make sure these fields get the respect and attention that they deserve.

Now, in a lot of ways, today is a celebration of the new. But the belief that we belong on the cutting edge of innovation — that’s an idea as old as America itself. I mean, we’re a nation of tinkerers and dreamers and believers in a better tomorrow. You think about our Founding Fathers — they were all out there doing experiments — and folks like Benjamin Franklin and Thomas Jefferson, they were constantly curious about the world around them and trying to figure out how can we help shape that environment so that people’s lives are better.

It’s in our DNA. We know that innovation has helped each generation pass down that basic American promise, which is no matter who you are, no matter where you come from, you can make it if you try. So there’s nothing more important than keeping that promise alive for the next generation. There’s no priority I have that’s higher than President — as President than this.

And I can’t think of a better way to spend a morning than with the young people who are here doing their part and creating some unbelievable stuff in the process. So I’m proud of you. I want you to keep up your good work.-President Barack Obama

 

 

Later in the day, Bill Nye, “The Science Guy,” Neil Tyson Degrasse and Tom Kalil participated in a live Twitter chat:

Expert Labs data: How does the @WhiteHouse drive engagement on Twitter? [INFOGRAPHIC]

Over at ExpertLabs, Andy Baio created a snazzy infographic of engagement around the White House’s Twitter account using data collected through the ThinkUp App.

There are lots of views into engagement on Twitter, but we have the data to give a unique view into what it looks like from the @whitehouse perspective.

We’ve tracked their activity for the last couple years using ThinkUp to analyze and publicly release large datasets. We decided it might be nice show how the White House engaged their audience last year — without resorting to cheap gimmicks like linkbait infographics.

As Baio points out, if you want to work some mojo on this data set, you can download the .CSV file and have some fun. Kudos to the Expert Labs team for making both the open data and visualization available to all.

Open Government News on Gov 2.0 TV: The Year in Review, SOPA and POTUS on Google+

On Thursday, I joined Edmonton-based social media consultant and digital strategist Walter Schwabe on “Gov 2.0 TV” to talk about what’s new in open government since our last interview.

Over the course of the show, we talked about the following stories:

4 reasons #40dollars resonated more with citizens on Twitter than #1000days

Yesterday, David Copeland reported at ReadWriteWeb that the GOP tried to replicate the success of the White House’s #40dollars social media campaign on Twitter with their own #1000days effort. As the Chicago Tribune reported, the GOP campaign sought to highlight an inauspicious milestone for the U.S. Senate. 1,000 since it passed a budget. Democrats, who control the Senate, last approved a budget in 2009.

Writes Copeland, “It’s clear that the digitial [sic] media campaigns had different goals, and #1000days was primarily aimed at emphasizing a point that was notably absent in President Obama’s State of the Union address last night. But if social media as it pertains to politics is truly about connecting with voters and constituents, score one for the Democrats.”

In this particular case, I mostly agree. The GOP’s efforts at gop.gov/sotu this year constituted an unprecedented use of the Web and social media by an opposition party to respond to a State of the Union, with smart integration of Twitter and YouTube. Citizens asked questions on the #GOPSOTU hashtag and Members of Congress responded using YouTube. As the Daily Dot reported, the #1000days hashtag has failed to spread beyond the Beltway. From what I’ve seen, the four reasons why #1000days hasn’t resonated in the same way break down into structural, tactical, and strategic issues:

1) Structural issue: Reach. Based upon the statistics I’ve seen, the @WhiteHouse has much more reach than than any single other “governance” account on Twitter. The GOP caucus in Congress, former Massachusetts governor@MittRomney and the @Heritage Foundation do have, in aggregate, an even or greater number of engaged followers. That said, the @BarackObama campaign account, which amplified the #40dollars conversation, has far greater reach, if lower engagement. Both metrics matter, in terms of the ability to involving and focusing more citizens in a given conversation around a #hashtag at a given time.

2) Tactical issue: Timing. The #1000days campaign was launched during the #SOTU, when the attention of politically engaged Americans was fractured between paying attention to the President’s speech itself, watching online (3m+ visits to wh.gov/sotu), reading the media organizations competing to report or fact check on the speech online, watching the TV networks and, of course, talking to one another.

3) Strategic issue: Adaptability. Agreeing upon and passing a budget is a fundamental, basic issue for the operations of any business, organization or government entity. Congress and the Obama administration have cobbled together a series of continuing resolutions and omnibus bills to fund itself over the past 3 years. While many Americans have to make and live by budgets in their personal lives and businesses, however, the #1000days campaign may be both too abstract and too constrained to a single message. The question about #40dollars, by contrast, asked citizens what it means to them, which is concrete, personal and invites creative answers.

4) Tactical issue: Engagement and Amplification. As Copeland reports, “Ahead of last night’s State of the Union address, Sen. John McCain, R-Ariz., GOP presidential candidate Mitt Romney and other Republicans started tweeting using the hashtag #1000days to accent the amount of time since Senate Democrats passed a federal budget.”

On Tuesday night, the top tweets for a search of the #1000days hashtag come from @MittRomney and Republican politicians. Neither Romney nor @SenJohnMcCain had retweeted any followers who have used the hashtag. @SpeakerBoehner has primarily retweeted the @GOPconference or other members of his caucus. The Heritage Foundation has only retweeted its own staff. That pattern is replicated throughout other participating accounts.

The @WhiteHouse, in contrast, continued its practice of resharing tweets from Twitter users who joined the conversation, sharing the voices of citizens with one another, not just other politicians. There’s a good lesson in this successful use to of Twitter that should extend well beyond citizen engagement and open government circles. One campaign amplified the messages of the representatives, the other channeled the voices of constituents responding to their elected issues on on a given issue back through the accounts coordinating the effort.

As I pointed out last year in an article on social media, politics and influence, it’s of note that the operators of the @WhiteHouse Twitter account now routinely natively retweet other accounts participating in #WHchats. While some of these Tweets will leave followers without context for the Tweet, the White House appears to have shifted its online strategy to one of engagement versus the lower risk style broadcasting that most politicians adopt online. To date, many of the president’s political opponents have not followed suit.

The challenges of these four issues look validated by the results to date: some 6,000 tweets per hour for #40dollars at the height of the campaign, as Ed O’Keefe wrote at the Washington Post. Keefe, on a talk on Monday, given by Kori Schulman, White House deputy director for digital strategy, “by 5 p.m., #40dollars was trending worldwide, Schulman said, and the hashtag was generating about 6,000 tweets per hour. At the height of the push, WhiteHouse.gov received about 5,000 responses per hour to the question.” In total, Schulman said the #40dollars campaign “generated 70,000 tweets, 46,000 submissions via the White House Web site, 10,000 related Facebook posts and contributions from 126,000 users.”

By way of contrast, according to the numbers in Topsy, the #1000days campaign has generated 3,862 tweets in the past week.

Agree? Disagree? What am I missing here.

Let’s get behind open data initiatives, says venture capitalist Fred Wilson

Writing on his widely read blog, influential New York City venture capitalist Fred Wilson urged developers to adopt the adopt the Green Button, the project that United States Chief Technology Officer Aneesh Chopra launched last week to unleash energy data. Chopra challenged the energy community to model the Green Button after the Blue Button, which enables veterans (and soon all federal workers) to download their personal health data. I quote from Wilson’s post, below. (Emphases are mine.)

This is the kind of innovation that gets me excited. The Green Button is like OAuth for energy data. It is a simple standard that the utilities can implement on one side and web/mobile deveopers can implement on the other side. And the result is a ton of information sharing about energy consumption and in all liklihood energy savings that result from more informed consumers.

The Green Button follows on the success of the Blue Button, a similar initiative that allows veterans to get at their medical data.

I’m a big fan of simplicity and open standards to unleash a lot of innovation. APIs and open data aren’t always simple concepts for end users. Green Buttons and Blue Buttons are pretty simple concepts that most consumers will understand. I’m hoping we soon see Yellow Buttons, Red Buttons, Purple Buttons, and Orange Buttons too.

Let’s get behind these open data initiatives. Let’s build them into our apps. And let’s pressure our hospitals, utilities, and other institutions to support them. I’m going to reach out to ConEd, the utility in NYC, and find out when they are going to add Green Button support to their consumers data. I hope it is soon.

This strikes me as an important data point, endorsement and call to action. Let’s see what happens. After a huge year of changes and progress for Gov 2.0 in 2011, open data looks poised to take off in 2012.

For more about the Green Button initiative, watch the video interview with the nation’s first CTO, below. (Hat tip PG & E.)

Rep. Smith pulls DNS provision from SOPA, Rep Issa postpones hearing, White House responds to epetition

The Friday night news dump lives on: at 12:30 AM last night, I received an email from the U.S. House Committee on Oversight and Government Reform: according to the release, Rep. Lamar Smith said he will remove the domain name provision from the Stop Online Piracy Act. Rep. Darrell Issa says he’ll suspend next week’s hearing with Reddit co-founder Alexis Ohanian & other Internet experts. As you may have heard, the United States Congress is considering anti-piracy bills that could cripple Internet industries that are engine of the dynamic economic growth all around the world: the Stop Online Piracy Act (SOPA) in the U.S. House of Representatives and the PROTECT IP Act in the U.S. Senate.

Here’s the release:

House Oversight and Government Reform Committee Chairman Darrell Issa today announced that a hearing scheduled for Wednesday, which was to examine the impact of Domain Name Service (DNS) and search engine blocking on the Internet, has been postponed following assurances that anti-piracy legislation will not move to the House floor this Congress without a consensus.

“While I remain concerned about Senate action on the Protect IP Act, I am confident that flawed legislation will not be taken up by this House. Majority Leader Cantor has assured me that we will continue to work to address outstanding concerns and work to build consensus prior to any anti-piracy legislation coming before the House for a vote,” said Chairman Issa. “The voice of the Internet community has been heard. Much more education for Members of Congress about the workings of the Internet is essential if anti-piracy legislation is to be workable and achieve broad appeal.”

“Earlier tonight, Chairman Smith announced that he will remove the DNS blocking provision from his legislation. Although SOPA, despite the removal of this provision, is still a fundamentally flawed bill, I have decided that postponing the scheduled hearing on DNS blocking with technical experts is the best course of action at this time. Right now, the focus of protecting the Internet needs to be on the Senate where Majority Leader Reid has announced his intention to try to move similar legislation in less than two weeks.”
http://www.keepthewebopen.com

This isn’t the end of the news, however: on the same night, this morning, the White House responded to the “We The People” epetition asking the President to veto the Stop Online Piracy Act & PROTECT IP Act. Cybersecurity coordinator Howard Schmidt, US CTO Aneesh Chopra and OMB intellectual property enforcement coordinator Victoria Espinel wrote it. While they don’t address the veto requested in the epetition, the White House did come out strongly against the DNS provisions in the bills.

Any effort to combat online piracy must guard against the risk of online censorship of lawful activity and must not inhibit innovation by our dynamic businesses large and small. Across the globe, the openness of the Internet is increasingly central to innovation in business, government, and society and it must be protected. To minimize this risk, new legislation must be narrowly targeted only at sites beyond the reach of current U.S. law, cover activity clearly prohibited under existing U.S. laws, and be effectively tailored, with strong due process and focused on criminal activity. Any provision covering Internet intermediaries such as online advertising networks, payment processors, or search engines must be transparent and designed to prevent overly broad private rights of action that could encourage unjustified litigation that could discourage startup businesses and innovative firms from growing.

We must avoid creating new cybersecurity risks or disrupting the underlying architecture of the Internet. Proposed laws must not tamper with the technical architecture of the Internet through manipulation of the Domain Name System (DNS), a foundation of Internet security. Our analysis of the DNS filtering provisions in some proposed legislation suggests that they pose a real risk to cybersecurity and yet leave contraband goods and services accessible online. We must avoid legislation that drives users to dangerous, unreliable DNS servers and puts next-generation security policies, such as the deployment of DNSSEC, at risk.

Taken in context with Senator Leahy’s statement on reconsidering DNS (albeit not removing it from the bill) and Rep. Lamar Smith saying he’ll remove a DNS provision from SOPA, one of the major concerns that the tech community appears to have been heard and validated. Read my past coverage of SOPA and PIPA at Radar for these concerns, including links to the bills and a white paper from Internet engineers.

The White House, however, did write that “existing tools are not strong enough” and that they want legislation to move forward. That could well be the OPEN Act supported by Senator Ron Wyden and Rep. Darrell Issa.

The MPAA has also weighed in on the Congressional moves. (PDF. Michael O’Leary, senior executive VP for global policy and external affairs for the MPAA:

“We fully support Chairman Smith in his efforts to protect U.S. workers, businesses and consumers
against online theft. We believe his announcement today regarding the Stop Online Piracy Act and
Senator Leahy’s earlier announcement regarding the PROTECT IP Act will help forge an even
broader consensus for legislative action, and we look forward to working with them and other
interested parties in passing strong legislation utilizing the remaining tools at our disposal to protect
American jobs and creativity. We continue to believe that DNS filtering is an important tool, already
used in numerous countries internationally to protect consumers and the intellectual property of
businesses with targeted filters for rogue sites. We are confident that any close examination of DNS
screening will demonstrate that contrary to the claims of some critics, it will not break the Internet.”

Gary Price, who forwarded the MPAA response, also notes that “on Thursday, the Library of Congress named a new Director of Communications. She starts at the end of this month. She was key in the founding of the Pro-SOPA Copyright Alliance and
also worked for the MPAA.

We’ll be seeing reactions to this all weekend. I’ll link to the best of them tomorrow from this story. For now, a couple of things seems clear:

1) The technical concerns of the Internet community appear to have been heard. It’s also likely that the federal government’s own cybersecurity experts, including Sandia Labs and Schmidt himself, influenced Congressional actions here. Senator Leahy, however, has not committed to remove DNS provisions entirely from PIPA, only to research them upon passage. That’s likely to be unsatisfactory to many concerned with the bills. “Trust us” to study it after passage is a tough sell.

2) The White House is supporting the arguments that online piracy is a a “real problem that harms the American economy, and threatens jobs for significant numbers of middle class workers.” That statement should have been supported with more evidence from the government’s research institutions.

3) The response from the White House has to be considered an open government win, with respect to an epetition resulting in a statement from the top IT officials in the country. That said, posting it on a Friday night Saturday morning, as opposed to a response from the President during his Friday news conference, buried* diminished the impact of the news and muted its political impact.

4) Most American citizens oppose government involvement in blocking access to content online, particularly when the word “censor” is accurately applied. When asked if ISPs, social media sites and search engines should block access — as they would under SOPA — only a third of Americans agree.

The White House stated that “we will not support legislation that reduces freedom of expression, increases cybersecurity risk, or undermines the dynamic, innovative global Internet.”

It will be up to the American people to hold them accountable for the commitment.

Update: Here’s Erik Cain, writing at Forbes on the White House response on SOPA:

This pretty clearly pits the Obama Administration against SOPA/PIPA. It also calls for more open and honest discussion about these bills and the problems they seek to address. Since there has been almost no discussion or debate until very recently on the legislation in question, this is a very welcome development.

I admit that while I’m pretty glad to see the administration come out with this sort of in-depth statement on the matter, I have a hard time trusting the president on these issues. His veto pen notably did not come out to quash the NDAA – a bill he vowed at one point to not let past his desk.

Then again, internet regulations may have wide, bipartisan support but still nowhere near the support that a defense funding bill has. Obama may have seen a political fight he couldn’t win, read the writing on the wall, and backed off of the NDAA rather than suffer a blow right before an election. The same does not apply to SOPA/PIPA.

So an executive veto on these bills seems much more likely, though at this point – with various congressmen starting to speak out, lots of companies threatening blackouts of their websites – including Wikipedia and Reddit – we may see the momentum behind these bills grind to a halt. The White House statement on the matter will only help push the conversation in congress. That’s a good thing.

Here’s Matt Yglesias, who writes at Slate that the Obama administration came out against SOPA and PIPA:

It increasingly looks like the SOPA/Protect IP fights are turning into an example of how the political system sometimes does work correctly after all. The con forces on these bills initially looked numerically overwhelmed in congress and hugely outspent. But opponents really mobilized vocally, got people and institutions who don’t normally focus on politics to write about this, and perhaps most important of all demonstrated that more people genuinely cared about this issue than most members of congress initially realized. Now the momentum has slowed incredibly and the White House technology policy team has come out against these bills.

To look a gift horse in the mouth for a second, however, I note that the White House statement does contain a “reasonable” to-be-sure line stating that “online piracy is a real problem that harms the American economy, and threatens jobs for significant numbers of middle class workers and hurts some of our nation’s most creative and innovative companies and entrepreneurs.”

Greg Sandoval and Declan McCullagh for CNET: DNS provision pulled from SOPA, victory for opponents:

Without the DNS provision, SOPA now looks a great deal more like the OPEN Act, a bill introduced by Rep. Darrell Issa (R-Calif.) and Sen. Ron Wyden (D-Ore.), which was designed to be an alternative to SOPA. A watered-down SOPA means Smith improves his chances of getting the bill through Congress but at this point, nothing is assured.

Late today came word that six Republican senators have asked Majority Leader Harry Reid to postpone a vote on Pro IP, also known as PIPA. The senators wrote: “Prior to committee action, some members expressed substantive concerns about the bill, and there was a commitment to resolve them prior to floor consideration.”

Leahy issued a statement which appears to be a reply to the request by those senators. He argued that the PIPA vote should go ahead as planned.

“Saying no to debating the [Pro IP Act] hurts the economy,” Leahy wrote. “It says no to the American workers whose livelihoods depend on intellectual property-reliant businesses. And it says yes to the criminals hiding overseas stealing American intellectual property…all Senators should agree that this is a debate we must have…and should support cloture on the motion to proceed on January 24.”

It sounds as if Leahy is trying to keep some of the bill’s supporters from bolting. There’s little question now that some SOPA and PIPA backers in Congress are in retreat and seeking some kind of compromise in the face of significant opposition.

Cory Doctorow at BoingBoing: Lamar Smith and Patrick Leahy blink, pull DNS-blocking out of PIPA and SOPA

After repeatedly insisting that establishing a national censoring firewall with DNS-blocking was critical to the Stop Online Piracy Act, the bill’s sponsor (and chair of the House Judicial Committee) Rep Lamar Smith has blinked. He’s agreed to cut DNS-blocking from the bill, in the face of a threat from rival Rep Darrell Issa, whose House Oversight and Government Reform Committee was preparing to hear expert testimony on the harm that this provision would do to national security and the Internet’s robustness against fraud and worse.

Even without its DNS provisions, SOPA remains terminally flawed, creating a regime that would be terminally hostile to any site that contains links and any site that allows the public to post comments on it. But attention has shifted to PIPA, the Senate version of the bill, which is nearly as bad, and which is rocketing towards an imminent vote.

Timothy Lee at ArsTechnica: Obama administration joins the ranks of SOPA skeptics:

Combine all those concerns, and the statement is a fairly sweeping condemnation of SOPA and PIPA in their current form. Espinel and her colleagues appear to have left enough wiggle room in the statement to allow the president to sign a future version of the bill that addresses some, but not all, of the critics’ concerns. But the bill’s sponsors are now going to have to work hard to satisfy critics and build a consensus in favor of passage.

Tim O’Reilly at Google+ on the White House response to the epetition on SOPA and PIPA:

I found myself profoundly disturbed by something that seems to me to go to the root of the problem in Washington: the failure to correctly diagnose the problem we are trying to solve, but instead to accept, seemingly uncritically, the claims of various interest groups. The offending paragraph is as follows:

“Let us be clear—online piracy is a real problem that harms the American economy, and threatens jobs for significant numbers of middle class workers and hurts some of our nation’s most creative and innovative companies and entrepreneurs. It harms everyone from struggling artists to production crews, and from startup social media companies to large movie studios. While we are strongly committed to the vigorous enforcement of intellectual property rights, existing tools are not strong enough to root out the worst online pirates beyond our borders.”

In the entire discussion, I’ve seen no discussion of credible evidence of this economic harm. There’s no question in my mind that piracy exists, that people around the world are enjoying creative content without paying for it, and even that some criminals are profiting by redistributing it. But is there actual economic harm?

In my experience at O’Reilly, the losses due to piracy are far outweighed by the benefits of the free flow of information, which makes the world richer, and develops new markets for legitimate content. Most of the people who are downloading unauthorized copies of O’Reilly books would never have paid us for them anyway; meanwhile, hundreds of thousands of others are buying content from us, many of them in countries that we were never able to do business with when our products were not available in digital form.

History shows us, again and again, that frontiers are lawless places, but that as they get richer and more settled, they join in the rule of law. American publishing, now the largest publishing industry in the world, began with piracy. (I have a post coming on that subject on Monday.)

Congress (and the White House) need to spend time thinking hard about how best to grow our economy – and that means being careful not to close off the frontier, or to harm those trying to settle it, in order to protect those who want to remain safe at home. British publishers could have come to America in the 19th century; they chose not to, and as a result, we grew our own indigenous publishing industry, which relied at first, in no small part, on pirating British and European works.

If the goal is really to support jobs and the American economy, internet “protectionism” is not the way to do it.

*The White House emailed me later in the morning to point out that the epetition response was posted on Saturday morning.