USAID and Open Data: An Overview

As some of you may be aware, USAID published their Open Data policy, and have begun including language in all contracts and cooperative agreements after October 1, 2014 to require partners to abide by the policy.

There have been many more questions than answers at this stage, which is to be expected when any new policy is rolled out. For those of us in the USAID contractor community, especially small businesses, unexpected change and lots of questions do not make us rest easy...

USAID Open Data Policy

Who do USAID's Open Data rules apply to?

All contracts and cooperative agreements awarded on or after October 1, 2014. Many contracts are having these modifications also rolled out retroactively via the IPN.

(from the ADS 579 page 10, emphasis mine)

USAID staff, as well as contractors and recipients of USAID assistance awards (e.g. grants and cooperative agreements) must submit any Dataset created or collected with USAID funding to the DDL in accordance with the terms and conditions of their awards.

This is in keeping with Executive Order 13642 and the OMB Open Data Policy (M-13-13) which states that an agency’s “public data listing may also include, to the extent permitted by law and existing terms and conditions, Datasets that were produced through agency -funded grants, contracts, and cooperative agreements.

What specific data do they want?

According to USAID, they want:

Organized collection of structured data, including data contained in spreadsheets, whether presented in tabular or non-tabular form (e.g. single spreadsheet, an extensible mark-up language (XML) file, a geospatial data file, or an organized collection of these).

Data Supporting “Intellectual Work”

Works that “document the implementation, monitoring, evaluation, and results of international development assistance activities.”

Examples

Baseline Household Surveys
School Attendance Data
Facility Surveys
Cluster Samples
Monitoring Data*
Economic Assessments

Does not include

Unstructured data (PDF, Word, graphics, emails)
Award administration data (financials, operations, management)
* Data already submitted to USAID via a different data repository (such as the Feed the Future Management System).

What do I need to do for my organization to be compliant?

If your contract or cooperative agreement includes language on open data, you will need to create an open data plan, created with the COR/AOR and the Data Steward for the funding Bureau or Mission.
Any datasets that are identified by this plan as needing to be published in the Development Data Library will need to be published in open data formats (i.e. machine readable) using common structures when available.
You may want to perform an open data audit of your data capture systems and processes you will use before starting the conversation with USAID, to make sure that you are not committing to publication of sensitive data or an expensive data publication method, as well making sure you are already using common data structures such as IATI.
Think about how you can use this data for internal and external purposes. Open data offers opportunities for data sharing among offices and organizations that did not exist before. There may be major benefits to your organization with having this sort of data easily available between teams.

What is Open Data

Open data has two parts - open and data.

Open: means publishing data in machine readable formats so that anyone can automatically pull that data into their software applications without a human being involved.

For an example, think Weather data - we all have it on our phones, right? Where does that data come from? Well, the raw data comes originally from NOAA, in most cases as an open data feed, and companies like the Weather Channel analyze it and come up with predictions. They then publish their own open data feeds, allowing app developers like me to write the code which displays today's weather on your phone.
Data: means data that is structured and in machine readable formats (XML or JSon are the two most popular). This could be weather data as listed above, demographic information such as from the census. It could be project data, such as published on the foreign assistance dashboard. It could be M&E data from a project. The key is that the data is structured and in machine readable formats.

Two other key components that are included in open data definitions:

The data has an API to allow a developer to write code that will pull specifically parts of the data they need without needing to talk to anyone (i.e. only data on Bangladesh, or for 2013).
The data must be free to be used. This means removing any data that might violate someone's privacy or cause a security incident. Remember that open data may be a subset of data that is used internally using the rest of the open data format - to allow internal systems to interoperate more easily.

Big Picture: Open Data

Is here to stay as it is part of a larger US Government commitment to Open Data, as well as the global committment to open data for development.
Is also part of the digital "big data" revolution we have been experiencing in all parts of our lives. As more and more organizations look to data-driven performance management, open data will be part of these solutions.
Is cheap when designed up front and expensive to do in retrospect.
Will be built into systems from now on. Just like Y2K, once everyone gets their systems "open data compliant" by default, open data will no longer be a huge burden on data gatherers.
Will be a huge resource for data analysts and researchers all around the world. Just like we cannot imagine working in a world before Google, we won't remember what limitations we had without all this data at our fingertips.
Is one key element in improving data access, but it is really only the start. There will still need to be translators, analysts, vizualizers, and archivists needed to manage the data.

Overwhelmed? Sonjara can help

We offer open data audits/evaluations, creation of open data plans and solution identification for every budget to help you keep in compliance with these new regulations.

More importantly, we can help you identify where complaince with these new regulations can actually be used to improve your organziations performance.

Give me a call (703 981-9982) or send me an email (sio @ sonjara dot com) for a free one hour overview/conversation on how you can move forward with open data.

Resources and Links

US Government

White House memorandum on open data policy
Data.gov (always updated list of US Government open data resources)
Foreign Assistance dashboard (US Government publication of all foreign assistance)
US Government Github site on open data
International Aid Transparency Initiative (IATI)

USAID

« Back to Sonjara Blog