Federal Environmental Web Tracker

Go to the tracker  

About the Federal Environmental Web Tracker  

Since January 2017, EDGI’s website monitoring team has monitored thousands of federal government webpages to track changes under the Trump administration. This tracker is a collection of documented changes to environment-related federal webpages that the team deemed noteworthy. 

The changes documented in this dataset can be traced across EDGI reports on the alteration of online climate change information, the removal of essential regulatory information from websites, in addition to other topics.

To find out more about this dataset, please see the sections below.

Why did EDGI create the Federal Environmental Web Tracker?

Documenting and analyzing changes to the federal government’s presentation of and access to environmental information through their websites is a core part of EDGI’s work. EDGI does this in the pursuit of public knowledge building and government transparency.

EDGI created the Federal Environmental Web Tracker in order to maintain a public record of changes made to environment-related federal government webpages of broad interest. EDGI hopes the publishing of this tracker spurs activists, journalists, and academic researchers to utilize this data in their own analyses.

Website changes are almost always precursors or responses to policy changes. Often, they influence policy. Website changes represent changes to the information the public is most likely to access before commenting on proposed rules and when learning about new environmental issues, which may influence their advocacy. See the timeline below of changes the website monitoring team has documented regarding climate change and changes to U.S. climate change policy. 

Another aim of this tracker is to enable others engaging in similar web monitoring efforts to build off of this work. This type of activism, centered around the surveillance of ephemeral information objects, is still in its infancy. EDGI hopes others can gain insights about where EDGI has succeeded and where it has fallen short.

What is the Federal Environmental Web Tracker? 

In the weeks following its founding in November 2016, EDGI’s website monitoring team compiled a list of 25,000 federal webpages related to climate, energy, and the environment through seeding a few thousand URLs and using an automated crawler to identify the rest from those seeds. This list includes webpages from 20 federal agencies, with the largest samples from the EPA, NOAA, and NASA. First using proprietary software and then building and using our own open source software, the EDGI website monitoring team has compared versions of webpages weekly since January 2017. 

Our software, Scanner, compares the HTML code of a webpage as it existed on two different dates, and identifies whether HTML code, visible text, and/or links have changed. The software generates lists of webpage changes for the website monitoring team to review, and offers a user interface that highlights the changes. The website monitoring team reviews these changes and systematically catalogs instances of webpage changes that are not simply administrative. Our process is described more in the graphic below. 

How EDGI’s website monitoring team works: Step 1. Curate list of pages to monitor based on monitoring experience and subject matter expertise. Step 2. Archive pages with our partner, the Internet Archive. Step 3. Compare the source code for different versions of the same webpage. Step 4. Creates metadata to guide analysts. Step 5. Render the comparison for visual analysis by the team. Step 6. Team meeting and discussion of changes viewed. Step 7. Publish reports about changes.

The tracker represents the documented changes that EDGI’s website monitoring team deemed important. Each entry represents a verified change that occurred as well as the team’s summary and characterization of that change. The tracker does not necessarily represent all important changes that have occurred across the 25,000 URLs monitored by EDGI. The website monitoring team manually reviews a portion of URLs each week, and the scope and scale of what the team reviews has varied along with team capacity. Thus, there may be important changes that have occurred on these webpages that were not documented and characterized by EDGI’s website monitoring team. 

EDGI’s website monitoring team will continue to update this database quarterly. At the time of publication, 41% of the changes in the dataset are from the EPA website, 16% from the DOE website, 13% from the DOI website, and smaller fractions from other agency websites. The most common topics include policies, rules and regulations (35%), energy (18%), and climate (16%). 

What is the criteria for including a change in the Tracker?

The tracker covers roughly one thousand federal webpage changes starting in January 2017 that EDGI’s website monitoring team deemed important. The team’s criteria for determining important webpage changes has changed over time, but for the tracker, an important change is one where:

  • Content related to the subject matter of the page has been added, removed, or substantively changed.  
  • Language has been altered in a way that affects the focus, tone, or emphasis of the page itself or issues discussed on the page.  
  • Certain links have been added, removed, or their destination or textual description changed.

Webpage changes that are insignificant, and thus are not included in this tracker, include changes that are mostly administrative, infrastructural (e.g. HTML source code changes that do not result in visible text changes), cosmetic, or happen regularly, such as updating links to recent news and events. 

What does each field/column in the Tracker mean?

Field/Column Explanation
Date Found  The date on which the change was captured by EDGI’s monitoring software system.
Agency The name of the agency overseeing the web page on which the change occurred. 
Page Name The title of the specific web page on which the change occurred. The page name is taken from the page’s <title> field.
URL The web address of the specific web page on which a change occurred. 
HTML link — prior to change  A link to the raw HTML file of the page prior to the change. 
HTML link — after change A link to the raw HTML file of the page after the change occurred. 
Internet Archive link — prior to change A link to the version of a page in the Internet Archive’s Wayback Machine that is the closest in time before the change occurred. 
Internet Archive link — after change A link to the version of a page in the Internet Archive’s Wayback Machine that is the closest in time after the change occurred. 
Is this primarily a content or access change (or both)? Describes if a change is primarily a change to the web page’s content (e.g. the text on a page) or overall accessibility (e.g. via the addition or removal of links to other pre-existing pages).
Topic 1 A term that describes the subject matter of the change and/or page.
Topic 2 A term that describes the subject matter of the change and/or page.
Notes The notes written by an analyst summarizing the change.

What do the selected topics mean? 

  • Toxics:  Toxic substances, including pesticides, hazardous air pollutants, heavy metals, and other poisonous chemicals and compounds.  
  • Air:  Air pollution, pollutants, policies, or programs. Excludes greenhouse gases. 
  • Water:  Water pollution and contaminants, or water-related policies, programs, modeling tools, etc. 
  • Conservation:  Focuses on protection of non-human species of animals, plants, and ecosystem dynamics.  
  • Remediation:  Cleaning up or otherwise mitigating health risks of contaminated areas.  
  • Energy:  Coal mining, coal fired power plants.. Energy efficiency and renewables (solar, wind, etc.). Can serve as a second topic for oil and gas-related activities and pollution. 
  • Oil and Gas:  Oil and gas exploration, production, infrastructure (e.g. processing and transportation); includes oil leases, and hydraulic fracturing.
  • Climate:  Covers climate change science, climate change indicators, climate change mitigation and adaptation, and general climate-related policy and science (including greenhouse gas (GHG), international climate agreements).
  • Weather:  Weather events such as droughts, hurricanes, floods, and wildfires. Also covers air temperature.
  • Agriculture:  Non-point source pollution (nutrient pollution and surface runoff), GMOs and bioengineered crops, CAFOs (Confined Animal Feeding Operations), water usage.
  • Economy:  Business and jobs, investment and finance, and budgetary and fiscal issues.
  • Policies, Rules, Regulations: Laws, regulatory actions, oversight and process.
  • Relations: Relationships and cooperation between states, municipalities, tribes, and the federal government. International relations and cooperation are also covered.
  • Science & Research:  Role of science, including its importance.
  • Health & Safety:  Human health effects from environmental pollution. Also covers occupational safety, public health, and the coronavirus.
  • Assistance & Aid:  Emergency response and relief efforts, including rebuilding assistance.
  • Educational Info:  Educational content for children, teachers, students, professionals, or the general adult public.
  • Administration References:  Mentions of either the Obama or Trump administration.

What acronyms and special terms are used?

  • Change: A webpage for which a new version and the previous version are not identical, based on their html source code.
  • Content Change: Information provided on a page has been altered substantively or the language used to describe information has changed the tenor or emphasis of that information.
  • Access Increase: There has been an improvement in access to information, such as adding links to further information.
  • Access Decrease: There has been a reduction in access to information, such as removing links or removing pages.
  • Agencies
    • Census: U.S. Census Bureau
    • CDC: Centers for Disease Control and Prevention
    • DOE: Department of Energy
    • DOI: Department of the Interior
    • DOJ: Department of Justice
    • DOT: Department of Transportation
    • EIA: Energy Information Administration
    • EOP: Executive Office of the President
    • EPA: Environmental Protection Agency
    • FEMA: Federal Emergency Management Agency
    • Forest Service: U.S. Forest Service
    • FWS: U.S. Fish and Wildlife Service
    • GAO: Government Accountability Office
    • DHHS: Department of Health and Human Services
    • NIH: National Institutes of Health
    • NASA: National Aeronautics and Space Administration
    • NOAA: National Oceanic and Atmospheric Administration
    • NPS: National Park Service
    • OSHA: Occupational Safety and Health Administration
    • State: U.S. State Department
    • USDA: U.S. Department of Agriculture
    • USGS: U.S. Geological Survey
    • Whitehouse: The White House  
  • Websites for interagency programs 
    • Coronavirus.gov: A collaboration among the CDC, the White House, and FEMA (formerly between CDC, the White House, and DHHS) to address the coronavirus pandemic.  
    • Data.gov:  A cross-agency open data portal.
    • Globalchange.gov: The website of the U.S. Global Change Research Program, which is a program mandated by Congress to coordinate federal research and investment in forces shaping the environment. There are 13 member agencies.  

How can I search and use the Tracker?

There are several ways to use this tracker. You can peruse changes and filter to those that are of most interest to you, view the before and after versions of individual changes, analyze types of changes across the tracker, and export HTML information.  

Each change detailed here includes the agency, type of change, and two topics related to the change. You can filter or sort by the fields Agency, Topic 1, Topic 2, and Content or Access Change to narrow the changes to those of interest to you, or to assess patterns to those changes. 

To learn more about a given change, read the analyst notes or search the text of the “Notes” column for keywords. Additionally, click the links to the “before” and “after” versions in the Internet Archive’s Wayback Machine to view the two different versions of the page. To better visualize the difference between the two versions, use the Wayback Machine’s “Changes” feature (built from EDGI’s open source website monitoring software) and load the date of the “before” version on the left and the “after” version on the right. Click here to see a guide on how to best use “Changes” in conjunction with the tracker.

Download the dataset in CSV, XLS, or other formats by clicking File > Downloads, and then selecting the preferred format. 

Each change includes a link to the source code of the before and after versions of the page, which can be downloaded to view the HTML, CSS, and other scripts. Learn more about EDGI’s database and processing tools at: https://github.com/edgi-govdata-archiving/web-monitoring.

Of Note

This is a living document as the EDGI Website Monitoring team continues to monitor federal websites. Please expect updates to this database approximately every three months starting in February 2021. 

If using data in this tracker for a paper or project, please include the following citation: 

EDGI, “Enviro Fed Web Tracker: Changes to Federal Environmental Information Under the Trump Administration,” URL: https://envirodatagov.org/enviro-fed-web-tracker/, Accessed on [Date]. 

Who contributed to the Federal Environmental Web Tracker?

EDGI website monitoring team members spent countless hours combing through changes to federal websites to document and categorize the changes presented in this database. Their efforts are deeply appreciated, and their names are listed here. Additionally, team members Alejandro Paz and Steven Gentry carried the bulk of the work to transform our internal documents to this public dataset, reviewing all changes and standardizing categorizations. Special thanks goes out to founding team members Rebecca Lave, Maya Anjur-Dietrich, Andrew Bergman, Toly Rinberg, and Gretchen Gehrke for designing the website monitoring program and Rob Brackett for developing the software to do it. 

Thank you to current team members: Alejandro Paz, Steven Gentry, Gretchen Gehrke, Marcy Beck, Rob Brackett, Aaron Lemelin, Eric Nost, Grace Poudrier, and Amy Wilson. 

Thank you to former team members and other contributors: Cole Adler, Dan Allan, Maya Anjur-Dietrich, Vangelis Banos, Chaitanya Bapat, Andrew Bergman, Madelaine Britt, Ed Byrne, Jesse Card, Ray Cha, Janak Chadha, Alize Clough, Morgan Currie, Justin Derry, Jesse DiValli, Jon Gobeil, Steve Hansen, Mike Hucka, Pamela Jao, Sara Johns, Abby Klionsky, Katherine Kulik, Rebecca Lave, Kevin Nguyen, Kendra Ouellette, Lindsay Poirier, Toly Rinberg, Sara Rubinow, Justin Schell, Lauren Scott, Jason Sherman, Miranda Sinnott-Armstrong, Andrew Smith, Chris Tirrell, Fotis Tsalampounis, Lizz Ultee, Julia Upfal, Tyler Wedrosky, Liz Williams, Adam Wizon, Jacob Wylie, Sara Wylie, Sarah Yu.

Who should I contact about this tracker?

For more information on the tracker and EDGI’s website monitoring work, please contact us at edgi.websitemonitoring@protonmail.com.