The DS1 and DS3 rates that the Ad Hoc Telecommunications Users filed publicly at the FCC are now accessible as an online, highly capable Needle domain (database). Needle is a data system that makes it easy to look at the data in different ways and to sort and filter it, all from within a web browser.
The original filings (here, here, and here) provide the data as pdf pages displaying tables with highly complex row and column structures. A human can read and page through the data as if it were text. That data format serves neither the reading capabilities of humans nor the data-processing capabilities of computers.
To make the Ad Hoc DS1/DS3 rate data more accessible, I extracted it from the pdf files and re-organized it into one, regular, comma-separated-value (CSV) file with 3698 data rows. I also put together some relevant data documentation. Analyzing the CSV file with a spreadsheet is possible but cumbersome. Since the CSV file has a simple tabular data form, it’s easy to analyze with a database program, if you have one. You would download the data, import it into the database program, and then set up and run a query that generated the data view that you seek.
Needle makes many different views of the data easily accessible to a web browser. Within Needle, a dataset is a graph of data nodes, where each data node is a single piece of data of a particular type. The Needle Ad Hoc DS1/DS3 domain shows (on the left under “Every:”) a linked list of every node type in that dataset. If you click on any of these node types, a table will appear that has as its leftmost column a list of all the data nodes of the clicked type. So, for example, if you click on “bandwidth,” you will see the nodes DS1 and DS3 in the left column of the table. The table also shows the number of attribute sets and the average circuit10 rate (a composite rate) across the DS1 and DS3 nodes, respectively. You can look at the circuit10 rates by clicking on the circuit10 link (node type) on the left. The resulting table shows all the circuit10 rates, in descending order, in the left column. Other columns of the table show other attributes associated with each circuit10 rate.
For any table that you see, you can filter, sort, and group the data. For example, to limit the table of circuit10 rates to DS1 rates, left click on the “bandwidth” column heading, select “filter by this column” in the pop-up menu, type DS1 into the box next to “show”, and then click on “do” just to the right of that box. The table will then contract to show just the DS1 circuit10 rates. A similar procedure produces filters for company, year, state, reg type, term, and zone. If you want to see the elements of each of these data types, click on that type on the left. Options on the pop-up menu also provide for sorting and grouping. Under “Index” on the top left, the “rates” and “rates subset” links show examples of tables made from grouping, filtering, and sorting the cn (attribute set) nodes. The “compare 2009 to 2006” and “compare 2009 to 2005” links under the index heading show tables that include circuit10 price ratios across the relevant years. You can sort and filter these tables like any other table.
Any subset of data can be extracted easily from Needle. At the bottom of each table are links “See this data as: Plain List · CSV · JSV · JSONa”. Just click on CSV to download a CSV file of the data. If the table has groups, you need to flatten the table (switch grouping to a regular data column) before exporting. Needle also offers API functionality that allows Needle to serve as a data repository for high-powered statistical analysis packages such as R or S.
Needle can do much more than what it is doing for the Ad Hoc DS1/DS3 dataset. Needle’s strengths include data acquisition, merging, and cleansing. In addition, Needle’s graph-based data organization can easily handle complex data structures that create nightmares in traditional relational databases, which require tabular data forms. Needle, for example, can easily handle variable-length lists of items. None of these strengths are applied to present the Ad Hoc DS1/DS3 dataset. Needle here merely makes the Ad Hoc DS1/DS3 data much more easily accessible, especially compared to data published as pages of tables in a pdf document.