Sometimes tables are messy not just in their data items, but also in the fields that define the table columns.[1] Various techniques help to deal with such “second order” messiness. Sorting table fields alphabetically or evaluating them with more powerful text similarity measures help to identify inadvertently duplicated fields. Sorting table fields by the … Continue reading exploring and remodeling table fields
Even in this age of big data, most persons collect data in spreadsheets. Two challenges are common with spreadsheet data, particularly spreadsheet data collected from a variety of sources. First, you need to understand what numbers you have. That means both the definition of a specific number and the presence or absence of particular numbers. … Continue reading describing and organizing spreadsheet data
Confidential documents submitted to government agencies have significant costs. Confidential documents don’t contribute to public knowledge. Persons face significant costs and complications to access confidential documents. Moreover, the receiving agency has to follow special, relatively expensive procedures for storing and archiving confidential documents. Both the cost of confidentiality to the public and to the … Continue reading confidential documents are costly
Creating a new, common language for machine-readable information allows information to be shared across organizations with disparate information systems and information formats. The Global Justice XML Data Model is a successful example of such a language. Its success prompted the development of a similar, but broader initiative called the National Information Exchange Model. Both models … Continue reading micro-consituencies support global information sharing
In a nineteenth century dataset, cabinets group birds at a high level of similarity. Each cabinet contains stuffed and mounted birds, arranged roughly in a grid. On the top row of this cabinet are three flycatchers followed by three kingbirds. On the second row are two kingbirds follwed by four flycatchers. The next row displays … Continue reading knowledge from the nineteenth century to now
Information sources reporting on studies, but not actually citing and linking to those studies, apparently don’t want to help readers to learn more. Not citing and linking has created well-justified frustration. Sources that don’t provide relevant citations and links should be judged less credible. You should be suspicious of their motives. You should be offended … Continue reading informational generosity
Huge archives of files containing U.S. local-exchange telephone companies’ service volumes, rates, and revenue from 1992 to 2009 are now available for collaborative reformatting, organizing, and analyzing. U.S. local-exchange telephone companies that the Federal Communications Commission regulates via price caps publicly file annual tariff review data. These data include service volume (demand) and rates for … Continue reading mountains of telecom data for crowd fun