Ready, Set… Go!
Ready, Set… Go!The Next Chapter in our Data Management Journey
We started datHere in 2020. The latest chapter in our journey that all started with us working out of Sami’s basement in 2011 for six months on our NYCBigApps entry. That gamble paid off when we won the Grand Prize and parlayed that win to starting Ontodia in NYU’s Varick Incubator in 2012.
Shortly after, we started building NYCpedia – organizing NYC’s raw open data into a semantic, temporal knowledgebase of NYC – computing hyperlocal, neighborhood Key Performance Indicators (KPIs). In the process of doing so, we needed to host the raw data we wrangled somewhere and found CKAN. Before long, we found ourselves working on CKAN not just for ourselves but for others as well who wanted a non-proprietary, standards-based, open-source data portal.
So we became CKAN’s first professional services partner in the US later that year – implementing data portals primarily in our backyard – Metro NYC and the Northeast US. BetaNYC’s Community Portal, NYC’s Department of Education, UNDP in NYC; Newark and Jersey City in New Jersey; WPRDC in Western Pennsylvania; and Boston – which we migrated from a proprietary data portal to CKAN – were notable implementations.
We also did a lot of data wrangling and consulting gigs, which largely informed the other products we launched – CivicDashboards, which was designed to compute KPIs at a national level for 35,000+ jurisdictions, being fed by our then nascent CKAN hosting service – pediacities, which allowed hosted data portals to additionally compute KPIs at the neighborhood level ala NYCpedia. We launched these products in Oct 2015 at the Code for America Summit.
For a small team of six, we accomplished a lot as we got a lot of support from New York City (NYCEDC and NYU in particular) and we stood on the shoulders of the open source community – not only CKAN, but numerous other open source projects and open data sources we leveraged to build our products and services.
Before we got to execute our combined CivicDashboards/Pediacities strategy in earnest however, Ontodia was acquired by OpenGov in 2016 and we went on to implement CKAN SaaS data portals across the country at all levels of government – from large state agencies to small municipalities.
Joel was appointed to the CKAN Steering Group, we sponsored a lot of open source work that was contributed to the core project, and we hosted the first CKANConUS in 2018.
READY
However, we felt that the promise of open data is still largely unrealized. As true believers, we knew that publishing raw data sets on an online data catalog was not what we signed up for when we embarked on this journey.
A data portal was not the goal, but just the beginning. It was the necessary foundational infrastructure to manage metadata at the enterprise level. If used properly, it can help you aggregate and “liquify” your Enterprise Data Assets and make fast, informed decisions regardless of the underlying technology behind each cataloged data source.
Ultimately, the entrepreneurial drive that got us started in 2011 harkened again, and we created datHere in late 2019 with the goal of launching our new startup – datHere, in the first half of 2020.
We wanted to re-embark on our unfinished data management journey – going beyond using CKAN as a traditional open data portal to a full-blown data management platform, launching enterprise-grade data management products and services, which will ultimately allow our customers to compute KPIs on demand, to make evidence-driven decisions based on insights hidden in the raw data – our “Answering People Interface”.
We wanted to apply the hard lessons we learned as we deployed CKAN across the country and recapture the camaraderie, high productivity, and fast development pace of Ontodia’s two-pizza team.
And then COVID happened…
SET.
It literally dashed starting up our startup. We had no choice but to pause our product/services plans, and we went into “semi-stealth” mode.
Thankfully, we were able to leverage our network and used the time to roll up our consulting sleeves, using professional services to fund product development and build out next-generation solutions informed by our customer’s challenges and requirements.
So we stuck our nose to the grindstone and worked on consulting gigs – productizing innovations as we went along. We created DataTables View, Quicksilver (qsv), and Datapusher+ and started building our envisioned DMS Platform-as-a-Service.
SET…
The silver lining of COVID was that it normalized “remote work” and allowed us to work on projects and tap mission-driven partners who shared our vision around the world!
Thanks to our customers/partners – Datopian, WPRDC, AppGeo, TWDB/TNRIS, BetaNYC, Mathematica, New Mexico Tech, and OpenGov we’ve been very busy on CKAN implementations despite COVID, SETting the stage (pun intended) for our upcoming offerings.
Our work was greatly accelerated by our collaboration with the University of Pittsburgh in the National Science Foundation (NSF) funded Pathways to Enable Open Source Ecosystems project to strengthen the CKAN ecosystem (https://civicdataecosystem.org/).
It allowed us to take the pulse of the CKAN ecosystem through several dozen interviews and a series of sense-making workshops attended by users from all over the world.
WPRDC just concluded Phase I of the project this summer, submitting a Phase II proposal last month that we co-created with CKAN stakeholders to scale up a robust, vibrant Civic Data Ecosystem powered by a panoply of open source tools and CKAN.
GO!!!
While we wait for the NSF to review the proposal, we’re launching a series of products and services this quarter.
We’ll be introducing them over the next few weeks, starting with Quicksilver (qsv) at https://qsv.dathere.com – a blazing-fast, open-source data wrangling toolkit that can make short-work of common data munging tasks.
We’ll be diving into it in detail tomorrow.
Twelve years in, we’re recommitting ourselves to this journey. We hope you can join us as we do our part to realize the true potential of Open Knowledge, CKAN, and open data – as we strive to make Data Useful, Usable, and Used.
Open Datanaut, Open Source contributor, SemTechie, Urbanist, Civic Hacker, Futurist, Humanist, Dad, Hubby & Social Entrepreneur on a 3BL mission.
I oversee the design, development, and implementation of innovative data solutions for clients. My expertise in data management, data quality, and data integration has been integral to driving data-driven decision-making. I am passionate about creating a culture of data-driven innovation that enables organizations to stay ahead of the competition.