Connect the Snowflake of CKAN tutorial CKAN to release to open data portal

The State of California leverages CKAN data management products to provide California citizens with extensive datasets from government agencies on the California Open Data Portal.These datasets are public and can be downloaded by anyone.The current state details how State can extract and load data through a python application.This small application requires an Azure virtual machine instance that needs to be brought online or manually turned on periodically to extract and load.This requires maintenance for small applications that they want to avoid.In this article, I detailed how to use an external function to push a dataset directly from Snowflake to CKAN.

Current Status

The current state requires managed servers that the infrastructure team must monitor.To avoid some of the work associated with this small server, the team is interested in establishing a more native Snowflake connection on the OpenGov-managed CKAN server.The deployed python code runs periodically and is triggered by a Windows OS task.

The python code checks for updates once a day and uploads data to CKAN.To do this, the python code extracts the data from Snowflake and puts it on the local file system as a csv.Then a stream of bytes pushes the data to CKAN.This means we have a file copy in Snowflake, a temporary data copy on the python server, and a data copy in CKAN.Every step in the process physically moves data.

Future Status

We want to build a pipeline that removes as much infrastructure from the equation as possible.So we're going to use Snowflake as our source of truth and connect with OpenGov in some direct way.as external