Data Warehousing and Database concepts- The basics
The basic concept of a Data Warehouse is to facilitate a single version of truth for a company for decision making and forecasting. A Data warehouse is an information system that contains historical and commutative data from single or multiple sources. Data Warehouse Concepts simplify the reporting and analysis process of organizations.
- Cloud Data Warehouse vs Traditional Data Warehouse Concepts
- Cloud Data Warehouse Concepts
- Conclusion: Traditional vs. Data Warehouse Concepts in Brief
- Data Warehouse Architecture: Traditional vs. Cloud
- New Data Warehouse Architectures
- Data Mart vs. Data Warehouse
- The Difference Between a Data Warehouse and a Database
- Data Lake vs. Data Warehouse
- 12 Datawarehouse cloud tools
- Data Warehouse Testing
Modern Data Warehouse
A modern data warehouse lets you bring together all your data at any scale easily, and to get insights through analytical dashboards, operational reports, or advanced analytics for all your users.
- BI and Data Warehousing
- Data Warehousing and Data Mining
- The New EDW: Meet the Big Data Stack
- Build An ETL Process with Examples
- ETL – load data into AWS Redshift
- Full vs Incremental Loading in ETL
- Best Cloud-Based and Open Source Tools
- Get Started with ETL
- ETL Testing
- Redshift Architecture, Pricing, and Performance
- Redshift Columnar Storage
- Redshift Cluster 101
- Snowflake and the Future of Data Warehousing
- Google BigQuery Architecture
Cloud Basics and Why choose the Cloud Warehouse
Due to their architecture, cloud-based data warehouse offers some major advantages over the traditional systems, such as: Scalability; Reliability; Security; Adaptability; Many organizations cite a lack of resources and expertise as barriers to implementing an on-site data warehouse solution. This is where cloud data warehouses become a preference.
- What is cloud computing?
- Cloud Computing Service Models – IaaS, PaaS, SaaS
- Cloud Computing Deployment Models – Public, Private & Hybrid
- Cloud Computing Basics – Compute
- Cloud Computing Basics – Storage
- Cloud Computing Basics – Network
- Cloud Computing Basics – Serverless
- What is a cloud datawarehouse?
Introduction To Snowflake
Introduction to the powerful and unique features that Snowflake provides for ensuring your data is protected, secure, and available.
Connecting To Snowflake
This guide will walk you through how to connect to Snowflake. You may need to contact the administrator of your Snowflake account if you lack some of the credentials discussed below.
Loading / Unloading Data Into/From Snowflake
These topics describe the concepts and tasks for Data Loading /unloading (i.e. exporting) data from Snowflake tables. Key concepts related to data unloading, as well as best practices.
- Overview of Data Loading/Unloading
- Data Loading/Unloading Considerations
- Preparing to Load/Unload Data
- Bulk Loading/Unloading from Amazon S3
- Bulk Loading/Unloading from a Local File System
- Bulk Loading /Unloading from Microsoft Azure
- Loading Continuously Using Snowpipe
- Loading Using the Web Interface (Limited)
- Querying Data in Staged Files
- Querying Metadata for Staged Files
- Transforming Data During a Load
Database Objects and Querying
A database object is any defined object in a database that is used to store or reference data. Anything which we make from create command is known as Database Object. Some of the examples of database objects are : view, table, sequence, indexes, etc.
Sharing Data In Snowflake
Snowflake data providers can share data that resides in different databases by using secure views. A secure view can reference objects such as schemas, tables, and other views from one or more databases, as long as these databases belong to the same account.
Managing Your Snowflake Account
These topics describe the administrative concepts and tasks associated with managing your account in Snowflake. These topics are intended primarily for administrators (i.e. users with the ACCOUNTADMIN, SYSADMIN, or SECURITYADMIN roles). Account Identifier.
Welcome to Advanced Topics. The goal of this course is to provide you with a deeper understanding of optimizing the usage of snowflake.
- Informatica Cloud Services and Snowflake Integration
- Informatica Snowflake Connector
- Informatica Snowflake JDBC Connection
- Informatica Snowflake Key Range Partitioning
- Informatica Snowflake Objects in Mappings
- Informatica Snowflake Sources in Mappings
- Informatica Snowflake Targets in Mappings
- Informatica Snowflake Lookups in Mappings
- Data Clustering
- Clustering Keys & Clustered Tables
- Using the Spark Snowflake Connector
Introduction to Data Sharing
Introduction to Secure Data Sharing¶
Secure Data Sharing enables sharing selected objects in a database in your account with other Snowflake accounts. The following Snowflake database objects can be shared:
Secure materialized views
Snowflake enables the sharing of databases through shares, which are created by data providers and “imported” by data consumers.
All database objects shared between accounts are read-only (i.e. the objects cannot be modified or deleted, including adding or modifying table data).
In this Topic:
How Does Secure Data Sharing Work?¶
With Secure Data Sharing, no actual data is copied or transferred between accounts. All sharing is accomplished through Snowflake’s unique
services layer and metadata store. This is an important concept because it means that shared data does not take up any storage in a consumer
account and, therefore, does not contribute to the consumer’s monthly data storage charges. The only charges to consumers are for the
compute resources (i.e. virtual warehouses) used to query the shared data.
In addition, because no data is copied or exchanged, Secure Data Sharing setup is quick and easy for providers and access to the shared data is instantaneous for consumers:
The provider creates a share of a database in their account and grants access to specific objects in the database. The provider can also share data from multiple databases, as long as these databases belong to the same account. One or more accounts are then added to the share, which can include your own accounts (if you have multiple Snowflake accounts).
For more details, see What is a Share? (in this topic).
On the consumer side, a read-only database is created from the share. Access to this database is configurable using the same, standard role-based access control that Snowflake provides for all objects in the system.
Through this architecture, Snowflake enables creating a network of providers that can share data with multiple consumers (including within their
own organization) and consumers that can access shared data from multiple providers:
Any full Snowflake account can both provide and consume shared data. Snowflake also supports third-party accounts, a special type of account that
consumes shared data from a single provider account. For more details, see Reader Accounts (in this topic).
Overview of Data Providers and Consumers¶
A data provider is any Snowflake account that creates shares and makes them available to other Snowflake accounts to consume. As a data provider, you
share a database with one or more Snowflake accounts. For each database you share, Snowflake supports using grants to provide granular access control
to selected objects in the database (i.e., you grant access privileges for one or more specific objects in the database).
Snowflake does not place any hard limits on the number of shares you can create or the number of accounts you can add to a share.
A data consumer is any account that chooses to create a database from a share made available by a data provider. As a data consumer, once you add
a shared database to your account, you can access and query the objects in the database just as you would with any other database in your account.
Snowflake does not place any hard limits on the number of shares you can consume from data providers; however, you can only create one database per
For more details, see Data Consumers.
Data sharing is only supported between Snowflake accounts. As a data provider, you might wish to share data with a consumer who does not already have
a Snowflake account and/or is not ready to become a licensed Snowflake customer.
To facilitate sharing data with these consumers, Snowflake supports providers creating reader accounts. Reader accounts (formerly known as
“read-only accounts”) provide a quick, easy, and cost-effective way to share data without requiring the consumer to become a Snowflake customer.
Each reader account belongs to the provider account that created it. Similar to standard consumer accounts, the provider account uses shares to
share databases with reader accounts; however, a reader account can only consume data from the provider account that created it:
Users in a reader account can query data that has been shared with it, but cannot perform any of the DML tasks that are allowed in a full account
(data loading, insert, update, etc.).
For more details, see Managing Reader Accounts.
Overview of the Product Offerings for Secure Data Sharing¶
Snowflake provides three product offerings for data sharing that utilize Snowflake Secure Data Sharing to connect providers of data with consumers.
In this Topic:
Snowflake Data Marketplace¶
Snowflake Data Marketplace is available to all Snowflake accounts hosted on non-VPS regions on all supported cloud platforms.
The Data Marketplace utilizes Snowflake Secure Data Sharing to connect providers of data with consumers.
You can discover and access a variety of third-party data and have those datasets available directly in your Snowflake account to query without transformation and join it with your own data. If you need to use several different vendors for data sourcing, the Data Marketplace gives you one single location from where to get the data.
You can also become a provider and publish data in the Data Marketplace, which is an attractive proposition if you are thinking about data monetization and different routes to market.
For more information, see Introduction to the Snowflake Data Marketplace.
Data Exchange is your own data hub for securely collaborating around data between a selected group of members that you invite. It enables providers to publish data that can then be discovered by consumers.
You can share data at scale with your entire business ecosystem such as suppliers, partners, vendors, and customers, as well as business units at your own company. It allows you to control who can join, publish, consume, and access data.
Once your Data Exchange is provisioned and configured, you can invite members and specify whether they can consume data, provide data, or both.
The Data Exchange is supported for all Snowflake accounts hosted on non-VPS regions on all supported cloud platforms.
For more information, see Data Exchange.