Terminal


Effective date: February 9, 2023

Deepnote is a collaborative data notebook trusted by the world’s best data teams. We’re on a mission to radically improve how teams think, explore and decide together. In order to do this, we must ensure that our product and services continue to operate smoothly, which includes protection against misuse or abuse by our users.

We've developed this Acceptable Use Policy to clarify what we mean when we say "misuse" or "abuse" and to assist us in identifying such offenses and responding appropriately. In accordance with this policy, we reserve the right to suspend or terminate any accounts that violate these guidelines. Even if actions are not explicitly included in the policy, Deepnote remains the right (in its sole discretion) to suspend or terminate any accounts that have violated this Acceptable Use Policy with or without notice.

## Prohibited materials

You may not, and you may not allow your users or any third party to, use Deepnote to display, store, process or transmit, or permit use of the SaaS to display, store, process or transmit, any of the following:

- Material that infringes or misappropriates a third party’s intellectual property or proprietary rights
- Hate-related or violent material, or material advocating discrimination against individuals or groups
- Obscene, excessively profane or otherwise offensive or objectionable material
- Material that enables, advocates or advances criminal hacking or phishing
- Material related to illegal drugs or paraphernalia
- Malicious or unlawful code, such as viruses, worms, time bombs, Trojan horses and other harmful or malicious files, scripts, agents or programs
- Material that violates, encourages or furthers conduct that would violate any applicable laws, including any criminal laws, or any third-party rights, including publicity or privacy rights

## Prohibited actions

- Generating or facilitating unsolicited commercial email (spam).
- Hacking, tampering, reverse-engineering or compromising the integrity of any systems. This prohibition does not apply to ethical security researchers working in accordance with our bug bounty policy to find vulnerabilities in our systems. Any such behavior must not result in degradation of service for other users of the platform.
- Overwhelming or attempting to overwhelm our infrastructure by imposing an unreasonably large load on our systems.
- Engage in cryptocurrency mining/farming or any similar behavior.
- Monitoring or crawling of any system that impairs or disrupts the system being monitored or crawled, or any other large scale harvesting or scraping .
- Operating open proxies, open mail relays, open recursive domain name servers, Tor exit nodes, or other similar network services.
- Interfering with the proper functioning of any system, including any deliberate attempt to overload a system by mail bombing, news bombing, broadcast attacks, flooding techniques, or conducting a denial of service (“DoS”) attack.
- Use Deepnote primarily as a remote desktop environment.

## HIPAA and working with PHI

- Deepnote permits the processing of PHI data under the condition of a signed BAA agreement with Deepnote.
- Do not use metadata, namely project names and notebook names, to store PHI.


Acceptable use policy

Deepnote AI is your own virtual AI collaborator helping you to solve hard problems. You can rely on it to kickstart a full analytical project, ask for assistance when you get stuck, and efficiently handle mundane coding tasks.

You can use Deepnote AI in two modes of operation:

- Auto AI is a fully autonomous agent that will not only add blocks, but also execute them and self-correct its way to the desired outcome.
- Generate mode will add code and text blocks to your notebook based on your instructions, but will wait for you to execute them.

One of the biggest challenges in coding can be getting started. Use Deepnote AI to create blocks from your natural language instructions. Simply use the **AI prompt bar** visible at the bottom of the notebook or click on **Generate with Deepnote AI** placeholder in an empty code block to enter you prompt in plain language. You can also use the `Cmd+Shift+A` shortcut to trigger AI in any block.

Let's explore the two modes of operation in more detail, starting with Auto AI.

### Auto AI

Auto AI is a fully autonomous mode of using Deepnote AI in your notebook. You can give it a task and Auto AI will devise a plan for best achieving your goals within the context of your work. It will then proceed to **write text, create and execute code & SQL blocks** until it gets the job done. It is even capable of self-correction and will attempt to fix any potential errors it encounters along the way.

To take advantage of Auto AI in your workspace, you need to have Deepnote AI enabled at **Settings & Members** >> **Project settings**.

<Callout status="info">As part of its task execution, Auto AI relies on block outputs as required context. The outputs can contain row-level data from your DataFrames. See more details at [Data Privacy](/docs/ai-data-privacy).</Callout>

![deepnote ai auto ai settings.png](https://media.graphassets.com/CjvK2ZNTSm60V4CIEoFQ)

Once the workspace setting is enabled, you can enable Auto AI mode by switching on the toggle in the AI prompt bar settings menu:

![CleanShot 2024-10-24 at 10.36.24.png](https://media.graphassets.com/Ep4w7izcQ6OT1E2jqPCy)

After you type in a prompt, Auto AI will execute the blocks it creates and will continuously adapt based on the outputs until the given task is completed. If your machine is off, Auto AI will spin it up when it needs to execute a code block.

Running Auto AI is like inviting a virtual collaborator to your project: it can operate in your notebook independently of your presence. This means that once Auto AI is given a task, it will continue running and executing blocks even if you close Deepnote.

![2024-10-24 10.30.46.gif](https://media.graphassets.com/HTQCWgKZR7SiMX80Sm8B)

If you want to stop Auto AI before it finishes its task, you can always do so by clicking on the **Stop AI Generation** button at the status bar on the top of your notebook. This will immediately cancel all block generation and will cease any block executions in progress.

![CleanShot 2024-03-11 at 16.21.19@2x.png](https://media.graphassets.com/SV18mnlQaaMnFphEBZeO)

#### SQL usage by Auto AI

If you have one or more SQL integrations connected, Auto AI sees them and can use them to generate and run SELECT queries to fetch data it needs to complete its given task.

To minimize risk when using AI to write and run SQL, we added 2 guardrails against this:

- AI is programmed to never execute any write queries
- Deepnote has a detection mechanism for write queries and will stop them from getting executed, even if the AI generates one

#### Settings

You can control what types of blocks get generated by Auto AI in the settings menu. You can turn on and off SQL block generation and also the AI's commentary and conclusions by turning off the text blocks.

![auto ai settings.png](https://media.graphassets.com/BWG6hQYoSwmO8bE1GeRW)

#### Limitations

Deepnote AI cannot yet use SQL with OAuth integrations or create chart blocks. We're working on improving the experience. If you'd like to provide feedback or be directly involved in shaping the future of Deepnote AI, see the [Feedback](/docs/deepnote-ai#feedback) section.

### Generate blocks without running them automatically

If you want to use Deepnote AI in generate mode to retain control over executing the blocks, make sure you have the Auto AI toggle set to **Auto AI off**.

![CleanShot 2024-10-24 at 10.41.15.png](https://media.graphassets.com/YSA3OCj0SCGW2asSj3pn)

Once you are happy with your typed in instructions, submit them and let Deepnote AI tackle your task. Deepnote AI will start generating **Python and/or text blocks**, it can even generate multiple blocks at once!

![2024-10-24 10.44.24.gif](https://media.graphassets.com/5lt95gUhSOC4CYRAj5F4)

The generation will stop automatically once Deepnote AI has finished with the given task, but you can always cancel the generation earlier from the status bar. The generated blocks are saved directly into your notebook and there is no need to explicitly accept them allowing you to immediately run them afterwards.

Remember that your collaborators in the notebook only see the AI-generated blocks and can't stop a generation which you've triggered.

If you're not fully satisfied with the output, you can:

- re-try the code generation using the same prompt by clicking on the **Try again** button. Try again will place you back into the prompt bar so you can modify your instructions for an improved result by providing further context or being more specific with your request.
- remove all generated blocks with the **Discard** button.

![CleanShot 2024-10-24 at 10.49.32.png](https://media.graphassets.com/A9iuxzXdSvCJm93fvGrn)

**Pro tip**: you don't always need elaborate instructions as Deepnote AI has access to the contents of your notebook and is fully conversational. You can tell it to **`Continue`** or to **`Do task X`** and it will respond appropriately.

![2024-10-24 10.52.54.gif](https://media.graphassets.com/V3lP15MSSSQVChE2PxC5)

Enhance your analytics with Deepnote AI's ability to generate and execute code effortlessly.

Generative analysis

Enhance productivity and streamline your workflow by leveraging the power of our for context-aware code suggestions and completions as you type, powered by [Codeium](https://www.google.com/url?q=https://codeium.com&sa=D&source=docs&ust=1714984444129190&usg=AOvVaw1G_ty1_eYZe0-RjO5IIUhQ). Whether you are cleaning data, crafting complex algorithms, or fine-tuning a machine learning model, our code completion is designed to assist you, cutting back on the monotony of writing repetitive code and speeding up your coding process.

<Callout status="info">
Deepnote AI code completion is available on **all** plans.
</Callout>

## Enabling code completion

In order to take advantage of code completions, you need to have this feature enabled for your workspace. Go to **Settings & Members** and click on **Project settings**. When **Deepnote AI code completion** is turned on, every editor and admin user in the workspace will see code completions displayed in all of the projects.

![project setting.png](https://media.graphassets.com/Gxq7HPf3Q5hFym9apHYP)

<Callout status="warning">
To provide code suggestions, Deepnote AI processes the content and metadata of the notebook. This can include code, text and metadata of variables (column names). None of this data is stored outside of Deepnote or used for model training purposes. If you have concerns about processing sensitive data via large language models, you can disable code completions.
</Callout>

## Toggling code completion for individual users

If you prefer to disable code completion for yourself only, you can do so in the block actions menu:

![CleanShot 2023-12-12 at 16.53.04@2x.png](https://media.graphassets.com/O59w9WuRK6EK9WI76WV4)

This will affect code completions in all your notebooks, but other workspace members will be unaffected.

## Working with completions

Once it's enabled, you can see code suggestions appearing real-time as you type in **code or SQL blocks**. You can accept suggestions by pressing `Tab`.

![outliers (1).gif](https://media.graphassets.com/C46TjJ8QPqLlhDH0Oi3U)

Suggestions are **context-aware**: the content in previous code and text blocks in the notebook will be taken into account to provide more relevant suggestions. You can try to prompt code suggestions by providing a textual description of what you want to achieve. You can put that in a text block or in a comment in the code as well.

![visualization (1).gif](https://media.graphassets.com/GO6HEOFFQOO659C6sxeh)

Suggestions work in **SQL blocks** as well. Similarly to Python code, SQL query completions also utilize your notebook as context, along with any relevant SQL blocks using the same integration in other notebooks within your workspace. This means that the more details you have in your notebook and the more work you do with the given data source, the more relevant the suggestions will become.

![2023-08-15 15.32.02 (1).gif](https://media.graphassets.com/L7sdHuk4TXy4n6yXweof)

_Watch it all come together in our 3-minute demo below._

<Embed url='https://www.loom.com/share/f9b7a0058b3648cf801431f761a84eaa?hide_owner=true&hide_share=true&hide_title=true&hideEmbedTopBar=true' />

Boost your efficiency with real-time code suggestions powered by Deepnote AI.

Code completion

Need to revise your code, refactor it, or make it more readable? Deepnote AI can help edit any existing code. Just click on **Open Deepnote AI** in the block sidebar and simply tell Deepnote AI how you want your code to be edited.

![docs-code_editing-medium.gif](https://media.graphassets.com/dnVYuRGoQveEYxSRgC0N)

Edited code will be displayed side by side of the original code in a diff view so you can easily see what's been changed. Press the **Run code** button to see the new code in action. If you're satisfied, press **Done** to accept the code. If the generated responses does not suit your needs, you can **Discard** all of the edits, returning to the state before you started editing with AI.

![CleanShot 2024-05-21 at 14.12.04@2x.png](https://media.graphassets.com/tuSvbyKZQLmX9Fpggfog)

<Callout status="info">**Pro tip**: When the diff view is presented by Deepnote AI, the active state of the block is automatically set to that of the new suggestion. Therefore, if you were to run the whole notebook, the AI suggestion would get executed even if you have not accepted the suggestion before. This behaviour is slightly different from other code editor applications, but rest assured, you can always Discard the change if you want to get back to your original state.</Callout>

### Iterating with follow-up prompts

![docs_code-iterative-editing-medium.gif](https://media.graphassets.com/Zg7wRnddSmmmHcyPEtVs)

If the suggested code does not suit your needs, you can immediately type in a new prompt as a follow-up to further edit it. Each prompt, including your follow-ups is added to the prompt edit history. If you wish to revert to a previous stage in your editing flow, you can simply click on the desired step from the list of your prompts, returning the block to that specific state. This allows you to effortlessly compare results across different prompts and continue with the most promising options.

<Callout status="info">**Pro tip**: With prompt history you are essentially having an isolated mini-chat with the AI, where all of the context of your prompts and AI outputs is preserved, so you can use short instructions as you would in a chat. For instance, if something doesn’t quite work, you can just say "try another option," and the AI will understand how to proceed.</Callout>

### SQL blocks

You can also use Deepnote AI to edit your SQL queries. Just click on Deepnote AI in a SQL block and tell Deepnote AI how it should edit your existing SQL code. Behind the scenes, Deepnote AI will inspect and use your database schema to better understand the written SQL code. [Read more](https://deepnote.com/docs/sql-generation)

![sql_ai-editing_sql.gif](https://media.graphassets.com/Tj1weG0TTtaBZrjjvpNK)

Refine your code efficiently with Deepnote AI's editing suggestions.

Code editing

### How is your data processed?

Deepnote AI processes the content and metadata of the notebook via its partner LLM providers (OpenAI and Codeium). This data can include **code**, **text**, **metadata of variables** (column names) within the notebook and your **database schema** for [SQL AI](https://deepnote.com/docs/sql-generation).

<Callout status="info">The processed content is **not used for model-training purposes** (your code cannot leak into suggestions displayed for other companies).</Callout>

### What about outputs?

You have the option to control whether you would like to share the **outputs of block executions** with OpenAI. If you turn on the '**Provide access to block outputs**' switch, Deepnote AI will be able to read the outputs in your notebook, greatly enhancing the relevance of its suggestions.

![deepnote ai outputs setting.png](https://media.graphassets.com/oSvfKdSTVu52VR599r9w)

<Callout status="warning">Please note that outputs can include **row-level data from your DataFrames** or other variables. If your outputs may include data you do not wish to process via an LLM provider, you can disable this option. You will still be able to take advantage of Deepnote AI features that do not depend on accessing outputs.</Callout>

### Where is your data stored?

The context sent can be processed in the US and worldwide. OpenAI may keep the context data for up to 30 days for safety monitoring purposes, after which it's automatically deleted. See more details [here](https://openai.com/policies/api-data-usage-policies).
Data processed by Codeium for real-time code suggestions is not stored at all.

### Working on sensitive projects?

If you're working on a sensitive project and have concerns about processing any code, text or data through LLMs, admins can disable Deepnote AI in **Settings & Members** → **Project settings**. This will prevent any data from being sent for processing and Deepnote AI features won't be available in your workspace.

Learn how Deepnote AI processes data securely, providing granular privacy controls.

Data privacy

## Chart AI

Not feeling like pointing and clicking today? We got you. As an alternative to manual setup, you can also ask Deepnote AI to create a chart for you. Open Chart AI by clicking **Chart AI** button on the top of the chart block, type in what you want to visualize, and hit **Submit**. In a few seconds, Deepnote AI will respond with a chart suggestion. You can accept it (**Done**) or reject (**Undo**) and adjust your description.

<VideoLoop src="https://media.graphassets.com/yCGeQETZqFODHTDsOFbw" />

The accepted chart will have all the configurations filled out for you. You can use it as a starting point and manually tweak things further. You can also request follow-up edits from Deepnote AI by typing in a new prompt to modify your existing chart.

<VideoLoop src="https://media.graphassets.com/CYey9W5tQyCcdV6zH4Jk" />

Visualize data faster in chart blocks with Deepnote AI.

Data visualization


When running into an error, you'll find a **Fix with Deepnote AI** button appearing right beneath your error message, ready to come to your aid. After you click it, Deepnote AI swings into action, delivering a suggested fix for your code.

![2024-03-11 17.47.00.gif](https://media.graphassets.com/j1KKoeM6QqIO02PSiEM3)

The suggested code will be displayed side by side of the original code in a diff view so. Press the **Run code** button to see the effect of the changes. If you're satisfied with the fixed code, press **Done** to accept it. If the suggested code fix does not suit your needs, you can **Discard** it, returning to your previous code.

![CleanShot 2024-05-21 at 16.46.46@2x.png](https://media.graphassets.com/ayk2tGuQAGLqgm9XsExw)

If you want to further edit the suggested fix, you can immediately type in a new prompt. Each prompt, including your follow-ups is added to the prompt edit history. If you wish to revert to a previous stage in your editing flow, you can simply click on the desired step from the list of your prompts, returning the block to that specific state.

![docs-fix-with-ai-medium.gif](https://media.graphassets.com/1SkHCdiRi6bridWa8i6A)


Quickly resolve coding errors with Deepnote AI's suggested fixes.

Error fixing


Code can be complicated. Whether it's long, uses an unfamiliar framework, or was written by someone else, Deepnote AI can help by providing a simple, understandable summary. Just open the block sidebar in a code block and select the **AI button**. Choose the **Explain code** option from the actions list and Deepnote AI will display the explanation for you.

For future reference, you may want to save this explanation in a new text block. To do this, just click on the **Add as text block** button.

![2024-03-11 17.06.19.gif](https://media.graphassets.com/M6VpdrIxS8CZoujrryxO)


Gain clarity on complex code with Deepnote AI's concise summaries.

Explaining code

![Deepnote Airtable (2).png](https://media.graphassets.com/OcNFT0CBSf2nToz4dBEc)
Deepnote can query data from Airtable bases. Follow these docs to create your own notebook to connect to Airtable, query data, and visualize it.

### Connect to Airtable from Deepnote

#### Create your Airtable token

[Follow these instructions](https://support.airtable.com/docs/creating-personal-access-tokens) to find your token.

#### Update the token in your notebook

Change the variable `AIRTABLE_TOKEN` to match your token. If you'd like to keep your data secure, consider storing the token as an [environment variable](/docs/environment-variables). Environment variables in Deepnote are encrypted and provide a secure way of storing sensitive data.

#### Find your Airtable base ID

Find the ID of the Airtable base you want to fetch data from [by following these instructions](https://support.airtable.com/docs/finding-airtable-ids).

#### Update the base ID in your notebook

Change the variable `AIRTABLE_BASE_ID` to match the base ID you just found.

#### Update the table name in your notebook

Change the variable `AIRTABLE_TABLE_NAME` to match the name of the table you want to fetch data from. This is the title of the tab in the Airtable UI.

![The name of the table is the name of the tab
](https://media.graphassets.com/TSFs7UY2T8inndmVfANK)

#### Set your keys in the notebook

```python
AIRTABLE_TOKEN = 'your-airtable-token'
AIRTABLE_BASE_ID = 'you-base-id'
AIRTABLE_TABLE_NAME = 'your-table-name'
```

#### Use environment variables instead

For more security in your notebook, we strongly recommend to use [ environment variables](https://docs.deepnote.com/environment/environment-variables)

![Screenshot 2024-07-18 at 16.23.22.png](https://media.graphassets.com/6PxGrLT9RHK5KFsCL4wA)

### Get your Airtable table from API

Since there was a change in Airtable API, the old way of using pyairtable will not work. But don't worry we got you covered. Use requests to fetch data from API.

**We will request the API with our base ID, table name, and token.**

```python
url = f'https://api.airtable.com/v0/{AIRTABLE_BASE_ID}/{AIRTABLE_TABLE_NAME}'

headers = {
 'Authorization': f'Bearer {AIRTABLE_TOKEN}',
}
response = requests.get(url, headers=headers)

if response.status_code == 200:
 records = response.json().get('records', [])
 data = [record['fields'] for record in records]
 df = pd.DataFrame(data)
 print(df)
```

**One more function to fetch Airtable table and we are good to go.**

```python
def fetch_airtable_data(base_id, table_name, access_token):
 url = f'https://api.airtable.com/v0/{base_id}/{table_name}'
 headers = {'Authorization': f'Bearer {access_token}'}
 all_records = []
 offset = None

 while True:
 params = {'offset': offset} if offset else {}
 response = requests.get(url, headers=headers, params=params)
 if response.status_code != 200:
 print(f"Failed to fetch data: {response.status_code}")
 break

 records = response.json().get('records', [])
 all_records.extend(records)
 offset = response.json().get('offset')

 if not offset:
 break

 data = [record['fields'] for record in all_records]
 return pd.DataFrame(data)
```

### Query Airtable data from Deepnote

The notebook will set up a `table` object that you can use to fetch any data from your table. For example, the code below fetches the data and converts it to a Pandas DataFrame.

```python
df = fetch_airtable_data(AIRTABLE_BASE_ID, AIRTABLE_TABLE_NAME, AIRTABLE_TOKEN)
df
```

<iframe title="Embedded cell output" src="https://embed.deepnote.com/89226e1d-380f-447a-924e-79271fc232ae/f0abc3c611164a2fbcb56b8cabaab7b5/edd266a97ca54f93b823779115fc2973?height=431"/>

### Analyze your data with SQL

You can run queries against our DataFrame using Deepnote's built-in SQL. [You can learn more about SQL blocks on our docs.](https://deepnote.com/docs/sql-cells)

```sql
SELECT
 "Estimated Value",
 "Priority",
 "Status"
FROM
 df
WHERE
 NOT ("Status" = 'Closed' OR "Status" = 'Lost')
```

<iframe title="Embedded cell output" src="https://embed.deepnote.com/89226e1d-380f-447a-924e-79271fc232ae/f0abc3c611164a2fbcb56b8cabaab7b5/a9e9543ce04d4cc88020a943d6382455?" />

### Visualize data

Deepnote can visualize data frames out of the box. You can learn more about [chart blocks on our docs](https://deepnote.com/docs/chart-blocks).
If you want to do something more sophisticated, you can use visualization libraries like Altair or Plotly.

### Duplicate the Airtable template

If you don't want to do it yourself, here is [template notebook](https://deepnote.com/workspace/Deepnote-Templates-71742312-24f2-4c10-9bf7-786d17280b92/project/Visualize-and-analyze-Airtable-data-in-Deepnote-89226e1d-380f-447a-924e-79271fc232ae/notebook/Notebook%201-f0abc3c611164a2fbcb56b8cabaab7b5)

![Duplicate the Airtable template](https://media.graphassets.com/YP7kc59TlSvtVCL42nwG)

Click the **Duplicate** button on the top right corner to do this. Once you've got the template in your Deepnote workspace, you can connect it to your Airtable base, don't forget to replace your own token and base id.

### What's next?

Now that you're querying data, you can share it with your team. You can even turn your charts [into a shareable dashboard](/docs/publish-projects).

Airtable

<Callout status="info">
Available on Team and Enterprise plans
</Callout>

Amazon Athena is a serverless, interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. When integrated with Deepnote, Athena allows you to run SQL queries against large datasets stored in S3 and visualize the results right in your notebooks.

To add a connection to Amazon Athena, go to **Integrations** via the **right-hand sidebar**, create a new Athena integration, and enter your credentials:

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2F8QGpvKLh2WvfCFVxSzTx%2FScreen%20Shot%202022-03-30%20at%2012.52.54%20PM.png](https://media.graphassets.com/vHgtxcwHTuSIcO4WsUg7)

After that's done, connect the new integration to your project using the Integrations sidebar and query it using [SQL cells](/docs/sql-cells).

## How to Get Started with Amazon Athena in Deepnote

Our setup will consist of these specific components:

1. **S3 Data Bucket** (`deepnote-athena-demo-data`): This bucket will store our NYC Airbnb dataset. S3 buckets provide scalable, secure storage that Athena can directly query.

2. **S3 Query Results Bucket** (`athena-demo-results`): Athena needs a dedicated location to save query results. This separation helps organize our analytics environment and manage costs.

3. **IAM User** (`deepnote-athena-demo-user`): This user will have precisely the permissions needed to access both our S3 buckets and our Athena setup. Creating a dedicated user with minimal permissions follows AWS security best practices.

4. **Athena Database and Table**: We'll create a database called `airbnb_demo` with a table that maps to our NYC Airbnb dataset. This will allow us to query this data using standard SQL.

5. **Deepnote Integration**: Finally, we'll connect all of these AWS resources to Deepnote, enabling interactive data analysis in a collaborative notebook environment.

### Step 1: Create S3 Buckets

We need two S3 buckets - one for our data and one for Athena query results.

#### Create the Data Bucket

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/)
2. Navigate to the S3 service
3. Click **Create bucket**
4. Enter the bucket name `deepnote-athena-demo-data`
5. Choose your preferred AWS Region
6. Leave default settings for most options
7. Click **Create bucket**

#### Create the Query Results Bucket

1. Click **Create bucket** again
2. Enter the name `athena-demo-results`
3. Choose the **same region** as your data bucket
4. Leave default settings for most options
5. Click **Create bucket**

### Step 2: Upload Sample Data to Your Data Bucket

We'll use the NYC Airbnb dataset for this guide:

1. First, download the Airbnb dataset from DeepNote by visiting http://deepnote.com/static/data/AB_NYC_2019.csv in your web browser
2. Save the file to your computer
3. In the AWS console, navigate to your `deepnote-athena-demo-data` bucket
4. Click **Upload**
5. Click **Add files** and select the downloaded CSV file
6. Click **Upload**
7. After the upload completes, you should see the file listed in your bucket

### Step 3: Create an IAM User for Athena Access

Now we'll create a user with permissions to access S3 and Athena:

1. Navigate to the IAM service in the AWS Console
2. In the sidebar, click **Users**
3. Click **Create user**
4. Enter the name `deepnote-athena-demo-user`
5. Click **Next: Permissions**
6. Click **Attach existing policies directly**
7. Search for and select these policies:
 - `AmazonAthenaFullAccess`
 - `AmazonS3ReadOnlyAccess`
8. Click **Next: Tags** (optional to add tags)
9. Click **Next: Review**
10. Click **Create user**
11. From the users list, select your newly created user `deepnote-athena-demo-user`
12. Click on the **Security credentials** tab
13. Scroll down to the **Access keys** section
14. Click **Create access key**
15. Select **Third-party service** as the use case
16. Check the acknowledgment box and click **Next**
17. (Optional) Add a description tag and click **Create access key**
18. **IMPORTANT**: You'll see a page with your **Access key ID** and **Secret access key**. Download the CSV file or copy both values immediately. You'll need these for Deepnote and you won't be able to see the secret key again after you close this page.

### Step 4: Setup Custom IAM Policy for S3 Write Access

The built-in policies don't provide write access to your results bucket, so let's create a custom policy:

1. In the IAM console, navigate to **Policies**
2. Click **Create policy**
3. Switch to the **JSON** tab
4. Paste the following policy with our specific bucket names:

```json
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:GetBucketLocation",
 "s3:GetObject",
 "s3:ListBucket",
 "s3:ListBucketMultipartUploads",
 "s3:ListMultipartUploadParts",
 "s3:AbortMultipartUpload",
 "s3:PutObject"
 ],
 "Resource": ["arn:aws:s3:::athena-demo-results", "arn:aws:s3:::athena-demo-results/*"]
 }
 ]
}
```

5. Click **Next: Tags** (optional)
6. Click **Next: Review**
7. Name the policy `AthenaResultsBucketAccess`
8. Click **Create policy**
9. Navigate back to **Users**
10. Select your `athena-deepnote-user`
11. Click **Add permissions**
12. Select **Attach existing policies directly**
13. Search for and select your new `AthenaResultsBucketAccess` policy
14. Click **Next** and then **Add permissions**

### Step 5: Configure Athena

Now we'll set up Athena to query our data:

1. Navigate to the Athena service in the AWS Console
2. If this is your first time using Athena, you'll need to set up a query result location:

 - Click **Settings**
 - Enter the S3 path to your results bucket: `s3://athena-demo-results/`
 - Click **Save**

3. Create a database:

 - In the query editor, run:

 ```sql
 CREATE DATABASE airbnb_demo;
 ```

4. Create a table for the NYC Airbnb data:

 - Run the following query:

 ```sql
 CREATE EXTERNAL TABLE airbnb_demo.nyc_listings (
 id BIGINT,
 name STRING,
 host_id BIGINT,
 host_name STRING,
 neighbourhood_group STRING,
 neighbourhood STRING,
 latitude DOUBLE,
 longitude DOUBLE,
 room_type STRING,
 price INT,
 minimum_nights INT,
 number_of_reviews INT,
 last_review DATE,
 reviews_per_month DOUBLE,
 calculated_host_listings_count INT,
 availability_365 INT
 )
 ROW FORMAT DELIMITED
 FIELDS TERMINATED BY ','
 STORED AS TEXTFILE
 LOCATION 's3://deepnote-athena-demo-data/'
 TBLPROPERTIES ('skip.header.line.count'='1');
 ```

 **NOTE**: We've set the table to look for data in the root of our `deepnote-athena-demo-data` bucket. Make sure your CSV file is uploaded to the root of this bucket.

5. Test a simple query:

 ```sql
 SELECT * FROM airbnb_demo.nyc_listings LIMIT 10;
 ```

 You should see the first 10 records from the dataset.

### Step 6: Connect Deepnote to Athena

Now let's set up Deepnote to query Athena. You'll need administrator privileges in Deepnote to create this integration:

1. Sign in to your [Deepnote](https://deepnote.com/) account
2. Navigate to the **Integrations** page
3. Click **+ Add integration**
4. Choose **Amazon Athena** from the options
5. Enter the following connection details:
 - **Name**: "Deepnote Athena Demo"
 - **Region**: The AWS region where you created your resources
 - **Output bucket URL**: `s3://athena-demo-results/`
 - **Access Key ID**: The IAM user access key you created earlier
 - **Secret Access Key**: The IAM user secret key you saved earlier
6. Click **Create integration**

![CleanShot 2025-03-11 at 10.29.27.png](https://media.graphassets.com/EMsLrPSSlmVENdEHSNSu)

After creating the integration, you need to connect it to your project:

1. Navigate to a new or existing project
2. Click on the "+" button to add a new block
3. Select **SQL cell**
4. From the dropdown menu, select your "Deepnote Athena Demo" integration
5. Now you can write and run SQL queries against your Athena database

### Step 7: Running Queries in Deepnote

Now you're ready to analyze the NYC Airbnb data:

1. In your Deepnote project, add a new SQL block
2. From the dropdown menu, select your "Deepnote Athena Demo" integration
3. Write a query in the SQL block, for example:
 ```sql
 SELECT
 neighbourhood_group,
 COUNT(*) AS listing_count,
 ROUND(AVG(price), 2) AS avg_price
 FROM airbnb_demo.nyc_listings
 GROUP BY neighbourhood_group
 ORDER BY avg_price DESC;
 ```
4. Click the "Run" button or press Shift+Enter to execute the query
5. Deepnote will display the results as a table below the SQL block
6. You can now analyze the results or create visualizations directly in Deepnote

![CleanShot 2025-03-11 at 10.58.49.png](https://media.graphassets.com/jOAJGgLQSbegUOonCyR4)

### Sample Queries to Try

Here are a few interesting queries to explore the Airbnb dataset:

#### Price Distribution by Room Type

```sql
SELECT
 room_type,
 COUNT(*) AS listings,
 MIN(price) AS min_price,
 ROUND(AVG(price), 2) AS avg_price,
 MAX(price) AS max_price
FROM airbnb_demo.nyc_listings
GROUP BY room_type
ORDER BY avg_price DESC;
```

#### Top 10 Neighborhoods by Average Price

```sql
SELECT
 neighbourhood,
 neighbourhood_group,
 COUNT(*) AS listings,
 ROUND(AVG(price), 2) AS avg_price
FROM airbnb_demo.nyc_listings
GROUP BY neighbourhood, neighbourhood_group
HAVING COUNT(*) >= 5
ORDER BY avg_price DESC
LIMIT 10;
```

#### Distribution of Reviews

```sql
SELECT
 CASE
 WHEN number_of_reviews = 0 THEN 'No reviews'
 WHEN number_of_reviews BETWEEN 1 AND 10 THEN '1-10'
 WHEN number_of_reviews BETWEEN 11 AND 50 THEN '11-50'
 WHEN number_of_reviews BETWEEN 51 AND 100 THEN '51-100'
 ELSE '100+'
 END AS review_bucket,
 COUNT(*) AS listing_count,
 ROUND(AVG(price), 2) AS avg_price
FROM airbnb_demo.nyc_listings
GROUP BY 1
ORDER BY CASE
 WHEN review_bucket = 'No reviews' THEN 1
 WHEN review_bucket = '1-10' THEN 2
 WHEN review_bucket = '11-50' THEN 3
 WHEN review_bucket = '51-100' THEN 4
 ELSE 5
END;
```

### Next Steps

- Try uploading your own datasets to S3
- Experiment with more complex SQL queries in Deepnote
- Create visualizations based on your query results
- Set up automated workflows combining Athena and Deepnote

Now you have a fully functional setup for analyzing large datasets with Amazon Athena and Deepnote. Happy querying!

Amazon Athena

<Callout status="info">
Available on the Enterprise plan
</Callout>

With Amazon ECR, you can run notebooks in your own custom environments using Docker containers.

## How to connect to Amazon ECR

To add a connection to ECR in Deepnote, go to **Integrations** in the **left-hand sidebar** and create a new Amazon ECR integration.

You will need the **Account ID** and the **Region** where your registry is located. You will also need an **Access key** and **Secret key** for an [IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) with the following permissions:

- ecr:BatchGetImage
- ecr:GetDownloadUrlForLayer
- ecr:GetAuthorizationToken

You can set the [policies on the registry level](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policy-examples.html) instead to limit access to specific repositories.

### Authenticating using an IAM role

Using an IAM role instead of a username and password to connect to your ECR is preferred for enhanced security and easier management. IAM roles offer temporary credentials that automatically rotate, reducing the risk of long-term credential exposure. This approach also simplifies permissions management, allowing for more granular access control to ECR images based on the principle of least privilege. Moreover, it eliminates the need to store static credentials, reducing the potential for security breaches.

Select the IAM role option in the "Authentication" section in the dropdown menu.

Enter the Amazon Resource Name (ARN) of the AWS role you wish to use for this integration. This role should be configured with the necessary permissions to access ECR and download images. For example, you can assign the policy named `AmazonEC2ContainerRegistryReadOnly`.

To allow Deepnote to access your ECR, you must update the role's trust policy. Copy the generated trust policy and update your role's trust relationship in the AWS IAM console.

## How to use Amazon ECR in Deepnote

After creating and connecting the integration, you can create a [custom environment](/docs/custom-environment) based on one of your Docker images.

Click the Environment dropdown in the Machine section in the left-hand sidebar to bring up the Environment selection modal. Click on "Set up a new Docker image," enter the URL for the image you want to use, and give it a name. Clicking "Add environment and apply" will restart your machine using your Docker image.

<Callout status="info">
**Your credentials are encrypted and securely stored in our database.** When you delete this integration, we also wipe your ECR credentials from our systems.
</Callout>

Do you have a private Docker image on Amazon ECR that you want to use in Deepnote? No problem. Follow these steps use your custom environment.

Amazon Elastic Container Registry

## What can you do with the Redshift integration?

The Redshift integration allows you to query databases and explore the structure of your warehouse. Think "SQL editor" but with all the modern productivity boosters that come with Deepnote's notebook. For example:

- Write native SQL and Python in the same notebook
- Search your entire warehouse instantly via the integrated schema explorer
- Get intelligent autocomplete for columns, tables, and databases
- Interactively explore data without writing any additional code

<Callout status="info">
Available on Team and Enterprise plans
</Callout>

## How to connect to Redshift

From the right-hand panel, under **Integrations**, click the **+** button and choose **Create new integration**.
![create_integration.png](https://media.graphassets.com/pIr2x2wQtyVUUAv6spVZ)

 

Select Redshift from the list of integrations or search for it using the search bar.
![redshift_create.png](https://media.graphassets.com/URRAjyZfS6O5tB97TbTa)

 

Fill out the fields in the pop-up form. You will need to know the Redshift hostname, which can be found on your AWS Redshift console.
![redshift_modal.png](https://media.graphassets.com/TVXyXrdSqa0ID0RB9skT)

 

<Callout status="success">
Please see our [security options](securing-connections) for configuring access to your Redshift data. This includes securing connections to Redshift via [SSH tunnels](/docs/secure-connections). Note that connections to Redshift will always be secured with SSL.
</Callout>

### Authenticating using an IAM role

Using an IAM role instead of a username and password to connect to your Redshift is preferred for enhanced security and making it easier to manage. IAM roles offer temporary credentials that automatically rotate, reducing the risk of long-term credential exposure. This approach also simplifies permissions management, allowing for more granular access control to Redshift resources based on the principle of least privilege. Moreover, it eliminates the need to store static credentials, reducing the potential for security breaches.

Select the IAM role option in the "Authentication" section in the dropdown menu.

Enter the Amazon Resource Name (ARN) of the AWS role you wish to use for this integration. This role should be configured with the necessary permissions to access the Redshift clusters with which Deepnote will interact.

To allow Deepnote to access your Redshift cluster, you will need to update the role's trust policy. Copy the generated trust policy and update your role's trust relationship in the AWS IAM console.

The Deepntote integration will create a database user called `IAMA:deepnote` that will be used to perform your queries. If you are using serverless Redshift, the user will have the same role as the IAM you used to authenticate. For example, if your IAM role is named `Deepnote-Redshift`, the user will be named `IAMA:Deepnote-Redshift`.

### Authenticating using Individual credentials

Suppose you want to control what data your users can access in your Redshift database. In that case, you can create unique database users for your users and use the Individual credentials option for authentication.

With Individual credentials, each user must authenticate with the database using their credentials before running their first query. The results of their queries will not be shared with other users in the notebook or project.

The user credentials are encrypted before they are stored in Deepnote's database and inaccessible to others.

<Embed url='https://www.loom.com/embed/b6ef8fc726d1485ba307f92da5138ef6?hide_owner=true&hide_share=true&hide_title=true&hideEmbedTopBar=true' />

### AWS PrivateLink

<Callout status="info">
Available only on Enterprise license with single tenant deployment
</Callout>

Deepnote supports connecting to Redshift instances via [AWS PrivateLink](https://docs.aws.amazon.com/redshift/latest/mgmt/security-private-link.html), which allows private connectivity between Deepnote and your Redshift instance without exposing your data to the public internet.

If you wish to try connecting to Redshift without an Enterprise license, we recommend:

- Using a temporary SSH bastion with a public IP address (see [SSH tunnels](/docs/secure-connections))
- Contacting our support team for an enterprise trial with PrivateLink configuration

## Connecting to Redshift with Python

Simple as installing the connector.
`pip install redshift_connector`

```python
import redshift_connector
conn = redshift_connector.connect(
 host='your_host',
 database='your_db',
 port=5439,
 user='your_user',
 password='your_password'
 )

# Create a Cursor object
cursor = conn.cursor()

# Query a table using the Cursor
cursor.execute("select * from book")

#Retrieve the query result set
result: tuple = cursor.fetchall()
print(result)
 (['One Hundred Years of Solitude', 'Gabriel García Márquez'], ['A Brief History of Time', 'Stephen Hawking'])
```

## Using Redshift in Deepnote

Now that you're connected to Redshift, you can do the following:

- Click the newly created integration in the **Integrations** section to open the schema browser. [Click here to learn more about the schema browser](schema-browser).

- Create an SQL block that points to your warehouse and queries your data. Autocomplete for columns, tables, and databases will be at your fingertips as you type. [Click here to learn more about SQL blocks](sql-cells).

- Explore column distributions as well as sorting and filtering capabilities on the results set. All results are displayed as an interactive pandas DataFrame. [Click here to learn about interactive DataFrame output](variable-explorer#interactive-dataframe-output).

- Pipe the results of your query into a chart block for rapid interactive data visualization. [Click here to learn more about chart blocks](chart-blocks).

Amazon Redshift is a fully managed, petabyte-scale data warehouse service. Deepnote integrates with Redshift seamlessly to bring the cloud data warehouse to your notebook.

Amazon Redshift

<Callout status="info">
Available on Team and Enterprise plans
</Callout>

With the S3 integration, you can access the files in your S3 buckets directly in Deepnote. The files will be accessible via the filesystem, so you can work with your data as if you had it locally.

## How to connect to S3

![s3-1.png](https://media.graphassets.com/aubHV6xDRCCGJMo0kkQ2)

First, you will need to [create a S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) or have access to an existing one.

To add a connection to an AWS S3 bucket in Deepnote, go to **Integrations** via the **left-hand sidebar**, then create a new S3 integration.

If the bucket is publicly accessible, you will only need to enter:

- A name for your integration
- The name of the S3 bucket

For private buckets, you will also need to add your:

- Access key ID
- Secret access key

You can learn how to create these credentials [here](https://repost.aws/knowledge-center/create-access-key).

### Authenticating using an IAM role

Using an IAM role instead of access keys to connect to an S3 bucket is preferred for enhanced security and making it easier to manage. IAM roles offer temporary credentials that automatically rotate, reducing the risk of long-term credential exposure. This approach also simplifies permissions management, allowing for more granular access control to S3 resources based on the principle of least privilege. Moreover, it eliminates the need to store static credentials, reducing the potential for security breaches.

![s3-2.png](https://media.graphassets.com/B5zOSnsQhulY5PzKUqpQ)

In the "Authentication" section, select the IAM role option in the dropdown menu.

Enter the Amazon Resource Name (ARN) of the AWS role you wish to use for this integration. This role should be configured with the necessary permissions to access the S3 bucket that Deepnote will interact with.

To allow Deepnote to access your S3 bucket, you will need to update the role's trust policy. Copy the generated trust policy and update your role's trust relationship in the AWS IAM console.

## How to use S3 in your notebook

Once you have connected the S3 integration to your project, you can access the files in your bucket under the path `/datasets/{integration name}`.

If the name of your integration is `sales-data`, you can list the files in the bucket by adding a code block with the following content:

```python
!ls /datasets/sales-data

```

You can access the files in your S3 bucket as if they were regular files.

```python
import pandas as pd

# Read the CSV file
df = pd.read_csv('/datasets/sales-data/june.csv')

# Display the DataFrame
df
```

Amazon S3

Understanding user interactions with your content is crucial for improving engagement and overall performance.

Analytics provide insights into the usage of your notebooks and apps, allowing you to track the most successful ones or identify those that are no longer in use and can be safely removed. You can also use the analytics to check the executions of the apps, which can help you justify hardware costs.

![analytics.gif](https://media.graphassets.com/pYS3fLSUQpuzzCRuYVrf)

### Available data

<Callout status="info">Data collection began on June 11, 2024; therefore, data from earlier periods are not available.</Callout>

- Notebook viewers
- App viewers
- App executors

The displayed time corresponds to when the user first opened a notebook or an app, not when they closed it. An anonymous user is tracked only once per session. If you see multiple anonymous users, it means there are multiple anonymous users with different sessions who opened your app or notebook.

### Viewing analytics

The analytics can be accessed from multiple places:

- Right-click on the notebook in the left sidebar and select `View analytics`.
- Click on the three-dot icon in the left corner of the project top bar and select `View analytics`.
- If the notebook is published as an app click on `View analytics` in the blue ribbon at the top of the notebook or in the right sidebar when the app settings are opened.

App & notebook usage insights

<Callout status="info">

This feature is available on the Enterprise plan.

</Callout>

Deepnote stores every user's action within a workspace, together with metadata associated with it. Workspace admins can export the audit log as a .csv file from the Workspace Settings page.

![audit-log-modal.png](https://media.graphassets.com/uKh6URs2TfKXNFcirDDa)

### SQL blocks audit log

When executing an [SQL block](/docs/sql-cells), Deepnote appends a SQL comment with metadata related to the context in which the query is executed. This helps administrators of a database to quickly understand when and by whom the query was executed. The metadata contains workspace, project and notebook ID, whether the query was executed by a user, scheduling or API, and a link to the project.

![Screenshot 2023-02-09 at 17.09.53.png](https://media.graphassets.com/dXu5IKt4SLC5ZF7p6lSB)

Audit log

Deepnote doesn't have integrated auto-grading solutions, but we've put together a list of useful packages that will do the job for you. Each library has a different setup and uses slightly different grading mechanisms.

The simplest solution is to include the tests in the student's assignments. This allows students to run the tests themselves. Keep in mind, it may also allow them to potentially use the tests to skirt the assignment.

<Callout status="info">
The auto-grading solutions below don't provide comprehensive support for grading multiple hundreds of assignments.
</Callout>

There are three types of tests: unit tests, hidden unit tests with immediate scores, and hidden unit tests with scores after submission. Sometimes you may want to test a variable value, not a function, so those are included first.

## 1. OKpy library

![ok.png](https://media.graphassets.com/a0ArnM3CTraGLVgFsEIB)

The ok.py software was developed for CS 61A at UC Berkeley. This library supports programming projects by running tests, tracking progress, and assisting in debugging. You can find the entire GitHub repository for the library [here!](https://github.com/okpy/ok)
<Callout status="info">
Check out our [Deepnote notebook with OKpy library](https://deepnote.com/workspace/Deepnote-classroom-template-ef3ec34d-be29-481f-a95d-9384fc1dd07e/project/1-OKpy-library-bce337dc-57c6-47a0-a293-e64920c2f515//OKpy%20testing%20Notebook%20.ipynb) template to learn more. You can duplicate it and use it for your own purposes.
</Callout>

## 2. Tests from imported files

![Group 3919.png](https://media.graphassets.com/PvaD3OscSa6LsCU65zus)

This is a simple test that functions with asserts in separate files. A useful thing about separate files is that they can be switched later.

<Callout status="info">
Check out our [Deepnote notebook with tests from imported files](https://deepnote.com/workspace/Deepnote-classroom-template-ef3ec34d-be29-481f-a95d-9384fc1dd07e/project/2-Tests-from-imported-files-4aa58a5c-31bf-4475-8f13-791880020c20/notebook/Test%20from%20imported%20files-d267139d9ca14595b7f3b05819f7361e) template to learn more. You can duplicate it and use it for your own purposes.
</Callout>

## 3. Inline tests

![Screenshot 2022-12-22 at 14.37.59.png](https://media.graphassets.com/reDZIR4yR2a1xW9X8OG4)

In-line tests are small pieces of code used to test a specific function or aspect of a more extensive program. They're often used to validate the correctness of code and ensure it's working as intended. In-line tests are usually written within the same file as the code being tested, hence their name.

<Callout status="info">
Check out our [Deepnote notebook with inline tests](https://deepnote.com/workspace/Deepnote-classroom-template-ef3ec34d-be29-481f-a95d-9384fc1dd07e/project/3-In-line-tests-fd5a8ed1-8f4e-4a30-a60b-64f57ed50221) template to learn more. You can duplicate it and use it for your own purposes.
</Callout>

## 4. Doc tests

![Screenshot 2022-12-22 at 14.30.05.png](https://media.graphassets.com/HcSdwO3KQPGwbVVMHqNM)

Doc tests are a way to embed test cases within the documentation for a piece of code. They allow developers to specify expected input and output for a given function or method, and then automatically verify that the code produces the expected results when run. Doc tests are often used to ensure that code examples in the documentation are accurate and up to date.

<Callout status="info">
Check out our [Deepnote notebook with doc tests](https://deepnote.com/workspace/Deepnote-classroom-template-ef3ec34d-be29-481f-a95d-9384fc1dd07e/project/4-Doc-tests-1e6f63a1-039d-493c-8d96-9b2ef1053a84) template to learn more. You can duplicate it and use it for your own purposes.
</Callout>

### 5.) Unit tests

![Screenshot 2022-12-22 at 14.32.21.png](https://media.graphassets.com/BdWLwJfySTCH1ymnKQGJ)

Unit tests are small, isolated tests that verify the correctness of a specific component or unit of code. They're used to make sure individual parts of a larger program are working correctly, and are typically written by developers as they're writing code. Unit tests are generally automated, meaning they can be run automatically by a testing framework or tool with the results being checked automatically as well.

<Callout status="info">
Check out our [Deepnote notebook with unit tests](https://deepnote.com/workspace/Deepnote-classroom-template-ef3ec34d-be29-481f-a95d-9384fc1dd07e/project/5-Unit-tests-23f45656-3ca5-4814-a8f8-f69c04f48165) template to learn more. You can duplicate it and use it for your own purposes.
</Callout>

Auto-grading

Deepnote can fetch blobs from Azure Blob Storage. Follow these docs to create your own notebook to connect to your storage container.

### 1. Duplicate the Azure Blob Storage template

[Open the template by following this link ->](https://deepnote.com/workspace/Deepnote-Templates-71742312-24f2-4c10-9bf7-786d17280b92/project/Fetch-blobs-from-Azure-in-Deepnote-4c7b1284-0ac9-48df-a35c-415db440d69c/%2Fnotebook.ipynb)

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FuBTyirnZvPcOcczBxtFa%2Fduplicate-notebook.png](https://media.graphassets.com/0ZPojAgrTtefUUQLT0JG)

Click the "duplicate" button on the top right corner to do this. Once you've got the template in your Deepnote workspace, you can connect it to your Synapse workspace.

### 2. Connect to Azure Blob Storage from Deepnote

#### Get the connection string for connecting to your container

To fetch blobs from an Azure storage container, you need a connection string for the container's storage account. Find your connection string [by following these instructions](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=environment-variable-windows#copy-your-credentials-from-the-azure-portal).

#### Update the connection string in your notebook

Change the variable `AZURE_BLOG_STORAGE_CONNECTION_STRING` to match the connection string. If you'd like to keep your data secure, consider storing the token as an [environment variable](/docs/environment-variables). Environment variables in Deepnote are encrypted and provide a secure way of storing sensitive data.

#### Update the container name in your notebook

Set `AZURE_BLOB_STORAGE_CONTAINER_NAME` to the name of the container you want to fetch blobs from.

### 3. Fetch Azure blobs from Deepnote

The notebook will set up a `container_client` that you can use to fetch any of the blobs in a container. For example, to get a list of all the blobs.

```python
# Fetch all the blobs and their names
blobs = [*container_client.list_blobs()]
blob_names = [blob.name for blob in blobs]
blob_names
```

When executing the code above, Deepnote will show the `blob_names` result as a list.

<Embed url='https://embed.deepnote.com/4c7b1284-0ac9-48df-a35c-415db440d69c/447156c5-d424-4958-ace2-a835d72b6989/139a8ae4f577454bb5c1497ad32fa209?height=229.5625' />

You can also fetch individual blobs. In our example, our blobs are images (of Mars!), so we generate a thumbnail.

```python
blob_download_stream = container_client.download_blob(blob_names[5])

image = Image.open(io.BytesIO(blob_download_stream.readall()))

image_thumbnail = image.copy()
image_thumbnail.thumbnail([500, 500])
image_thumbnail.show()
```

When executing this, Deepnote will fetch the blob and display the thumbnail.

<Embed url='https://embed.deepnote.com/4c7b1284-0ac9-48df-a35c-415db440d69c/447156c5-d424-4958-ace2-a835d72b6989/dcc96c6f0d7d4c72b60268f53690e682?height=451' />

### What's next?

Now that you're fetching blobs, you can share it with your team. You can even turn your charts [into a shareable dashboard](/docs/publish-projects).

For more information on how to use the Azure SDK, [you can read their docs](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=environment-variable-windows).

Available to teams and users on all plans

Microsoft Azure Blob Storage

It is possible to import existing code or notebooks from an Azure Repos git repository to Deepnote and sync your code back to the repository. While there is no built-in integration for Azure Repos yet, setting up and using Azure Repos in Deepnote is already fairly straightforward.

<Callout status="info">If you require a full Azure Repos integration to use Deepnote, please let us know!
If you require a full Bitbucket integration to use Deepnote, please let us know by [submitting your request](https://portal.productboard.com/deepnote/1-deepnote-product-portal/tabs/1-under-consideration/submit-idea), or contact us directly. We are happy to help!</Callout>

## How to set it up

Every project in Deepnote comes with a filesystem and a terminal. This makes it possible to clone and use any git repository the same way as you would do locally.

First, you need to get credentials for your repository. In the Azure Repo, navigate to the clone section:

<ImageBorder variant="blue">![azure-deepnote-01.png](https://media.graphassets.com/FsMXfkbaQvC75zP4j7qj)</ImageBorder>

Click on "Generate git credentials". This will generate a one-time access token that will be used for this repository by Deepnote.

<ImageBorder variant="blue">![azure-repos-01.png](https://media.graphassets.com/x0B3tpvRjSA7y6LEyzJ6)</ImageBorder>

Copy the repository URL and the access token.

In Deepnote, open a terminal and clone the repository the same way you would do it locally.

```bash
git clone https://youraccount@dev.azure.com/youraccount/repository
```

When asked for a password, use the access token that you've just generated.

That's it! You can now use files in the repository, edit/open them from your notebook or from the File Explorer. If you wish to commit and push/pull the files to the remote, you can use terminal to issue corresponding git CLI commands.

Azure Repos


We all love a good bar chart or a scatter plot — they're great for deep dives into data. But sometimes, you need to cut to the chase and highlight a single, crucial metric for your stakeholders. That's where our big number chart comes in. It's the simplest way to put your most essential numbers front and center in your apps and dashboards.

![CleanShot 2024-09-17 at 13.18.26@2x.png](https://media.graphassets.com/jrloPDWaTkSYpdCk3lfq)

You can add a big number block by pressing the **add block (+)** button between blocks and selecting the "Big number" option from the menu.

![2024-09-18 13.44.02.gif](https://media.graphassets.com/I4in5gzTQsCyUsXGxswx)

The title block supports interpolating Python variables using Jinja-style brackets. For example: _"Sales in &#123;&#123;month&#125;&#125;"_.

You can choose any variable available inside the notebook for the value you want to display.

The big number chart has different formatting options, including:

- Number
- Currency
- Percent
- Scientific
- Financial

## Comparisons

If you want to show progress or change over time, you can also add a comparison to visualize how this number differs from a previous value, making trends instantly noticeable.

There are a few different ways to do the comparison:

- Absolute change: The difference between the current and previous values.
- Percentage change: The percentage difference between the current and previous values.
- Absolute value: The current value compared to a fixed reference value.


Big number blocks

## What can you do with BigQuery's Google OAuth?

<Callout status="info">
Available on Enterprise plan
</Callout>

With BigQuery's Google OAuth authentication you can give every member of your Deepnote workspace their own set of credentials. This ensures greater security by using short-lived tokens and enabling the use of multi-factor authentication. Follow the principle of least privilege and use granular access control for various BigQuery resources to ensure users can only access the data they need.

### Setting up OAuth for BigQuery in Deepnote

To set up authentication to BigQuery via OAuth, navigate to the [Google Cloud console](https://console.cloud.google.com) and complete the following steps:

1. Click **API & Services** Section and select **Credentials** from the left side bar
2. Click **Create Credentials** and select **OAuth Client ID**
3. If this is the first one, you will need to **Configure Consent Screen** by following the steps. Pick options that make sense for your organization. If you are not sure, you can pick these:
 1. Select user type: **Internal**
 2. Set **Your company OAuth** as name, and add your own email as support email and developer contact information
 3. You do not need to add any scopes
 4. Click **Save and continue**, and return to creating credentials
4. For Application type, choose **Web Application**
5. Fill in the **Name** field, for example `Deepnote OAuth`
6. In Redirect URIs, click on Add URI and enter

 ```
 https://deepnote.com/auth/bigquery/google-oauth-callback
 ```

 as redirect URI

7. Click **Create**
8. From the newly created Client, copy `CLIENT_ID` and `CLIENT_SECRET`

👉 Now head back to Deepnote and add the Client ID, Client Secret, and Project ID to the Google Big Query integration pop up. You will of course need to make sure you've selected Google OAuth as the authentication method.
![gbq_oauth_fill.png](https://media.graphassets.com/nStDWWqlQgGp73sYYZ6Y)

### Querying data with user-specific authentication

Since every user will have to authenticate using their own Google account with this method, the notebook provides a few ways to accomplish authentication (also see the image below). Any of the following actions will initiate authentication for a given user:

- Pressing the **Authenticate** button on a SQL block
- 2. Clicking **Authenticate integration** from three-dot menu of the integration in the right sidebar
- 3. Responding to a request for authentication when executing an SQL block

![gbq_authenticate.png](https://media.graphassets.com/oQtF3hYJShWVLELDxB3T)

When you create an app from a notebook that uses the integration, every app user will need to authenticate with their own account. They will be prompted to sign in using Google during the execution of the app. The results they see will depend on the permissions they have.

<Callout status="info">
- User authentication is also required to browse the BigQuery schema.
- Scheduling a project with a BigQuery OAuth integration connected is not supported, since it requires an interactive authentication flow. To use scheduling, create a BigQuery integration with [service account authentication.](google-bigquery#authenticating-with-a-service-account)
</Callout>

Deepnote allows each user to authenticate to Google BigQuery using their own credentials.

BigQuery with Google OAuth


Deepnote offers a simple mechanism to keep the costs for pay-as-you-go machines predictable and under control.

## Billing alerts

![billing-alerts.png](https://media.graphassets.com/2BVOSUvTBawcdUufGpuP)

You can set up a billing alert in the **Billing & usage** section of the workspace settings. The dollar limit will apply for calendar months, which are the billing period for the pay-as-you-go machines. When the alert limit is reached, all admins in the workspace will receive an email notification.

The usage data is guaranteed to be correct to 30 minutes. It is possible that at the time the email is sent, the real charges are actually higher. The upper bound of the difference should be 30 minutes of whatever pay-as-you-go machines are running in that moment.

## Controlling the use of pay-as-you-go machines

![disable-machines.png](https://media.graphassets.com/11sgzGPqTnSxY3lcfd8x)

By default, the editors and admins in the workspace can pick stronger pay-as-you-go machines for their projects. This enables them to complete more complex analysis on larger datasets.

If you need to make sure to stay in your budget, you can choose to disable the pay-as-you-go machines in the **Billing & usage** section of the workspace settings. Changing the setting will immediately:

- Stop any running projects with stronger machine settings, and change them to use Basic machines on the next start
- Disable the options for pay-as-you-go machines in the project settings

Be careful – enabling the option may interrupt the work of your colleagues.

### Setting billing limits

If you want to set a limit for your charges, we recommend you set a billing alert for that amount. When you receive the notification, you can follow the link the email and disable the pay-as-you-go machines in your workspace.


Billing alerts and limits

Deepnote allows you to both import existing code from Bitbucket (as you would in a local development setting) and sync your code and notebooks with Bitbucket. While there is no built-in integration for Bitbucket yet, setting up and using a Bitbucket repository in Deepnote is fairly straightforward and quick to implement.

<Callout status="info">
If you require full Bitbucket integration to use Deepnote, please let us know by [submitting your request](https://portal.productboard.com/deepnote/1-deepnote-product-portal/tabs/1-under-consideration/submit-idea) or contacting us directly. We are happy to help you integrate Deepnote into your stack!
</Callout>

## Steps to connect to Bitbucket (via ssh)

Every project in Deepnote comes with a filesystem and a terminal. This makes it possible to clone and use any Git repository the same way you would locally. The only challenge is authorization, which we will solve with an SSH key in this tutorial.

If you are fine with passing an access token or password for every Git operation you perform (for example, if you are working on a hobby project or just trying out Deepnote), feel free to skip these steps and clone the repository as you normally would.

1. First, generate a private/public SSH key pair. You can find a tutorial on how to do this in the [Bitbucket documentation](https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/). Upload the public key to Bitbucket, and use the private key to create an SSH key integration in Deepnote, connecting the integration to your project. The process of creating an SSH integration is described [here](/docs/ssh-key). Connecting the SSH key integration to a project means that the private key will be available in an environment variable.

2. In a terminal (in your Deepnote project), execute `source <(/compute-helpers/code/print-integration-env-vars.py)`. This will ensure that your environment variables are available in the terminal as explained in the documentation.

3. In a terminal, enter `eval "$(ssh-agent)" > /dev/null && echo "$BITBUCKET_SSH_KEY_SSH_KEY" | ssh-add - > /dev/null`. This will add your private SSH key to the machine.

4. Then, in a terminal, clone your repository. For example, use the command `git clone git@bitbucket.org:allan_campopiano/robust_statistics_simulator.git`.

5. That's it! Now you can make changes to the code and push them back to Bitbucket as usual.

Bitbucket

Chart blocks allow you to create charts from Pandas DataFrames without using code. They are perfect for fast exploratory analysis, or for those who are not familiar with how to make charts using Python code.

To get started, you can watch our quick demo on chart block configuration.

<Embed url='https://www.loom.com/share/6a6f531acf514df88da25656e144e78e'/>

<Callout status="info">

For practical examples of how to use charts for exploratory analysis, check out this [notebook](https://deepnote.com/workspace/deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/Showcase-fbb28234-0514-42c2-965a-85ce2cdec982/notebook/Notebook%201-17cb089a528f488ab2931c4ac944790a).

</Callout>

## Adding a chart

<VideoLoop src="https://media.graphassets.com/zfPIemgaSuerKkxnvbVL" />

You can add a chart block to your notebook by clicking on the **Visualize** button in the upper right corner of a Dataframe output table.

Alternatively, you can also add a chart block by pressing the **Add block (+)** button in between blocks and selecting the **Chart** option from the menu.

![Screenshot 2022-11-08 at 9.23.06 (1).png](https://media.graphassets.com/3WSqMFGWTJabStyp9day)

## Setting up the data

<VideoLoop src="https://media.graphassets.com/CQysjgaRWfuPpuRfg6sA" />

To start, you need to select a DataFrame to visualize in the upper left corner. By default, your chart will be simple column chart, if you want to switch to different type (e.g. line chart or scatter), you can do so using dropdown in the top left corner.

Once you picked a DataFrame, you need to configure which data to display on the chart. Depending on selected chart type this can be either X/Y axes or segment's color and size for circular charts. Both can be adjusted from the first two sections of the sidebar. Just select a column you want to visualize and adjust settings if needed (we'll go over those in a minute).

In this doc we'll be referring to those setting as dimension and measure. Dimension describes categories (structure) of your chart. For example, in "Monthly revenue" column chart this is "monthly" part and will be displayed on X axis. Dimension settings are always located in the first section of the sidebar. Measure is quantitative value which you want to analyze or summarize on a chart. Following our previous example, this is the "revenue" part and it will be charted on Y axis.

Dimension and measure can correspond to different axes, depending on chart type. For example, in column chart dimension is X axis and measure is Y axis. But they are swapped in bar chart. In pie and donut charts, dimension is color of the segment, while measure is its size.

## Chart AI

Not feeling like pointing and clicking today? We got you. As an alternative to manual setup, you can also ask Deepnote AI to create a chart for you. Open Chart AI by clicking **Chart AI** button on the top of the chart block, type in what you want to visualize, and hit **Submit**. In a few seconds, Deepnote AI will respond with a chart suggestion. You can accept it (**Done**) or reject (**Undo**) and adjust your description.

<VideoLoop src="https://media.graphassets.com/yCGeQETZqFODHTDsOFbw" />

The accepted chart will have all the configurations filled out for you. You can use it as a starting point and manually tweak things further. You can also request follow-up edits from Deepnote AI by typing in a new prompt to modify your existing chart.

<VideoLoop src="https://media.graphassets.com/CYey9W5tQyCcdV6zH4Jk" />

## Adjusting how data is interpreted

Depending on chart type and data type, there are couple of additional settings that change how data is interpeted and charted. You'll find available settings next to the column dropdown.

### Data type

Depending on the chart type, you might be able to manually change the data type of selected column. Deepnote supports 3 data types.

- Nominal — categorical data without a specific order among values. For example, product categories in sales report.

- Temporal — data related to time and dates. Following our sales report example, order date will have temporal data type.

- Quantitative — numerical data that can be measured and analyzed mathematically, e.g. product prices or store's revenue.

### Aggregation

Quantitative and nominal fields can have an aggregation. Aggregation allows us to combine multiple records into single summary value.

Nominal fields support two integration: `count` which reduces set or records to single number representing their count, and `distinct` which returns number of unique values of selected column among matching set of records.

But aggregations really shine with quantitative fields. In addition to `count` and `distinct` you can calculate `sum`, `min`, `max`, `average`, and `median`.

### Binning

In addition to aggregation, it's possible to bin quantitative dimension field into set number of ranges. Those ranges work as categories for data, so you can, for example, chart average customer satisfaction score depending on their age group.

### Time resolution

For temporal fields you can select how precise you want your chart to be. All records will be grouped by selected field into groups depending on the selected resolution. For example, setting resolution to "Year, month" allows you to analyze monthly revenue over time. Or you could set it to "Week" to analyze seasonality of sales across multiple years.

## Group by

Grouping allows you to add another dimension to your chart. But instead of making it 3D and hard to read, it adds color. To add grouping to series click **+ Group by** and select column which should be used for grouping. Don't see it? That means selected chart type doesn't support grouping, try another chart type.

Results will vary depending on selected chart type and data type of the column used for grouping. Grouping by nominal column will split chart into multiple elements, one for each value of the grouping column. If you selected stacked chart type, those elements will be stack one on top of another, otherwise they will be displayed side-by-side. Grouping by nominal field, for example, allows you to see monthly revenue for each store location instead of single total value.

![Group by nominal field](https://media.graphassets.com/Yb5neGfRQeCCzv3ykNwK)

Grouping by quantitative field with applied aggregation will change color of the element depending on the value of the column. For example, if you have "Sales by location" chart, you can add grouping by average value of order.

![Group by numerical field](https://media.graphassets.com/jLihFDeTTqgInO4vcfWC)

To change color palette you can click on color/palette badge next to the series name and select one from the built-in palettes.

<VideoLoop src="https://media.graphassets.com/HOHBGWkRHC812iq2GNmC" />

## Filtering

You can slice and dice your data right from the charts with interactive filtering.
Select data points to filter in/out in one of two ways.

- Highlight a range of data points with mouse-select.
- Click on individual legend entries of grouped series.

Once you selected some data, press the **Keep only** or **Exclude** buttons to filter in/out the selected data points on your chart. You can also combine multiple filtering steps to drill down even further.

<VideoLoop src="https://media.graphassets.com/KC6hlTXPTCuIlpiWdNQV" />

You can also apply **conditional filters** on your charts, giving you more precision and the ability to apply more complex filtering logic. Simply click the **Filter** button in the block actions menu, select the column you wish to filter, choose an operator and a value for your condition, and press **Apply**.

<VideoLoop src="https://media.graphassets.com/p7WogC9QmqIYj1exF9VQ" />

Applied filters are always added to the Filters list, making them easy to remove or modify. You can also combine multiple conditional filters — just press the plus button in the Filters list to add a new filter for more complex filtering logic.

<VideoLoop src="https://media.graphassets.com/oQd8gnR22ohfu3Q2hBqw" />

Conditional filters are also available in Apps! By utilizing filters, your audience can easily slice and dice the plotted data themselves, enabling them to focus on particular aspects of interest. This greatly extends the self-serve capability of your Apps, making them more dynamic and eliminating the need to anticipate all filtering needs in advance.

<VideoLoop src="https://media.graphassets.com/zfPIemgaSuerKkxnvbVL" />

## Combo charts

Sometimes, one series is simply not enough. Like peanut butter with jelly, some visualizations are miles better when combined. In Deepnote you can add multiple series to your charts, use dual axis, and switch series type.

To add new series, click **+ Add series**. Some chart types (like donut or histogram) don't support multiple series, so if you don't see the button try changing the chart type in the top left corner.

In addition to all the usual series settings, you can switch series type. Click on the icon right to the column dropdown and select one of compatible chart types. Additional series can be switched to use seconday measure axis by changing **Axis** settings. This is very handy to show correlation between series with values of different magnitudes.

<VideoLoop src="https://media.graphassets.com/giLHonoFTyiZEhE8liU8" />

## Custom tooltips

Ever needed to show just a bit more information on a chart without cluttering the entire visualization? With custom tooltips you can enrich your charts by selecting additional columns from your plotted DataFrame to display in the tooltip — not just those configured on the chart.

It’s perfect for highlighting more information for an unusual data point on a scatterplot or adding context with an extra metric on your bar charts, all without overwhelming your audience.

Simply go to the **Tooltip** section and select which column you'd like to display.

![Custom tooltip settings](https://media.graphassets.com/v7a8m59sRZezm7c256j6)

## Visual settings

With main data-related settings covered, let's go over how you can adjust look and feel of your chart.

### Titles

You can change chart title, as well as axis titles (if applicable) in the **Titles** section (no surprises here) of the sidebar. For axis titles you can also hide them completely, or, if you leave them empty, default value will be used.

### Value labels

In the **Value labels** section you can enable displaying value labels for a series. Value labels can be displayed as numbers, percentages, or in scientific notation. You can also control number of decimals.

For pie, donut, and normalized chart types it's possible to switch from displaying of absolute values to percentages of total. This is controlled by **Show as** setting in the Value labels section.

For pie and donut charts you can also select if you want to display value labels inside or outside of the chart.

In the **Style** section you can set formatting for labels on quantitative axes.

<VideoLoop src="https://media.graphassets.com/sraOXSaDSMq80vG3ZOD0" />

### Resizing charts

Once you’ve finalized your chart, you can collapse the configuration sidebar to make the chart fill out the whole block. If you wish to present charts on larger displays, you can switch to full-width mode for your notebook. You can also vertically resize the chart by dragging the resize handle at the bottom of the chart canvas.

<VideoLoop src="https://media.graphassets.com/OKU5JTVwSzadIOqC8BUs" />

## Customize charts with code

If you need more flexibility to customize your charts, you can **duplicate your chart block into code** by selecting the option in the **block actions** menu.

This adds a new Python code block to your notebook, containing the configuration of your chart in the [Vega-lite](https://vega.github.io/vega-lite/) specification format. Vega-lite is very powerful and fairly easy to learn so it’s a great option if you need to create a finely customized or super advanced visualisation.

<VideoLoop src="https://media.graphassets.com/GGVOfg2mSISnWCjbp6jy" />

## Limitations

Chart blocks can display a **maximum of 10 thousand rows** of data. If you plot datasets larger than this, Deepnote will sample random 10k rows. You can aggregate or filter the data prior to visualization to get under this limit.

Don't really want all those dimensions, measures, series? Just want to show important metrics in your app? Check out our [Big number](big-number-blocks) block! It looks excellent on dashboards.

No-code charts for fast exploratory analysis

Chart blocks

<Callout status="info">
**Want to get started right away?** Jump right into Deepnote and start exploring using our ClickHouse template [here](https://deepnote.com/launch?template=ClickHouse%20and%20Deepnote).
</Callout>

With the [ClickHouse](https://clickhouse.com/) integration, you can leverage the performance and scalability that comes with ClickHouse's open-source column-oriented DBMS right from within Deepnote. ClickHouse allows users to handle thousands of sub-second queries per second on petabyte-scale datasets. If you need to run fast queries against (very) large datasets, ClickHouse is for you.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FXbgKCh6PkMKJPId0qLPD%2FCleanShot%202022-05-31%20at%2011.28.08%402x.png](https://media.graphassets.com/mwvnyIVZRVCzds7ZXk2E)

Deepnote's ClickHouse integration allows data teams to efficiently query very large datasets, extract relevant data, and start analyzing and modeling in the comfort of their known notebook environment.

### How to connect

To create a ClickHouse integration in Deepnote, open up the integrations overview and click on the ClickHouse tile.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FvlXvYvUYtEz0sTbDBcWt%2FScreen-Recording-2022-05-31-at-2%20(1).gif](https://media.graphassets.com/EbN8UwFNTLaUdL6mYlGM)

To create the integration, you'll need a few things:

- **Hostname**: The hostname of the server you are trying to connect to. [Check out this section](https://clickhouse.com/docs/en/sql-reference/functions/other-functions/#hostname) of ClickHouse's docs for more details.
- **Port**: The port on the server of interest you are trying to connect to. Luckily, [ClickHouse's docs ](https://clickhouse.com/docs/en/sql-reference/functions/other-functions/#getserverport)offer concrete steps for this one as well.
- **Username**: Your username. More in ClickHouse's docs [here](https://clickhouse.com/docs/en/operations/settings/settings-users/#user-namepassword).
- **Password**: The password for the specified username. More details [here](https://clickhouse.com/docs/en/operations/settings/settings-users/#user-namepassword).
- **Database**: The name of the database you would like to connect to.

<Callout status="info">
If your connection is protected, you might need to allowlist Deepnote's IP addresses. [Read more here](/docs/authorize-deepnote-ip-addresses).
</Callout>

### How to use

Once created, you'll be able to connect the ClickHouse integration to any project within your workspace through the right-hand sidebar. The ClickHouse integration comes with custom ClickHouse SQL blocks that help streamline your analytics efforts. You can also convert any existing [SQL block](/docs/sql-cells) to a ClickHouse block.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FXEq6NlDbyPYBkypornU9%2FMy-Moviee.gif](https://media.graphassets.com/GwYoEpMdRYiuJyACFz9T)

As with all SQL blocks, the query results will be saved as a DataFrame and stored in the variable specified in the SQL block.

### Next steps

Jump right into Deepnote &[ learn more about SQL blocks in this A/B testing template](https://mail.deepnote.com/e/c/eyJlbWFpbF9pZCI6IlJJU0VCZ01BQVgxX0E5QWFNWGJ0R3Q2YzFGVXlrZz09IiwiaHJlZiI6Imh0dHBzOi8vZGVlcG5vdGUuY29tL2xhdW5jaD90ZW1wbGF0ZT1BL0IlMjBUZXN0aW5nXHUwMDI2dXRtX3NvdXJjZT1wcm9kdWN0X3VwZGF0ZXNcdTAwMjZ1dG1fbWVkaXVtPWVtYWlsXHUwMDI2dXRtX2NhbXBhaWduPXRlbXBsYXRlc19wcm9tbyIsImludGVybmFsIjoiODQ4NDA2MDFiNTFjZTY4NjAyIiwibGlua19pZCI6NDc1fQ/e578c32db11e3a419a2b3185b2cc15dade67e66d74254451171a351513b18bfa)[.](https://deepnote.com/workspace/Deepnote-Templates-71742312-24f2-4c10-9bf7-786d17280b92/project/Machine-Learning-With-SQL-c626fd30-cb3f-4eae-ac70-c9c4a4f626eb/%2Fmindsdb_demo.ipynb) You can also save yourself some setup work by hitting the `Duplicate` button in the top-right corner to start exploring on your own!

### Secure connections

Deepnote supports securing connections to ClickHouse via optional [SSH tunnels](/docs/secure-connections).

ClickHouse

Deepnote comes with code intelligence tools that are common in traditional IDEs. As you write code or SQL, we make it easy for you to focus on the problem at hand rather than having to memorize function signatures and other code-specific details.

#### Code autocomplete

As you type, we show you the relevant functions, methods, and defined variables. You can also trigger the autocomplete any time by hitting 'Tab' key.

![Autocomplete](https://media.graphassets.com/KrkSnm8iSGn7hjKrV6iB)

#### Function documentation and parameter names

When you hover over a function, we surface its documentation if it is provided by the associated module. In addition, while filling out function parameters, you will see hints for the parameters you'll need. You can also trigger these hints by hitting <Keyboard>Shift + Tab</Keyboard>

![function documentation and parameter names](https://media.graphassets.com/PkgJdy0TEaE7fEoxTP1g)

#### SQL autocomplete

When writing SQL, the built-in _Intellisense_ will offer relevant suggestions for your cursor position. This includes entities in your schema such as databases, tables or columns but also aliases or CTEs that you may have defined in your query. The autocomplete will open automatically as you type. In addition to that, you can trigger it manually by using one of these keyboard shortcuts <Keyboard>Control + Space</Keyboard>, <Keyboard>Option + Space</Keyboard> or <Keyboard>⌘ + I</Keyboard>

Code intelligence


Enter Deepnote, the AI-powered data workspace designed to enhance collaboration, increase productivity, and simplify code reviews. We'll explore how to make the most of code reviews in Deepnote. From real-time collaboration to version control, we've got you covered.

## Workflows for Code Reviews in Deepnote

### Comments

Deepnote allows users to add comments directly within notebooks. This feature is essential for providing feedback during the review process. Reviewers can highlight specific lines of code or sections of the notebook and leave detailed comments or suggestions. This real-time feedback mechanism ensures that issues are addressed promptly, fostering a collaborative environment.

For example, if you spot a potential bug in a data processing block, you can immediately leave a comment, allowing the author to rectify it quickly. Comments will show in real-time, but the mail response will take some time. This interactive review process helps maintain code quality and reduces the chances of errors slipping through.
![Capture Aug 21.png](https://media.graphassets.com/AOtaKcF9RsqUcoC2P3JQ)

### History

The history feature in Deepnote tracks changes over time, allowing reviewers to see how a notebook has evolved. This is crucial for understanding the context of changes and ensuring that new code adheres to the project's standards. By reviewing the history, teams can ensure that all changes are deliberate and justified.

With a clear historical record, it's easy to identify when and why certain changes were made, enhancing transparency and accountability in the development process.
![Deepnote Screenshot Aug 21.png](https://media.graphassets.com/H87SmPZ3RFKQ1EzC5pYo)

### Notebook locking

Notebook locking prevents multiple users from editing the same notebook simultaneously, reducing the risk of conflicting changes. This feature is particularly useful during code reviews, as it ensures that the notebook remains stable while under review, and only authorized users can make changes.

This helps prevent situations where multiple reviewers might inadvertently overwrite each other's feedback or edits, ensuring a smoother and more coherent review process.
![Deepnote Screenshot Aug 21 (1).png](https://media.graphassets.com/rG54iTPSPuOVSy8qXGRT)

### Folder structure for workflow stages

Organize your projects using a folder structure to indicate their status in the review process. In Deepnote, we organize code reviews using the following folder structure:

- 🟡 **In progress:** notebooks are actively being worked on by the team.
- 🔄 **Under review:** notebooks are currently under code review.
- 🟢 **Live (or production, or scheduled):** approved notebooks ready for deployment or execution.
- 🏛️ **Archive:** completed notebooks that are no longer active but need to be retained for reference.

By moving projects between these folders, you can create a clear, visual workflow that tracks the status of each notebook.

### Versioning

Deepnote’s versioning capabilities allow you to track and manage different versions of a notebook. This is particularly useful when multiple people are contributing to the same project. Versioning ensures that you can always revert to previous versions if necessary and helps reviewers see the evolution of the code.

This feature is crucial for maintaining a clear record of changes and ensuring that any issues introduced by recent changes can be quickly identified and resolved.
![Aug 21 Screenshot from Notion.png](https://media.graphassets.com/Z1APPl5fQ1uvE8OoSdC4)

### Pair programming with real-time collaboration

In addition to traditional code reviews, consider using Deepnote’s real-time collaboration features for pair programming. This approach allows two or more team members to work on the same notebook simultaneously, making it easier to spot issues early and speeding up the development process. Real-time collaboration can serve as a dynamic form of code review, with peers providing immediate feedback and suggestions.

This collaborative approach fosters a sense of teamwork and accelerates problem-solving, leading to more efficient project completion.

### Code intelligence tools

Deepnote’s built-in code intelligence tools further enhance the review process. Features like autocomplete, lining, and code formatting help maintain code quality and reduce the number of errors. These tools can also assist reviewers in understanding the code more quickly, making the review process more efficient.

By leveraging these tools, your team can ensure that their code adheres to best practices and industry standards, resulting in cleaner, more reliable code.

## Integrating with GitHub, GitLab, Bitbucket, and Azure DevOps

For teams using version control systems like [GitHub](https://deepnote.com/docs/github-export), [GitLab](https://deepnote.com/docs/gitlab), [Bitbucket](https://deepnote.com/docs/bitbucket), or [Azure DevOps](https://deepnote.com/docs/azure-repos), Deepnote provides seamless integration. These integrations allow you to push and pull notebooks directly from your repositories, ensuring that your code is always in sync with your version control system. These integrations simplify the process of managing code changes and collaborating across different platforms, enhancing the overall efficiency of your development workflow.

## Using nbdime in Deepnote

For teams that need a more detailed comparison of notebook versions, nbdime is an excellent tool. It provides a clear, side-by-side comparison of notebook changes, making it easier to spot differences in code, outputs, and Markdown blocks. Deepnote’s integration with nbdime ensures that you can review changes efficiently without leaving the Deepnote environment.

This tool is particularly useful for comprehensive reviews, ensuring that every change is thoroughly vetted before being merged into the main project.
![Screenshot 2024-08-27 at 10.14.46.png](https://media.graphassets.com/IgZkGOgzT8OZWvhpbbMD)

## Combining Deepnote with GitHub and ReviewNB

To further enhance your code review workflow, consider integrating Deepnote with GitHub and [ReviewNB](https://www.reviewnb.com/). ReviewNB is a specialized tool for reviewing Jupyter Notebooks, and it provides a user-friendly interface for commenting on notebooks within GitHub pull requests. By combining these tools, you can create a seamless review process that leverages the strengths of each platform. This combination provides a robust framework for managing notebook reviews, ensuring that all feedback is captured and addressed efficiently.
![Aug 27 Screenshot from Code Reviews.png](https://media.graphassets.com/g3aRnhWkTnWFQZ1j9oZd)

From real-time collaboration to detailed version tracking and seamless integrations with popular version control systems, Deepnote offers everything you need to maintain high code quality and streamline your development process. Ready to take your code reviews to the next level? Start using Deepnote today and experience the difference in your team's productivity and code quality.


Code reviews


# Comet.ml

[Comet.ml](http://comet.ml/) is meta machine learning platform built to handle the requirements of deep learning. It’s used to track, compare, and optimize the ML and data science experiments. Comet provides tools for logging the process, visualizing results, and collaborating with teams, all within a single interface. The platform supports various machine learning frameworks and allows seamless integration into existing workflows. Simply put, Comet is doing for ML what GitHub does for code.

### Examples Repository

For a better understanding of how to work with [Comet.ml](http://comet.ml/), you can explore the [Example repository](https://github.com/comet-ml/comet-examples?tab=readme-ov-file#comet-for-machine-learning-experiment-management) on GitHub. This repository is packed with examples demonstrating how to use Comet with a wide array of Machine Learning Python libraries, including Fastai, Torch, Scikit-learn, Chainer, Caffe, Keras, TensorFlow, MXNet and more.

The repository serves as a valuable resource for understanding how to integrate Comet into your ML workflows, regardless of the tools you’re using.

### Documentation

For those looking to dive deeper, the full documentation and additional training examples are available on the [Comet documentation page](http://www.comet.com/docs/v2). There can be found everything to master Comet.ml. With clear guides, detailed references, and practical examples, it’s ultimate resource for making the most out of this platform.


Comet.ml


You can open and close the command palette by pressing `cmd + P` on a Mac or `ctrl + P` on Windows. It provides quick access to all your files and most popular actions (e.g. run all, deleting a cell).

![command palette animation](https://media.graphassets.com/fP53g5n1QJqZFfX4q3eK)

You can view all of our keyboard shortcuts [here](/docs/keyboard-shortcuts).


Command palette


In Deepnote, you can work with others [real time](/docs/real-time-collaboration), or asynchronously by leaving comments for your collaborators, available now in both notebooks and apps.

Open the Comments sidebar by clicking on the Comments icon in the top menu on the right side.

### Add a comment

To add a comment, click on the manilla colored comment button in the upper right-hand corner of a block. Type your comment and hit `Post`.

![add a comment](https://media.graphassets.com/pGrzRQypQIObyDaOHCj5)

### Edit or delete a comment

To edit or delete a comment click on the comment and click`Edit` or `Delete`.

![delete a comment](https://media.graphassets.com/uQBEsKgTQGd2Gl6R6ou1)

### Resolve or unresolve threads

Once you comment on a thread, you will be subscribed to that thread and get updates via email. When a discussion is finished, you can close or open a thread by clicking "resolve" or "unresolve", respectively.

In the Comments sidebar, use the toggle switches to search through previous resolved and unresolved comments. Clicking on a comment will take you to the associated block in the notebook.

![resolve or unresolve threads](https://media.graphassets.com/6ohKa3eLTsG89C2s3G62)

### Add a mention in comment

Tag a collaborator in a comment by typing '@' and choosing them from the list. They will receive an update via email.

### Add a mention in notebook

Tag a collaborator or stakeholder in the [rich text](docs/text-editing) of the notebook which will also send an email update.


Work with others real time or asynchronously by leaving comments for your collaborators.

Comments

[Anaconda Inc.](https://www.anaconda.com/) maintains a repository of thousands of curated packages for data science.

Deepnote comes with a pre-built conda environment which allows users to instantly take advantage of the Anaconda ecosystem. This includes being able to use conda—a widely used CLI tool for package management which will include packages from the [Miniconda Installer](https://docs.conda.io/en/latest/miniconda.html).

## How to use the conda environment Deepnote

Simply select “Anaconda - Python 3.8” from Deepnote’s environment dropdown (as shown below). You now have access to the pre-installed packages listed below as well the conda CLI.

![conda.gif](https://media.graphassets.com/Oyygz8pnS7K7ibNWoyZA)

### Installing packages from the Anaconda repository

Use the conda CLI from within the notebook to install packages. For example, to install Altair, type the following into a code block:

```python
!conda install altair
```

See this [template notebook](https://deepnote.com/workspace/deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/Anaconda-Template-Weather-a2c95362-b4c4-4f22-8bc9-0b0f12ee5c34) for an example.

### Listing the installed packages in the conda environment

To list the currently installed packages, type the following into a code block:

```python
!conda list
```

### How does a `requirements.txt` file work with the conda environment?

Upon startup, Deepnote will install any packages listed in a `requirements.txt` file; however, currently `pip` will be used to install packages into the conda environment in this case.

### Which packages are pre-installed in the miniconda environment?

Here is a list of packages that are pre-installed when using the initial conda environment (as described above, you can use `!conda install <package name>` to install additional packages):

- \_libgcc_mutex
- \_openmp_mutex
- altair
- attrs
- brotlipy
- ca-certificates
- certifi
- cffi
- charset-normalizer
- colorama
- conda
- conda-content-trust
- conda-package-handling
- cryptography
- entrypoints
- idna
- jinja2
- jsonschema
- ld_impl_linux-64
- libffi
- libgcc-ng
- libgomp
- libstdcxx-ng
- markupsafe
- ncurses
- numpy
- openssl
- pandas
- pip
- pyarrow
- pycosat
- pycparser
- pyopenssl
- pyrsistent
- pysocks
- python
- python-dateutil
- pytz
- readline
- requests
- ruamel_yaml
- setuptools
- six
- sqlite
- tk
- toolz
- tqdm
- tzdata
- urllib3
- wheel
- xz
- yaml
- zlib

Conda environment


Creating an app is a great way to present your story in a more polished way, while enabling multiple users to simultaneously use the app without interfering with one another. Learn more about the concept and how to create an app [here](/docs/data-apps).


Creating apps

Custom AI Instructions is a new feature in Deepnote that allows you to personalize how Deepnote AI responds to your queries. These instructions are user-specific and provide a way to customize the AI's behavior according to your preferences.

<Embed url='https://www.loom.com/share/bfbca44467664ffdae27d5eccde77a74?sid=c86eaca5-dff2-4a32-91c0-b3f3555dbfaa'/>

## **Accessing custom instructions**

1. Navigate to your Account Settings
2. Look for the AI custom instructions input field
3. Enter your desired instructions
4. Click "Update" to save your changes

## **Key features**

- **Personal customization**: Custom instructions are specific to your account and are not shared with your team members
- **Persistent behavior**: Once set, the instructions will apply to all your interactions with Deepnote AI across notebooks
- **Flexible configuration**: You can modify instructions to specify:
 - Preferred plotting libraries (e.g., Seaborn)
 - Response formatting
 - Interaction style
 - Code styling preferences
 - Or just making your docstrings sounds uwu

## **Example use case**

Let's look at a practical example:

**Without custom Instructions:**

- When asked to plot data, DeepnoteAI uses default plotting libraries
- Responses follow standard formatting

**With custom instructions:**

# Example instruction:

"When you're going to be plotting multiple charts, always use plotly and subplots to plot them in one output. Also I want all of the charts to be green!"

![CustomAIInstructionsJustGif.gif](https://media.graphassets.com/YZ7UwPGUQGOUQnJOcEmd)

### **Best practices**

1. Be specific about your preferences
2. Consider including:
 - Preferred libraries and frameworks
 - Coding style guidelines(suggested to refer to PEP)
 - Communication preferences
 - Output formatting requirements

> Custom Instructions is a powerful way to maintain consistency in your AI interactions and streamline your workflow according to your specific needs and preferences.

Custom AI instructions

If [default environments](https://deepnote.com/docs/default-environment) don't meet your project's requirements, you can create **custom environments** in Deepnote. These can be either imported into your project or defined directly in Deepnote itself.

## How does it work?

At its core, each project's runtime environment is a **Docker image**, giving you extensive customization options. Anything you can define in a Dockerfile can be implemented in your Deepnote project. This is particularly powerful for:

- Building libraries from source with specific versions
- Installing custom binary dependencies
- Managing Python packages through requirements.txt

Custom environments are especially valuable for complex projects where dependency installation is time-consuming and needs to be reused across multiple notebooks.

## Requirements

We recommend building your custom Dockerfile on top of our `deepnote:python3.x` base image to ensure compatibility with all Deepnote features. If you're creating an environment from scratch, your image must meet these requirements:

- Python (versions 3.8-3.11) installed and accessible via the `python` command
- Built for the `linux/amd64` platform (M1 Mac users: use `-platform linux/amd64` flag)
- Functioning `pip` installation that can install packages to Python's path
- `bash` and `curl` installed

### Package Installation

All pip installations must use our constraints file:

```bash

pip install tensorflow -c https://tk.deepnote.com/constraints<PYTHON_VERSION>.txt

```

This constraints file helps prevent library conflicts and ensures compatibility with Deepnote's features like the variable explorer and interactive DataFrame views. Don't worry about installing additional dependencies for these features—we'll automatically detect and load them at runtime.

## How to create a custom environment?

### Local Dockerfile

<aside>
Building a local Dockerfile in Deepnote is a premium feature available only on **Team** and **Enterprise** plans.
</aside>

You can customize your environment by placing a `Dockerfile` in your project's root directory. Deepnote handles all the image building and hosting for you.

The most common approach is to extend one of our default environments. For example, here's how to add [OpenCV](https://opencv.org/) dependencies to our Python 3.9 base image:

```
FROM deepnote/python:3.9

RUN apt update && apt install -y ffmpeg libsm6 libxext6

```

To use your local Dockerfile, simply select **Dockerfile** from the Environment dropdown.

![CleanShot 2024-11-28 at 10.00.45.png](https://media.graphassets.com/CzaaCxT1QO5mqxUtXavA)

When building from a local Dockerfile, only the following Docker instructions are supported:

- **FROM**: Start with any public base image or [Deepnote environment](https://hub.docker.com/u/deepnote)
- **RUN**: Execute commands as root to install libraries and tools
- **ENV**: Set environment variables or define default Jupyter kernels

Local Dockerfile builds are available on **Team** and **Enterprise** plans with the following constraints:

- Maximum build time: 60 minutes
- For longer builds, we recommend building outside of Deepnote (see below)

### Hosted docker image

**Public images**

You can use any publicly available Docker image from repositories like [Docker Hub](https://hub.docker.com/), giving you access to thousands of pre-configured environments.

**Private Repositories**

**Team** and **Enterprise** users can connect to private image repositories from:

- [Docker Hub](https://www.notion.so/docs/docker-hub)
- [Google Container Registry](https://www.notion.so/docs/google-container-registry)
- [Amazon ECR](https://www.notion.so/docs/amazon-ecr)

To use private repositories:

1. Create an integration with your repository provider
2. Add your credentials
3. Connect the integration to projects where you need your private images

![CleanShot 2024-11-28 at 10.14.03.png](https://media.graphassets.com/m8a18CpYRaCyRNQEF0bk)

## Keep environment settings

When you duplicate a project, all environment configurations come with it - saving you from setting everything up again.
Note: This doesn't apply when duplicating to a different workspace. In that case, you'll need to reconfigure your environment.

Pro tip: Need the same environment across many projects? Create a [project template](https://deepnote.com/docs/projects#project-templates) - it's the most efficient way to standardize your setup.

Custom environments


Deepnote supports dark mode. You can find the settings under “Preferences” > “Theme.”

Alternatively, you can switch to dark mode by opening the command palette and searching for the “Switch to dark mode” command.


Dark mode

![Apps 2.png](https://media.graphassets.com/6ecwQuqREaUWb33uWO8g)

### How to create an app

To publish a project as an app, open the notebook you would like to publish and select **Create app** in the top right corner.

Next, click the **Add all blocks** button to add all blocks to the app, or add blocks individually by using the **Add block to app** button in the top right corner of your blocks.

For executable blocks you can choose to only display the output, the code, or both.

In the App settings, you can use toggles to **Hide all code blocks**, **Make app full-width**, or **Show table of contents**.

<Callout status="info">
Note there can be only **one** app per project. In order to create more apps split your work across multiple projects.
</Callout>

### App sharing settings

Just like you can set the [sharing settings for projects](/docs/share-projects), you can set the access controls for your app. For example, it is possible to have a private project with a public app. In this case, no one will be able to open and edit your project; however, the app will be visible to anyone.

Visibility of the app can be set through the **Permissions** dropdown in the _App settings._

The following access levels are available:

- **Workspace & collaborators.** The only people who can access the app are the ones who have access to the original project. This will depend on your [project sharing settings](/docs/share-projects).
- **Anyone with a link.** Only people with whom you share the link can access the app. It is hidden from your public profile and search engines.
- **Public.** Your app is visible in your public profile and can be indexed by search engines. This is useful if you would like others to come across your work.

It is important to note that these settings apply to all the content within the notebooks, regardless of the visibility of individual blocks. Even if certain blocks are hidden within the notebook, the content of these hidden blocks remains accessible to those who have access to the app. This means that sensitive or private data within hidden blocks is not excluded from the access permissions set for the app. You should be mindful of this when deciding what information to include in notebooks linked to apps with broader access settings.

### Live updates

Changes in the notebook are now automatically saved to your app to improve your development speed.

There are three ways to make changes to already published apps:

1. **Small updates.** You can simply edit the notebook to make immediate changes to the app. The app viewers will need to refresh the tab in order to see them.
2. **Duplicating the notebook and bringing changes over.** For larger changes, you may wish to first prepare them in a copy of the notebook, and once you are satisfied, you can paste them over to the original notebook.
3. **Duplicating the notebook and republishing it.** If you want to ensure that your app is in a working state at all times, prepare them in a copy of the notebook, and once you are satisfied, publish the new notebook. Block visibility and other configuration options will be preserved, you only need to check how you want to display any new blocks you added. The link to the app will remain the same.

### Modifying the layout

The layout of an app can be modified independently of the notebook layout via the app preview pane. Simply drag and drop multiple blocks next to each other to organize your work. Changing the layout of an app does not affect the notebook layout.

Don't forget that you can organize blocks into columns! First create a row of blocks by dropping them next to each other. Then, drag and drop other blocks below the cells to create a column. This is useful for creating layouts that should render side-by-side on large screens, but stack vertically on smaller screens, like smartphones.

You can change the width of a block or column by dragging the border between the blocks.

Complex layouts work best when combined with the "full-width" setting - see an example of an app on the picture below:

<ImageBorder variant="blue">![Screenshot 2024-05-03 at 13.56.00.png](https://media.graphassets.com/RnGEydvTmKDP405EsMZk)</ImageBorder>

### Show notebook outputs in app

By default, your app users start with a clean slate, and all block outputs are empty. If you'd like to show the results from the last notebook execution (for example a scheduled run, or something you ran manually), you can enable the setting **Show notebook outputs in app**.

Note, that when your app has multiple input blocks, your users might need to provide a value for each one before the notebook executes without any null values.

When the setting is enabled, the app users will see the values of input blocks and all block outputs currently visible in the notebook.

### Embedding

By embedding your app, you can render it within tools that support it - like Notion, Confluence, or Salesforce, or on your website. You can specify the width and height of the app by using the width and height attributes in the iframe HTML tag. The Deepnote app is responsive and will adjust to fit.

Once your app is embedded, you can parameterize it the same way as a normal app - by adding input values in the URL. The \_\_run parameter can be attached to the URL to make the app run automatically right after it's loaded with the current parameters.

To embed an app, click on the "Embed" button in the Sharing section of App settings.

<ImageBorder variant="blue">![embed-app-settings.png](https://media.graphassets.com/yI96GF5SzO1WvGIUQawl)</ImageBorder>

In the modal window, turn on embedding. Once the setting is enabled, Deepnote will allow the app to be rendered in an iframe, and the preview of the embedded app will render on the left side of the modal.

<ImageBorder variant="colorful">![embed-app-modal.png](https://media.graphassets.com/Qd7LB2LiRcSgBuCi5rcI)</ImageBorder>

To embed the app within tools like Notion, Confluence, or Medium, use the **Embed url**. These tools will automatically use the link to render an iframe. If you want to embed your app directly within a website where you have control over the HTML markup, you can use the Embed code link. You can adjust the width and height attributes of the iframe to get the desired size of the app.

#### Embedding apps in Notion

Type `/embed` in a Notion page and paste the embed link into the input. The Deepnote app should appear. You can resize the embedded app to make it look nicer (for example, not collapse columns if it's too narrow).

#### Embedding apps in Confluence

To embed an app in Confluence, use `/iframe` command, and paste the link to **URL** field. We also recommend to set the **frameborder** parameter to "hide", since Deepnote app already renders a frame around itself.

Keep in mind that you can embed individual blocks as well! You can read more about how to do it [here](https://deepnote.com/docs/sharing-and-embedding-blocks).

### App interactivity

The viewers can refresh the app results by clicking the **Run** button in the top-right corner, or by selecting a different value for any of the input blocks. This executes the notebook from top to bottom, or, with the [Reactivity](#reactivity) option turned on and kernel already existing for the session - only the blocks affected by the inputs.

The apps are interactive by default, unless you disable the setting _Allow viewers to run the app_.

#### How does interactivity work?

- Whenever a user interacts with an [Input block](/docs/input-blocks) (e.g. a dropdown menu, a text input) and then clicks on **Apply,** a kernel is created (unless it already exists for the session), and the blocks are executed with the new input values.
- The kernel (which contains the app state) is kept alive for 10 minutes after the last execution, or unless the user closes the browser window for the app. Every run after 10 minutes of inactivity will result in the creation of a new kernel.
- Every run is stateless - users interacting with a published project don't interfere with the project's notebook or app state of other users.
- App runs use the same machines as the project. Execution of the app will wake up the project's machine and use its resources, and the machine will be shut down [after a period of inactivity](/docs/long-running-jobs#how-deepnote-decides-when-to-turn-off-hardware-of-a-project). If you expect a lot of concurrent users in your app, you can upgrade to a stronger machine in your environment settings, and change the project's inactivity settings.
- The app will use the same environment settings (such as custom Docker images) as the underlying project.

<Callout status="info">
If the machine is not running when the app is run, the first execution may take longer due to machine startup time and execution of the initialization notebook.
</Callout>

#### Reactivity

Reactivity can be enabled by turning on "Allow reactivity" settings in App settings. After any input block value is changed and the app is re-run, instead of executing all blocks in the app, only blocks that depend on the values of changed input blocks are executed. This makes app re-runs much quicker, by skipping the blocks that do not influence what the consumer sees in the app.

The analysis of dependencies between blocks is performed by examining the Abstract Syntax Tree (AST) of the Python code in your notebook, also considering the order in which they appear in the notebook. The resulting dependencies can be reviewed by clicking the "Open Dependency Graph" button in the app editor, located directly below the reactivity settings.

<Callout status="info">
Note that reactivity works only if a kernel with the app's state already exists. The first execution of the app, or an execution after 10 minutes of inactivity, will always require running all the blocks.
</Callout>

### Advanced usage

#### Automatically run the app on load

If you add `__run==true` parameter to the app's url, it will run automatically on load. This is especially useful if you want to run an app with inputs in the url automatically. An example of how the app's url with inputs and auto-run could look like:

```
https://deepnote.com/app/your-company/Movie-Recommendation-8dd173b0-d6bc-4212-b1b5-7c89dc0db780/?min_rating=1&max_rating=5&__run=true
```

#### Letting users download files from the project filesystem

A common use case for apps is preparing some results as a CSV file, and letting the end user download it.

When you publish your project as app, you can access the contents of the filesystem using a URL of this format:

```
https://deepnote.com/publish/<project_id>/file?path=<file_path>
```

You can find the `project_id` in the URL, it's in the form of a UUID. The general structure of the URL when editing a notebook is

```
https://deepnote.com/workspace/<workspace_name>-<workspace_id>/project/<project_name>-<project_id>/notebook/<notebook_name>-<notebook_id>
```

You can also find it in an environment variable `DEEPNOTE_PROJECT_ID` if you want to generate it programmatically.

Here's an [example notebook](https://deepnote.com/@deepnote/Download-files-in-app-03df4fac-84e9-45c0-aff6-691016698a7b) demonstrating how to do that.

The access controls of the published app are still respected – only people with access to the app can access the files. Note: all files in the /work directory (what you see in the left sidebar) can be accessed this way, so make sure you are not exposing anything sensitive.

Keep in mind that parallel executions of the app (two users at the same time) could overwrite each other's files. If that is a concern, consider generating unique filenames at execution time, and printing the generated URL for each user.

_Watch it all come together in this 1-minute demo:_

<Embed url='https://www.loom.com/share/4f7f5397f7a3481189102f3743f9afdf?hide_owner=true&hide_share=true&hide_title=true&hideEmbedTopBar=true' />

### Deleting apps

You can delete an app at any time by going to _App settings_ and selecting **Delete app**.

Some analyses are best presented with the journey to get there, while others call for a more polished interface. You may wish to hide some code blocks so they don’t distract from the story you are trying to tell, or parametrize the notebook with a couple of inputs so your audience knows which knobs to turn. With Deepnote, you can now create stunning apps that will impress your CEO.

Data apps


Data cataloging is a game-changer in modern data management. At the forefront of this revolution is Deepnote, an AI-powered data workspace. Let’s dive into data cataloging, the unique capabilities of Deepnote, and how it empowers data scientists, analysts, and engineers to achieve greater insights and efficiency.

**What is a data catalog?** A data catalog is a comprehensive inventory of an organization's data assets. It uses metadata aka data about data, to provide detailed descriptions, helping users quickly locate, evaluate, and understand the data they need. This inventory includes structured data like tables and charts, unstructured data such as documents and images, and machine learning models and visualizations. Key components of a data catalog:

- **Metadata management**: collects and organizes metadata to describe data assets.
- **Search functionality**: facilitates easy discovery of data through keyword search, tagging, and filters.
- **Data evaluation**: provides context and quality metrics to assess data suitability for specific uses.
- **Access control**: ensures secure access to data, maintaining compliance with regulations.

## Why data cataloging is crucial

Implementing a robust data catalog offers numerous benefits:

- **Improved efficiency**: streamlines data discovery and reduces time spent searching for data.
- **Enhanced data context**: provides comprehensive metadata that helps users understand data relevance and quality.
- **Reduced errors**: ensures data accuracy by maintaining detailed data lineage and quality information.
- **Better decision-making**: facilitates quicker and more informed decisions by providing easy access to high-quality data.

## Deepnote's approach to data cataloging

Deepnote takes data cataloging to the next level by offering a seamless, collaborative, and intuitive platform specifically designed for data professionals.

**Documenting and describing datasets**: use Markdown blocks to document each dataset, including metadata such as source, creation date, and data schema. Detailed descriptions maintain clarity and context.

**Data profiling and cleaning**: identify and correct missing values, outliers, and inconsistencies through data profiling. This ensures the quality and integrity of your datasets.

**Incorporating visualizations and summary statistics**: add visualizations using libraries like Seaborn to provide a quick overview of the data. This helps in understanding data patterns and making informed decisions.

**Collaborating with team members**: share your Deepnote project and set permissions for team members to ensure effective teamwork and data integrity, or create a [Deepnote app](https://deepnote.com/docs/data-apps) that you embedded in a document for your coworkers.

**Organizing and managing datasets**: use folders and tags to organize your project. This ensures a streamlined workflow and easy retrieval of datasets.

**Maintaining an up-to-date data catalog**: regularly updating documentation and metadata. Use version control to track changes and set periodic reviews to ensure data accuracy and relevance.

Deepnote offers several unique ways that make it an ideal platform for data cataloging:

- **Seamless integration**: integrates with various data sources, allowing easy import and management of datasets.
- **Collaborative environment**: enables real-time teamwork with robust sharing and permission-setting features.
- **Interactive visualizations**: provides tools to create visualizations and summary statistics for quick data insights.
- **Enhanced documentation**: utilizes Markdown blocks for comprehensive documentation, ensuring clarity and context.
- **Interactive data applications**: allows the creation of interactive data applications within Deepnote, streamlining the cataloging process.

By integrating Deepnote into your data management workflow, you can significantly enhance your ability to catalog, manage, and analyze datasets. This leads to better decision-making, increased efficiency, and a deeper understanding of your data assets.

- **Empowers users**: Deepnote's intuitive interface and powerful features empower data professionals to take control of their data cataloging processes.
- **Promotes collaboration**: encourages teamwork and knowledge sharing, fostering a collaborative data culture within organizations.
- **Drives innovation**: by simplifying complex processes, Deepnote allows data professionals to focus on innovation and derive actionable insights from their data.

## Example of data cataloging in Deepnote

We prepared an [example notebook](https://deepnote.com/workspace/Deepnote-Templates-71742312-24f2-4c10-9bf7-786d17280b92/project/Data-catalog-34f035b4-bb25-410b-ac79-5c60c6ccc5cb/notebook/%F0%9F%93%9D%20Introduction%20-f00a6fdd779148238e351b26646bcfe7) with [Atlan API](https://atlan.com/) that you can duplicate, play around, or use for your data set. There are multiple levels that you can explore depends on your seniority and use case.

## Collaborating and sharing your data catalog

**Share your notebook**

- Click on the “Share” button in the top-right corner of the notebook.
- Set permissions for your collaborators (e.g., view, edit).
- Share the link with your team.
- Create a [data app](https://deepnote.com/docs/data-apps) from your data catalog

**Collaborate in real-time**

- Collaborate with your team by allowing them to add, edit, or comment on the notebook.
- Use comments to discuss dataset details or flag issues that need attention.

## Maintaining and updating your data catalog

**Regular updates**: set a schedule for reviewing and updating your data catalog. This ensures that all information is current and accurate.

Example: Every month, review the datasets and update any changes in schema, data sources, or descriptions.

**Version control:** utilize Deepnote’s version control features to track changes in your notebook.

This helps maintain a history of updates and allows you to revert to previous versions if needed.

In the rapidly evolving landscape of data management, having a robust data catalog is no longer optional—it's essential. Deepnote not only makes data cataloging more accessible but also enhances it with features tailored to the needs of data professionals. By leveraging Deepnote's capabilities, you can transform your data cataloging process, driving better decision-making and unlocking the full potential of your data.

Ready to streamline your data cataloging process? Sign up for Deepnote today and experience the benefits firsthand. Empower your data teams, enhance collaboration, and drive innovation with Deepnote—the leading platform for data cataloging.

For those interested in further exploring the capabilities of Deepnote and how it can revolutionize your data management processes, here are some additional resources:

- [**Getting started with Deepnote**](https://deepnote.com/docs): Learn the basics of setting up and using Deepnote for your data projects.
- [**Deepnote integrations**](https://deepnote.com/integrations): Discover how to connect Deepnote with your existing data sources and tools.
- [**Case studies**](https://deepnote.com/customers): See how other organizations are using Deepnote to enhance their data practices and drive success.

Feel free to reach out to our support team if you have any questions or need assistance getting started with Deepnote. Happy data cataloging!


Data catalogs and documenting data with Deepnote

<Embed url='https://www.loom.com/share/0538c44bf30a4098979f0fc31d1958ba'/>

Our Data table isn’t a separate block type, but a part of SQL and Code blocks used to display DataFrame contents.

The data table interface provides powerful tools to explore, analyze, and present your data. Through intuitive controls for sorting, filtering, cell formatting, and column customization, you can make your data easier to understand and highlight important bits.

<Callout status="info">
The manipulations we cover here don’t modify underlying data. So you can adjust the data table as you like without fear of messing up the original data.
</Callout>

## Sorting

To sort DataFrame rows by particular column, click on its name, and from the column menu, select if you want to sort ascending or descending.

<Callout status="info">
Clicked on column name, but got nothing? It seems you can’t modify this DataFrame. This might be the case for data tables in apps without the “Allow app to be run” setting enabled, or if the data table is inside an embedded cell.
</Callout>

<VideoLoop src="https://media.graphassets.com/K1ymyNmqTW2V5g1lGzlt" />

## Filtering

To filter your DataFrame, click on **Filter** in the column menu and fill in required data. Alternatively, you can use the **Filter button** from block actions.

If DataFrame has at least one filter applied to it, you’ll see a list of applied filters on top of the data table. From there, you can edit, remove, and add new filters.

<VideoLoop src="https://media.graphassets.com/rRUPVPP7QfW7gDSl3aaY" />

## Cell formatting

To make data easier to digest, you can highlight important bits with a pop of color. To open cell formatting configuration, click the **Format** button at the bottom of the data table.

Here you can select if you want to apply single color formatting to cells that match a certain condition or if you want to apply color scale to all cells in a column.

<VideoLoop src="https://media.graphassets.com/6qngCrpGQ4WvOa7DUyej" />

## Renaming column

To change column name, click **Rename** in the column menu and enter a new name. Pressing Enter will rename the column, and pressing Esc will cancel renaming.

For columns with a custom name, there will be a **Reset name** button in the column menu that restores the original column name.

<VideoLoop src="https://media.graphassets.com/GE4agCuMQcyD54NoOxYN" />

## Hiding column

In the same menu, you can click **Hide column** to hide it from the data table. To see a list of hidden columns, click on the **eye icon** in the top left corner of the data table. Clicking on a column in this list will make it visible again.

<VideoLoop src="https://media.graphassets.com/AylhYjQkQFCnMR8lOfQS" />

## Reordering columns

Columns can be rearranged just by dragging them around.

<VideoLoop src="https://media.graphassets.com/ugEACzBDQtSb5QyKGmdE" />

## Text wrap

By default, cells in the data table render data only in single line, so longer texts get trimmed. With **Wrap cells** enabled (you can find it in the column menu), text will be wrapped instead (up to 10 lines per cell).

<VideoLoop src="https://media.graphassets.com/tSoJC51fR5uJkdMRHpl7" />

## Copying cell content

You can always select a portion of text in a cell and press `Cmd + C` or `Ctrl + C` to copy it. To copy the whole content of the cell, you need to select it first by triple-clicking on it. Now, if you press `Cmd + C` or `Ctrl + C` (or use the context menu to copy), the whole content of the cell will be copied instead.

<VideoLoop src="https://media.graphassets.com/DnkXoyQRcKiRsd8FfMFP" />

Sort, filter, format, organize, and view tabular data

Data tables

<Callout status="info">
Beta version available on the Enterprise plan - please talk to our support to get it enabled
</Callout>

With the Databricks integration, you can leverage the capabilities of Deepnote's SQL blocks to query your warehouses at lightning-fast speed.

Deepnote's Databricks integration allows data teams to efficiently query their data, extract relevant data, and start analyzing and modeling in the comfort of their known notebook environment.

### How to connect to Databricks

![databricks_modal.png](https://media.graphassets.com/Qz23S2MLSYe3PlSeTEjv)

To create the integration, you'll need a few things. Follow [Databricks guide](https://docs.databricks.com/integrations/jdbc-odbc-bi.html#get-connection-details-for-a-sql-warehouse) to get the connection details for a SQL warehouse.

- **Host**: Databricks server hostname. The hostname of the server you are trying to connect to.
- **Port**: The port on the server of interest you are trying to connect to. The default is port 433.
- **HTTP Path**: Databricks compute resources URL.
- **Token**: Your personal access token. For instructions about how to generate a token, see [Databricks docs](https://docs.databricks.com/dev-tools/auth.html#pat).
- **Schema**: Default schema used by integration. This is equivalent to running `USE <schema_name>`. If you provide no schema the default is `default`.
- **Catalog**: Default catalog used by integration. If you provide no name the default is `hive_metastore`.

### How to use

Once created, you'll be able to connect the Databricks integration to any project within your workspace through the right-hand sidebar. The Databricks integration comes with custom Databricks SQL blocks that help streamline your analytics efforts. You can also convert any existing SQL block to a Databricks block.

As with all SQL blocks, the query results will be saved as a Pandas DataFrame and stored in the variable specified in the SQL block.

In order to enable the SQL blocks to work with Databricks integration, you have to install `sqlalchemy-databricks` in your project. Run the following command in a block to get it installed:

```
!pip install sqlalchemy-databricks
```

Or, alternatively, add it to local Dockerfile and it will get automatically installed and cached for faster project starts.

### Next steps

Jump right into Deepnote & [learn more about SQL blocks in this A/B testing template](https://deepnote.com/launch?template=A/B%20Testing). You can also save yourself some setup work by hitting the `Duplicate` button in the top-right corner to start exploring on your own!

### Secure connections

Deepnote supports securing connections to Databricks via optional [SSH tunnels](/docs/secure-connections).

Databricks

dbt empowers analytical engineers to transform data within their data warehouses by simply writing select statements. It automatically converts these select statements into tables and views.

dbt specializes in the Transform step of the ELT (Extract, Load, Transform) process. While it doesn't handle data extraction or loading, it excels at transforming data already present in your warehouse.

## **Integrating dbt with Deepnote**

In Deepnote, you can use the following adapters for your dbt development workflow:

- Snowflake
- BigQuery
- Redshift
- Postgres
- Databricks
- Clickhouse

<Callout status="info">
If you wish to use dbt for its Semantic Layer (MetricFlow) capabilities, please see your options [here](https://deepnote.com/docs/semantic-layer).
</Callout>

### **Configuring dbt**

To set up dbt with Deepnote, follow these steps:

1. `pip install dbt-core dbt-[adapter]`
2. Create a new init notebook and add `!export DBT_PROFILES_DIR=/work/` into new block
 ![Screenshot 2025-01-17 at 17.37.44.png](https://media.graphassets.com/VsUcB4H8S0GqbbH72AfD)

3. Create profiles.yml and add your environment variables there, e.g. snowflake

 ```bash
 default:
 outputs:
 dev:
 account: "{{ env_var('SNOWFLAKE___DBT_ACCOUNTNAME') }}"
 user: "{{ env_var('SNOWFLAKE___DBT_USERNAME') }}"
 password: "{{ env_var('SNOWFLAKE___DBT_PASSWORD') }}"
 database: "{{ env_var('SNOWFLAKE___DBT_DATABASE') }}"
 role: "{{ env_var('SNOWFLAKE___DBT_ROLE') }}"
 warehouse: "{{ env_var('SNOWFLAKE___DBT_WAREHOUSE') }}"
 type: snowflake
 schema: TYPEFORM
 target: dev
 ```

4. create dbt_project.yml and add your project info, e.g.

 ```bash
 # Name your project! Project names should contain only lowercase characters
 # and underscores. A good package name should reflect your organization's
 # name or the intended use of these models
 name: 'default'
 version: '1.0.0'

 # This setting configures which "profile" dbt uses for this project.
 profile: 'default'

 # These configurations specify where dbt should look for different types of files.
 # The `model-paths` config, for example, states that models in this project can be
 # found in the "models/" directory. You probably won't need to change these!
 model-paths: ["models"]
 analysis-paths: ["analyses"]
 test-paths: ["tests"]
 seed-paths: ["seeds"]
 macro-paths: ["macros"]
 snapshot-paths: ["snapshots"]

 clean-targets: # directories to be removed by `dbt clean`
 - "target"
 - "dbt_packages"

 # Configuring models
 # Full documentation: https://docs.getdbt.com/docs/configuring-models

 # In this example config, we tell dbt to build all models in the example/
 # directory as views. These settings can be overridden in the individual model
 # files using the `{{ config(...) }}` macro.
 models:
 test:
 # Config indicated by + and applies to all files under models/example/
 example:
 +materialized: view
 ```

## Example usage

In our [example notebook](https://deepnote.com/workspace/Deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/dbt-demo-b0c962f8-96a6-4a1e-9182-9ba3b34bb5ae/notebook/1-dbt-coffee-analysis-024eae4504cc48018d88b11461e06bb0) we demonstrate how to use dbt in Deepnote to analyze coffee preferences by data role using the COFFEE_RESPONSES dataset. We automate data transformations and ensure data quality with scheduled tasks.

### **Creating models**

We developed a model, `coffee_preferences_by_role.sql,` to identify the most popular coffee for each data role and calculate its order percentage. This simplifies complex SQL into reusable components. To run the model we use `dbt run --models coffee_preferences_by_role` or just simply `dbt run` . Here is an [example usage](https://deepnote.com/workspace/Deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/dbt-demo-b0c962f8-96a6-4a1e-9182-9ba3b34bb5ae/notebook/1-dbt-coffee-analysis-024eae4504cc48018d88b11461e06bb0)

```
%%writefile /work/models/coffee_preferences_by_role.sql

WITH coffee_counts AS (
 SELECT
 data_role,
 ordered_coffee,
 COUNT(*) AS coffee_count
 FROM DEEPNOTE_DEVELOPMENT.TYPEFORM.COFFEE_RESPONSES
 GROUP BY data_role, ordered_coffee
),

ranked_coffees AS (
 SELECT
 data_role,
 ordered_coffee,
 coffee_count,
 ROW_NUMBER() OVER (PARTITION BY data_role ORDER BY coffee_count DESC) AS rank
 FROM coffee_counts
)

SELECT
 data_role,
 ordered_coffee AS most_popular_coffee,
 coffee_count,
 ROUND((coffee_count / SUM(coffee_count) OVER (PARTITION BY data_role)) * 100, 2) AS percentage
FROM ranked_coffees
WHERE rank = 1
```

**Implementing tests**

We created `not_null_coffee_data.yml` to validate our data, ensuring no nulls in key columns and unique entries per data role. To run the test we use `dbt test`

```
%%writefile /work/tests/not_null_coffee_data.yml

version: 1

models:
 - name: coffee_data
 tests:
 - not_null:
 column_name: data_role
 - not_null:
 column_name: ordered_coffee

 - name: coffee_preferences_by_role
 tests:
 - unique:
 column_name: data_role
 - not_null:
 column_name: most_popular_coffee
```

## Organizing and cataloging

If you want to share your catalog and organize it with Deepnote you can run `!dbt doc` , then create streamlit of Flask app, if you want to use streamlit you can see the the [example](https://deepnote.com/workspace/Deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/dbt-demo-b0c962f8-96a6-4a1e-9182-9ba3b34bb5ae/dbt_catalog.py) if you want use Flask you need to serve the 'index.html' on `0.0.0.0`, then switch on incoming connection

<aside>
💡Note that this will open your connection to entire internet
</aside>

![Screenshot 2025-01-24 at 15.55.06.png](https://media.graphassets.com/UdEXzNKjTCWqT8tD1sFm)

Then you can share the url with your teammates or iframe it in the notebook.

![Screenshot 2025-01-24 at 16.03.39.png](https://media.graphassets.com/9jKWMgxsRcWIewFqTGg7)

### Github integration

If you are using Github for storing the models you can use [Github integration](https://deepnote.com/docs/github), if wish to see how to set it up [click here
](https://deepnote.com/workspace/Deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/dbt-demo-b0c962f8-96a6-4a1e-9182-9ba3b34bb5ae/notebook/5-Github-integration-aebe8acded9c49edb58e681999499173)
![Screenshot 2025-02-10 at 14.53.36.png](https://media.graphassets.com/7WyNNzDWR3eKwbakIUs2)

### **Scheduling and automation**

- **Automate execution:** Use Deepnote's scheduled notebook to run dbt models and tests regularly.
- **Continuous monitoring:** Keep your data insights updated automatically.
- **Advanced alerts:** Use Deepnote's Slack integration to receive alerts if model tests fail.

![Screenshot 2025-01-20 at 18.20.41.png](https://media.graphassets.com/uPWmW6kRS2yZPqTmzBQH)

Explore the complete [models](https://deepnote.com/workspace/Deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/dbt-demo-005f82aa-d184-435e-9909-f2328681ca78/models/coffee_preferences_by_role.sql?project-sidebar=project&integrationId=9731b859-597b-4c7e-8b5f-720840f62afe) and [tests](https://deepnote.com/workspace/Deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/dbt-demo-005f82aa-d184-435e-9909-f2328681ca78/tests/not_null_coffee_data.yml?project-sidebar=project&integrationId=9731b859-597b-4c7e-8b5f-720840f62afe) in this project to see dbt in action. This example illustrates how dbt, combined with Deepnote, Snowflake and [Slack integration](https://deepnote.com/docs/slack) empowers data teams to streamline data transformations and maintain data quality. By automating these processes, you can focus on deriving insights and making data-driven decisions efficiently.

Deepnote AI is designed to make your data work more efficient and productive. Ask questions in plain language, and Deepnote AI will craft or revise code according to your instructions. It can also provide concise explanations for complex pieces of code, enhancing your understanding. Whether you’re a Python pro or a coding beginner, Deepnote AI assists you every step of the way to gain the insights you need.

<Callout status="info">
Deepnote AI is available on **Pro**, **Team** and **Enterprise** plans.
</Callout>

<Callout status="warning">
Deepnote AI utilizes generative AI technology (powered by OpenAI and Codeium) to provide answers based on your instructions and offer code completion suggestions. The generated outputs such as code and text can include errors, biases and inaccuracies. Please always review code before running it.
</Callout>

## Enabling Deepnote AI

To make use of Deepnote AI, you must enable this feature in your workspace. Navigate to **Settings & Members** and click on **Project settings**. With Deepnote AI turned on, every editor and admin user in your workspace will gain access to features like code generation, editing, and explanation.

<Callout status="info">Deepnote AI utilizes third-party providers (OpenAI and Codeium) to process the content of your notebook. Refer to our [Data Privacy](/docs/ai-data-privacy) section for more information about data usage.</Callout>

![deepnote ai main settings.png](https://media.graphassets.com/kkhgsstdTiSqqTyIyJT2)

## Feedback

We're always looking to improve Deepnote AI, and your feedback is invaluable!
The easiest way to leave feedback on the quality of Deepnote AI's outputs is by giving a thumbs up/down when you're reviewing the AI-generated blocks. If a particular output didn't work for you, just press 👎 and you will be able to share more context about it in a survey window. The more details you put in here, the better we can act upon it to make it better for you!

![CleanShot 2024-10-24 at 09.54.55.png](https://media.graphassets.com/Gto2HcFgTOC7q8vCKm3U)

You can also share your ideas and suggestions about Deepnote AI on our [Product Portal](https://portal.productboard.com/deepnote/1-deepnote-product-portal/c/110-deepnote-ai?utm_medium=social&utm_source=portal_share).

If you want to be directly involved in shaping the future of Deepnote AI, we're seeking research partners for our upcoming AI projects. To chat more, please book a time with our product team via this [link](https://calendly.com/gabor-deepnote/30min-1).

Use the power of Deepnote AI to generate, edit and explain code.

Deepnote AI

<Callout status="info">
The API is available on Team and Enterprise plans.
</Callout>

The Deepnote API provides you with an endpoint to programmatically execute an existing notebook. This enables various automation use cases from customized scheduling to integrating notebooks deeper into your workflows together with other applications.

## Authorization

To use the API, first you need to create an API key in your workspace's Settings & members > General settings.

![Screenshot 2022-09-05 at 15.06.57.png](https://media.graphassets.com/50a8WERkR5ivtYVpUQO0)

After creating your API key, you can use it to post requests to the Deepnote API. To use it, send it as bearer token in the `Authorization` header of your requests.

```
Authorization: Bearer INSERT_API_KEY
```

## Execute notebook

```
POST https://api.deepnote.com/v1/projects/{project_id}/notebooks/{notebook_id}/execute
```

If the project's machine is offline, this starts the machine. If the notebook is already running, triggering this endpoint has no effect.

To find the project ID and notebook ID, open a Deepnote notebook in your browser and inspect the URL, it should be in the following format:

```
https://deepnote.com/workspace/{workspace name}-{workspace ID}/project/{project name}-{project ID}/notebook/{notebook name}-{notebook ID}
```

The project ID is a UUID with dashes (e.g. `25fcb3b2-cf3d-4c08-9b24-4306f1518caa`) and the notebook ID is a UUID without dashes (e.g. `abaf726ac4c34589961a588de29cd665`).

#### Parameters

| Parameters | Example | Description |
| --------------------- | ------------------ | ------------------------------------------------------ |
| **Path parameters** |
| `project_id` | | The ID of the project that the notebook is located in. |
| `notebook_id` | | The ID of the notebook. |
| **Header parameters** | | |
| `Authorization` | `Bearer {API_KEY}` | Your API key |

#### Responses

| Response code | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `202(Accepted)` | The request was successfully received and the notebook was staged for execution. |
| `401(Unauthorized)` | No API key was provided, or it's not associated with the workspace that owns the project, or the workspace is not on one of the required billing plans. |

Deepnote API


## **Connecting to your data**

Deepnote integrates with all major data warehouses and databases, as well as common file storage services. Dropping a CSV straight into the notebook also works like a charm.

### Working with a CSV

To work with a CSV, simply drag it onto the notebook.

![Import_CSV_Squared.jpg](https://media.graphassets.com/N5sICTLi0cSODwkEXkwe)

An SQL block with a sample query will be created for you and the file will be uploaded to Deepnote's file system. The results are saved to a Pandas DataFrame.

![Query_it_with_SQL.jpg](https://media.graphassets.com/UVu8NCRzREm1LUKLk8dy)

Pandas can also be used to read the uploaded CSV into the notebook's memory.

![Query_it_with_python.jpg](https://media.graphassets.com/LKQbquJRqGEZmuH1sivy)

### Connecting to a database

Click "Create new" from the **Integrations** section in the Project sidebar on the left.

Choose the database integration you want (e.g., Snowflake, BigQuery, PostgreSQL). You'll be asked to add your credentials.

![Connect_database_From_project.jpg](https://media.graphassets.com/ghICx4WQgqqhiMrLDIUq)

Once you've connected the database to a project (It'll appear in your sidebar), you can click it to preview its schema and use SQL blocks to query your data. The results are saved to a Pandas DataFrame.

![Preview_Schema and Query.jpg](https://media.graphassets.com/Aw5nXRIZTmo2DXhUGYhg)

## Inviting your team members

It's dangerous to go alone. Take this link.

### Links, email invites, and business domains

From the **Settings & members** section in the left-hand panel, you'll find links that you can send to your team in order to invite them to the workspace.

![Members_Page.jpg](https://media.graphassets.com/GTEICRTtGtfHZareqjnA)

Different links provide different access controls.

![Invite_Link.jpg](https://media.graphassets.com/RZeTuVcnTwiOHJbJWGul)

Alternatively, you may enter email addresses and assign access controls that way (note the toggle switch that allows anyone with your business domain to join the workspace).

![Alternatively_Invite.jpg](https://media.graphassets.com/R151o7DRta8abJfIn7W1)

## Analyzing your data

Deepnote is a fully collaborative SQL and Python environment with a suite of no-code tools to help you move fast.

### SQL blocks

Create an SQL block and write native SQL queries against your CSVs and databases. Mix in Python to get the best of both languages. Results are saved to a Pandas DataFrame (am I repeating myself?).

![Combine_SQL&Python.jpg](https://media.graphassets.com/hxQB0czRAixz87ASLcOA)

### Python blocks

You probably expected Python blocks, but there's more to it than that. Use the preinstalled libraries, `pip install,` whatever you want — you can even define your environment with Docker.

![Install_Libraries.jpg](https://media.graphassets.com/eCOg7oQJRvGW5I24jruC)

### Moving fast with no-code tools

Visualize any Pandas DataFrame with [chart blocks](https://deepnote.com/docs/chart-blocks).

![Leverage_Vizualizations.jpg](https://media.graphassets.com/fh48Ob32QOSM05NbMkD7)

Parameterize your notebook with [input blocks](https://deepnote.com/docs/input-blocks).

![Leverage_Input_Blocks.jpg](https://media.graphassets.com/y3cf1J4Tc2XOba86JvPX)

Communicate with [rich text blocks](https://deepnote.com/docs/text-editing).

![Rich_Text.jpg](https://media.graphassets.com/DnwLKlQ2aXIv1vEKId6g)


You want to get up and running right away, so let's cover the basics - Connecting to data, inviting team members, and analyzing data.

First steps

## What can you do with the shared datasets integration?

When you don't have a bucket of your own (or your files are too large for the integrated file system), the shared datasets integration is a good option. Under the hood, it's simply a Google Cloud Storage bucket that Deepnote manages for you. Since this integration is a bucket, it means that the files stored here will be available for your whole team across all projects.

<Callout status="info">
Available on Team and Enterprise plans
</Callout>

## How to use the shared datasets integration

From the right-hand panel, under **Integrations**, click the **+** button and choose **Create new integration**. Select **Upload large files or datasets**.
![shared_data_select.png](https://media.graphassets.com/ztkuOdWHRzyFoskumQGA)

 

Upload your files into a shared dataset by dragging and dropping them into the drop zone. Give your integration a name and you're good to go.
![datasets_modal.png](https://media.graphassets.com/slsVwOhTQbuAIOpHvpJ3)

 

Your files are now available on the path `/datasets/{integration name}` in your notebook as shown below (query any CSVs with Python and SQL). You can also edit the data stored by adding, deleting, or modifying it directly in the mounted integration folder.
![datasets_frame.png](https://media.graphassets.com/miLOoSenSR6hvhOEZ4OK)

<Callout status="info">
- The shared dataset integration may cause slower performance for a large number of files (e.g., 50+ images). It's better suited for a smaller number of large files.
- The shared datasets integration does not support uploading folders.
</Callout>

The shared datasets integration can be used to store your team's large data files and make them available across all projects.

Shared datasets


Deepnote offers a variety of preconfigured environments, all maintained by our engineering team and ready to use out of the box. These environments are primarily built on Debian Linux and the official Python repository, with Python being our primary supported language. While you can also use other languages like R or Stata, these environments have limited functionality and don't support all Deepnote features.

Default environments launch in seconds thanks to pre-warmed instances and cached images. These environments are battle-tested, used by the majority of Deepnote projects, and come with comprehensive support from our team. They include all common binary dependencies pre-installed, minimizing potential setup issues and helping you get started faster.

Our engineering team regularly updates the default environments to ensure all system components stay current, including Linux security patches and binary dependencies. However, if your project requires specific versions of system binaries or kernels, we recommend using a [custom environment](https://deepnote.com/docs/custom-environment) to maintain consistent behavior and prevent potential compatibility issues.

![CleanShot 2024-11-28 at 09.23.46.png](https://media.graphassets.com/4RENoyheQwa27hZtVDTz)

## Managed environments

| Environment name                   | Image tag                                                                   | Description                                                                                                                                                                                                                                               |
| ---------------------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Python                             | [deepnote/python:3.x](https://hub.docker.com/r/deepnote/python)             | This lightweight Python environment includes essential binary dependencies while letting you start with a clean slate for Python packages. It's ideal for projects that need a few specific libraries or for building custom environments on top of this. |
| Python with data science libraries | [deepnote/python:3.x-datascience](https://hub.docker.com/r/deepnote/python) | This environment comes pre-loaded with popular data science libraries, offering a ready-to-use workspace for those who want to start coding immediately without package installation.                                                                     |
| Conda                              |                                                                             | This environment comes pre-loaded with popular data science libraries via Conda, offering a ready-to-use workspace for those who prefer Conda package management.                                                                                         |
| Tensorflow GPU                     |                                                                             | This environment comes with TensorFlow configured for GPU acceleration, offering a ready-to-use workspace for deep learning projects that require high-performance computing.                                                                             |
| R                                  | [deepnote/ir:x](https://hub.docker.com/r/deepnote/ir)                       | This experimental environment enables R programming within Deepnote notebooks.                                                                                                                                                                            |
| R with data science libraries      | [deepnote/ir-with-libs:x](https://hub.docker.com/r/deepnote/ir-with-libs)   | This environment comes with popular R data science libraries pre-installed.                                                                                                                                                                               |

Need something more specific? You can create [custom environments](https://deepnote.com/docs/custom-environment) tailored to your project's needs. Build your own environment with exactly the dependencies you require, then share it across your entire workspace for consistent development.


Default environments


Deepnote provides an environment for training machine learning models similar to a standard Jupyter notebook or any local Python setup. It offers secure integration with third-party services like Comet.ml or Weights and Biases, enabling you to adhere to best practices throughout the model development lifecycle. For more details, refer to the section on [Training machine learning models in Deepnote](https://deepnote.com/docs/training-machine-learning-models).

## Model deployment options

Once your model is trained and optimized, Deepnote offers various deployment options, allowing you to seamlessly move from data preprocessing to production deployment within the same platform. Here are some deployment methods supported by Deepnote:

- **Directly from Deepnote**: The simplest way to host your model is to create a dedicated notebook that loads your trained model and runs inference against desired input values. Read more on this below.
- **Cloud storage**: Save your model to cloud storage services like [Amazon S3](/docs/amazon-s3), [Google Cloud Storage (GCS)](/docs/google-cloud-storage), or [Google Drive](/docs/google-drive) using Deepnote's native integrations.
- **Docker Containers**: Package your model and its dependencies into a Docker container. You can then deploy this container to repositories such as [DockerHub](https://deepnote.com/docs/docker-hub), [Google Container Registry](https://deepnote.com/docs/google-container-registry) (GCR), or [Amazon Elastic Container Registry](https://deepnote.com/docs/amazon-ecr) (ECR).
- **Hugging Face**: Push your models to the Hugging Face model repository and use Hugging Face Inference Endpoints for hosting deployments.
- **Modelbit**: Create a hosted deployment environment and push models directly from Deepnote to Modelbit. You can also point Modelbit deployment to run on Snowflake. Read more about [deploying ML models to Snowflake with Deepnote & Modelbit](https://deepnote.com/blog/deploy-ml-models-to-snowflake-modelbit) on our blog.
- **Beam**: Use Beam to set up a hosted environment and deploy models directly from Deepnote. [Learn more about packaging models into Bean endpoints](https://docs.beam.cloud/v2/endpoint/invocation).
- **AWS SageMaker**: If your infrastructure is based on AWS, deploy models directly to AWS SageMaker environments, leveraging Deepnote's integration for a streamlined deployment process. Check our our template project for Creating training jobs on [AWS SageMaker from Deepnote](https://deepnote.com/workspace/deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/Training-on-Sagemaker-from-Deepnote-dc3eb58f-8a56-4431-976b-69c87995db96/notebook/66be62fbac9e4577bace85b457c87aa6) and our [example data app](https://deepnote.com/app/deepnote/Sagemaker-training-and-inference-from-Deepnote-dc3eb58f-8a56-4431-976b-69c87995db96) serving inference from a model deployed on Sagemaker.
- **FastAPI**: Use the FastAPI framework to build a backend service around your model. You can then deploy this service using hosting platforms like [Porter](https://docs.porter.run/guides/fastapi/deploy-fastapi), [Coherence](https://docs.withcoherence.com/#fast-api-example), or [Platform.sh](https://github.com/platformsh-templates/fastapi).

These options provide flexibility and ease of use, enabling you to choose the deployment method that best fits your project's needs.

### Serve directly from a notebook

The simplest way to host your model is to create a dedicated notebook that loads your trained model and runs inference against desired [input values](https://deepnote.com/docs/input-blocks). You can even create a [beautiful data app](https://deepnote.com/docs/data-apps), that sits on top of the inference notebook and executes the model with any provided input inside this app. Think of it as a simple dashboard connected to your trained model inside Deepnote which also uses Deepnote hardware to run. These runs are also cached, can be [launched on page load](https://deepnote.com/docs/data-apps#automatically-run-the-app-on-load) and can also be used to [host generated files](https://deepnote.com/docs/data-apps#letting-users-download-files-from-the-project-filesystem) or even whole models.

Check our our example project on [💳 Customer churn modeling](https://deepnote.com/workspace/deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/Customer-churn-modeling-1096d967-46f3-4233-8500-19b888b80b1d/notebook/2.%20Building%20a%20TensorFlow%20model-33c813c0beae4210ab3bdf55fd6e5a50). Notice we have 3 notebooks inside the project

1. [Exploratory data analysis](https://deepnote.com/workspace/deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/Customer-churn-modeling-1096d967-46f3-4233-8500-19b888b80b1d/notebook/1.%20Exploratory%20data%20analysis-40744b0586bc494cb470f45b5e71aa98) - where we explore our dataset and possibly perform any necessary data cleaning or augmentation.
2. [Building a TensorFlow model](https://deepnote.com/workspace/deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/Customer-churn-modeling-1096d967-46f3-4233-8500-19b888b80b1d/notebook/2.%20Building%20a%20TensorFlow%20model-33c813c0beae4210ab3bdf55fd6e5a50) - where we conduct all our training
3. [Application](https://deepnote.com/workspace/deepnote-8b0ebf6d-5672-4a8b-a488-2dd220383dd3/project/Customer-churn-modeling-1096d967-46f3-4233-8500-19b888b80b1d/notebook/3.%20Application-88c92d2dc77a4242add2f64cb590f989) - where we load the trained model, and using Deepnote’s input blocks we allow other users (colleagues or other stakeholders) to run churn prediction according to their needs.

On top of our **Application** notebook we have built [this data app](https://deepnote.com/app/deepnote/Customer-churn-modeling-1096d967-46f3-4233-8500-19b888b80b1d), that simplifies the user experience for all stakeholders, especially the non-technical ones. This app can also be [embedded](https://deepnote.com/docs/data-apps#embedding) into 3rd party tools like Notion or Confluence.


Deploying machine learning models

<Callout status="info">
Public docker images are available on the Free plan, and private docker images are available on Team and Enterprise plans
</Callout>

1. Enter your Docker Hub credentials. You can also use your [Personal Access Token](https://docs.docker.com/docker-hub/access-tokens/) instead of your password. Docker Hub does not allow using email address for login -- you will need to use the username.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FPRasVN2hCbXoa2wc17y2%2FScreen%20Shot%202022-03-30%20at%201.02.15%20PM.png](https://media.graphassets.com/RfnqWM9YQZ6DtnErGPxX)

2\. After **creating** and **connecting** the integration, you can use Docker images from your private Docker Hub repository as a [custom environment](/docs/custom-environment) for your project.

**Your credentials are encrypted and securely stored in our database.** When you delete this integration, we also wipe your Docker Hub credentials from our systems.

Add credentials to use your private Docker images in your projects. Available only for teams.

Docker Hub


When you cancel your subscription or downgrade your plan, you will lose access to some features which are not available on the Free plan. Below we describe how each feature will be downgraded.

1. **Editor seats.** The workspace will revert to a member limit of 3. The most recently added admins and editors above this limit will be given Viewer permission.

1. **Projects.** You will keep all projects that are in your workspace. If the workspace has reached or exceeded the project limit of the Free tier, you will not be able to create any new projects.
1. **Folders.** The workspace will keep any existing folders, but new folders cannot be created.
1. **Revision history.** You will not be able to open any versions older than 7 days. They will still be visible in the Activity Log.
1. **Workspace member permissions.**
   - Any workspace members who are not Admins will be given Viewer permissions.
   - Any workspace members that join via an Editor invite link will join as Viewers.
1. **Project permissions.**
   - Any project collaborators with a permission that is not _Edit_ will be set to _View_.
   - Any project public links with a permission that is not _Edit_, will be set to _View_.
   - Any project with workspace permissions that is not _Full Access_ will be set to _View_.
1. **Published projects.** Any published projects that are shared with Workspace & collaborators will be unpublished. You can always publish these projects again with a different sharing settings.
1. **Team integrations.** Integrations that are not part of the Free plan will be disconnected from all projects. You will still be able to see these integrations, but you will not be able to connect them to your projects. You can find a full list of these integrations on our [Pricing page](https://deepnote.com/pricing).
1. **Private docker images.** You will not be able to use private Docker Hub, ECR, or GCR images as your environment.
1. **Scheduled notebooks.** If you have more than 1 scheduled notebook, the most recently scheduled notebooks will be un-scheduled.
1. **Notebook API.** Any API keys that were generated will be invalidated.
1. **Additional machines.** All projects that use paid machines will revert to using free Basic machines.
1. **Static IPs.** You will no longer be able to connect to Deepnote via these static IPs.
1. **Additional machines.** All workspace projects will revert to using the default machine. Any additional machine hours that you haven’t paid for yet will be charged.
1. **Hardware shutdown time.** All projects will revert to a 15-minute hardware shutdown time.


Downgrading or canceling

<Callout status="info">
Available on Enterprise plan
</Callout>

With the Dremio integration, you can leverage the capabilities of Deepnote's SQL blocks to query your Dremio instances at ludicrous speed.

Deepnote's Dremio integration allows data teams to efficiently query their data, extract relevant data, and start analyzing and modeling in the comfort of their known notebook environment. In addition, switching between Dremio and other distributed SQL engines won't require you to rewrite any SQL statements; all that's needed is switching the SQL block's assigned integration.

### How to connect to Dremio

![Dremio_setup.png](https://media.graphassets.com/MyA8JXY1Q5GPsb8CBjJp)

To create the integration, you'll need a few things:

- **Hostname**: The hostname of the server you are trying to connect to. `data.dremio.cloud` for the US control plane, `data.eu.dremio.cloud` for the European control plane.
- **Port**: The port on the server of interest you are trying to connect to. The default port is 443.
- **Schema**: Database schema to use by default when a schema is not specified in a query. However, this does not prevent queries from being issued for other schemas. Such queries must explicitly include the schema.
- **Token**: Personal access token to use when authenticating to Dremio. See Dremio's [docs](https://docs.dremio.com/cloud/security/authentication/personal-access-token/#creating-a-token) for more details.

At the moment this integration only supports connecting to Dremio Cloud, and not to Dremio Software.

### How to use

Once created, you'll be able to connect the Dremio integration to any project within your workspace through the right-hand sidebar. The Dremio integration comes with custom Dremio blocks that help streamline your analytics efforts. You can also convert any existing [SQL block](/docs/sql-cells) to a Dremio block.

As with all SQL blocks, the query results will be saved as a Pandas DataFrame and stored in the variable specified in the SQL block.

### Next steps

Jump right into Deepnote & [learn more about SQL blocks in this A/B testing template](https://deepnote.com/launch?template=A/B%20Testing). You can also save yourself some setup work by hitting the `Duplicate` button in the top-right corner to start exploring on your own!

### Secure connections

Deepnote supports securing connections to Dremio via optional [SSH tunnels](/docs/secure-connections). The Dremio integration always connects via SSL.

Dremio


Deepnote does not yet offer a native way to integrate, but it's easy enough to use the Python SDK.

### 1. Register a Dropbox API app and create an integration

To use the Dropbox API, you'll need to register a new app in the [App Console](https://www.dropbox.com/developers/apps).

First, choose your app's permissions (before you generate the token). You'll probably need at least `files.metadata.read`, `files.metadata.write`, `files.content.read` and `files.content.write`.

You'll need to use the access token created with this app to access API v2. You can do it on the Settings tab, under OAuth2 section, with **Generate access token**.

Create a new Environment variable integration in Deepnote, and add the access token.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FEemnuVUJ7L37bsgsNTd3%2FScreen%20Shot%202022-03-30%20at%2010.59.24%20AM.png](https://media.graphassets.com/eOOCJnXaRW2l3gQMCh73)

### 2. Try it out

First, install the Python SDK with `pip install dropbox` . Then modify and add this snippet:

```python
import os
import dropbox

dbx = dropbox.Dropbox(os.environ.get("DROPBOX_ACCESS_TOKEN"))

# list files in the root folder
for entry in dbx.files_list_folder('').entries:
    print(entry.name)

# download a file from Dropbox
dbx.files_download_to_file('my_file_in_deepnote.csv', '/my_folder/my_file_in_dropbox.csv')
```


Dropbox

Deepnote is a collaborative data notebook revolutionizing how people approach data science. It provides a powerful, cloud-based workspace that allows users to easily explore, collaborate on, and share data, create interactive charts and dashboards, and build and deploy machine learning models.

Deepnote is completely free for students, teachers, and researchers. Anyone interested in learning about data science or developing their skills in this field can access a top-notch tool without worrying about the cost.

Designed to be intuitive and user-friendly, Deepnote gives you a wide range of features that make it easy to get up and running quickly, including pre-built templates, integrations with popular data science libraries and tools, and a collaborative environment that allows multiple users to work on the same project at the same time.

<Callout status="info">
Our Education plan is **100% free** for students, teachers, and researchers who connect their accounts to school email addresses. The free Education plan provides access to most of the Team plan features other than advanced computing machines, Scheduling and Deepnote AI.
</Callout>

![CTA@2x.jpg](https://media.graphassets.com/6oUQY8vzRmg2n5omQiyg)

## The challenges we're tackling

Teaching and learning data science is often harder than it needs to be. Deepnote helps learners and educators alike overcome common classroom challenges so they can focus on what matters most.

##### Remote teaching

Making sure everyone is on the same page can be difficult, especially in an online environment. Thanks to Deepnote’s real-time collaboration, educators can see what others are doing and keep them focused on the task at hand.

##### Classroom setup

Students typically lose precious time learning how to set up their environment by installing Python, Jupyter, and other necessary software. Deepnote is fully browser- based, which means students can get started instantly without any instructions. Just sign up, create your classroom, and invite your students.

##### Collaboration

Data science is a team sport. With Deepnote, students can work together in the same notebook. They can also be invited to your notebook as Viewers so they can follow along and ask questions.

##### Accessibility

Because Deepnote runs in the cloud, students don’t require powerful hardware. They can also access Deepnote from any computer, be it their personal laptop or a PC at their school's library.

## Features for teachers

Teaching data science doesn't have to be difficult. Here are some of the key features both students and instructors can count on.

##### Real-time collaboration

Deepnote offers real-time collaboration options (just like Google Docs), which simplifies both teaching and group projects. Collaborating in Deepnote is as easy as copy-pasting a URL.

##### Shared environments

Students can create duplicates of a lecturer’s notebook with the required environment already pre-set. No more time wasted on environment preparation.

##### Zero setup

Deepnote is fully browser-based and requires zero setups. No more fussing with terminals or package installations. You're ready to go in two clicks.

##### Comments & reviews

Provide assistance, leave feedback, or grade notebooks using comments. Receive notifications whenever a new comment is added.

##### Classroom management

Manage all classroom material — from lectures to assignments — all in one workspace. Customize permissions for students, instructors, and TAs to match your classroom structure.

##### Easy submissions

Deepnote allows all students to easily [export their work](/docs/submissions) into the desired format and submit all their assignments, homework, and group work to their instructors.

## Deepnote glossary

##### Workspace

[Workspaces](/docs/workspaces) are the home for all your notebooks. You can organize your notebooks in folders and create integrations that allow you to connect to your data. Once this is done, you can then access it from every notebook. You can also invite your classmates/colleagues to a workspace and collaborate together. In other words, a workspace is your classroom.

##### Folder

Deepnote helps you keep your work sorted and organized. Each Deepnote classroom can have multiple folders with many projects. Thanks to folders, you can divide your students into multiple teams while keeping them part of the same classroom.

##### Project

A project in Deepnote contains code and data, and it all runs in one environment. Each project can represent one lecture or class exercise.

##### Classroom

The space where all your projects, students, data, and integrations are located. A classroom can also be referred to in different terminology (i.e., [workspace](/docs/workspaces)).

##### Access levels

Deepnote has two access levels: project base, which only affects people in the same project as you, and workspace base, which affects all the members of the workspace. This feature gives the owner of the classroom more control. For more information, visit [workspace permissions.](/docs/team-permissions)

Education overview

Deepnote is built from the bottom up to help people work together. Educational institutions around the globe use Deepnote to teach data science and foster collaboration between both students and instructors.

The free Education plan provides even more collaboration options, giving users access to all Deepnote features and the ability to invite an unlimited number of educators and students.

## How does verification of the Education plan work?

There are three ways to verify your Education plan:

### 1.) Sign up with your school email

When signing up to Deepnote for the first time, enter your valid school email address. A link will be sent to you from Deepnote asking you to verify your email address. Once you verify your email, you can create your first Deepnote workspace and access all your free benefits. 

![EDU_verification_email.png](https://media.graphassets.com/1GNGPwOOSUeL7WIC3PXG)

### 2.) Sign up with GitHub Student Developer Pack

You can access Deepnote’s free Education plan by signing up with your GitHub account and accessing your GitHub Student Developer Pack. We’ll automatically verify your eligibility and link your GitHub account to your Deepnote account. Visit the GitHub Student Developer Pack to learn more. 
![EDU_Github_signup.png](https://media.graphassets.com/3AFRhxYRCbbdWKoh4twO)

<Callout status="info">
 Sign up using your [GitHub Student account!](https://deepnote.com/github-student-pack)
</Callout>

### 3.) Verify your Education plan directly in the app

If you already signed up using your personal or private email address, you can still get your free Education plan directly in the Deepnote app. Go to **Settings** -> **Upgrade** -> **Education plan** 

![In_app_EDU_verification.png](https://media.graphassets.com/2mhOb7BpSMuT9rQy9Zwx)

## FAQs:

#### Am I eligible for Deepnote's free Education plan?

You’re eligible if:

- You’re a student or educator at an accredited college, university, or K-12.
- You're part of another educational institution, MOOC, or NGO.
- You have a school email address (we don’t accept student IDs or other documentation for verification).

You’re not eligible if:

- Your Deepnote account is associated with a personal email address, such as a Gmail or Outlook address.

<Callout status="info">
**Deepnote's free Education plan** is available to all students, teachers, and researchers with access to their school email addresses. But we want to help all seekers of data science knowledge and individuals looking to do social good.

Are you part of another community, MOOC, NGO, or learning institution and don't have a school email? You might still qualify for the free Education plan. [Get in touch with us!](mailto:help@deepnote.com)
</Callout>

#### What should I do if I can't verify my email address?

If your school email address has not been verified automatically, and you believe you're eligible for Deepnote's free Education plan, please get in touch with us at help@deepnote.com. We'll verify your request manually.

#### What should I do if I haven't received an automatic verification email?

If you haven't received your verification email, please check your spam folder and wait a moment. It might take up to 15 minutes to receive your confirmation email. After 15 minutes, please request us to resend it.

#### How long does it take to verify my free Education plan manually?

If your school email address is not automatically recognized and you decide to contact us directly at help@deepnote.com, it might take a couple of days to complete your verification.

Get free Education plan

Dig into the questions and answers below to find out more about our free Education plan. And don't forget to visit our [Education page](/education) to discover more helpful content, including our teaching templates!

<Callout status="info">
Our Education plan is **100% free** for students, teachers, and researchers who connect their accounts to school email addresses. The free Education plan provides access to most of the Team plan features other than advanced computing machines, Scheduling & Deepnote AI.
</Callout>

#### Does the education plan include free computing hours of Plus, Performance, high-memory, or GPU machines?

Each member of the education workspace gets an unlimited number of hours of the Basic machine (up to 5GB RAM and 2vCPU) for free. If you want to use Pro, Performance, high-memory, or GPU machines, you must add a payment method.

#### Can I get an education plan even though I don't have a school email address?

Deepnote's free education plan is offered to all students, teachers, and researchers with access to their school email addresses. However, we also provide it to users who wish to learn more about data science or do social good, even if they don't have school email addresses (e.g., MOOCs, NGOs, etc.). Learn more in our [Education plan verification page.](/docs/edu-verification)

#### How can I upgrade my machines while I'm on the free Education plan?

You can easily add your payment option and change the machine you want. Learn more in our [Selecting hardware page.](/docs/selecting-hardware)

#### How do I use Deepnote for educational purposes?

Deepnote makes it easy to create and share assignments with your class or work on group assignments collaboratively. All your projects are private and only visible to you by default. With the free Education plan, you can create a Deepnote workspace that allows you to share and manage projects with students and colleagues.

#### How can I migrate my accounts between personal and school email addresses?

In case you wish to migrate your accounts or merge the work on your accounts, please [contact us.](mailto:help@deepnote.com)

#### What should I do if my education account has been suspended?

Deepnote's free education plan is only for learning and teaching purposes. Any abuse of usage will result in a permanent banning of your account. Learn more in our [terms of service.](/terms) If you believe you haven't breached our policy and the suspension was unauthorized, [contact our support team.](mailto:help@deepnote.com)

#### Does Deepnote support learning management system (LMS) integrations?

No, Deepnote doesn't currently offer integrations with any LMS. We focus on creating an open and collaborative environment for data scientists to explore their ideas. As such, we don't have plans to add LMS support in the near future. However, you can easily export your work from Deepnote into PDFs or other formats that can be imported directly into most LMS systems.

<Callout status="info">
If you don't find the answer to your question, please don't hesitate to [contact us!](mailto:help@deepnote.com)
</Callout>

FAQs

You can set environment variables for a project by adding a special type of Integration. Go to **Integrations** in the **right-hand sidebar**, and then either modify an existing "Environment variables" integration or add a new one:

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2Fz86MzyybMGq4fmuWK23c%2FScreen%20Shot%202022-03-30%20at%203.37.14%20PM.png](https://media.graphassets.com/7HB8rqRSXuPSu4BfS96a)

Environment variables are a secure way to store secrets. They are stored encrypted, and decrypted only before mounting them into your project.

Don't forget to connect your project(s) to the newly created "Environment variable" from within the project itself. Once added to the project, you can easily use the Environment Variable from within your notebook:

```python
import os
os.environ.get("VARIABLE_NAME")
```

### Connecting to any service or integration

You might be storing your data in a database that does not yet have a native integration in Deepnote. Or perhaps you need to use a third-party library that issues an API key (e.g. [Mailgun](https://www.mailgun.com/)).

In this case, you can add an environment variable which will let you access the remote integration or third party service. An example is Deepnote's [neptune.ai integration](https://docs.neptune.ai/integrations-and-supported-tools/ide-and-notebooks/deepnote).

<Callout status="warning">
Some integrations can't be used just by including environment variables. [Let us know](https://deepnote.typeform.com/to/agk6aN) if you are looking to use a service like this in Deepnote.
</Callout>

### Environment variables in Terminal

The environment variables from the Integrations are not available in Terminal sessions out of box.

You can use following command in the Terminal to set the environment variables in your Terminal session:

```
source <(/compute-helpers/code/print-integration-env-vars.py)
```

Please note that the environment variables are not automatically injected into the running Terminal sessions when a new integration is connected to your running project. You need to rerun the command above to set the variables.

### Deepnote-specific environment variables

Deepnote automatically provides several environment variables in notebooks, terminals, and Streamlit apps that can be useful for authentication or other workflows:

- **DEEPNOTE_PROJECT_ID**: The unique identifier of the current project. This can be used to generate URLs or for other integrations where project identity is needed.

- **DEEPNOTE_PROJECT_OWNER_ID**: The unique identifier of the project owner. Useful for authentication workflows.

- **DEEPNOTE_RUNTIME_UUID**: A unique identifier for the current runtime session.

- **DEEPNOTE_CPU_COUNT**: The number of CPUs allocated to the project's hardware.

You can access these variables in your notebooks or scripts using:

```python
import os

project_id = os.environ.get("DEEPNOTE_PROJECT_ID")
owner_id = os.environ.get("DEEPNOTE_PROJECT_OWNER_ID")
```

Environment variables


Ensuring seamless data integration across various platforms is pivotal for any organization. Enter data transformation processes. What if you could charge these processes with the power of Deepnote? Deepnote is an AI-powered data workspace that revolutionizes the way you handle data pipelines, making it an essential tool for data scientists, data engineers, analysts, and data enthusiasts.

## Why Deepnote for ETL pipelines

Deepnote provides several options for creating and managing ETL pipelines:

- **Automation and scheduling**: automate your ETL pipelines to keep your data current.
- **Version control**: track changes, revert to previous versions and maintain a clear development history.
- **Seamless integrations**: connect effortlessly with various data sources like Snowflake, Amazon S3, Google BigQuery, and more.
- **Collaborative environment**: enable real-time teamwork for easier development and debugging.
- **Interactive notebooks**: utilize powerful data manipulation and visualization libraries within the same environment.

One of Deepnote's standout features is its ability to connect effortlessly to a variety of data lakes and data warehouses such as Snowflake, BigQuery, PostgreSQL, and many more. Deepnote supports over 80 data sources, ensuring that no matter where your data resides, you can access and manage it with ease. Deepnote leverages GPT-4 and [CoduimAI](https://codeium.com/) to provide AI-powered suggestions (not just for Python but SQL queries as well), code generation, and real-time data analysis. This facilitates quicker and more accurate data processing, ensuring you get the insights you need when you need them. Work together with your team in real time, share insights instantly, and keep everyone on the same page with robust version control and commenting features. Deepnote's collaborative tools make it easier to develop, debug, and refine your data pipelines collectively.

## Understanding data pipelines and their importance

Data pipelines are essential for moving data from various sources to a destination for analysis. Deepnote's interactive notebooks are ideal for managing these pipelines, allowing for exploration, transformation, and visualization of data within a single environment.

ETL (Extract, Transform, Load) is a process where data is extracted from source systems, transformed into a suitable format, and loaded into a target system. This approach is beneficial when data needs to be cleaned, enriched, or restructured before storage.

**Extract data**: Pull data from source systems.

**Transform data**: Modify the data as needed.

**Load data**: Store the transformed data in a target system.

### Diving deeper into ETL

In real-world scenarios, data often resides in warehouses. Here's how you can set up an ETL pipeline in Deepnote using Snowflake and Amazon S3.

**Connect to Snowflake**: navigate to your project's settings in Deepnote. Go to the "Integrations" tab and add a new integration for Snowflake with your credentials.

**Extract data from Snowflake**: create a new notebook in Deepnote and use the Snowflake integration to extract data.

**Transform data**: for example, increase the salary by 10%.

**Load data to Amazon S3**: save the transformed data to an S3 bucket.

### ELT: Extract, Load, Transform

ELT (Extract, Load, Transform) is a variation where data is extracted and loaded into the target system first and then transformed using the computational power of the target system. This method is advantageous for handling large data volumes and leveraging the scalability of modern data warehouses. Here's how to set up an ELT pipeline in Deepnote using Snowflake.

**Extract and load data to Snowflake**: use the Snowflake integration to extract and load data.

**Transform data in Snowflake**: use SQL capabilities to transform the data.

### Transformations using dbt

The data community increasingly relies on dbt (data build tool) for the transformation layer in ETL pipelines. dbt allows for programmatic data transformations, testing, and documentation.

**Programmatic transformations**: write SQL transformations in dbt models.

**Testing and documentation**: use dbt's features to test data quality and document transformations.

For more information, check out the [dbt documentation](https://docs.getdbt.com/).

## Example of Deepnote project for ETL pipeline

Deepnote provides an intuitive interface for creating and managing notebooks. Here's an [example notebook](https://deepnote.com/workspace/Deepnote-Templates-71742312-24f2-4c10-9bf7-786d17280b92/project/ETLELT-pipeline-d0cd4ccf-eede-4a0d-a59d-af64946f9c06/notebook/%F0%9F%91%BE%20Example%201-e77960b4460d401dacf2435a173da647) that encapsulates your ETL process.

Invite team members to collaborate in real time. Use comments to discuss specific parts of the pipeline. Share the notebook via a link, create a data app for visualizations and dashboarding, or export it as a PDF.

Deepnote provides a comprehensive and user-friendly platform for developing and managing ETL/ELT pipelines. By following this guide, you can import data, craft ETL/ELT procedures, utilize version control, automate execution, and monitor performance effectively.

**Which approach to choose?**

**ETL:** ideal for complex data transformations that require preprocessing before loading.

**ELT:** best for large datasets, utilizing the computational power of the target system for transformations.

Ready to enhance your data workflows? Start leveraging Deepnote today and reach out to our support team for assistance. For more tutorials, visit our guides page. Happy data wrangling!


ETL/ELT pipelines

<Embed url="https://www.loom.com/embed/e8f122e1ba4f47a19c7dfdb75f936edd?sid=922f1ba5-097d-4c5f-b563-0bf728136471"/>

Deepnote offers two modes of execution:

- **Block**: This runs only the selected block (or multiple blocks in the case of multi-selection).

- **Block and dependent blocks**. This runs the selected block and other blocks below it that depend on that block.

The default execution mode for a block in the notebook can be changed next to the "Run notebook" button.

<ImageBorder variant="blue">![Screenshot 2024-04-23 at 10.15.40.png](https://media.graphassets.com/VHvdkG1DT862UHkNdg9t)</ImageBorder>

The current execution mode is displayed within the "Run block" icon in the block actions. You can always override the default execution mode by selecting "Run this block only" or "Run dependent blocks" in the block actions.

<ImageBorder variant="colorful">![Screenshot 2024-04-19 at 12.52.41.png](https://media.graphassets.com/vRTSDtYbSoWZiZemv3R9)</ImageBorder>

## How are block dependencies detected?

In some cases, this process is straightforward; for example, a visualization block depending on a DataFrame, or a text input block defining Python variables used by other blocks.

However, with code blocks containing arbitrary user code, the problem becomes more complicated. These blocks can define new variables, reassign, or mutate them, potentially affecting other blocks.

### Directed Acyclic Graph (DAG)

Every code block, including code segments within SQL blocks using Jinja templates, is parsed into an Abstract Syntax Tree (AST). We traverse this AST to compile lists of variables used and defined by each block. By compiling these lists for every block in the notebook, along with their position in the notebook, we construct a Directed Acyclic Graph (DAG) of blocks.

When a block is executed, the DAG is traversed from the requested block downward. All blocks encountered during this traversal are executed.

You can view the DAG for a notebook anytime by clicking on the "Open dependency graph" button.

Execution modes


# Exploring JAX

JAX is a linear algebra library that's quick for numerical computations on high-end machines like GPUs and TPUs. Python already has a great XLA (accelerated linear algebra) library named NumPy. The code snippets below show examples for both NumPy and JAX. In the article below, the core features of JAX are explained in ![jax.jpg](https://media.graphassets.com/75vRxf8FRhmHJJz2Mohl)
For more information on setup and installation, here is the link to the official JAX website.
![jax numpy.png](https://media.graphassets.com/gefeaPCfRpKmeIVWVZJj)

## XLA, automatic differentiation, and just-in-time compilation

The title JAX came from:

- **XLA** (Accelerated linear algebra): JAX uses XLA, Google’s optimized compiler for linear algebra, to speed up calculations and make it run efficiently on different hardware.
- **A** (Autograd): JAX was originally built by the Autograd library and can automatically calculate derivatives. This makes it perfect for tasks like optimization, neural networks, and machine learning, as it helps compute how functions change, enabling gradient-based optimization.
- **JIT**(Just-in-time compilation): In JAX, functions are converted into basic operations (JAXPR) and evaluated like a small functional programming language. Here's an example of how JAX transforms a simple function:
  ![jax code1.png](https://media.graphassets.com/jTl922N6QfBrPTwazCT2)

## Get started:

The JAX can be installed based on the target hardware:

```
pip install "jax[cpu]"
pip install "jax[cuda12]"  # For Nvidia GPUs
pip install "jax[tpu]"  # For TPUs
```

Each installation customize JAX to use device's strengths. The cuda12 option is for Nvidia GPU users, while tpu is for Tensor processing units.

## Example: immutable arrays in JAX

Unlike NumPy, JAX uses immutable arrays. For instance, when the array element is updated, to set values the .at[] is used to set values in a new array, preserving the original:
![jax2.png](https://media.graphassets.com/Gjsuxr9aS4mWR3mKKnMU)

## Automatic differentiation in JAX

Automatic differentiation (AD) is useful in real-world applications because it quickly calculates gradients, which are key for optimization and machine learning. In machine learning, JAX automates this process, speeding up model training and eliminating the need for manually computing derivatives, making it ideal for complex tasks like neural networks and scientific simulations.

For instance, a Python function could compute the height of a physical system at a specific time, whereas JAX’s grad function facilitates automatic differentiation to determine the rate of change.
![jax3.png](https://media.graphassets.com/Vbq8xd8wTxyRO9OrjAcu)
The `grad` function can also handle arrays, producing a gradient array with partial derivatives. Because grad itself returns a function, it can be applied it multiple times to get higher-order derivatives, making it essential in complex machine learning and optimization.

## Real-world example: modeling population growth

JAX’s automatic differentiation excels at modeling dynamic systems, especially those where tracking the change of parameters over time is essential. For instance, considering modeling the growth of a city’s population over time. A function might be created to estimate the population at any given year, but it can also be essential to know the growth rate at specific points to make predictions or policy decisions.

Using JAX’s `grad`, one can differentiate this function to understand how fast the population is growing at any given time. If the function is adapted to accept a range of parameters (such as birth rate, death rate, migration rate), grad will return an array of partial derivatives, each representing the effect of one variable on the growth rate.
![jax4.png](https://media.graphassets.com/UNHMEPMNRrCfjbX2uDAb)
Gradients are essential in machine learning, where they guide adjustments to model parameters, helping decrease error rates and improve performance. With libraries like `Flax` and `nnx` built on JAX, it’s possible to create neural networks that automatically use this gradient-based optimization for faster, more efficient training.

## Next steps for learning more

- Explore the official [JAX](https://jax.readthedocs.io/en/latest/index.html#) websites.

- Here is the [data app](https://deepnote.com/app/deepnote/JAX-93f79619-9557-4a82-b58e-bfe548891f0b), where the real-life use of JAX can be explored.

- Here is a comparison of [JAX vs. PyTorch.](https://deepnote.com/docs/jax-vs-pytorch)


Exploring JAX


To export a notebook as a `.ipynb` file, right-click on the notebook in the file tree (in the right-hand sidebar menu) and click 'Download'.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2F0keCWiqlWFSDNOshzpj5%2FCleanShot%202022-03-30%20at%2014.53.21.gif](https://media.graphassets.com/azGGNHSpTYCtGis0UO6O)


Export a notebook


To export your notebook, follow these steps:

1. Open the notebook, and [create an app](/docs/creating-apps) from it by clicking the "Create app" button in the top right corner.
2. In the app editor, choose which blocks should be included in the PDF.
3. After you are done, open your app in a new tab by clicking on the "View app" button in the right sidebar.
4. Click on the "Export to PDF" button in the top right corner. This will open your browser's native print dialog to create the PDF. Elements like the top bar or sidebars are hidden in the printed version.
5. (Optional) Delete the app.


Export a PDF


To export a project, click the ellipses (`...`) to the right of the Share & publish button and then click on Download. The downloaded file is in zip format and contains your notebooks (in `.ipynb` format) as well as your other assets.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FwbD1MjjX59V9qqjgh7kc%2Fdown.gif](https://media.graphassets.com/f5pdf2VPRqWDvwDM74aY)


Export a project


Deepnote is the AI workspace for data professionals, designed to simplify data exploration, accelerate analysis, and quickly deliver actionable insights for you and your team. Unlike outdated tools such as Jupyter, Deepnote is built with the next decade in mind. Deepnote is built to give anyone working with data superpowers. It unifies your data workflow through an integrated semantic layer, preparing your data for advanced AI applications. You can also leverage our AI data copilot to chat with your data, create charts, and write code, or turn your AI notebooks into fully-fledged data dashboards or apps. Combine data, SQL or Python code, and visualizations side-by-side on a flexible canvas - enhanced with cutting-edge AI reasoning models.

## Deepnote can help you to:

### 🤖 Analyze with AI

- [**Describe and visualize:**](https://deepnote.com/docs/ai-data-visualization) Generate visualizations and code simply by describing your goal.
- [**Auto AI:**](https://deepnote.com/docs/deepnote-ai) Write, execute, and debug code with AI.
- **Code faster:** Use our AI data copilot’s context-aware code suggestions to ship faster.

### 🔗 Unify

- [**Seamlessly connect:**](https://deepnote.com/docs/integrations) Integrate with 60+ data sources (like BigQuery, Snowflake, and PostgreSQL).
- **Ship code for production:** Combine Python and SQL effortlessly within the same notebook.
- [**Reuse modules:**](https://deepnote.com/docs/modules) Build reusable components for ETL processes, analytics, and trusted metrics—write once, reuse everywhere.
- [**Define a semantic layer:**](https://deepnote.com/docs/semantic-layer) Create a shared understanding of your data with reusable definitions and metrics, ensuring consistency across your organization.

### ⚖️ Scale

- **Supercharge:** Boost compute power if needed, instantly. We come with better compute for free than other players like Google Colab.
- [**Schedule & notify:**](https://deepnote.com/docs/scheduling) Run analytics tasks in the background or by scheduling them and to receive fresh insights immediately at the pace that works for you.
- [**Work in a team:**](https://deepnote.com/docs/real-time-collaboration) Structure your analyses clearly within projects and folders for improved team alignment.
- **Integrate via REST API:** Programmatically manage projects, notebooks, and workflows to fit seamlessly into your existing infrastructure.

### 🚀 Launch

- [**Transform:**](https://deepnote.com/docs/data-apps) Notebooks → powerful dashboards or data apps natively or with our [Streamlit integration](https://deepnote.com/docs/streamlit).
- **Explore:** Allow viewers to play around with your data and simulate outcomes with customizable inputs.
- **Share:** One-click, secure distribution of your live apps.

## Join 500,000+ data professionals

From startups to Fortune 500s, data teams trust Deepnote to move from exploration to insight faster.
**📬 Contact us anytime at [help@deepnote.com](mailto:help@deepnote.com)**


Your AI workspace for data analysis, exploration, and machine learning

Welcome to Deepnote

<Callout status="info">
Available on Team & Enterprise plans
</Callout>

You can easily export all your notebooks in open-sourced format from Deepnote to GitHub without touching the terminal.

Exporting notebooks to GitHub unlocks new possibilities and workflows, such as:

- Archiving or backing up your notebooks.
- Leveraging GitHub to trigger automated testing and deployment pipelines whenever changes are made to your notebooks.
- Tracking changes made to your notebooks over time, reviewing them, and reverting them as needed.
- Enhancing collaboration, accessibility, and reproducibility of your work.

## Adding a repository

First, click on the plus icon in the sidebar Versions section.

![git-export-9.png](https://media.graphassets.com/ZlumhnGRpicHNVPiwG86)

Deepnote connects to GitHub through a [GitHub app](https://github.com/apps/deepnote).

Click on **"Add GitHub repositories"** to install the Deepnote GitHub app on your organization's or personal GitHub account.

![git-export-10.png](https://media.graphassets.com/Obi25Q1ASdefjNFmg0fM)

In GitHub, you can configure the GitHub App to have access to all your organization's repositories or specific ones. This process only needs to be completed once per GitHub account or organization. Please note that if you use multiple workspaces, you have to complete the process for every workspace.

In case you want to add repositories from another Github organization, open the repository dropdown and click on the "Add GitHub repositories" button.

Once the GitHub App is installed, you can select the repository you want to add to Deepnote. In addition to the repository, you need to specify the path to which all the project notebooks will be exported, and the branch to which the changes will be committed & pushed. The branch you specify must exist in the repository.

![git-export-11.png](https://media.graphassets.com/Z5ZA7y1JQAaRCnYv2sij)

## Exporting notebooks

To export all project notebooks to GitHub click the **Commit & push** button.

![git-export-12.png](https://media.graphassets.com/9QIAh7UzS6uGCi65v1HZ)

Add the commit message with a description, and export it.

Once exported, all project notebooks will be in GitHub. The notebooks are saved in open-source .ipynb format. Note that the block outputs, which may contain sensitive data, are also included. If you wish to not commit the output of a specific block, you can clear the output of that specific block before committing.

The commits that Deepnote creates are based on the latest commit in the remote branch, and no files other than the notebooks are overwritten. It is possible to use the same GitHub repository for multiple Deepnote projects - and also for other tools your might be using to export your code.

GitHub exports are also integrated into Deepnote's [native versioning functionality](/docs/history). Each commit creates a new version of the project, which you can quickly preview or restore from the project version history.

![git-export-13.png](https://media.graphassets.com/BMWHZYtQtemHMdD7d6SA)

#### Troubleshooting

##### Repositories not loading

If you don't see your repositories in the list after finishing the authorization process in Github, your GitHub organization may already be connected to another workspace in Deepnote. To fix this:

1. Trigger the authorization process again from the new workspace
2. Click on "Configure" button for your organization in GitHub
3. Add the relevant repository in "Repository access" section
4. Click on **Save**. This will connect your GitHub organization to the new workspace.

##### Unable to commit & push

To export the notebook to GitHub, the repository must contain the specified branch, and it can't be empty (it has at least one initial commit).

## Revoking access

If you no longer want Deepnote to access some or all of your repositories, you can either revoke access to some repositories or uninstall the Deepnote GitHub application completely in your organization's GitHub settings at `https://github.com/organizations/<your-organization>/settings/installations`.

<Callout status="info">
Follow this [guide](/docs/github) to import packages to GitHub.
</Callout>

Deepnote lets you export your notebooks to GitHub, opening up new development workflows and improving collaboration within your team or wider community.

Export notebooks to GitHub

<Embed url="https://www.loom.com/embed/aba5f2f4101243ad8da43ad7ed65e038?sid=ebabb492-285b-49bd-a977-5c7ed0746d5d"/>

<Callout status="info">
Available on Team & Enterprise plans
</Callout>

If you want to include a private or public repository, you can add it to your Deepnote project. This allows you and all your project collaborators to sync files from GitHub without touching the terminal.

Remember that when it comes to collaboration, Deepnote provides many native solutions. For example, consider the following use cases for collaboration that don't require GitHub:

- [Sharing notebooks with access controls](/docs/workspaces)
- [Real-time](/docs/real-time-collaboration) and [asynchronous](/docs/comments) notebook collaboration
- [Native versioning of notebooks](/docs/history)

However, Deepnote's GitHub integration makes the following workflows simple:

- Using a private Python module in Deepnote notebooks
- Augmenting local library/package development
- Importing `.ipynb` files from GitHub to Deepnote's native representation

## Adding a repository

Deepnote connects to GitHub through a [GitHub app](https://github.com/apps/deepnote).

First, Click **"Add GitHub repositories"** to install the Deepnote GitHub app on your organization's or personal GitHub account.

![GitHub1.png](https://media.graphassets.com/1lYJO7WvQIGtuwQzrOaD)

In GitHub, you can configure the GitHub App to have access to all your organization's repositories or specific ones. This process only needs to be completed once per GitHub account or organization. Please note that if you use multiple workspaces, you have to complete the process for every workspace.

In case you want to add repositories from another Github organization, open the repository dropdown and click on the "Add GitHub repositories" button.

Once the GitHub App is installed, you can select the repository you want to add to Deepnote. You can also specify a branch or commit that you want to use.

![GitHub2.png](https://media.graphassets.com/p1SHGzKlTuq9575czGCC)

#### Troubleshooting

If you don't see your repositories in the list after finishing the authorization process in Github, your GitHub organization may already be connected to another workspace in Deepnote. To fix this:

1. Trigger the authorization process again from the new workspace
2. Click on "Configure" button for your organization in GitHub
3. Add the relevant repository in "Repository access" section
4. Click on **Save**. This will connect your GitHub organization to the new workspace.

#### Public repositories

To use code from a public repository as an integration, fork the repository to your GitHub organization and use that repository.

Alternatively, it is also possible to clone a public repository via a [terminal](/docs/terminal).

### Git workflow

There are multiple actions you can execute on the connected GitHub repository in the UI. To see them in action, you can view the video at the beginning on this page.

- Changing the current branch. Just click on the dropdown with list of the branches and select a branch!
- Pulling the current branch from the remote
- Commit & Push to the remote (this will trigger a modal that will allow you to specify the commit message)
- Re-Clone - this will clone the repository from scratch into the same directory. Watch out, you might loose your local changes!

After executing any git command, you'll notice that output from the underlying git command that was executed.

<Callout status="info">
Keep in mind that you can execute any git command within the repository - just open a terminal, navigate to the repository and execute any `git xxx` command. Your Git commands will be augmented with access token automatically.
</Callout>

### Authentication under the hood

We use short-lived [access tokens](https://docs.github.com/en/developers/apps/authenticating-with-github-apps) to authenticate `git` commands. To be able to use this approach, we access the repository via HTTPS protocol rather than SSH. The token is valid for 1 hour and only grants access to the repository linked to the given project. We use the [Custom Git Credential Helper](https://git-scm.com/docs/gitcredentials#_custom_helpers) to pass the token to the `git` commands.

## Revoking access

If you no longer want Deepnote to access some or all of your repositories, you can either revoke access to some repositories or uninstall the Deepnote GitHub application completely in your organization's GitHub settings at `https://github.com/organizations/<your-organization>/settings/installations`.

We also verify if the person who linked the GitHub repository to a Deepnote project still has access to the GitHub repository. This means that if somebody leaves your organization or no longer has access to the repository in question, they'll no longer be able to access it through Deepnote.

<Callout status="info">
Follow this [guide](/docs/github-export) to export notebooks to GitHub.
</Callout>

Deepnote allows you to import public and private repositories from GitHub, just as you would with your local computer.

Importing code from GitHub

<Callout status="info">
**Want to get started right away?** Jump right in and explore [this hands-on example](https://deepnote.com/@deepnote/GitLab-b44caee9-23c0-49fe-a31c-15820d4ae453) of using GitLab in Deepnote.
</Callout>

Deepnote allows you to both import existing projects from GitLab (as you would in a local development setting) and sync your code and notebooks to GitLab. While there's no built-in integration for GitLab, setting up and using GitLab in Deepnote is fairly straightforward and quick to implement!

![GitLab workflow](https://media.graphassets.com/1hGbe5udR92VZq6AsMTL)

### How to set it up

There are three core steps to starting to work with GitLab in Deepnote. Let's go through them one by one.

1. **Generate an SSH key**
 - This can be done either directly in Deepnote or somewhere else and then uploaded to Deepnote. The SSH key should be stored somewhere in the `~/work` directory to ensure it does not get deleted after a hardware reboot. We will store it in `~/work/git-ssh-key`.
 - If you'd like to create the SSH key in Deepnote, you can run `!ssh-keygen -t rsa -N "" -C "Deepnote key" -f ~/work/git-ssh-key`
2. **Create a deploy key in GitLab**
 - Retrieve the contents of the public key we just created in `~/work/git-ssh-key.pub`
 - Take the output of the above cell and create a new [Deploy Key](https://docs.gitlab.com/ee/user/project/deploy_keys/) at [https://gitlab.com/YOUR_ORG/YOUR_REPO/-/settings/repository](https://gitlab.com/YOUR_ORG/YOUR_REPO/-/settings/repository). Depending on whether you only want to `pull` the code in Deepnote or also have the capability to `push` new code, check the `Grant write permissions to this key` in GitLab's UI.
3. **Create environment variables**
 - Set the `GIT_SSH_COMMAND` environment variable to `ssh -i ~/work/git-ssh-key` using [Environment Variables](/docs/environment-variables). Environment variables in Deepnote are encrypted and offer a secure option to store sensitive data.

### How to use

Now that you've set up everything required to use GitLab in Deepnote, getting started is as easy as cloning a repository from GitLab and performing the workflows you're used to on your local machine!

### Next steps

Jump right in and explore [this hands-on example](https://deepnote.com/@deepnote/GitLab-b44caee9-23c0-49fe-a31c-15820d4ae453) of using GitLab in Deepnote. You can also save yourself some setup work by hitting the `View source` button first before clicking on `Duplicate` in the top-right corner to start exploring on your own!

GitLab


Deepnote can interact with your Google AlloyDB instances. Follow these docs to create your own notebook that connects to your Google AlloyDB instance.

Prerequisites are that that you have [setup an Google AlloyDB instance](https://cloud.google.com/alloydb/docs/instance-primary-create), and you have [created a Compute VM instance](https://cloud.google.com/compute/docs/instances/create-start-instance) that you can access your Google AlloyDB instance from.

### 1. Getting set up

You will need a notebook and terminal open. We will use the terminal to create an SSH tunnel between Deepnote project and the VM instance.

### 2. Set up SSH keys

**Create your SSH key**

In your Deepnote Terminal, create an SSH key. Your user name must match the username in the Compute VM instance.

```
mkdir ./.ssh
ssh-keygen -t rsa -f ./.ssh/[key_name_here] -C [user_name_here]
cd .ssh
chmod 400 [key_name_here]
cat [key_name_here].pub
```

**Add your SSH key to your VM Compute instance**

Copy the resulting key, and paste it into Google Cloud -> VM Instances -> [INSTANCE_NAME] -> SETTINGS -> METADATA -> SSH KEYS

### 3. Connect to your Compute VM Instance and start the Auth Proxy

Connect to your Compute VM Instance

```
ssh -i ./[key_name_here] [user_name_here]@[vm_instance_external_ip_here]
```

On your compute VM, you now need to run your [AlloyDB Auth Proxy](https://cloud.google.com/alloydb/docs/auth-proxy/connect), that will allow you to connect to your AlloyDB instance. Follow the instructions listed. After installing and Authenticating the Auth Proxy, you can run it using:

```
./alloydb-auth-proxy projects/[project_id_here]/locations/[region_id_here]/clusters/[cluster_id_here]/instances/[instance_id_here]
```

### 4. Create your SSH tunnel

In another terminal session, create the SSH tunnel using the keys we created and whitelisted before.

```
ssh -L 127.0.0.1:9999:127.0.0.1:5432 -i ./.ssh/[key_name_here] [user_name_here]@[vm_instance_external_ip_here]
```

### 5. Connect to the Google AlloyDB instance from your Notebook

Now that your SSH tunnel is running, you may simply add our built in AlloyDB integration. FIll out the host and port of your SSH tunnel, and the database information of your Google AlloyDB instance. Viola, you may now query with Deepnote's SQL blocks.

![Screenshot 2023-06-14 at 15.33.50.png](https://media.graphassets.com/WjsF3LoREKCCvYVGhTrh)

### What's next?

This is a new integration, and we are still working out some kinks.
We are working on reducing the SSH connection steps, and implementing our schema browser for Google AlloyDB.

But for now query away!

Now that you're querying data, you can share it with your team. You can even turn your charts [into a shareable dashboard](/docs/publish-projects).


Google AlloyDB

## What can you do with the Google BigQuery integration?

As with all of Deepnote's SQL-related integrations, the Google BigQuery integration allows you to query databases and explore the structure of your warehouse. Think "SQL editor" but with all the modern productivity boosters that come with Deepnote's notebook. For example,

- Write native SQL and Python in the same notebook
- Search your entire warehouse instantly via the integrated schema explorer
- Get intelligent autocomplete for columns, tables, and databases
- Interactively explore data without writing any additional code
- Run queries powered by BigQuery DataFrames

<Callout status="info">
Available on Team and Enterprise plans
</Callout>

### How to connect to Google BigQuery

From the right-hand panel, under **Integrations**, click the **+** button and choose **Create new integration**.
![create_integration.png](https://media.graphassets.com/pIr2x2wQtyVUUAv6spVZ)
 

Select Google BigQuery from the list of integrations or search for it using the search bar.
![google_select_int.png](https://media.graphassets.com/L0AQ9K9RSn2NCuUbtwPR)
 

Fill out the fields in the pop up form. Importantly, you will need to select the desired authentication method and supply the related credentials (described below).
![gbq_modal.png](https://media.graphassets.com/rKvTSWW8TvaPqaCJdsVb)

#### Authenticating with a service account

A service account will provide a shared connection to Google BigQuery. That is, all collaborators with least **Editor** privileges will be able run queries against databases provisioned in the service account.

To use the service account authentication, you will need to supply a JSON service account key. [Click here for a guide on creating a JSON service account key](https://cloud.google.com/docs/authentication/getting-started). Your service account key will be encrypted and stored in Deepnote's database.

<Callout status="warning">

**Make sure to enable BigQuery API for your GCP project.** The service account needs sufficient permissions for the resources you want to use. If the authorisation process fails, we recommend visiting BigQuery's [access control page](https://cloud.google.com/bigquery/docs/access-control).

**Grant sufficient permissions to your service account.** The minimum required permissions are: _BigQuery Job User_, _BigQuery Read Session User_, and _BigQuery Data Viewer_. The best practice is to only grant this last role for specific datasets or tables you wish to explore in Deepnote.

</Callout>

#### Authenticating to BigQuery with Google OAuth

<Callout status="info">
Available on the Enterprise plan
</Callout>

With BigQuery's Google OAuth authentication you can give every member of your Deepnote workspace their own set of credentials. This ensures greater security by using short-lived tokens and enabling the use of multi-factor authentication. Follow the principle of least privilege and use granular access control for various BigQuery resources to ensure users can only access the data they need. [Click here to learn how to set up BigQuery's Google OAuth authentication in Deepnote](bigquery-oauth).

### Working with data from Google BigQuery

Now that you are connected to your Google BigQuery can do the following actions in Deepnote:

- Click the newly created integration in the **Integrations** section to open the schema browser. [Click here to learn more about the schema browser](schema-browser).

- Create an SQL block that points to your warehouse and begin querying your data. Autocomplete for columns, tables, and databases will be at your fingertips as you type. [Click here to learn more about SQL blocks](sql-cells).

- Explore column distributions as well as the sorting and filtering capabilities on the results set. All results are displayed as an interactive Pandas DataFrame. [Click here to learn about interactive DataFrame output](variable-explorer#interactive-dataframe-output).

- Pipe the results of your query into a chart block for rapid interactive data visualization. [Click here to learn more about chart blocks](chart-blocks).

### Using pure Python to connect to BigQuery

To go beyond querying (like listing tables, creating datasets, etc.), you may need to use the official Python client library ([docs](https://googleapis.dev/python/bigquery/latest/index.html)).

Use this code snippet to authenticate the python client using the integration's service account:

```python
import json
import os
from google.oauth2 import service_account
from google.cloud import bigquery

bq_credentials = service_account.Credentials.from_service_account_info(
 json.loads(os.environ['INTEGRATION_NAME_SERVICE_ACCOUNT']))
client = bigquery.Client(credentials=bq_credentials,
 project=bq_credentials.project_id)
```

Just replace the `INTEGRATION_NAME` with an uppercased, underscore-connected name of your integration. If you have trouble finding it, run this one-liner to list environment variables that contain service accounts:

```python
[var for var in os.environ if '_SERVICE_ACCOUNT' in var]
```

Once the BigQuery `client` is initialized, you can use it to run queries and materialize the results as dataframes like this:

```python
sql = """
SELECT *
FROM `bigquery-public-data.usa_names.usa_1910_current`
LIMIT 1000
"""

df = client.query(sql).to_dataframe()
```

### Working with Multiple Projects

Deepnote's BigQuery integration supports querying data over multiple projects, offering a flexible and powerful way to handle diverse datasets across your Google Cloud Platform (GCP) environment. This capability is particularly useful for users managing or analyzing data across various GCP projects.

To leverage this feature, you must set up the necessary access permissions within GCP. Ensure your service account or Google OAuth credentials are configured to access all the projects you intend to work with. For detailed guidance on managing access and permissions in GCP, you can refer to the [GCP documentation on access control](https://cloud.google.com/bigquery/docs/access-control).

Once you have the required access, querying data from different projects is straightforward. In your SQL queries, use the `projectId.dataset.table` format to specify exactly where your data resides. This format allows you to seamlessly switch between datasets hosted in different projects without changing your connection settings.

![big-query-1.png](https://media.graphassets.com/TSkzUZdIQfeK9229xSTj)

Deepnote's schema explorer enhances your ability to browse through various projects and tables. With this tool, you can visually inspect the structure of your datasets, including the columns and their data types, across different GCP projects. This feature simplifies the process of understanding and navigating through your data, especially when dealing with multiple projects.

![big-query-2.png](https://media.graphassets.com/XcgsadbwQUCt0q7cYUWt)

By utilizing these features, Deepnote users can effectively manage and analyze data across multiple GCP projects, making it a robust solution for complex data environments.

Google BigQuery is a fully managed, serverless data warehouse with built-in machine learning capabilities.

Google BigQuery


## What can you do with Google Cloud SQL

Google Cloud SQL is a fully managed, relational database service offered by Google Cloud Platform. It enables users to deploy, maintain, and scale relational databases like MySQL, PostgreSQL, and SQL Server without the need to manage underlying infrastructure. With built-in security, high availability, and automated backups, it simplifies database management and allows developers to focus on building applications.

And you are now available to connect in Deepnote!

### How to connect to Google Cloud SQL

#### Google Cloud Platform

1. Enable `Cloud SQL`, and `Cloud SQL Admin API`.
2. Create an instance on your project, Deepnote supports all available options (postgres, mysql, sql-server) in Cloud SQL.
3. Create a database and a user and password with access to that DB
4. Create a Service account with access to your Project. [Click here for a guide on creating a JSON service account key](https://cloud.google.com/docs/authentication/getting-started).

#### Authenticating with a service account

A service account will provide a shared connection to Google Cloud SQL. That is, all collaborators will be able run queries against databases provisioned in the service account. Your service account key will be encrypted and stored in Deepnote's database.

**Make sure to enable Cloud SQL and Cloud SQL API Admin for your GCP project.** The service account needs sufficient permissions for the resources you want to use. If the authorisation process fails, we recommend visiting the Cloud SQL [access control page](https://cloud.google.com/sql/docs/mysql/iam-permissions).

### Connecting to your Instance

Now that you have connected our integration, simply query the database with Python.

You still need to install an appropriate the appropriate [Google Cloud SQL Connector](https://github.com/GoogleCloudPlatform/cloud-sql-python-connector).

**After installing your connector, you must restart your notebook**

MySQL:

`pip install "cloud-sql-python-connector[pymysql]"`

Postgres:

`pip install "cloud-sql-python-connector[pg8000]"`

SQL Server:

`pip install "cloud-sql-python-connector[pytds]"`

After installing, especially with SQL server, we may have some dependency mismatches. The usual suspects can be solved with:

`!pip install python-tds`

`!pip install pyopenssl --upgrade`

`!pip install sqlalchemy-pytds`

Replace all connection string information with appropriate values.

Replace the `YOUR_INTEGRATION_NAME` with an uppercased, underscore-connected name of your integration.

MySQL

```python
from google.cloud.sql.connector import Connector
  import sqlalchemy
  import pymysql
  from google.oauth2 import service_account
  from google.cloud import spanner
  import json

  cloud_sql_credentials = service_account.Credentials.from_service_account_info(
      json.loads(os.environ['YOUR_INTEGRATION_NAME_SERVICE_ACCOUNT']))

  # initialize Connector object
  connector = Connector(credentials=cloud_sql_credentials)

  # function to return the database connection
  def getconn() -> pymysql.connections.Connection:
      conn: pymysql.connections.Connection = connector.connect(
          "project:region:instance",
          "pymysql",
          user="my-user",
          password="my-password",
          db="my-db-name"
      )
      return conn

  # create connection pool
  pool = sqlalchemy.create_engine(
      "mysql+pymysql://",
      creator=getconn,
  )
```

Postgres

```python
from google.cloud.sql.connector import Connector
  import sqlalchemy
  import pymysql
  from google.oauth2 import service_account
  from google.cloud import spanner
  import json

  cloud_sql_credentials = service_account.Credentials.from_service_account_info(
      json.loads(os.environ['YOUR_INTEGRATION_NAME_SERVICE_ACCOUNT']))

  # initialize Connector object
  connector = Connector(credentials=cloud_sql_credentials)

  # function to return the database connection
  def getconn() -> pymysql.connections.Connection:
      conn: pymysql.connections.Connection = connector.connect(
          "project:region:instance",
          "pg8000",
          user="my-user",
          password="my-password",
          db="my-db-name"
      )
      return conn

  # create connection pool
  pool = sqlalchemy.create_engine(
      "postgresql+pg8000://",
      creator=getconn,
  )
```

SQL Server

```python
import os
from google.cloud.sql.connector import Connector
from google.oauth2 import service_account
import pytds
import json
import sqlalchemy

cloud_sql_credentials = service_account.Credentials.from_service_account_info(
    json.loads(os.environ['YOUR_INTEGRATION_NAME_SERVICE_ACCOUNT']))


def connect_with_connector() -> sqlalchemy.engine.base.Engine:

    connector = Connector(credentials=cloud_sql_credentials)

    def getconn() -> pytds.Connection:
        conn = connector.connect(
            "project:region:instance",
            "pytds",
            user='sqlserver',
            password='password',
            db='db-name',

        )
        return conn

    pool = sqlalchemy.create_engine(
        "mssql+pytds://",
        creator=getconn,
    )
    return pool

pool = connect_with_connector()


```

### Working with data from Google Cloud SQL

Now that you are connected to your Google Cloud SQL you can do the following actions in Deepnote:

- Explore column distributions as well as the sorting and filtering capabilities on the results set. All results are displayed as an interactive Pandas DataFrame. [Click here to learn about interactive DataFrame output](variable-explorer#interactive-dataframe-output).

- Pipe the results of your query into a chart block for rapid interactive data visualization. [Click here to learn more about chart blocks](chart-blocks).


Google Cloud SQL is a fully managed, relational database service offered by Google Cloud Platform. It enables users to deploy, maintain, and scale relational databases like MySQL, PostgreSQL, and SQL Server without the need to manage underlying infrastructure. With built-in security, high availability, and automated backups, it simplifies database management and allows developers to focus on building applications.

Google Cloud SQL

<Callout status="info">
Available on Team and Enterprise plans
</Callout>

### Connecting to a public bucket

<Callout status="warning">
Make sure the [Cloud Storage API](https://cloud.google.com/endpoints/docs/openapi/enable-api) is enabled in your Google project.
</Callout>

If you are connecting a public bucket, simply fill out the bucket name in the integration modal. After that your bucket will be mounted into your filesystem.

_If you need help with making a bucket public,_ [_here is a tutorial_](https://cloud.google.com/storage/docs/access-control/making-data-public)\_\_

### Connecting to a private bucket

<Callout status="warning">
Make sure the [Cloud Storage API](https://cloud.google.com/endpoints/docs/openapi/enable-api) is enabled in your Google project.
</Callout>

1. In Google Cloud Platform, create a service account and download the key ([guide](https://cloud.google.com/docs/authentication/getting-started)) or re-use an existing one.
2. In Deepnote, create a new Google Cloud Storage integration and enter the name of the bucket. Then copy the contents of the service account into the empty field.
3. After the integration is created, you can connect it to any of your projects. The bucket will be mounted to your file system where you can easily access it.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FkAqRN3pXX0uFOQjZZiiD%2FScreen%20Shot%202022-03-30%20at%2012.55.12%20PM.png](https://media.graphassets.com/Gzv9fdXRQwqlzAkoiWxR)

Google Cloud Storage (GCS)

<Callout status="info">
Available on the Enterprise plan
</Callout>

1. First, enter your hostname (one of `gcr.io` , `us.gcr.io` , `asia.gcr.io` or `eu.gcr.io`).

2. Then, authenticate using your [JSON service account file](https://cloud.google.com/docs/authentication/getting-started)

 ![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FdA5DpFcg1u6WCvDsYSFc%2FScreen%20Shot%202022-03-30%20at%201.01.20%20PM.png](https://media.graphassets.com/RiZbC2WcRkuQDt7PgNAc)

3. After **creating** and **connecting** the integration, you can use Docker images from your GCR repository as a [custom environment](/docs/custom-environment) for your project.

**Your credentials are encrypted and securely stored in our database.** When you delete this integration, we also wipe your GCR credentials from our systems.

Google Container Registry

## How to use the Google Drive integration

There is a two-step process for setting up the Google Drive integration:

1. Grant Deepnote permission to access the files.
2. Connect the integration to your desired project.

From there you will be able to use Python to read the files into your notebook.

As with all integrations, once you set up the Google Drive integration, members of your workspace will be able to connect it to any project (if they have the appropriate permissions to do so).

Let's set it all up and look at some examples of working with files from Google Drive.

### Authorizing & connecting

First, click the **Integrations** section in the left panel and find the Google Drive entry in the list of native integrations (you can also create new integrations in the **Integrations** area in the right panel).

![choose_gd.png](https://media.graphassets.com/D43WydjvRT6fAPBdq2cW)

You will be taken through the authorization flow (via OAuth). Once Deepnote is authorized to access the files, connect the Google Drive to your desired project as shown below.

![connect_gd_to_proj.png](https://media.graphassets.com/9ukMIblxTKGLZT0Wb55R)

<Callout status="success">

The access credentials are encrypted and stored securely in our database. You can revoke access by revisiting the Google Drive integration pop-up and clicking **Revoke access**.

</Callout>

### Accessing a Shared Drive

To access a Shared Drive within your workspace, set the Team Drive ID in the Integration settings. You may
get the Drive ID from the URL of the Drive.

### Working with files in Google Drive

Once Google Drive is connected to your project, it will be mounted to the `/datasets/` folder. It is now a simple matter to use pandas (or other standard Python commands) to read the files into your notebook and begin working with them.

Here's an example using pandas to read a CSV from Google Drive into the notebook:

![gd_data.png](https://media.graphassets.com/tmKxV9zJRMOMZqbqQHz0)

<Callout status="info">

💡 You can also use the Google API to read Google Sheets directly into the notebook. Please see these examples to learn how to read a [public](https://deepnote.com/workspace/Subspace-Analytics-3e79b326-0cf6-4bff-8b41-99d9feb9b992/project/Read-from-public-Google-Sheet-25da9f7e-82e2-416c-827a-514b802bb1a1/notebook/notebook-2044ad29f11c47adae4ed1baf900262f) and [private](https://deepnote.com/workspace/edison-insights-3e79b326-0cf6-4bff-8b41-99d9feb9b992/project/Read-from-a-private-Google-Sheet-faa77a5d-72a1-46cb-9a20-7ee446de7f58/notebook/Notebook%201-4428c941a1f14b6a9e222828ff260fd5) Google sheet into Deepnote.

</Callout>

If you store spreadsheets and CSVs in Google Drive, our native integration makes it easy to access these files from within Deepnote.

Google Drive


## What can you do with the Google Spanner integration?

As with all of Deepnote's SQL-related integrations, the Google Spanner integration allows you to query databases and explore the structure of your database. Think "SQL editor" but with all the modern productivity boosters that come with Deepnote's notebook. For example,

- Write native SQL and Python in the same notebook
- Get intelligent autocomplete for columns, tables, and databases
- Interactively explore data without writing any additional code

### How to connect to Google Spanner

From the panel, under **Integrations**, click the **+** button and choose **Create new integration**.

Select Google Spanner from the list of integrations or search for it using the search bar.

Fill out the fields in the pop up form.

#### Authenticating with a service account

A service account will provide a shared connection to Google Spanner. That is, all collaborators will be able run queries against databases provisioned in the service account.

To use the service account authentication, you will need to supply a JSON service account key. [Click here for a guide on creating a JSON service account key](https://cloud.google.com/docs/authentication/getting-started). Your service account key will be encrypted and stored in Deepnote's database.

**Make sure to enable Spanner API for your GCP project.** The service account needs sufficient permissions for the resources you want to use. If the authorisation process fails, we recommend visiting the Spanner [access control page](https://cloud.google.com/spanner/docs/iam).

**Grant sufficient permissions to your service account.**
For Data Boost please enable `spanner.databases.useDataBoost`

In order to use our built in MYSQL blocks, you must run the following code in your notebook (also available from How to modal):

```
import os

!printf "%s" "$YOUR_INTEGRATION_NAME_HERE_SERVICE_ACCOUNT" > /tmp/google

os.environ['GOOGLE_APPLICATION_CREDENTIALS']="/tmp/google"
```

Make sure to replace YOUR_INTEGRATION_NAME_HERE with your a snakecased uppercased version of your integration name. Or simply add the code it from the How to Modal.

### Working with data from Google Spanner

Now that you are connected to your Google Spanner you can do the following actions in Deepnote:

- Create an SQL block that points to your database and begin querying your data. Autocomplete for columns, tables, and databases will be at your fingertips as you type. [Click here to learn more about SQL blocks](sql-cells).

- Explore column distributions as well as the sorting and filtering capabilities on the results set. All results are displayed as an interactive Pandas DataFrame. [Click here to learn about interactive DataFrame output](variable-explorer#interactive-dataframe-output).

- Pipe the results of your query into a chart block for rapid interactive data visualization. [Click here to learn more about chart blocks](chart-blocks).

![Screenshot 2023-06-29 at 15.53.55.png](https://media.graphassets.com/BpfgyDXQwyPxdtXMz7Rw)

### Using pure Python to connect to Google Spanner

To go beyond querying (like listing tables, creating datasets, etc.), you may need to use the official Python client library ([docs](https://cloud.google.com/python/docs/reference/spanner/latest)).

Use this code snippet to authenticate the Python client using the integration's service account:

```python
import os
from google.oauth2 import service_account
from google.cloud import spanner

spanner_credentials = service_account.Credentials.from_service_account_info(
    json.loads(os.environ['INTEGRATION_NAME_SERVICE_ACCOUNT']))
spanner_client = spanner.Client(credentials=spanner_credentials,
    project=spanner_credentials.project_id)

instance = spanner_client.instance('YOUR_INSTANCE_HERE')
database = instance.database('YOUR_DB_HERE')
```

Just replace the `INTEGRATION_NAME` with an uppercased, underscore-connected name of your integration. If you have trouble finding it, run this one-liner to list environment variables that contain service accounts:

```python
[var for var in os.environ if '_SERVICE_ACCOUNT' in var]
```

Once the Google Spanner `client` is initialized, you can use it to run queries and view the results as dataframes like this:

```
with database.snapshot() as snapshot:
results = snapshot.execute_sql("SELECT * from YOUR_TABLE_HERE")

    for row in results:
        print(row)"""
```

### Data Boost in Google Spanner

You can make use of Google Spanner's [Data Boost](https://cloud.google.com/spanner/docs/databoost/databoost-overview) in Deepnote. Ensure that the service account that runs the application has the `spanner.databases.useDataBoost` Identity and Access Management (IAM) permission. For more information, see Access control with IAM.

Then it is as simple as passing `data_boost_enabled=True` to valid queries to use Data Boost.


Cloud Spanner is a highly scalable database that combines unlimited scalability with relational semantics, such as secondary indexes, strong consistency, schemas, and SQL providing 99.999% availability in one easy solution. Hence, it’s suitable for both relational and non-relational workloads.

Google Spanner

<Callout status="info">

**Want to get started right away?** Jump right into Deepnote and view a sample workflow using Great Expectations [here](https://deepnote.com/@Deepnote-lessgreater-GE/Great-Expectations-in-Deepnote-5119d502-592c-43bd-b99d-244e09f7080a).

</Callout>

Great Expectations (GE) is a tool for data testing and documentation. Onboarding to GE, however, usually comes with a few challenges for newcomers such as switching between multiple notebooks, using the terminal, and hosting documentation. This guide will fast-forward you through any pain points and enable you to bring the software development discipline of automated testing to your data science team.

### How to set it up

All default Python environments in Deepnote come with preinstalled Pandas (learn more about all preinstalled packages [here](/docs/pre-installed-packages)). GE can be installed through a simple `!pip install great_expectations` . All that is needed to get started with GE, then, is to initialize GE via `!great_expectations --yes --v3-api init`. Note that both of these statements could also be run in a terminal within your Deepnote project (without the `!`, of course).

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FPus8irVJgvyrswZjsMRK%2FPUZZLE.png](https://media.graphassets.com/bYtaxTjqTo6i0LAR2LRk)

### How to use

Once initialized, you can start using Great Expectations within your notebooks. In the example below, three [Expectations](https://greatexpectations.io/expectations/) (tests) are defined on the fictitious `df_pass` Pandas DataFrame. In simple terms,

1. the `skill` cannot contain null values
2. the `runner` column must contain unique values
3. the `total_time` column must have values between 70 and 100

```python
# import pandas and great_expectations
import pandas as pd
import great_expectations as ge

# initialize a Pandas DataFrame
df_pass = ge.from_pandas(df_pass)

# define Expectations
df_pass.expect_column_values_to_not_be_null('skill')
df_pass.expect_column_values_to_be_unique('runner')
df_pass.expect_column_values_to_be_between('total_time', 70, 100)
```

### Next steps

Jump right into Deepnote & take a look at [this thorough walkthrough of Great Expectations in Deepnote](https://deepnote.com/@Deepnote-lessgreater-GE/Great-Expectations-in-Deepnote-5119d502-592c-43bd-b99d-244e09f7080a). You can also save yourself some setup work by hitting the `View source` button first before clicking on `Duplicate` in the top-right corner to start exploring on your own!

Great Expectations

Deepnote is a fantastic platform for group work because it combines the power of a collaborative code editor with the convenience of a cloud-based workspace. This means multiple students can work on the same project simultaneously without worrying about version control or merging conflicts.

Deepnote's intuitive interface makes it easy for everyone to get up to speed quickly, regardless of their level of experience with programming. It also offers a wide range of features to help streamline the collaborative process, such as real-time code execution and sharing, customizable workspaces, and seamless integration with popular tools and services.

## Setup

Each group project can be shared with multiple students working on the assignment together. The easiest way to share the assignment with multiple students is to go to the project, click on the share button, select the desired access level, and distribute the link among the students in the group. 

![EDU_group_work.gif](https://media.graphassets.com/a188WyxTzaT225UERnbX)

Group work

It’s safe to say that most hackathons are organized and attended online. Deepnote is a great tool to help you encourage better collaboration among participants and mentors!

Deepnote provides a wide range of powerful features that make it easy for participants to collaborate, analyze data, and build and deploy machine learning models. Its cloud-based workspace allows team members to easily access and edit their code and data, even if they're not physically in the same location.

<Callout status="info">
**Deepnote's free Education plan** is available to all students, teachers, and researchers with access to their school email addresses. But we want to help all seekers of data science knowledge and individuals looking to do social good.

If you're planning to use Deepnote for your hackathon and don't have access to a school email, [get in touch with us!](mailto:help@deepnote.com)
</Callout>

## Benefits

Some of the key benefits of Deepnote for hackathons include:

#### Collaboration

Deepnote's collaborative workspace allows multiple participants to work on the same project simultaneously, making it easy to share ideas and work together.

#### Easy setup

With Deepnote, you don't need to spend valuable time setting up your environment for the participants. You can simply sign up, create your project, invite the rest of your teammates, and get to work.

#### Data analysis

Deepnote offers a wide range of tools and integrations that make it easy to import and explore data, whether it's stored in a database, spreadsheet, or other format.

#### Machine learning

Deepnote provides a wide range of machine learning libraries and tools that make it easy for participants to build and deploy models. This can be especially useful for hackathons focusing on data science or artificial intelligence.

#### Interactive reports and dashboards

Deepnote allows participants to create interactive reports and dashboards that can help them visualize and communicate their findings to judges and other stakeholders.

Hackathons


Deepnote lets you preview and restore older versions of notebooks in the _Project version history_. You can also view everything that has happened in a project in the _Versions_ sidebar.

You will need edit- or admin-level access to the project in order to view its history.

### Project version history

You can find the Project version history in the three-dot menu in the top-right corner of a project. You can preview older version of all notebooks inside of a project.

![project version history](https://media.graphassets.com/EYHGtgSOarcr4CilcDQq)

#### Version creation

New version are created in two ways:

- **Automatic**. When you're working on a project, a new version is automatically created every 10 minutes. Another version will be created when you stop—after 2 minutes of inactivity.
- **Manual**. You can open the Versions sidebar from the top left part, and press a plus button in the sidebar to create a named version.

#### Restoring a version

You can navigate to an older version of the project and restore the notebooks to their previous state. Restoring is a safe operation, and performs several steps:

- Create a snapshot of the current state. This makes it possible to return back without losing any code.
- Overwrite all notebooks with their old versions. Notebooks created since then will be removed, and removed notebooks will be recovered.

Since notebooks sometimes depend on each other, restoring a version always restores all notebooks at the same time.

If you'd like to keep multiple versions of a project, duplicate it at various points in time.

### Versions

Open the Versions sidebar by clicking on the Versions icon in the top menu on the right side.

You can see actions done by all collaborators and significant system events.

If an action log is associated with a specific notebook, you can click on the action and you will be focused on the notebook where the change was made.

### Filesystem snapshots (deprecated)

Deepnote no longer versions the rest of the files in your project. If you were relying on this feature and need to restore some project files, please reach out to Deepnote's support.


Version history

## How to import`.ipynb` files

There are two ways to import `.ipynb` files into a Deepnote project:

1. Drag and drop `.ipynb` files into the **NOTEBOOKS** section in the left panel.

![Drag_And_Drop notebooks.jpg](https://media.graphassets.com/Jlgt96IgRLejNtaNvkjA)
 

2.1 Click the **+** button in the **FILES** section and select the **Upload file** option.

![Upload Notebooks.jpg](https://media.graphassets.com/D2qW41uThSaTGWASHi8L)

2.2 After uploading the .ipynb file, open it and select Move to Notebooks. Once it's in the notebooks section, you can open it and begin your work.

![Move Notebooks.jpg](https://media.graphassets.com/9iG6ikWrTNqFNw0ZMOzV)
 

## How to export `.ipynb` files

To export a notebook as a `.ipynb` file, click the three dots next to the notebook in the right panel and select **Export as .ipynb**.

![Export_Notebooks.jpg](https://media.graphassets.com/hydWm3T3RNOROOzvq969)

<Callout status="info">

While we do our best to maintain compatibility with Jupyter, specialized blocks in Deepnote cannot be represented exactly in Jupyter (e.g., SQL, chart, input, and rich text blocks). Upon export, queries in SQL blocks will be converted to a string representation and rich text blocks will be converted to Markdown.

</Callout>

Deepnote is compatible with core Jupyter notebook functionalities.

Working with Jupyter notebooks

## How to upload data files to Deepnote

You can upload files into the integrated file system by dragging them from your computer into the **FILES** section in the right sidebar. You can also click the **+** button to upload files from your computer or from a URL.

![upload files.png](https://media.graphassets.com/ZG2hzoBoTbyNZImXNz8Q)

 

<Callout status="info">

Note that Deepnote is compatible with core Jupyter functionality. Read our section on [importing and exporting Jupyter notebooks](https://deepnote.com/docs/importing-and-exporting-jupyter-notebooks) to learn more.

</Callout>

Pro tip: If you drag a CSV file into the notebook (either from your computer or from the file system), an [SQL block](/docs/sql-cells) with a prepopulated query will be created for you.

![sql_csv.png](https://media.graphassets.com/OPw0PxhLRXe9FOGO72nq)

<Callout status="info">
Note that the limit for uploading a file via a web interface is 100MB.
</Callout>

## Remote storage

If your files live in a remote service or bucket, you can simply use one of the native [Deepnote integrations](https://deepnote.com/docs/amazon-s3) to access the files. We have integrations with Amazon S3, Google Cloud Storage, Google Drive, Google Sheets, Microsoft OneDrive, Box, Dropbox, and Azure Blob Storage.

## High performance file system

The default working directory (`/work`) is backed by object storage and it is not expected to provide high enough throughput required for working with large amount of smaller files.

If you require high throughput access to your data (i.e. training of ML models using a GPU or unzipping a file) you can switch to a fast ephemeral storage available in `/tmp`. It is recommended to first copy the necessary files from the integrated storage (`/work`) to the ephemeral storage of the project (`/tmp/subdirectory`) before executing the notebook.

Tip: You can do this on [project initialization](https://deepnote.com/docs/project-initialization) by adding a copy statement in the init notebook of your project:

```
!cp -fr /work/<data> /tmp
```

Please note that any files stored in the fast ephemeral storage will be lost when the machine restarts. Don't forget to copy them back to the integrated storage if you want to persist them between restarts.

Deepnote has an integrated file system and native integrations to help you work with data files.

Integrated file system

Many tools, such as [Tensorboard](https://www.tensorflow.org/tensorboard) and [Streamlit](https://streamlit.io/), require a local webserver to be spun up to serve docs, dashboards, and other web interfaces. They often prompt you to open an address like `http://localhost:8080)`.

When you enable incoming connections, Deepnote exposes port `8080` on the internet under an address such as `b9d13315-f12a-4931-857c-ed6b4c59dcad.deepnoteproject.com`. You can use this address to access services running on port `8080` on your Deepnote machine.

<Callout status="warning">
Enabling incoming connections means that anyone on the internet will be able to access port 8080 on your machine.
</Callout>

### How to allow Incoming connections

You can allow incoming connections from the **Project** tab in the right sidebar. From there, click on the arrow next to Environment to be taken to the and then toggle "Allow incoming connections" at the bottom.

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FHR5AyvqdfZ5LRpaHithm%2Finc.gif](https://media.graphassets.com/kq80vCWRYu217reZ4pnu)

If you want to try it out, paste `!python -m http.server 8080` into a block and run it. If incoming connections have been enabled as described above, you should be able to paste the provided link into a browser to connect to the running webserver.

### What if I need to expose a different port?

Right now we only support exposing port 8080. To expose other ports, you can either reconfigure your tool, or use utilities like `socat` which can forward traffic from port `8080` to the port of your choosing. You can use the example below to set up port forwarding with `socat` in the terminal:

```
apt update && apt install socat
socat tcp-l:8080,fork,reuseaddr tcp:127.0.0.1:YOUR_PORT
```

### Running Flask

You can also use Deepnote to prototype a simple Flask server. When using its development server, don't forget to set the `host` to reach it from outside the local machine.

Example:

```python
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
 return "Hello, World!"

app.run(host='0.0.0.0', port=8080)
```

Incoming connections

See how **Deepnote helps over 100 000 data analysts and scientists** worldwide from startups to Fortune 500 companies in this 5 minutes demo.

<iframe width="720" height="405" src="https://www.youtube.com/embed/mLPUNo4O7aA?si=DwlP2Cts1pPVYvfW" title="YouTube video player" frameBorder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowFullScreen></iframe>

## What makes Deepnote a go-to tool for so many data teams?

### 1. It's more than just a notebook

![Deepnote_Infographics.png](https://media.graphassets.com/BVxG2tMFTzCSmgwqILlM)

- **Collaboration at its core:** Deepnote's cloud-based environment enables real-time collaboration and commenting, supported by robust version control, making teamwork seamless and efficient.

- [**SQL support:**](https://deepnote.com/docs/sql-cells) easily switch between SQL, Python and R within a single notebook.

- [**Over 50 secure built-in integrations:**](https://deepnote.com/integrations) Connect effortlessly to popular data sources, including Snowflake, BigQuery, and Postgres, with secure, built-in integrations that get your team up and running instantly.

- **Accessible to all:** With no-code tools and AI-superpowers, coding isn’t a barrier—it’s a bridge. Deepnote makes data science accessible, inviting everyone into the world of data insights.

### 2. Deep AI integration to give users superpowers

Explainable and transparent AI assistant — Deepnote has a deep understanding of your projects, data warehouses, and metadata. It provides precise and auditable AI assistance for your data work. Our notebook structure ensures that every AI suggestion is turned into transparent and reproducible results allowing data teams to trust in AI like never before.

![AI.png](https://media.graphassets.com/GXT55Lu8RKuMoFygbDPN)

- [**Speak naturally, code flawlessly:**](https://deepnote.com/docs/deepnote-ai) Picture this: you ask your data questions just like you'd chat with a colleague, and watch as the code materializes. Deepnote AI goes beyond writing code—it explains and corrects it, enabling your team to achieve wonders, even those new to coding.

- [**AI Code completion:**](https://deepnote.com/docs/ai-code-completion) Context-aware suggestions as you type, reducing repetition and accelerating your projects.

### 3. From notebooks to data apps in seconds

Deepnote features a built-in app functionality that allows you to arrange, format, and publish a selection of blocks as a dashboard, report or interactive app. [Explore what you can build in our library here](https://deepnote.com/explore)

![App.png](https://media.graphassets.com/eKZQkqZ3THqjoKFjgTTF)

- **Arrange your notebook into columns:** Put your charts side-by-side, group your inputs next to each other, and tailor your story for the best viewing experience.

- **Curate what’s visible:** From the app builder you can select which inputs, text, code and SQL blocks will be visible for your audience.

- **Make your apps interactive with Input blocks:** Transform your apps into interactive tools using Deepnote's input blocks. Let your audience actively engage with your analysis by filtering and exploring the data on their own.

If you encounter any issues, please contact [help@deepnote.com](mailto:help@deepnote.com).

Deepnote revolutionizes the way your team work with data, combining Python, SQL, and no-code analysis, natively connecting to over 50 data sources. Enhanced by GPT-4, it offers code generation, editing, explanation, auto-completion, and ability to convert your notebooks into dynamic data apps and dashboards instantly.


Distribute your homework and assignments among your students, let them complete their tasks, and collect them back — all within the same environment.

Each of your workspaces has individual private spaces visible only to you. You can create multiple projects in this space and share each of them with individual students for individual assignments.

![Group 3920.png](https://media.graphassets.com/yoIWrmfpTIaoa0SxdPkn)

![Picture2.png](https://media.graphassets.com/8CDPMV3TQumVTwPgixsS)

You can create as many private projects as you want, each dedicated to a single student. Once you create all of the desired projects, you can invite students individually by [sharing a project](/docs/share-projects). Since projects are located in a private section of the workspace, students will not be able to see each other's work.

Once your students complete their assignments, you can either [collect the work](/docs/submissions) or immediately check the progress by going directly into the project.


Individual assignments

Deepnote can query your time series data in InfluxDB v2. Follow these docs to create your own notebook to connect to your InfluxDB instance.

### 1. Duplicate the InfluxDB template

[Open the template by following this link ->](https://deepnote.com/workspace/Deepnote-Templates-71742312-24f2-4c10-9bf7-786d17280b92/project/Query-InfluxDB-from-Deepnote-46097291-95c8-4a64-a247-541975d8aadc/%2Fnotebook.ipynb)

![spaces%2FtfH69m1V6bYYvquUay8O%2Fuploads%2FuBTyirnZvPcOcczBxtFa%2Fduplicate-notebook.png](https://media.graphassets.com/VoHNl6juQ5qUFurTn0DR)

Click the "duplicate" button on the top right corner to do this. Once you've got the template in your Deepnote workspace, you can connect it to your InfluxDB instance.

### 2. Connect to InfluxDB from Deepnote

To connect to SQL from Deepnote, you'll need credentials to connect to a database and the endpoint where that database is located.

**Create an InfluxDB token**

[Follow these instructions](https://docs.influxdata.com/influxdb/cloud/security/tokens/create-token/) to create a token that has permission to access the bucket you're interested in.

#### Update the token in the notebook

Change the variable `INFLUXDB_TOKEN` to the token you made in the previous step.

#### Update the organization and URL details in the notebook

Change the variable `INFLUXDB_ORG` to the name of your organization, [here is how you can find it](https://docs.influxdata.com/influxdb/v2.2/organizations/view-orgs/).

Change the variable `INFLUXDB_URL` to the URL where your InfluxDB instance is located.

### 3. Query InfluxDB from Deepnote

The notebook will set up a `client` object from the InfluxDB Python library. You can create a query client with this object, and query InfluxDB directly to a Pandas DataFrame. In the example below, we query [InfluxDB's built-in sample data](https://docs.influxdata.com/influxdb/v2.2/reference/sample-data/#air-sensor-sample-data).

```python
query_api = client.query_api()

query = """import "influxdata/influxdb/sample"

sample.data(set: "airSensor")
 |> filter(fn: (r) => r._field == "temperature")
 |> range(start: -45m)
 |> aggregateWindow(every: 1m, fn: mean)"""

# influxdb_client gives us Pandas dataframes out-of-the-box
df = query_api.query_data_frame(query)

df
```

Running the code above will have Deepnote query the InfluxDB instance, save the result into a DataFrame, and visualize it in an interactive table like the one below.

<Embed url="https://embed.deepnote.com/f1bda07e-1467-4298-955b-41b55e97fc75/299ab3c5-886e-4c29-9b86-04bda8b30e8f/0b9e56b351314e729b9cd10168c4e665?height=577" />

### What's next?

Now that you're querying data, you can share it with your team. You can even turn your charts [into a shareable dashboard](/docs/publish-projects).

For more information on connecting to InfluxDB from Python, [you can read the influxdb-client-python docs](https://github.com/influxdata/influxdb-client-python).

InfluxDB

### Adding input blocks

Input blocks are a great way to add different user inputs to be used as variables within your code and SQL blocks. Button block also gives you the ability to control the execution in your notebook or app.

<Callout status="info">

Input blocks can also be used to create [interactive apps and dashboards](https://deepnote.com/docs/publish-projects#interactivity-of-published-projects) from your published projects.

</Callout>

You can easily add an input block to your notebook by clicking the block adder between blocks. Alternatively, you can add input blocks by clicking the Input button at the bottom of your notebook or using the command palette.

![input-adder.png](https://media.graphassets.com/BRAbFKI2RLqYfVC7kX7A)

Once you've added an input block, you can adjust various settings like variable name and display name. Explore more options for the given input type.

### Using input variables in notebook

You can use the inputs as variables within your code. For example, you can use the date input variable in the code block to filter DataFrame data.

![date-input-example.png](https://media.graphassets.com/tGrNcUi1QzHmz33PhoHR)

You can also pass on input variables as parameters in your SQL blocks.

![input-sql-example.png](https://media.graphassets.com/wEHOtXL1ShynEXVt1hY4)

### Using input variables in app

Even though all inputs work great both in the notebook and in the app some are particularly powerful in the app.

#### File input

The file input block allows users to easily upload files to your app for further processing.

![file-in-app-example.png](https://media.graphassets.com/4Mh9g5dTQRWDEqtKCp78)

After uploading the file, the input variable will contain the path to the file. All files are stored in the _file_input_uploads_ folder within your projects and the maximum file size is 5GB. It's also possible to specify allowed file extensions and MIME types.

![file-input-options.png](https://media.graphassets.com/iin2KOZSCCNdZd8alqAg)

#### Button block

After adding a button block to your notebook, you can customize its color, label, and behavior when a user clicks it. There are two types of behaviors available:

**1) Set Variable**

With this behavior selected, you can specify a name for a variable, which the button will interact with. When the button is clicked, this variable is set to the boolean value “True,” and any blocks that depend on this variable are executed. You can identify which blocks depend on this variable by inspecting the notebook’s [DAG](https://deepnote.com/docs/execution-modes#directed-acyclic-graph-dag). After the dependent blocks are executed, the variable is reset to “False”.

The “Set Variable” behavior enables you to create more dynamic apps, where outputs are rendered based on user actions. By adding conditions into your Python code blocks based on the button variable’s value , you can control when certain code blocks are executed and which block outputs are displayed.

In the demo below, the ‘Sales trends’ button produces a timeseries chart, while the ‘Reviews’ button displays correlation data.

![2024-11-06-2010.37.21.gif](https://media.graphassets.com/VVeLdkVeSom54VKrAyU8)

To enable this type of workflow, all you need to do is configure the button with a variable (eg. 'trends') and then use this variable in a code block as a condition.

![CleanShot 2024-11-07 at 10.27.50 3.png](https://media.graphassets.com/i3Hk9Z7aTMCCrUwDxtN8)
![CleanShot 2024-11-07 at 10.28.49 3.png](https://media.graphassets.com/YYd8ETBXTcSrl3mlBgQO)

When building apps with **multiple input blocks that depend on each other**, you can use the Run button to create more controlled workflows. The button allows you to delay the execution of downstream code until you have all the necessary input values collected from the user - you just need to add a button variable condition to the code block.

![CleanShot 2024-11-07 at 10.38.32.png](https://media.graphassets.com/afdn2ewrTxWzjXp81QKD)

![2024-11-06-2011.13.27.gif](https://media.graphassets.com/F2UWLLiTK6pg4TxJL5Hy)

**2) Run Notebook**

When the user clicks the button, the entire notebook is run. The same applies when the button block is used within an app—all blocks are executed. This behavior is useful for building forms where an explicit “Submit” button is needed, for example.

![button-example.png](https://media.graphassets.com/OzDzFLCjQc2K9x8SDXj4)

### Dynamic input variables

In some cases, you may want to dynamically generate the list of available values in select inputs instead of using pre-defined values. You can do so by clicking on input settings and select the 'From variable' option at the Values section. In the dropdown you will need to pick one of your existing variables (supported python types that can be used are: `list`, `ndarray`, `DataFrame`, and `Series`).

Once you select the variable, the values in the list will be then made available as user inputs in the select input dropdown.

![2022-09-20 13.07.41.gif](https://media.graphassets.com/Kl3C7JoQKeefDuO73MDw)

Limits for using dynamic input variables:

1. Only variables with 1,000 elements or fewer are supported.
2. Total characters of all elements cannot exceed 10,000.

Input blocks

There are 3 major ways of installing system-level dependencies and Python packages in Deepnote:

- installing them ad-hoc in your notebook;
- using an initialization script (Init notebook);
- or by changing the underlying environment image.

### 1. Installing new packages via ad-hoc installation (pip or apt)

Simply run `pip install <package_name>` or `apt install <library>`, either in the terminal or with the `!` prefix in a notebook.

![CleanShot 2024-11-28 at 09.10.08.png](https://media.graphassets.com/YxhpDdlTSCF37EZzcZTQ)

If you are in a [conda environment](https://deepnote.com/docs/conda-environment), you can also run `conda install <package_name>`.

This will immediately install the library for you to use.

Note that when the hardware restarts, the environment will revert back to the default state.

### Installing packages through a proxy

If you have packages pulled from a custom registry which needs to have an IP allowlisted, Deepnote can route your pip download through a proxy. Simply add the `proxy` argument with our IP `http://34.68.180.189:3128/` to the pip install.

```
pip install -r requirements.txt --extra-index-url=http://something.com --proxy <http://34.68.180.189:3128>

```

### 2. Initialization script (Init notebook)

Deepnote provides an optional feature to run custom Python initialization code every time your project hardware boots up. To utilize this capability, you will need to **include an 'Init' notebook** in your project.

To create an 'Init' notebook:

Navigate to your project's **Machine** section in the project sidebar.

- Click on the environment selection menu.
- Click on the **Create initialization notebook** button.

![CleanShot 2024-11-28 at 08.58.46.png](https://media.graphassets.com/JFHU4qtsSdCDWINAo66C)

After creating it, the 'Init' notebook will be conveniently located at the top of your Notebook list. The contents of this notebook will automatically execute each time your project starts up, providing an ideal place to set up your preferred environment.

By default, the 'Init' notebook installs any packages listed in **./requirements.txt**. However, you are free to customize this according to your project's needs.

For a more detailed explanation of project initialization, follow this [link](https://deepnote.com/docs/project-initialization).

### 3. Environments

Deepnote environments are customizable runtime configurations powered by **Docker images**. Each environment defines the complete stack - from the Linux operating system to Python packages and binary dependencies.

While Deepnote provides several pre-configured environments for common use cases, you can also create your own custom environment using any Docker image, giving you full control over your runtime setup.

Read more about [default](https://deepnote.com/docs/default-environment) and [custom](https://deepnote.com/docs/custom-environment) environments.

Installing dependencies

Deepnote natively integrates with a substantial collection of databases, data warehouses, and other services. All integrations are plug-and-play, meaning they can be connected to any project in the workspace they were created in on an as-needed basis (click [here](/docs/team-permissions) to read more about integration permissions).

Once created, integrations greatly simplify using Deepnote in conjunction with other tools in your data stack through, for instance, custom [SQL blocks](/docs/sql-cells). On top of all of that, Deepnote also takes care of credential management for you by encrypting and securely storing all integration-related credentials.

### Supported integrations

<Callout status="info">
Missing an integration? [Let us know through Deepnote's Product Portal!](https://portal.productboard.com/deepnote/1-deepnote-product-portal/tabs/1-under-consideration/submit-idea)
</Callout>

Learn more about any of Deepnote's currently supported integrations by selecting a specific integration from the left-hand sidebar or browsing integrations by category and jumping into a detailed walkthrough of connecting and using integrations.

### Creating & managing integrations

To edit your existing integrations (and create new ones), open up the integrations overview from the left-hand sidebar. Alternatively, you can also find your way to the integrations overview via the integrations menu in the right-hand sidebar within each project.

![Opening the integrations overview in Deepnote](https://media.graphassets.com/bJNx6rBXRJilCYxYcdbD)

As mentioned above, each integration can be connected to multiple projects within your workspace. It is important to remember that deleting an integration that is connected to multiple projects will remove the specified integration from all projects in that workspace.

![Managing integrations in Deepnote](https://media.graphassets.com/DF2YWWwSne9pSULPMSVA)

<Callout status="info">
Some integrations require your machine to restart (don't fret, you will be warned in advance). Note that this will terminate all running processes in the project.
</Callout>

To keep an integration available in your workspace and only remove it from a specific project, open up the right-hand sidebar and edit the integration in question in the integrations menu.

### Integrations & collaboration

Integrations are shared amongst all team members and can be connected to any of the projects in the workspace.

### Integrations by category

Explore integrations by category and dive deeper into any of the listed integrations.

#### Databases

- [PostgreSQL](/docs/postgresql)
- [MySQL](/docs/mysql)
- [MariaDB](/docs/mysql)
- [MongoDB](/docs/mongodb)
- [MindsDB](/docs/mindsdb)
- [Supabase](/docs/supabase)
- [ClickHouse](/docs/clickhouse)
- [SQL Server (Azure SQL)](/docs/sql-server)

#### Data warehouses

- [AWS Redshift](/docs/redshift)
- [Google BigQuery](/docs/google-bigquery)
- [Snowflake](/docs/snowflake/)

#### Object storage & querying services

- [Amazon S3](/docs/amazon-s3)
- [Amazon Athena](/docs/amazon-athena)
- [Google Cloud Storage](/docs/google-cloud-storage)

#### Drive storage services

- [Google Drive](/docs/google-drive)
- [Dropbox](/docs/dropbox)

#### Container registries

- [Amazon Elastic Container Registry](/docs/amazon-ecr)
- [Google Container Registry](/docs/google-container-registry)
- [DockerHub](/docs/docker-hub)

#### Version control

- [GitHub](/docs/github)

#### Deepnote utilities

- [Environment variables](/docs/environment-variables)
- [Shared datasets for large files](/docs/deepnote-shared-datasets)
- [SSH keys](/docs/ssh-key)

### Next steps

Jump right into Deepnote and [take a look at SQL blocks and MindsDB in action.](https://deepnote.com/workspace/Deepnote-Templates-71742312-24f2-4c10-9bf7-786d17280b92/project/Machine-Learning-With-SQL-c626fd30-cb3f-4eae-ac70-c9c4a4f626eb/%2Fmindsdb_demo.ipynb) You can also save yourself some setup work by hitting the `Duplicate` button in the top-right corner to start exploring on your own!

Integrations


The code you write on Deepnote is yours and Deepnote does not make any ownership claims. If you make your project public (either by sharing it with friends or by making it accessible by link), the people you share with will be able to access your project and see your code. It is your responsibility to make sure this is ok.

If you are making the project public and sharing the link to it on some online platform (eg. a blog), we recommend you to include a LICENSE file to mitigate any possible disputes.


Intellectual property


Deepnote is a cloud-based data platform that transforms how data teams work together. Built for modern data science workflows, it combines the power of Jupyter notebooks with enterprise-grade collaboration features in a secure environment.

A common tool many data scientists use is **IPywidget,** [an interactive widget library](https://ipywidgets.readthedocs.io/en/8.1.2/) that allows users to build rich, interactive GUIs in Jupyter notebooks. However, IPywidgets are not supported in Deepnote, and here is why, what alternatives you can use to enhance your productivity

## Why IPywidgets are not supported in Deepnote

The primary reason Deepnote doesn't support IPywidgets is security. The platform had to disable IPywidgets functionality due to potential security vulnerabilities that could arise from executing arbitrary JavaScript code in notebooks.

## Alternatives to IPywidgets in Deepnote

While IPywidgets provide an interactive experience, Deepnote offers **native features** and **visualization options** that can achieve the same goals but in a more streamlined manner. Below are some of the ways you can add interactivity and visualization to your Deepnote notebooks:

### **Interactive charts and visualizations**

Deepnote natively supports interactive libraries like **Plotly**, **Altair**, **Seaborn**, and **Matplotlib**, which allow for the creation of dynamic and interactive charts directly in notebooks. These libraries are lightweight and integrate seamlessly with Deepnote’s interface.

- **Plotly** provides powerful charting tools that allow you to zoom, pan, and hover over data points for additional insights.
- **Altair** is another declarative visualization library that automatically renders interactive charts with minimal code.
- **Matplotlib** and **Seaborn** can be used for static charts, but with a bit of customization, you can add basic interactive elements.

### **Deepnote's built-in blocks**

Deepnote has its **blocks**(chart blocks, big numbers, input bloks and more) for interactivity, such as sliders, input boxes, and dropdowns, that can be added to notebooks. These blocks allow you to dynamically control parameters and variables without writing complex code, making them simpler alternatives to IPywidgets.

**Example: Adding a slider to control variables**

Instead of using IPywidgets, you can add a native Deepnote slider to your notebook, which can adjust your data visualization dynamically.
![Nov-29-2024 20-03-45.gif](https://media.graphassets.com/CcwUq7chTTWjVFytIFoB)

In Deepnote, you can insert a **slider input block** next to the code blocks and use that to control elements like the frequency of the sine wave or the range of the x-axis, checkout [app with example](https://deepnote.com/app/deepnote/Slider-example-adb7a9d5-5f07-44a2-9364-8b34714521ba?utm_source=app-settings&utm_medium=product-shared-content&utm_campaign=data-app&utm_content=adb7a9d5-5f07-44a2-9364-8b34714521ba).

### **Parameterize notebook with Deepnote input blocks**

Deepnote’s blocks parameters feature is a powerful way to create reusable and interactive notebooks without needing external libraries like IPywidgets. You can make a parameterised notebook that accepts different inputs to change the analysis or visualization dynamically.

For example, by using dropdowns or text inputs, you can parameterize blocks to automatically update based on user selections.

### **Native forms for collaboration**

For data science teams collaborating in Deepnote, the platform allows users to build **forms** with inputs such as text boxes, sliders, and dropdowns, providing an easy way to make a notebook interactive for both technical and non-technical users. This is particularly useful for teams who may not want to dig into the code but still need to tweak parameters or adjust analyses.

### **Deepnote’s text blocks**

Text blocks in Deepnote allow you to embed **interactive tables, links, and visualizations** directly in your notebooks, making it easy to share your work with stakeholders. While Markdown isn't an exact replacement for IPywidgets, it does provide a clean, simple way to present interactive content without adding code-heavy widgets.

### **Output blocks**

Deepnote notebooks support **dynamic blocks outputs**, which allow you to show or hide certain results or outputs based on block execution. This can be used to create a pseudo-interactive environment similar to what you might achieve with IPywidgets.

While IPywidgets are a popular tool for adding interactivity to Jupyter notebooks, Deepnote decided to do it differently with focus on simplicity, performance, and collaboration. Deepnote provides several powerful alternatives that allow you to build interactive and engaging notebooks and apps without needing to rely on third-party widget libraries.

Whether you are building interactive charts with Plotly or creating parameterized blocks Deepnote ensures that you can add interactivity and dynamic content without sacrificing performance or ease of use.

By exploring Deepnote's built-in tools, you’ll find that you can achieve all the interactivity you need—without the complexity of IPywidgets!


We do not support IPywidgets but we offer native alternatives like built-in input blocks for enhanced interactivity in data science notebooks and apps.

IPyWidgets in Deepnote


# JAX vs. PyTorch

The JAX vs. PyTorch is compared in this article - their performance, and the ideal use cases, helping to choose the best framework for machine learning projects.

Specialized libraries make it easier to build complex models by handling the tough math and speeding up data processing with GPU power. Supported by vibrant communities, libraries like JAX and PyTorch bring people together, making it simple to experiment, collaborate, and drive AI forward.

## PyTorch

PyTorch is an open-source platform developed by Facebook's AI Research lab, famous for its user-friendly interface and powerful capabilities, making it ideal for researchers and developers. Users can create and modify neural networks in real-time thanks to its adaptable computation graph, which facilitates rapid experimentation and streamlines the debugging procedure.
![pytorch_logo.png](https://media.graphassets.com/WuKqX0yJQemluAz2LOcG)
The library focuses on performance by using GPU-accelerated performance and a wide array of tools, making it a go-to choice for areas like computer vision and natural language processing. PyTorch's dynamic computation graphs enable making changes to neural network architectures in real-time, it's highly beneficial for research as it allows for rapid improvements in model performance, particularly in natural language processing and computer vision tasks.

### Ideal use cases:

- **Research and development:** Quick creation of cutting-edge algorithms through rapid prototyping.
- **Computer vision:** Sophisticated image processing applications enabled by libraries such as torchvision.
- **Natural language processing:** Effective management of ordered information for tasks such as sentiment analysis.

### Example:

This example shows how to build a simple neural network in PyTorch, using a fully connected layer to transform input data of size 10 to an output of size 1. A forward pass on random data calculates the mean squared error loss against target data.
![jax_exemple_code.png](https://media.graphassets.com/DMXCsccYQCq4tijj1yl0)

## JAX

JAX is a open-source library created by Google that makes high-speed numerical computing and machine learning more accessible. It focuses on automatic differentiation and composability, allowing developers to build complex models easily. With its ability to work smoothly with NumPy and take advantage of GPUs and TPUs, JAX boosts performance during model training and inference.
![jax.jpg](https://media.graphassets.com/75vRxf8FRhmHJJz2Mohl)

### Ideal use cases:

- **Scientific research:** Speeding up simulations and model development in areas such as physics and biology.

- **Machine learning:** Applying state-of-the-art algorithms with effective automatic differentiation.

- **High-performance computing:** Utilizing JAX for complex computations that require enhanced performance.

This example shows a basic neural network in JAX using a functional approach, where a dot product computes output, random input and target data are generated, and mean squared error loss is calculated, highlighting JAX's efficiency.
![jax_ex_2.png](https://media.graphassets.com/Y8wxWgOBSCiw0wyhu6IW)

## PyTorch vs. JAX: A quick comparison

This table outlines the key differences and strengths of PyTorch and JAX to help you choose the best fit for your deep learning project.
