Choose AI provider & PySpark data tables

Native table view for PySpark DataFrames

Spark users, this one's for you! PySpark DataFrames now work seamlessly with our interactive table views, giving you a proper way to explore your massive datasets beyond the basic df.show() output. See how it works in this quick demo.

If you've been working with Spark, you know the drill - df.show() gives you a quick minimalistic peek at your data, but that's about it. Not exactly ideal when you're trying to understand what you're working with. Now your PySpark DataFrames get the full Deepnote treatment: just return your DataFrame on the last line of your code block, and you'll see it displayed in our rich table view with sorting, filtering, column management, and conditional formatting.

Here's the smart part: we pull 10,000 records into a preview mode so you can explore quickly, but when you apply transformations like filtering or sorting, we push those operations back to Spark to work with your complete dataset. This means you get snappy exploration of massive data while still having the power to manipulate everything when needed.

This update is also paving the way for something we're really excited about: native charting support for arbitrarily large datasets. Soon you'll be able to create visualizations that run aggregations on the backend, so you can chart massive datasets without worrying about row limits or pre-filtering. We're bringing this to both Spark and Pandas DataFrames.

Choose your AI provider

We're giving workspace admins on paid plans more control over their AI experience! You can now set the default AI provider for your entire workspace, choosing between OpenAI and Anthropic to best fit your team's needs.

Here's how it works: head to Settings & members → Project Settings to select your preferred provider. Once set, all new AI interactions in your workspace will use your chosen default. This means you can standardize your team's AI experience while still maintaining the flexibility to switch providers when needed.

For new paying customers, we've made Anthropic the default AI provider. Anthropic offers enhanced capabilities that we think will make your data science workflows even more powerful. If you're an existing paying customer and want to try Anthropic, you can easily make the switch in your workspace settings.

This is just the beginning of giving you more choice and control over your AI tools. We're planning to add more providers to this list, so you'll have even more options to choose from based on your team's specific needs.