[dagster-k8s] The PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process.
[ui] The launch asset backfill modal now offers a preview dialog showing the targeted asset partitions and their backfill policies when partition mapping or varying backfill policies are present.
[asset-checks] New load_asset_checks_from_modules functions for loading asset checks in tandem with load_assets_from_modules.
Previously, the daemon process would terminate with an error if it believed that a thread might be hanging, which sometimes caused undesirable terminations when doing large backfills or auto-materializing many assets. Now, the daemon process will emit a warning instead of terminate.
[dagster-dbt] dagster-dbt project scaffold now uses ~/.dbt/profiles.yml if a profiles.yml is not present in the dbt project directory.
[dagster-dbt] @dbt_assets now support PartitionMapping using DagsterDbtTranslator.get_partition_mapping.
[dagster-dbt] Self dependencies can now be enabled for dbt models that are represented by @dbt_assets. To enable this, add the following metadata to your dbt model’s metadata in your dbt project:
Fixed an issue where Dagster imports would throw errors when using pydantic<2.0.0 but having pydantic-core installed.
Previously, asset backfills that targeted partitioned assets with a LastPartitionMapping dependency would raise an error. This has been fixed.
Fixed a multipartitions partition mapping bug where a instance is not available to load partitions error was raised.
[asset-checks] Fixed an issue with conflicting op names when using build_asset_with_blocking_check
[ui] Viewing run logs containing very large messages no longer causes the UI to crash in Safari on macOS
[ui] Drilling into the graph of a graph-backed asset with 4+ inputs or outputs no longer causes the asset graph to crash with a rendering error.
[ui] On the backfill details page, clicking to a specific asset correctly shows the partitions that were materialized for that asset when partition mapping is in use.
[ui] The Asset > Partition page now loads partition details more quickly in cases where calculating the staleness of the partition took a significant amount of time.
Fixed a bug introduced in 1.5.0 where instances that haven’t been migrated to the latest schema hit an error upon run deletion.
[auto-materialize] Previously, if an asset partition was skipped on a tick for one reason, and then processed and skipped on a later tick for an additional reason, only the most recent skip reason would be tracked. Now, all skip reasons are tracked.
[dagster-dbt] Fixed an issue where if an exclude that didn’t match any dbt nodes was used in @dbt_assets, an error would be raised. The error is now properly handled.
[dagster-dbt] When invoking DbtCliResource.cli(...) in an op, AssetMaterialization's instead of Output are now emitted.
Dagster now supports using Pydantic 2 models for Config and Resources. Pydantic 1.10 continues to be supported.
Added a report_asset_observation REST API endpoint for runless external asset observation events
Dramatically improved the performance of partition-mapping, for basic hourly and daily partitions definitions
[ui] When viewing a list of runs, you can quickly add the tag in the “Launched by” column as a filter on the list view. Hover over the tag to see the “Add to filter” button.
[helm] The env vars DAGSTER_K8S_PG_PASSWORD_SECRET and DAGSTER_K8S_INSTANCE_CONFIG_MAP will no longer be set in all pods.
[dagster-pyspark] build_pyspark_zip now takes an exclude parameter that can be used to customize the set of excluded files.
[ui] Links beginning with http://, https:// in unstructured run logs (via context.log) are automatically converted to clickable links
Added a report_asset_check REST API endpoint for runless external asset check evaluation events. This is available in cloud as well.
The config argument is now supported on @graph_multi_asset
[ui] Improved performance for global search UI, especially for deployments with very large numbers of jobs or assets.
[dagster-pipes] Add S3 context injector/reader.
[dagster-dbt] When an exception when running a dbt command, error messages from the underlying dbt invocation are now properly surfaced to the Dagster exception.
[dagster-dbt] The path to the dbt executable is now configurable in DbtCliResource.
Fixed a bug introduced in 1.5.3 that caused errors when launching specific Ops in a Job.
Fixed a bug introduced in 1.5.0 that prevented the AssetExecutionContext type annotation for the context parameter in @asset_check functions.
Fixed an issue where the Dagster scheduler would sometimes fail to retry a tick if there was an error reloading a code location in the middle of the tick.
[dagster-dbt] Fixed an issue where explicitly passing in profiles_dir=None into DbtCliResource would cause incorrect validation.
[dagster-dbt] Fixed an issue where partial parsing was not working when reusing existing target paths in subsequent dbt invocations.
[ui] Fixed an issue where the job partitions UI would show “0 total partitions” if the job consisted of more than 100 assets
[dagster-duckdb] The DuckDBResource and DuckDBIOManager accept a connection_config configuration that will be passed as config to the DuckDB connection. Thanks @xjhc!
Added events in the run log when a step is blocked by a global op concurrency limit.
Added a backoff for steps querying for open concurrency slots.
Auto-materialize logic to skip materializing when (1) a backfill is in progress or (2) parent partitions are required but nonexistent are now refactored to be skip rules.
[ui] Added 2 new asset graph layout algorithms under user settings that are significantly faster for large graphs (1000+ assets).
Running multiple agents is no longer considered experimental.
When the agent spins up a new code server while updating a code location, it will now wait until the new code location uploads any changes to Dagster Cloud before allowing the new server to serve requests.
Alert policies can now be set on assets + asset checks (currently experimental). Check out the alerting docs for more information.
Added a new flag --live-data-poll-rate that allows configuring how often the UI polls for new asset data when viewing the asset graph, asset catalog, or overview assets page. It defaults to 2000 ms.
Added back the ability to materialize changed and missing assets from the global asset-graph. A dialog will open allowing you to preview and select which assets to materialize.
Added an experimental AMP Timeline page to give more visibility into the automaterialization daemon. You can enable it under user settings
Added a report_asset_materialization REST API endpoint for creating external asset materialization events. This is available in cloud as well.
[dbt] The @dbt_assets decorator now accepts a backfill_policy argument, for controlling how the assets are backfilled.
[dbt] The @dbt_assets decorator now accepts a op_tags argument, for passing tags to the op underlying the produced AssetsDefinition.
[pipes] Added get_materialize_result & get_asset_check_result to PipesClientCompletedInvocation
[dagster-datahub] The acryl-datahub pin in the dagster-datahub package has been removed.
[dagster-databricks] The PipesDatabricksClient now performs stdout/stderr forwarding from the Databricks master node to Dagster.
[dagster-dbt] The hostname of the dbt API can now be configured when executing the dagster-dbt-cloud CLI.
[dagster-k8s] Added the ability to customize how raw k8s config tags set on an individual Dagster job are merged with raw k8s config set on the K8sRunLauncher. See the docs for more information.
[pipes] A change has been made to the environment variables used to detect if the external process has been launched with pipes. Update the dagster-pipes version used in the external process.
[pipes] The top level function is_dagster_pipes_process has been removed from the dagster-pipes package.
Previously, asset backfills targeting assets with multi-run backfill policies would raise a "did not submit all run requests" error. This has been fixed.