The InputContext passed to an IOManager ’s load_input function when invoking the output_value or output_for_node methods on JobExecutionResult now has the name "dummy_input_name" instead of None.
[dagster-ui] Asset materializations can now be reported from the dropdown menu in the asset list view.
[dagster-dbt] DbtProject is adopted and no longer experimental. Using DbtProject helps achieve a setup where the dbt manifest file and dbt dependencies are available and up-to-date, during development and in production. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject.
[dagster-dbt] The —use-dbt-project flag was introduced for the cli command dagster-dbt project scaffold. Creating a Dagster project wrapping a dbt project using that flag will include a DbtProject .
[dagster-ui] The Dagster UI now loads events in batches of 1000 in the run log viewer, instead of batches of 10000. This value can be adjusted by setting the DAGSTER_UI_EVENT_LOAD_CHUNK_SIZE environment variable on the Dagster webserver.
Asset backfills will now retry if there is an unexpected exception raised in the middle of the backfill. Previously, they would only retry if there was a problem connecting to the code server while launching runs in the backfill.
Added the ability to monitor jobs which have failed to start in time with the RunFailureReason.START_TIMEOUT run monitoring failure reason. Thanks @jobicarter!
[experimental] Introduced the ability to attach code references to your assets, which allow you to view source code for an asset in your editor or in git source control. For more information, see the code references docs: https://docs.dagster.io/guides/dagster/code-references.
[ui] Performance improvements to loading the asset overview tab.
[ui] Performance improvements for rendering gantt charts with 1000’s of ops/steps.
[dagster-celery] Introduced a new Dagster Celery runner, a more lightweight way to run Dagster jobs without an executor. Thanks, @egordm!
Fixed a bug that caused tags added to ObserveResult objects to not be stored with the produced AssetObservation event.
Fixed a bug which could cause metadata defined on SourceAssets to be unavailable when accessed in an IOManager.
For subselections of graph-backed multi-assets, there are some situations where we used to unnecessarily execute some of the non-selected assets. Now, we no longer execute them in those situations. There are also some situations where we would skip execution of some ops that might be needed. More information on the particulars is available here.
Fixed the @graph_asset decorator overload missing an owners argument, thanks @askvinni!
Fixed behavior of passing custom image config to the K8sRunLauncher, thanks @marchinho11!
[dagster-dbt] Fixed an issue with emitting column lineage when using BigQuery.
[dagster-k8s] Added additional retries to execute_k8s_job when there was a transient failure while loading logs from the launched job. Thanks @piotrmarczydlo!
[dagster-fivetran] Fixed an issue where the Fivetran connector resource would sometimes hang if there was a networking issue while connecting to the Fivetran API.
[dagster-aws] Fixed an issue where the EMR step launcher would sometimes fail due to multiple versions of the dateutil package being installed in the default EMR python evnrionment.
[ui] The “Create date” column in the runs table now correctly shows the time at which a run was created instead of the time when it started to execute.
[ui] Fixed dark mode colors in run partitions graphs.
[auto-materialize] Fixed an issue which could cause errors in the AutoMaterializeRule.skip_on_parent_missing rule when a parent asset had its PartitionsDefinition changed.
[declarative-automation] Fixed an issue which could cause errors when viewing the evaluation history of assets with AutomationConditions.
[declarative-automation] Previously, AutomationCondition.newly_updated() would trigger on any ASSET_OBSERVATION event. Now, it only triggers when the data version on that event changes.
[dagster-dbt] The cli command dagster-dbt project prepare-for-deployment has been replaced by dagster-dbt project prepare-and-package.
[dagster-dbt] During development,DbtProject no longer prepares the dbt manifest file and dbt dependencies in its constructor during initialization. This process has been moved to prepare_if_dev(), that can be called on the DbtProject instance after initialization. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject.prepare_if_dev.
Passing GraphDefinition as the job argument to schedules and sensors is deprecated. Derive a job from the GraphDefinition using graph_def.to_job() and pass this instead.
Updated dagster-plus CLI in the sidenav to correctly be dagster-cloud CLI.
Thanks to Tim Nordenfur and Dimitar Vanguelov for fixing a few typos!
Introduced guides to migrate Airflow pipelines to Dagster that leverage the TaskFlow API or are containerized and executed with an operator like the KubernetesPodOperator.
Fixed instructions on setting secrets in Kubernetes Dagster deployments, thanks @abhinavDhulipala!
A history of code location deploys can now be viewed on the Code Locations tab under the Deployment view. Previously deployed versions can now be restored from history.
[ui] Various improvements have been made to the asset health dashboard, which is now no longer experimental.
[ui] Fixed issues in per-event asset insights where bar charts incorrectly displayed events in reverse order, and with UTC timestamps.
Fixed a recent regression where creating an alert that notifies asset owners that are teams raises an error.
[ui] Improved performance for loading assets that are part of big asset graphs.
[ui] Improved performance for loading job backfills that have thousands of partitions
[ui] The code location page can now be filtered by status
[agent] K8s and ECS agent main loop writes a sentinel file that can be used for liveness checks.
[agent][experimental] ECS CloudFormation template with private IP addresses using NAT Gateways, security groups, IAM role separation, tighter permissions requirements, and improved documentation.
Ephemeral asset jobs are now supported in run status sensors (thanks @the4thamigo-uk)!
In AssetsDefinition construction, enforce single key per output name
Fixed a bug where freshness checks on assets with both observations and materializations would incorrectly miss a materialization if there’s no observation with dagster/last_updated_timestamp.
Fixed a bug with anomaly detection freshness checks where “not enough records” result would cause the sensor to crash loop.
Fixed a bug that could cause errors in the Asset Daemon if an asset using AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron() rule gained a new dependency with a different PartitionsDefinition.
[ui] Fixed an issue that caused the backfill page not to be scrollable.
[ui] Fixed an issue where filtering by partition on the Runs page wouldn’t work if fetching all of your partitions timed out.
[dagster-dlt] Fixed bug with dlt integration in which partitioned assets would change the file name when using the filesystem destination.
[ui] Fixed an issue where an erroring code location would cause multiple toast popups.
Allow a string to be provided for source_key_prefix arg of load_assets_from_modules. (thanks @drjlin)!
Added a missing debug level log message when loading partitions with polars (thanks Daniel Gafni)!
Set postgres timeout via statement, which improves storage-layer compatibility with Amazon RDS (thanks @james lewis)!
In DBT integration, quote the table identifiers to handle cases where table names require quotes due to special characters. (thanks @alex launi)!
remove deprecated param usage in dagster-wandb integration (thanks @chris histe)!
Add missing QUEUED state to DatabricksRunLifeCycleState (thanks @gabor ratky)!
Fixed a bug with dbt-cloud integration subsetting implementation (thanks @ivan tsarev)!
The dagster-cloud ci init CLI will now use the --deployment argument as the base deployment when creating a branch deployment. This base deployment will be used for Change Tracking.
The BigQuery dbt insights wrapper dbt_with_bigquery_insights now respects CLI arguments for profile configuration and also selects location / dataset from the profile when available.
[experimental feature] Fixes a recent regression where the UI errored upon attempting to create an insights metric alert.
Asset search results now display compute and storage kind icons.
Asset jobs where the underlying assets have multiple backfill policies will no longer fail at definition time. Instead, the backfill policy for the job will use the minimum max_partitions_per_run from the job’s constituent assets.
[dagstermill] asset_tags can now be specified when building dagstermill assets
[dagster-embedded-elt] Custom asset tags can be applied to Sling assets via the DagsterSlingTranslator
[dagster-embedded-elt] dlt assets now automatically have dagster/storage_kind tags attached
tags passed to outs in graph_multi_asset now get correctly propagated to the resulting assets.
[ui] Fixed an issue in the where when multiple runs were started at the same time to materialize the same asset, the most recent one was not always shown as in progress in the asset graph in the Dagster UI.
The “newly updated” auto-materialize rule will now respond to either new observations or materializations for observable assets.
build_metadata_bounds_checks now no longer errors when targeting metadata keys that have special characters.
Updated the Schedule concept page to be a “jumping off” point for all-things scheduling, including a high-level look at how schedules work, their benefits, and what you need to know before diving in
[experimental] The backfill daemon can now store logs and display them in the UI for increased visibility into the daemon’s behavior. Please contact Dagster Labs if you are interested in piloting this experimental feature.
Added a --read-only flag to the dagster-cloud ci branch-deployment CLI command, which returns the current branch deployment name for the current code repository branch without update the status of the branch deployment.
Dagster will now display a “storage kind” tag on assets in the UI, similar to the existing compute kind. To set storage kind for an asset, set its dagster/storage_kind tag.
You can now set retry policy on dbt assets, to enable coarse-grained retries with delay and jitter. For fine-grained partial retries, we still recommend invoking dbt retry within a try/except block to avoid unnecessary, duplicate work.
AssetExecutionContext now exposes a has_partition_key_range property.
The owners, metadata, tags, and deps properties on AssetSpec are no longer Optional. The AssetSpec constructor still accepts None values, which are coerced to empty collections of the relevant type.
The docker_executor and k8s_job_executor now consider at most 1000 events at a time when loading events from the current run to determine which steps should be launched. This value can be tuned by setting the DAGSTER_EXECUTOR_POP_EVENTS_LIMIT environment variable in the run process.
Added a dagster/retry_on_asset_or_op_failure tag that can be added to jobs to override run retry behavior for runs of specific jobs. See the docs for more information.
Improved the sensor produced by build_sensor_for_freshness_checks to describe when/why it skips evaluating freshness checks.
A new “Runs” tab on the backfill details page allows you to see list and timeline views of the runs launched by the backfill.
[dagster-dbt] dbt will now attach relation identifier metadata to asset materializations to indicate where the built model is materialized to.
[dagster-graphql] The GraphQL Python client will now include the HTTP error code in the exception when a query fails. Thanks @yuvalgimmunai!
Fixed sensor logging behavior with the @multi_asset_sensor.
ScheduleDefinition now properly supports being passed a RunConfig object.
When an asset function returns a MaterializeResult, but the function has no type annotation, previously, the IO manager would still be invoked with a None value. Now, the IO manager is not invoked.
The AssetSpec constructor now raises an error if an invalid owner string is passed to it.
When using the graph_multi_asset decorator, the code_version property on AssetOuts passed in used to be ignored. Now, they no longer are.
[dagster-deltalake] Fixed GcsConfig import error and type error for partitioned assets (Thanks @thmswt)
The asset graph and asset catalog now show the materialization status of External assets (when manually reported) rather than showing “Never observed”
The ability to set a custom base deployment when creating a branch deployment has been enabled for all organizations.
When a code location fails to deploy, the Kubernetes agent now includes additional any warning messages from the underlying replicaset in the failure message to aid with troubleshooting.
Serverless deployments now support using a requirements.txt with hashes.
Fixed an issue where the dagster-cloud job launch command did not support specifying asset keys with prefixes in the --asset-key argument.
[catalog UI] Catalog search now allows filtering by type, i.e. group:, code location:, tag:, owner:.
New dagster+ accounts will now start with two default alert policies; one to alert if the default free credit budget for your plan is exceeded, and one to alert if a single run goes over 24 hours. These alerts will be sent as emails to the email with which the account was initially created.