If you have any feedback reach out to us from, in our discord channel or #tools-fal channel of the dbt slack community.
Milestone Name | Description | T-shirt size | Depends on | Status | Owner | Related github issue |
---|---|---|---|---|---|---|
1) Introduce fal run --before to add ability to run before scripts in fal run |
Same behavior as fal today but instead runs only the before scripts, |
this command supports fal selectors
this command ignores artifacts runs all before scripts by default.
See example 1 below for config design.
| M | Nothing | Done | @Anonymous | ‣ |
| 2) Introduce fal flow
command | Invoke fal flow
with all dbt cli params, with no regression in dbt *
behavior. This should also work for all other dbt commands, dbt test, build ect.. probably for now it is just a pass through for fal and it runs fal run
as we know of today
example:
fal flow run -—select modela
runs:
dbt run —-select modela && fal run
(today, fal scripts look at the artifacts so this only runs fal run
under the hood..)
dont support this yet in this step:
fal flow run -select script
| M | Nothing | Done | @Anonymous | ‣ |
| 3) Introduce ability to add before scripts in fal flow
| Same as “Introduce fal flow command” but this time invoke fal run --before
before the dbt run
command
example:
fal flow run -—select modela
example runs:
fal run —-before -—select modela && dbt run -—select modela && fal run
dont support this yet in this step:
fal flow run -select scripty
now we need to understand what to do and rewrite the dbt command
| M | 1, 2 | Done | @Anonymous | ‣ |
| 4) Create internal representation of the whole dag including fal scripts | Create a data structure that represents the whole dag. for example I can find out the children of a script or parents of a sql node including before scripts.
Similar to how we wrap other dbt data structures we can wrap the dbt graph created internally and add additional functionality
This will unlock usecases such as starting the dag with just a fal script, for example fal flow run --select scriptA+
should keep running the dbt nodes after the origin fal script is done. | M | Nothing | Done | @Anonymous | ‣ |
| 5) Make fal flow
work with script selectors | example:
fal flow run —select +terminalscript
example runs: val [modelA, modelB] = find parents of terminalScript.py val [before1, before2] = ask graph for before scripts of terminalScript.py
fal run —before —select before1 before2
dbt run --select +modelc +modelB
fal run
example
fal flow run —select originScript+
example runs: val [modelA, modelB, .. ] = find first children of originscript.py
fal run —before originScript
dbt run —select modelA+ modelB+
fal run
https://about.gitlab.com/handbook/business-technology/data-team/platform/dbt-guide/#command-line-cheat-sheet | M | 1, 2, 3, 4 | In Progress | @Anonymous | ‣ | | 6) Standalone documentation website | Using https://docusaurus.io/ | L | Nothing | Done | @Anonymous | | | 7) Fal feature store dbt package | A dbt only feature store implementation | L | Nothing | Done | @Anonymous | ‣ |
models:
- name: zendesk_ticket_descriptions
description: zendesk ticket descriptions
config:
materialized: table
meta:
fal:
scripts:
before:
- fal_scripts/postgres.py
after:
- fal_scripts/slack.py