We're planning to use Dagster for Fundraising Analytics. We would like to run this on a baremetal Debian server. We need to develop install, operation and maintenance, and security monitoring and maintenance.
Basic install to a python venv, from the docs:
source dagster_venv/bin/activate pip install dagster dagster-dbt dagster-webserver dbt-duckdb dbt-mysql
We're proposing to use a daily pip-audit run to monitor for security updates, this should report on everything installed by pip.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Open | None | T367817 Investigate Dagster packaging, install, security monitoring. |
The basic install of dagster and its dependencies has been puppetized. Puppet also configures a pip-audit script to check for updates in the virtualenv and emails with success (clean audit) or an alert (patches are available). There's no project configuration in puppet yet, this part will be developed with BDC.
I think a single project makes sense for our context. I've seen two approaches to importing the project, one where it's installed to the venv and the other where a .py is called at startup like "-f test_project.py". I'm not sure for our case which is preferred or how frequently this config-as-code will change. We could template it all from puppet (infrequent changes), or we could use frdeploy tools (frequent changes) plus some puppet+pip processing.
For Debian Bullseye, the following pip packages get installed to the Dagster venv:
Package Version ------------------ -------------- ABBA 0.1.0 acme 1.12.0 appdirs 1.4.4 argon2-cffi 18.3.0 astroid 2.5.1 attrs 20.3.0 backcall 0.2.0 bleach 3.2.1 blinker 1.4 cachetools 4.2.1 certbot 1.12.0 certifi 2020.6.20 chardet 4.0.0 click 7.1.2 colorama 0.4.4 ConfigArgParse 1.2.3 configobj 5.0.6 cov-core 1.15.0 coverage 5.1 croniter 0.3.34 cryptography 3.3.2 cycler 0.10.0 decorator 4.4.2 defusedxml 0.6.0 distlib 0.3.1 distro 1.5.0 Django 2.2.28 entrypoints 0.3 fasteners 0.14.1 filelock 3.0.12 geoip2 2.9.0 gevent 20.9.0 greenlet 0.4.17 html5lib 1.1 httplib2 0.18.1 idna 2.10 importlib-metadata 1.6.0 iniconfig 0.0.0 iotop 0.6 ipykernel 5.4.3 ipython 7.20.0 ipython-genutils 0.2.0 isort 5.6.4 jedi 0.18.0 Jinja2 2.11.3 joblib 0.17.0 josepy 1.2.0 jsonschema 3.2.0 jupyter-client 6.1.11 jupyter-console 6.2.0 jupyter-core 4.7.1 kiwisolver 1.3.1 lazy-object-proxy 0.0.0 logilab-common 1.8.1 MarkupSafe 1.1.1 matplotlib 3.3.4 maxminddb 2.0.3 mccabe 0.6.1 mistune 0.8.4 monotonic 1.5 more-itertools 4.2.0 mypy-extensions 0.4.3 mysqlclient 1.4.4 natsort 7.1.0 nbconvert 5.6.1 nbformat 5.1.2 nltk 3.5 nose2 0.9.2 notebook 6.2.0 numpy 1.19.5 oauthlib 3.1.0 packaging 20.9 pandas 1.1.5 pandocfilters 1.4.3 parsedatetime 2.6 parso 0.8.1 pathspec 0.8.1 pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.1.2 pip 20.3.4 pluggy 0.13.0 process-control 1.0.5 prometheus-client 0.9.0 prompt-toolkit 3.0.14 protobuf 3.12.4 py 1.10.0 pycurl 7.43.0.6 Pygments 2.7.1 PyJWT 1.7.1 pylint 2.7.2 PyMySQL 0.9.3 pyOpenSSL 20.0.1 pyparsing 2.4.7 pyRFC3339 1.1 pyrsistent 0.15.5 PySimpleSOAP 1.16.2 pytest 6.0.2 pytest-cov 2.10.1 python-apt 2.2.1 python-dateutil 2.8.1 python-debian 0.1.39 python-debianbts 3.1.0 pytz 2021.1 PyYAML 5.3.1 pyzmq 20.0.0 regex 2020.11.13 reportbug 7.10.3+deb11u1 requests 2.25.1 requests-oauthlib 1.0.0 requests-toolbelt 0.9.1 scikit-learn 0.23.2 scipy 1.6.0 seaborn 0.11.1 Send2Trash 1.6.0b1 setuptools 52.0.0 six 1.16.0 SQLAlchemy 1.3.22 sqlparse 0.4.1 terminado 0.9.2 testpath 0.4.4 toml 0.10.1 tornado 6.1 tqdm 4.57.0 traitlets 5.0.5 typing-extensions 3.7.4.3 tzlocal 2.1 uritemplate 3.0.1 urllib3 1.26.5 virtualenv 20.4.0+ds wcwidth 0.1.9 webencodings 0.5.1 wheel 0.34.2 wrapt 1.12.1 yamllint 1.26.0 zipp 1.0.0 zope.component 4.3.0 zope.event 4.4 zope.hookable 5.0.1 zope.interface 5.2.0
Updated list:
agate==1.7.1
alembic==1.13.2
aniso8601==9.0.1
annotated-types==0.7.0
anyio==4.4.0
attrs==23.2.0
Babel==2.15.0
backoff==2.2.1
boolean.py==4.0
CacheControl==0.14.0
certifi==2024.6.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
coloredlogs==14.0
croniter==2.0.5
cyclonedx-python-lib==7.4.1
dagster==1.7.10
dagster-dbt==0.23.10
dagster-graphql==1.7.10
dagster-mysql==0.23.10
dagster-pipes==1.7.10
dagster-webserver==1.7.10
dbt-core==1.7.17
dbt-duckdb==1.7.5
dbt-extractor==0.5.1
dbt-mysql==1.7.0
dbt-semantic-interfaces==0.4.4
defusedxml==0.7.1
docstring_parser==0.16
duckdb==1.0.0
exceptiongroup==1.2.1
filelock==3.15.4
fsspec==2024.6.0
gql==3.5.0
graphene==3.3
graphql-core==3.2.3
graphql-relay==3.2.0
greenlet==3.0.3
grpcio==1.64.1
grpcio-health-checking==1.62.2
h11==0.14.0
html5lib==1.1
httptools==0.6.1
humanfriendly==10.0
idna==3.7
importlib-metadata==6.11.0
isodate==0.6.1
Jinja2==3.1.4
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
leather==0.4.0
license-expression==30.3.0
Logbook==1.5.3
Mako==1.3.5
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mashumaro==3.13.1
mdurl==0.1.2
minimal-snowplow-tracker==0.0.2
more-itertools==10.3.0
msgpack==1.0.8
multidict==6.0.5
mysql-connector-python==8.4.0
networkx==3.2.1
orjson==3.10.5
packageurl-python==0.15.1
packaging==24.1
parsedatetime==2.6
pathspec==0.11.2
pendulum==3.0.0
pip==24.1
pip-api==0.0.33
pip-requirements-parser==32.0.1
pip_audit==2.7.3
pkg_resources==0.0.0
protobuf==4.25.3
py-serializable==1.0.3
pycparser==2.22
pydantic==2.7.4
pydantic_core==2.18.4
Pygments==2.18.0
pyparsing==3.1.2
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2024.1
PyYAML==6.0.1
referencing==0.35.1
requests==2.32.3
requests-toolbelt==1.0.0
rich==13.7.1
rpds-py==0.18.1
setuptools==70.1.1
shellingham==1.5.4
six==1.16.0
sniffio==1.3.1
sortedcontainers==2.4.0
SQLAlchemy==2.0.31
sqlglot==25.3.2
sqlglotrs==0.2.8
sqlparse==0.5.0
starlette==0.37.2
structlog==24.2.0
tabulate==0.9.0
text-unidecode==1.3
time-machine==2.14.1
toml==0.10.2
tomli==2.0.1
toposort==1.10
tqdm==4.66.4
typer==0.12.3
typing_extensions==4.12.2
tzdata==2024.1
universal_pathlib==0.2.2
urllib3==1.26.19
uvicorn==0.30.1
uvloop==0.19.0
watchdog==4.0.1
watchfiles==0.22.0
webencodings==0.5.1
websockets==12.0
wheel==0.43.0
yarl==1.9.4
zipp==3.19.2