我创建了一个docker compose-file
,其中定义了Spark
、Airflow
、Postgres
和Flower
。在我的docker-compose up -d
中执行cmd
之后,我得到以下输出:
[+] Running 11/11
- Network airflow_spark_default Created 0.8s
- Container airflow_spark-redis-1 Started 3.7s
- Container airflow_spark-spark-worker-1 Started 2.9s
- Container airflow_spark-postgres-1 Started 3.8s
- Container airflow_spark-spark-1 Started 4.1s
- Container airflow_spark-airflow-init-1 Started 9.5s
- Container airflow_spark-airflow-triggerer-1 Started 26.7s
- Container airflow_spark-flower-1 Started 27.4s
- Container airflow_spark-airflow-worker-1 Started 26.4s
- Container airflow_spark-airflow-scheduler-1 Started 26.4s
- Container airflow_spark-airflow-webserver-1 Started 27.0s
一切似乎都运行正常。所以,当我冲浪到localhost:5555
时,我得到了UI
of Flower
,当我冲浪到localhost:8080
时,我得到了Airflow
的UI
。但当我在localhost:4141
上冲浪时,我会得到谷歌Chrome版的This page is not working
。火花是怎么回事?
Spark
服务是如何在我的docker-compose file
中定义的
#Spark
spark:
image: docker.io/bitnami/spark:3
environment:
- SPARK_MODE=master
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
ports:
- '4141:4141'
volumes:
- ./spark-apps:/opt/spark-apps
spark-worker:
image: docker.io/bitnami/spark:3
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark:7077
- SPARK_WORKER_MEMORY=1G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
发布于 2022-01-29 13:37:17
星火图像在端口8080上公开web接口,因此要访问它,必须将端口8080映射到主机端口之一。例如:- 8888:8080
在ports:
部件下将web-ui映射到localhost:8888。
https://stackoverflow.com/questions/70891552
复制相似问题