๋ณธ๋ฌธ ๋ฐ”๋กœ๊ฐ€๊ธฐ
๐Ÿ’ป Programming/Apache Airflow

[Airflow] ์—์–ดํ”Œ๋กœ์šฐ ์„ค์น˜ ๋ฐ ์›น ์ธํ„ฐํŽ˜์ด์Šค ์‹คํ–‰ํ•˜๊ธฐ

by ๋ญ…์ฆค 2023. 11. 20.
๋ฐ˜์‘ํ˜•

1. Airflow ์„ค์น˜

pip install apache-airflow

 

 

2. Airflow ์„ค์ •

cd airflow
airflow db init
mkdir dags
  • ๋งŒ๋“ค์–ด์ง„ airflow ํด๋”๋กœ ๋“ค์–ด๊ฐ€์„œ
  • db๋ฅผ init ํ•ด์ฃผ๊ณ 
  • dags ํด๋”๋ฅผ ์ƒ์„ฑ
airflow users create -u admin -p admin -f Clueless -l Coder -r Admin -e admin@admin.com
  • ๊ด€๋ฆฌ์ž ๊ณ„์ • ์ƒ์„ฑ

 

 

3. Airflow ์‹คํ–‰

airflow webserver -p 8080
  • 8080 ํฌํŠธ๋กœ ์—์–ดํ”Œ๋กœ์šฐ ์‹คํ–‰
  • 'localhost:8080' ๋กœ ์ ‘์†

 

 

4. Airflow ์›น ์ธํ„ฐํŽ˜์ด์Šค

Apache Airflow ์›น์„œ๋ฒ„๋Š” Airflow ์›Œํฌํ”Œ๋กœ์šฐ์˜ ์‹œ๊ฐํ™”, ๋ชจ๋‹ˆํ„ฐ๋ง ๋ฐ ๊ด€๋ฆฌ๋ฅผ ์œ„ํ•œ ์‚ฌ์šฉ์ž ์ธํ„ฐํŽ˜์ด์Šค๋ฅผ ์ œ๊ณตํ•œ๋‹ค. ๋‹ค์–‘ํ•œ ๋ทฐ๋ฅผ ์ œ๊ณตํ•˜๊ณ  DAG๋ฅผ ์‹œ๊ฐํ™”ํ•˜์—ฌ ์‹คํ–‰ ์ค‘์ธ ์ž‘์—…์„ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๊ฑฐ๋‚˜ ์Šค์ผ€์ค„๋Ÿฌ๋ฅผ ์ œ์–ดํ•  ์ˆ˜๋„ ์žˆ๋‹ค. 

Airflow ์›น ์ธํ„ฐํŽ˜์ด์Šค ๋กœ๊ทธ์ธ ํŽ˜์ด์ง€
Airflow ์›น ์ธํ„ฐํŽ˜์ด์Šค ๋ฉ”์ธ ํŽ˜์ด์ง€

 

 

  • ๋ฉ”์ธ ํŽ˜์ด์ง€์—์„œ ์›Œํฌํ”Œ๋กœ ์ด๋ฆ„, ์Šค์ผ€์ค„, ์ตœ๊ทผ ์‹คํ–‰ํ•œ ์›Œํฌํ”Œ๋กœ ํƒœ์Šคํฌ์˜ ์ƒํƒœ ๋“ฑ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค.

 

 

 

Graph view

 

  • Graph view๋Š” DAG์˜ ํƒœ์Šคํฌ ๋‚ด์šฉ๊ณผ ํƒœ์Šคํฌ ๊ฐ„์˜ ์˜์กด์„ฑ์„ ๋ณด์—ฌ์ค€๋‹ค.

 

๋ฐ˜์‘ํ˜•