[Airflow] ์—์–ดํ”Œ๋กœ์šฐ ์„ค์น˜ ๋ฐ ์›น ์ธํ„ฐํŽ˜์ด์Šค ์‹คํ–‰ํ•˜๊ธฐ

2023. 11. 20. 14:16ยท๐Ÿ’ป Programming/Apache Airflow
๋ฐ˜์‘ํ˜•

1. Airflow ์„ค์น˜

pip install apache-airflow

 

 

2. Airflow ์„ค์ •

cd airflow
airflow db init
mkdir dags
  • ๋งŒ๋“ค์–ด์ง„ airflow ํด๋”๋กœ ๋“ค์–ด๊ฐ€์„œ
  • db๋ฅผ init ํ•ด์ฃผ๊ณ 
  • dags ํด๋”๋ฅผ ์ƒ์„ฑ
airflow users create -u admin -p admin -f Clueless -l Coder -r Admin -e admin@admin.com
  • ๊ด€๋ฆฌ์ž ๊ณ„์ • ์ƒ์„ฑ

 

 

3. Airflow ์‹คํ–‰

airflow webserver -p 8080
  • 8080 ํฌํŠธ๋กœ ์—์–ดํ”Œ๋กœ์šฐ ์‹คํ–‰
  • 'localhost:8080' ๋กœ ์ ‘์†

 

 

4. Airflow ์›น ์ธํ„ฐํŽ˜์ด์Šค

Apache Airflow ์›น์„œ๋ฒ„๋Š” Airflow ์›Œํฌํ”Œ๋กœ์šฐ์˜ ์‹œ๊ฐํ™”, ๋ชจ๋‹ˆํ„ฐ๋ง ๋ฐ ๊ด€๋ฆฌ๋ฅผ ์œ„ํ•œ ์‚ฌ์šฉ์ž ์ธํ„ฐํŽ˜์ด์Šค๋ฅผ ์ œ๊ณตํ•œ๋‹ค. ๋‹ค์–‘ํ•œ ๋ทฐ๋ฅผ ์ œ๊ณตํ•˜๊ณ  DAG๋ฅผ ์‹œ๊ฐํ™”ํ•˜์—ฌ ์‹คํ–‰ ์ค‘์ธ ์ž‘์—…์„ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๊ฑฐ๋‚˜ ์Šค์ผ€์ค„๋Ÿฌ๋ฅผ ์ œ์–ดํ•  ์ˆ˜๋„ ์žˆ๋‹ค. 

Airflow ์›น ์ธํ„ฐํŽ˜์ด์Šค ๋กœ๊ทธ์ธ ํŽ˜์ด์ง€
Airflow ์›น ์ธํ„ฐํŽ˜์ด์Šค ๋ฉ”์ธ ํŽ˜์ด์ง€

 

 

  • ๋ฉ”์ธ ํŽ˜์ด์ง€์—์„œ ์›Œํฌํ”Œ๋กœ ์ด๋ฆ„, ์Šค์ผ€์ค„, ์ตœ๊ทผ ์‹คํ–‰ํ•œ ์›Œํฌํ”Œ๋กœ ํƒœ์Šคํฌ์˜ ์ƒํƒœ ๋“ฑ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค.

 

 

 

Graph view

 

  • Graph view๋Š” DAG์˜ ํƒœ์Šคํฌ ๋‚ด์šฉ๊ณผ ํƒœ์Šคํฌ ๊ฐ„์˜ ์˜์กด์„ฑ์„ ๋ณด์—ฌ์ค€๋‹ค.

 

๋ฐ˜์‘ํ˜•

'๐Ÿ’ป Programming > Apache Airflow' ์นดํ…Œ๊ณ ๋ฆฌ์˜ ๋‹ค๋ฅธ ๊ธ€

[Airflow] DB ์ฟผ๋ฆฌ์˜ ๊ฒฐ๊ณผ๋ฅผ ๊ฐ์‹œํ•˜๊ณ , ํŠน์ • ์กฐ๊ฑด์ด ์ถฉ์กฑ๋  ๋•Œ๊นŒ์ง€ ์ž‘์—…์„ ์ผ์‹œ ์ค‘์ง€ํ•˜๋Š” ๊ธฐ๋Šฅ | SqlSensor  (1) 2023.11.20
[Airflow] Python ํ•จ์ˆ˜ ์‹คํ–‰ํ•˜๊ธฐ | PythonOperator ์‚ฌ์šฉ  (0) 2023.11.19
[Airflow] ์‰˜ ์Šคํฌ๋ฆฝํŠธ, ๋ช…๋ น์–ด ์‹คํ–‰ํ•˜๊ธฐ | BashOperator ์‚ฌ์šฉ  (0) 2023.11.19
[Airflow] ์ผ์ •ํ•œ ๊ฐ„๊ฒฉ์œผ๋กœ DAG ์‹คํ–‰ํ•˜๊ธฐ (์Šค์ผ€์ค„๋ง) | schedule_interval | cron ๊ธฐ๋ฐ˜ ์Šค์ผ€์ค„  (0) 2023.11.19
[Airflow] Airflow & DAG ์„ค๋ช…  (0) 2023.11.19
'๐Ÿ’ป Programming/Apache Airflow' ์นดํ…Œ๊ณ ๋ฆฌ์˜ ๋‹ค๋ฅธ ๊ธ€
  • [Airflow] DB ์ฟผ๋ฆฌ์˜ ๊ฒฐ๊ณผ๋ฅผ ๊ฐ์‹œํ•˜๊ณ , ํŠน์ • ์กฐ๊ฑด์ด ์ถฉ์กฑ๋  ๋•Œ๊นŒ์ง€ ์ž‘์—…์„ ์ผ์‹œ ์ค‘์ง€ํ•˜๋Š” ๊ธฐ๋Šฅ | SqlSensor
  • [Airflow] Python ํ•จ์ˆ˜ ์‹คํ–‰ํ•˜๊ธฐ | PythonOperator ์‚ฌ์šฉ
  • [Airflow] ์‰˜ ์Šคํฌ๋ฆฝํŠธ, ๋ช…๋ น์–ด ์‹คํ–‰ํ•˜๊ธฐ | BashOperator ์‚ฌ์šฉ
  • [Airflow] ์ผ์ •ํ•œ ๊ฐ„๊ฒฉ์œผ๋กœ DAG ์‹คํ–‰ํ•˜๊ธฐ (์Šค์ผ€์ค„๋ง) | schedule_interval | cron ๊ธฐ๋ฐ˜ ์Šค์ผ€์ค„
๋ญ…์ฆค
๋ญ…์ฆค
AI ๊ธฐ์ˆ  ๋ธ”๋กœ๊ทธ
    ๋ฐ˜์‘ํ˜•
  • ๋ญ…์ฆค
    CV DOODLE
    ๋ญ…์ฆค
  • ์ „์ฒด
    ์˜ค๋Š˜
    ์–ด์ œ
  • ๊ณต์ง€์‚ฌํ•ญ

    • โœจ About Me
    • ๋ถ„๋ฅ˜ ์ „์ฒด๋ณด๊ธฐ (198)
      • ๐Ÿ“– Fundamentals (33)
        • Computer Vision (9)
        • 3D vision & Graphics (6)
        • AI & ML (15)
        • NLP (2)
        • etc. (1)
      • ๐Ÿ› Research (64)
        • Deep Learning (7)
        • Image Classification (2)
        • Detection & Segmentation (17)
        • OCR (7)
        • Multi-modal (4)
        • Generative AI (6)
        • 3D Vision (2)
        • Material & Texture Recognit.. (8)
        • NLP & LLM (11)
        • etc. (0)
      • ๐ŸŒŸ AI & ML Tech (7)
        • AI & ML ์ธ์‚ฌ์ดํŠธ (7)
      • ๐Ÿ’ป Programming (85)
        • Python (18)
        • Computer Vision (12)
        • LLM (4)
        • AI & ML (17)
        • Database (3)
        • Apache Airflow (6)
        • Docker & Kubernetes (14)
        • ์ฝ”๋”ฉ ํ…Œ์ŠคํŠธ (4)
        • C++ (1)
        • etc. (6)
      • ๐Ÿ’ฌ ETC (3)
        • ์ฑ… ๋ฆฌ๋ทฐ (3)
  • ๋งํฌ

  • ์ธ๊ธฐ ๊ธ€

  • ํƒœ๊ทธ

    ๊ฐ์ฒด๊ฒ€์ถœ
    OpenCV
    ์ปดํ“จํ„ฐ๋น„์ „
    ํ”„๋กฌํ”„ํŠธ์—”์ง€๋‹ˆ์–ด๋ง
    CNN
    nlp
    segmentation
    GPT
    OCR
    ๊ฐ์ฒด ๊ฒ€์ถœ
    ๋”ฅ๋Ÿฌ๋‹
    3D Vision
    AI
    Computer Vision
    pandas
    multi-modal
    ๋„์ปค
    object detection
    VLP
    ChatGPT
    ํŒŒ์ด์ฌ
    OpenAI
    deep learning
    Python
    material recognition
    airflow
    LLM
    Text recognition
    pytorch
    Image Classification
  • ์ตœ๊ทผ ๋Œ“๊ธ€

  • ์ตœ๊ทผ ๊ธ€

  • hELLOยท Designed By์ •์ƒ์šฐ.v4.10.3
๋ญ…์ฆค
[Airflow] ์—์–ดํ”Œ๋กœ์šฐ ์„ค์น˜ ๋ฐ ์›น ์ธํ„ฐํŽ˜์ด์Šค ์‹คํ–‰ํ•˜๊ธฐ
์ƒ๋‹จ์œผ๋กœ

ํ‹ฐ์Šคํ† ๋ฆฌํˆด๋ฐ”