Homework 2: Docker, ArXiv API, Multi-Container Pipelines

Overview

This assignment introduces containerization using Docker. You will build and run containers, manage data persistence through volumes, and create multi-container applications using Docker Compose.

Getting Started

Download the starter code: hw2-starter.zip

unzip hw2-starter.zip
cd hw2-starter

Problem 1: Docker Basics – HTTP Data Fetcher

Write a Python application that fetches data from HTTP endpoints, processes the responses, and outputs structured results. You will containerize this application using Docker.

Use only Python standard library modules: sys, json, time, datetime, os, re, urllib.request. Do not use the requests library or any other third-party package.

Part A: Python HTTP Fetcher

Create a file fetch_and_process.py that fetches data from URLs and computes statistics about the responses.

Your script must accept exactly two command line arguments:

1. Path to an input file containing URLs (one per line)
2. Path to output directory

For each URL in the input file, your script must:

1. Perform an HTTP GET request to the URL
2. Measure the response time in milliseconds
3. Capture the HTTP status code
4. Calculate the size of the response body in bytes
5. Count the number of words in the response (for text responses only)

Your script must write three files to the output directory:

File 1: responses.json - Array of response data:

[
  {
    "url": "[URL string]",
    "status_code": [integer, or null on transport failure],
    "response_time_ms": [float],
    "content_length": [integer, or null on transport failure],
    "word_count": [integer, or null],
    "timestamp": "[ISO-8601 UTC]",
    "error": [null, or error message string on transport failure]
  },
  ...
]

File 2: summary.json - Aggregate statistics:

{
  "total_urls": [integer],
  "successful_requests": [integer, count of 2xx responses],
  "failed_requests": [integer, everything else],
  "average_response_time_ms": [float, over successful requests only],
  "total_bytes_downloaded": [integer, sum of content_length over all HTTP responses, any status],
  "status_code_distribution": {
    "200": [count],
    "404": [count],
    ...
  },
  "processing_start": "[ISO-8601 UTC]",
  "processing_end": "[ISO-8601 UTC]"
}

File 3: errors.log - One line per transport failure:

[ISO-8601 UTC] [URL]: [error message]

Requirements:

• Every URL produces exactly one entry in responses.json. An HTTP response (any status, including 404/500) is recorded with its status code. A transport failure (DNS failure, connection refused, timeout) has no status: record null for status_code, content_length, and word_count, put the message in error, and continue with the remaining URLs
• HTTP error statuses are data in responses.json; only transport failures appear in errors.log
• successful_requests counts 2xx responses; everything else counts as failed
• Set a timeout of 10 seconds for each request
• For word counting, a word is any maximal sequence of alphanumeric characters
• If the response Content-Type header contains “text”, perform word count; otherwise set word_count to null
• All timestamps must be UTC in ISO-8601 format with ‘Z’ suffix

Part B: Dockerfile

Create a Dockerfile that packages your Python application.

FROM python:3.11-slim
WORKDIR /app
COPY fetch_and_process.py /app/
RUN mkdir -p /data/input /data/output
ENTRYPOINT ["python", "/app/fetch_and_process.py"]
CMD ["/data/input/urls.txt", "/data/output"]

The Dockerfile must:

• Use python:3.11-slim as the base image (no other base image permitted)
• Set working directory to /app
• Copy your script to the container
• Create input and output directories at /data/input and /data/output
• Use ENTRYPOINT for the Python interpreter and script
• Use CMD for default arguments (can be overridden at runtime)

Part C: Building and Running

Build your container image:

docker build -t http-fetcher:latest .

Run your container with input/output volume mounts:

docker run --rm \
    -v "$(pwd)/test_urls.txt":/data/input/urls.txt:ro \
    -v "$(pwd)/output":/data/output \
    http-fetcher:latest

The volume mounts connect your host filesystem to the container:

• -v "$(pwd)/test_urls.txt":/data/input/urls.txt:ro mounts your input file read-only
• -v "$(pwd)/output":/data/output mounts the output directory for results
• --rm removes the container after execution

On Windows, replace $(pwd) with the full path or %cd% (cmd) / ${PWD} (PowerShell).

Part D: Testing

The starter code provides test_urls.txt:

http://httpbin.org/status/200
http://httpbin.org/delay/2
http://httpbin.org/delay/12
http://httpbin.org/status/404
http://httpbin.org/json
http://httpbin.org/html
https://www.example.com
http://httpbin.org/status/500
http://invalid.url.that.does.not.exist.com

Your application must handle all these cases correctly:

• Successful responses (200)
• Delayed responses within the timeout (/delay/2)
• Timeouts (/delay/12 exceeds the 10-second limit)
• Client errors (404)
• Server errors (500)
• JSON responses (Content-Type: application/json)
• HTML responses (Content-Type: text/html)
• Invalid URLs / DNS failures

Validation

docker build -t http-fetcher:latest .
docker run --rm \
    -v "$(pwd)/test_urls.txt":/data/input/urls.txt:ro \
    -v "$(pwd)/output":/data/output \
    http-fetcher:latest

Your container must not require network configuration beyond Docker defaults.

Deliverables

See Submission. Your README should explain:

• How you classify outcomes (2xx, non-2xx, transport failure) and where each is recorded
• How you detect timeouts

Problem 2: arXiv Paper Metadata Processor

Build a containerized application that fetches paper metadata from the arXiv API, processes it, and generates structured output.

Use only Python standard library modules: sys, json, urllib.request, xml.etree.ElementTree, datetime, time, re, os. Do not use the requests library or any other third-party package.

Part A: arXiv API Client

Create a file arxiv_processor.py that queries the arXiv API and extracts paper metadata.

Your script must accept exactly three command line arguments:

1. Search query string (e.g., “cat:cs.LG” for machine learning papers)
2. Maximum number of results to fetch (integer between 1 and 100)
3. Path to output directory

If the arguments are missing, malformed, or out of range, print a usage message to stderr and exit with code 1.

Your script must perform the following operations:

1. Query the arXiv API using the search query
2. Fetch up to the specified maximum number of results
3. Extract and process metadata for each paper
4. Generate text analysis statistics
5. Write structured output files

arXiv API endpoint: http://export.arxiv.org/api/query

Query parameters:

• search_query: Your search string
• start: Starting index (0-based)
• max_results: Maximum results to return

Example API call:

http://export.arxiv.org/api/query?search_query=cat:cs.LG&start=0&max_results=10

The API returns XML (parsing guide). You must parse this XML to extract:

• Paper ID (from the <id> tag, extract just the ID portion after the last ‘/’)
• Title (from <title>)
• Authors (from all <author><name> tags)
• Abstract (from <summary>)
• Categories (from all <category> tags’ term attribute)
• Published date (from <published>)
• Updated date (from <updated>)

Part B: Text Processing

Definitions, used throughout:

• Word: a maximal sequence of alphanumeric characters; lowercase before counting
• Sentence: abstract text split on '. ', '! ', '? '
• Technical terms are extracted from the original abstract text, before lowercasing and word splitting—case and hyphens must still be intact:
  • Uppercase term: contains an uppercase letter at any position other than the first character (“LSTM”, “ResNet50”—not “The”)
  • Numeric term: contains at least one digit (“3D”, “ResNet50”)
  • Hyphenated term: contains an internal hyphen with alphanumeric characters on both sides (“state-of-the-art”, “pre-trained”)

Per abstract (reported in papers.json under abstract_stats):

• Total and unique word counts
• Sentence count and average words per sentence
• Average word length
• Longest and shortest sentence, as word counts
• Top 20 most frequent words, excluding stopwords

Across all abstracts (reported in corpus_analysis.json):

• Top 50 words by total frequency, excluding stopwords, with document counts
• Technical terms, deduplicated and sorted
• Category distribution

Use the stopwords list provided in the starter code:

"""
Stopwords for text analysis.

Do not modify.
"""

STOPWORDS = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for',
             'of', 'with', 'by', 'from', 'up', 'about', 'into', 'through', 'during',
             'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had',
             'do', 'does', 'did', 'will', 'would', 'could', 'should', 'may', 'might',
             'can', 'this', 'that', 'these', 'those', 'i', 'you', 'he', 'she', 'it',
             'we', 'they', 'what', 'which', 'who', 'when', 'where', 'why', 'how',
             'all', 'each', 'every', 'both', 'few', 'more', 'most', 'other', 'some',
             'such', 'as', 'also', 'very', 'too', 'only', 'so', 'than', 'not'}

Part C: Output Files

Your script must write three files to the output directory:

File 1: papers.json - Array of paper metadata:

[
  {
    "arxiv_id": "[paper ID]",
    "title": "[paper title]",
    "authors": ["author1", "author2", ...],
    "abstract": "[full abstract text]",
    "categories": ["cat1", "cat2", ...],
    "published": "[ISO-8601 UTC]",
    "updated": "[ISO-8601 UTC]",
    "abstract_stats": {
      "total_words": [integer],
      "unique_words": [integer],
      "total_sentences": [integer],
      "avg_words_per_sentence": [float],
      "avg_word_length": [float],
      "longest_sentence_words": [integer],
      "shortest_sentence_words": [integer],
      "top_20_words": [
        {"word": "[word]", "count": [integer]},
        ...
      ]
    }
  },
  ...
]

File 2: corpus_analysis.json - Aggregate analysis across all papers:

{
  "query": "[search query used]",
  "papers_processed": [integer],
  "processing_timestamp": "[ISO-8601 UTC]",
  "corpus_stats": {
    "total_abstracts": [integer],
    "total_words": [integer],
    "unique_words_global": [integer],
    "avg_abstract_length": [float, in words],
    "longest_abstract_words": [integer],
    "shortest_abstract_words": [integer]
  },
  "top_50_words": [
    {"word": "[word1]", "frequency": [count], "documents": [count]},
    ...
  ],
  "technical_terms": {
    "uppercase_terms": ["TERM1", "TERM2", ...],
    "numeric_terms": ["term1", "term2", ...],
    "hyphenated_terms": ["term-1", "term-2", ...]
  },
  "category_distribution": {
    "cs.LG": [count],
    "cs.AI": [count],
    ...
  }
}

File 3: processing.log - Processing log with one line per event:

[ISO-8601 UTC] Starting ArXiv query: [query]
[ISO-8601 UTC] Fetched [N] results from ArXiv API
[ISO-8601 UTC] Processing paper: [arxiv_id]
[ISO-8601 UTC] Completed processing: [N] papers in [X.XX] seconds

Part D: Error Handling

Your script must handle the following error conditions:

1. Network errors: If the arXiv API is unreachable, write the error to the log and exit with code 1
2. Invalid XML: If a paper’s entry is malformed, log the error, skip that paper, and continue with the remaining papers—do not exit
3. Missing fields: If a paper lacks required fields, skip it and log a warning
4. Rate limiting: If you receive HTTP 429, wait 3 seconds and retry (maximum 3 attempts)

Requirements:

• All word processing must be case-insensitive for frequency counting
• Preserve original case in technical terms
• Handle Unicode (arXiv abstracts contain mathematical symbols)

Part E: Dockerfile

Create a Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY arxiv_processor.py stopwords.py /app/
RUN mkdir -p /data/output
ENTRYPOINT ["python", "/app/arxiv_processor.py"]

Part F: Building and Running

Build your container image:

docker build -t arxiv-processor:latest .

Run your container:

docker run --rm \
    -v "$(pwd)/output":/data/output \
    arxiv-processor:latest \
    "cat:cs.LG" 10 /data/output

The arguments are passed directly to your Python script:

• "cat:cs.LG" - the search query
• 10 - maximum results to fetch
• /data/output - output directory inside the container

On Windows, replace $(pwd) with the full path or %cd% (cmd) / ${PWD} (PowerShell).

Part G: Testing

Test your container with various queries:

# Machine Learning papers
docker run --rm -v "$(pwd)/output_ml":/data/output \
    arxiv-processor:latest "cat:cs.LG" 5 /data/output

# Search by author
docker run --rm -v "$(pwd)/output_author":/data/output \
    arxiv-processor:latest "au:LeCun" 3 /data/output

# Search by title keyword
docker run --rm -v "$(pwd)/output_title":/data/output \
    arxiv-processor:latest "ti:transformer" 10 /data/output

Validation

docker build -t arxiv-processor:latest .
docker run --rm \
    -v "$(pwd)/output":/data/output \
    arxiv-processor:latest \
    "cat:cs.LG" 10 /data/output

Your container must respect arXiv’s rate limits and terms of service. Do not make more than 1 request per 3 seconds to avoid being blocked.

Deliverables

See Submission. Your README should explain:

• How your tokenization implements the Part B definitions, and any edge cases (hyphens, Unicode)
• How you distinguish fatal from non-fatal errors

Problem 3: Multi-Container Text Processing Pipeline with Docker Compose

Build a multi-container application that processes web content through sequential stages. Containers coordinate through a shared filesystem, demonstrating batch processing patterns used in data pipelines.

Use only Python standard library modules in process.py and analyze.py: json, os, re, sys, time, datetime, collections. Do not use BeautifulSoup, lxml, or any other HTML parsing library—extraction is regex-based.

Architecture

Three containers process data in sequence:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   fetcher   │────▶│  processor  │────▶│  analyzer   │
└─────────────┘     └─────────────┘     └─────────────┘
       │                   │                   │
       ▼                   ▼                   ▼
    /shared/            /shared/            /shared/
    └── raw/           └── processed/      └── analysis/
    └── status/        └── status/         └── status/

Containers communicate through filesystem markers:

• Each container monitors /shared/status/ for its input signal
• Processing stages write completion markers when finished
• Data flows through /shared/ subdirectories

Definitions, used by both stages:

• Word: a maximal sequence of alphanumeric characters; lowercase before counting
• Sentence: text split on '. ', '! ', '? '
• Paragraph: one <p> element in the source HTML

Part A: Container 1 - Data Fetcher

The fetcher is provided in the starter code. Study this code to understand the coordination pattern.

#!/usr/bin/env python3
"""
Data Fetcher - downloads URLs and writes to shared volume.

Do not modify.
"""
import json
import os
import sys
import time
import urllib.request
from datetime import datetime, timezone

def main():
    print(f"[{datetime.now(timezone.utc).isoformat()}] Fetcher starting", flush=True)

    # Wait for input file
    input_file = "/shared/input/urls.txt"
    os.makedirs("/shared/input", exist_ok=True)
    while not os.path.exists(input_file):
        print(f"Waiting for {input_file}...", flush=True)
        time.sleep(2)

    # Read URLs
    with open(input_file, 'r') as f:
        urls = [line.strip() for line in f if line.strip()]

    # Create output directory
    os.makedirs("/shared/raw", exist_ok=True)
    os.makedirs("/shared/status", exist_ok=True)

    # Fetch each URL
    results = []
    for i, url in enumerate(urls, 1):
        output_file = f"/shared/raw/page_{i}.html"
        try:
            print(f"Fetching {url}...", flush=True)
            with urllib.request.urlopen(url, timeout=10) as response:
                content = response.read()
                with open(output_file, 'wb') as f:
                    f.write(content)
            results.append({
                "url": url,
                "file": f"page_{i}.html",
                "size": len(content),
                "status": "success"
            })
        except Exception as e:
            results.append({
                "url": url,
                "file": None,
                "error": str(e),
                "status": "failed"
            })
        time.sleep(1)  # Rate limiting

    # Write completion status
    status = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "urls_processed": len(urls),
        "successful": sum(1 for r in results if r["status"] == "success"),
        "failed": sum(1 for r in results if r["status"] == "failed"),
        "results": results
    }

    with open("/shared/status/fetch_complete.json", 'w') as f:
        json.dump(status, f, indent=2)

    print(f"[{datetime.now(timezone.utc).isoformat()}] Fetcher complete", flush=True)

if __name__ == "__main__":
    main()

Create fetcher/Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY fetch.py /app/
CMD ["python", "-u", "/app/fetch.py"]

The -u flag disables output buffering so log lines appear as they are written.

Part B: Container 2 - HTML Processor

Create processor/process.py that extracts and analyzes text from HTML files.

Required processing operations:

1. Wait for /shared/status/fetch_complete.json
2. Read all HTML files from /shared/raw/
3. Extract text content using regex (not BeautifulSoup)
4. Extract all links (href attributes)
5. Extract all images (src attributes)
6. Count words, sentences, paragraphs
7. Save processed data to /shared/processed/
8. Create /shared/status/process_complete.json

Text extraction requirements:

def strip_html(html_content):
    """Remove HTML tags and extract text."""
    # Remove script and style elements
    html_content = re.sub(r'<script[^>]*>.*?</script>', '', html_content, flags=re.DOTALL | re.IGNORECASE)
    html_content = re.sub(r'<style[^>]*>.*?</style>', '', html_content, flags=re.DOTALL | re.IGNORECASE)

    # Extract links before removing tags
    links = re.findall(r'href=[\'"]?([^\'" >]+)', html_content, flags=re.IGNORECASE)

    # Extract images
    images = re.findall(r'src=[\'"]?([^\'" >]+)', html_content, flags=re.IGNORECASE)

    # Remove HTML tags
    text = re.sub(r'<[^>]+>', ' ', html_content)

    # Clean whitespace
    text = re.sub(r'\s+', ' ', text).strip()

    return text, links, images

Count paragraphs from the markup, before tags are stripped—strip_html collapses the whitespace and tags that mark paragraph boundaries.

Output format for each processed file (/shared/processed/page_N.json):

{
    "source_file": "page_N.html",
    "text": "[extracted text]",
    "statistics": {
        "word_count": [integer],
        "sentence_count": [integer],
        "paragraph_count": [integer],
        "avg_word_length": [float]
    },
    "links": ["url1", "url2", ...],
    "images": ["src1", "src2", ...],
    "processed_at": "[ISO-8601 UTC]"
}

Create processor/Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY process.py /app/
CMD ["python", "-u", "/app/process.py"]

Part C: Container 3 - Text Analyzer

Create analyzer/analyze.py that performs corpus-wide analysis.

Required analysis operations:

1. Wait for /shared/status/process_complete.json
2. Read all processed files from /shared/processed/
3. Compute global statistics:
   • Word frequency distribution (top 100 words)
   • Document similarity for every document pair (Jaccard similarity)
   • Top 20 bigrams and top 20 trigrams (consecutive words in each document’s word sequence)
   • Average sentence length and average word length, as global totals: all words over all sentences, all characters over all words
4. Save to /shared/analysis/final_report.json

Similarity calculation, over each document’s set of unique words:

def jaccard_similarity(doc1_words, doc2_words):
    """Calculate Jaccard similarity between two documents."""
    set1 = set(doc1_words)
    set2 = set(doc2_words)
    intersection = set1.intersection(set2)
    union = set1.union(set2)
    return len(intersection) / len(union) if union else 0.0

Report every unordered pair once, with doc1 before doc2 in sorted filename order. In top_100_words, frequency is count divided by total_words.

Final report structure (/shared/analysis/final_report.json):

{
    "processing_timestamp": "[ISO-8601 UTC]",
    "documents_processed": [integer],
    "total_words": [integer],
    "unique_words": [integer],
    "top_100_words": [
        {"word": "the", "count": 523, "frequency": 0.042},
        ...
    ],
    "document_similarity": [
        {"doc1": "page_1.json", "doc2": "page_2.json", "similarity": 0.234},
        ...
    ],
    "top_bigrams": [
        {"bigram": "machine learning", "count": 45},
        ...
    ],
    "top_trigrams": [
        {"trigram": "convolutional neural network", "count": 12},
        ...
    ],
    "readability": {
        "avg_sentence_length": [float],
        "avg_word_length": [float]
    }
}

Create analyzer/Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY analyze.py /app/
CMD ["python", "-u", "/app/analyze.py"]

Part D: Docker Compose Configuration

The docker-compose.yaml is provided in the starter code:

# Pipeline services. Do not modify.
services:
  fetcher:
    build: ./fetcher
    container_name: pipeline-fetcher
    volumes:
      - pipeline-data:/shared
    environment:
      - PYTHONUNBUFFERED=1

  processor:
    build: ./processor
    container_name: pipeline-processor
    volumes:
      - pipeline-data:/shared
    environment:
      - PYTHONUNBUFFERED=1
    depends_on:
      - fetcher

  analyzer:
    build: ./analyzer
    container_name: pipeline-analyzer
    volumes:
      - pipeline-data:/shared
    environment:
      - PYTHONUNBUFFERED=1
    depends_on:
      - processor

volumes:
  pipeline-data:
    name: pipeline-shared-data

depends_on orders container startup; it does not wait for a container to finish. Your scripts must wait for their inputs themselves.

Part E: Orchestration Script

The run_pipeline.sh orchestration script is provided in the starter code. It handles building containers, starting the pipeline, injecting URLs, monitoring for completion, and extracting results.

#!/bin/bash
#
# Pipeline orchestration script. Do not modify.
#

if [ $# -lt 1 ]; then
    echo "Usage: $0 <url1> [url2] [url3] ..."
    echo "Example: $0 https://example.com https://wikipedia.org"
    exit 1
fi

echo "Starting Multi-Container Pipeline"
echo "================================="

# Clean previous runs
docker compose down -v 2>/dev/null

# Create temporary directory
TEMP_DIR=$(mktemp -d)
trap "rm -rf $TEMP_DIR" EXIT

# Create URL list
for url in "$@"; do
    echo "$url" >> "$TEMP_DIR/urls.txt"
done

echo "URLs to process:"
cat "$TEMP_DIR/urls.txt"
echo ""

# Build containers
echo "Building containers..."
docker compose build --quiet

# Start pipeline
echo "Starting pipeline..."
docker compose up -d

# Wait for containers to initialize
sleep 3

# Inject URLs
echo "Injecting URLs..."
docker exec pipeline-fetcher mkdir -p /shared/input
docker cp "$TEMP_DIR/urls.txt" pipeline-fetcher:/shared/input/urls.txt

# Monitor completion
echo "Processing..."
MAX_WAIT=300  # 5 minutes timeout
ELAPSED=0

while [ $ELAPSED -lt $MAX_WAIT ]; do
    # The analyzer exits when finished; docker cp works on a stopped container.
    if [ "$(docker inspect -f '{{.State.Status}}' pipeline-analyzer 2>/dev/null)" = "exited" ]; then
        echo "Pipeline complete"
        break
    fi
    sleep 5
    ELAPSED=$((ELAPSED + 5))
done

if [ $ELAPSED -ge $MAX_WAIT ]; then
    echo "Pipeline timeout after ${MAX_WAIT} seconds"
    docker compose logs
    docker compose down
    exit 1
fi

# Extract results
mkdir -p output
docker cp pipeline-analyzer:/shared/analysis/final_report.json output/
docker cp pipeline-analyzer:/shared/status output/

# Cleanup
docker compose down

# Display summary
if [ -f "output/final_report.json" ]; then
    echo ""
    echo "Results saved to output/final_report.json"
    python3 -m json.tool output/final_report.json | head -20
else
    echo "Pipeline failed - no output generated"
    exit 1
fi

On Windows, you will need to run this script using WSL2, Git Bash, or translate the commands to PowerShell.

Part F: Testing

URLs are passed to the pipeline as script arguments:

# Single URL
./run_pipeline.sh https://www.example.com

# Multiple URLs
./run_pipeline.sh https://www.example.com https://www.wikipedia.org https://httpbin.org/html

Include a failing URL to confirm the pipeline completes when a fetch fails.

Debugging

To diagnose pipeline issues:

1. View container logs:

   docker compose logs fetcher
   docker compose logs processor
   docker compose logs analyzer
   

2. Inspect shared volume:

   docker run --rm -v pipeline-shared-data:/shared alpine ls -la /shared/
   

3. Check container status:

   docker compose ps
   

4. Enter a container (while it is running):

   docker exec -it pipeline-fetcher /bin/bash
   

Validation

docker compose build
./run_pipeline.sh https://www.example.com https://httpbin.org/html

Deliverables

See Submission. Your README should explain:

• How each stage waits for its input signal, and what happens if the signal never arrives
• How you count paragraphs and where in your extraction order it happens

README.md
q1/
├── fetch_and_process.py
├── Dockerfile
├── test_urls.txt
└── README.md
q2/
├── arxiv_processor.py
├── stopwords.py
├── Dockerfile
└── README.md
q3/
├── docker-compose.yaml
├── run_pipeline.sh
├── README.md
├── fetcher/
│   ├── Dockerfile
│   └── fetch.py
├── processor/
│   ├── Dockerfile
│   └── process.py
└── analyzer/
    ├── Dockerfile
    └── analyze.py