seth-public/ws4kp-to-hls
1
0
Fork 0
Code Activity

Initial Commit

Browse Source
This commit is contained in:
Seth Van Niekerk
2025-11-07 13:43:26 -05:00
commit b2c65553a6
9 changed files with 1906 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.dockerignore Normal file
View File

@@ -0,0 +1,5 @@
node_modules
.git
.env
*.log
.DS_Store

4
.gitignore vendored Normal file
View File

@@ -0,0 +1,4 @@
node_modules
*.log
.DS_Store
.env

33
Dockerfile Normal file
View File

@@ -0,0 +1,33 @@
FROM ghcr.io/mwood77/ws4kp-international:latest
# Install FFmpeg, Chromium, wget, and unzip
RUN apk add --no-cache \
chromium \
ffmpeg \
font-noto-emoji \
wget \
unzip
# Download and extract Weatherscan music
RUN mkdir -p /music && \
wget -O /tmp/weatherscan.zip "https://archive.org/compress/weatherscancompletecollection/formats=OGG%20VORBIS&file=/weatherscancompletecollection.zip" && \
unzip -j /tmp/weatherscan.zip "*.ogg" -d /music && \
rm /tmp/weatherscan.zip
# Set Puppeteer to use installed Chromium
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser \
WS4KP_PORT=8080 \
MUSIC_PATH=/music
# Install our streaming app
WORKDIR /streaming-app
COPY package.json yarn.lock* ./
RUN npm install -g yarn && yarn install --frozen-lockfile || yarn install
COPY index.js ./
# Expose both ports
EXPOSE 3000 8080
# Start both services
CMD cd /app && node index.js & cd /streaming-app && yarn start

175
README.md Normal file
View File

@@ -0,0 +1,175 @@
# Webpage to HLS Stream
Simple Dockerized service that converts any webpage to an HLS stream using Puppeteer and FFmpeg. Includes built-in ws4kp weather display.
**Docker Image**: `ghcr.io/sethwv/ws4kp-to-hls:latest` (multi-platform: AMD64, ARM64)
## Quick Start
### Using Docker Compose (recommended)
```bash
# Build and start
docker-compose up --build
# Or pull pre-built image
docker-compose pull
docker-compose up
# The service runs on http://localhost:3000
# ws4kp interface available at http://localhost:8080
```
### Using Docker directly
```bash
docker run -p 3000:3000 -p 8080:8080 ghcr.io/sethwv/ws4kp-to-hls:latest
```
**Note**: Initial build takes an additional ~90 seconds due to downloading the Weatherscan music collection (~500MB).
## Usage
### Stream any webpage
```bash
# Basic usage
http://localhost:3000/stream?url=http://example.com
# With custom dimensions and framerate
http://localhost:3000/stream?url=http://example.com&width=1280&height=720&fps=25
```
### Stream weather (ws4kp)
The service includes a built-in ws4kp weather display with automatic geocoding and background music:
```bash
# Default location (Toronto, ON)
http://localhost:3000/weather
# Custom location (automatically geocoded via OpenStreetMap)
http://localhost:3000/weather?city=New+York,NY,USA
http://localhost:3000/weather?city=London,UK
http://localhost:3000/weather?city=Tokyo,Japan
# With imperial units (Fahrenheit, mph, miles, inHg)
http://localhost:3000/weather?city=Miami,FL,USA&units=imperial
# With 12-hour time format
http://localhost:3000/weather?city=New+York,NY,USA&units=imperial&timeFormat=12h
# Show only current weather and radar (hide other forecasts)
http://localhost:3000/weather?city=Miami,FL,USA&showHourly=false&showExtendedForecast=false
# Minimal display (just current weather)
http://localhost:3000/weather?city=Toronto,ON,Canada&showHourly=false&showRadar=false&showExtendedForecast=false&showLocalForecast=false
# With custom resolution
http://localhost:3000/weather?city=Miami,FL,USA&width=1280&height=720&fps=25
# Hide the ws4kp logo
http://localhost:3000/weather?city=Toronto,ON,Canada&hideLogo=true
```
**City Format**: Use `City,State,Country` format for best accuracy (e.g., `Toronto,ON,Canada` or `Miami,FL,USA`). The service automatically geocodes the city using OpenStreetMap's Nominatim API and falls back to Toronto coordinates if geocoding fails.
**Units**: Use `units=metric` (default) for Celsius/kph/km/mb or `units=imperial` for Fahrenheit/mph/miles/inHg.
**Time Format**: Use `timeFormat=24h` (default) for 24-hour time or `timeFormat=12h` for 12-hour AM/PM format.
**Forecast Sections**: Control which sections are displayed using `show*` parameters (set to `true` or `false`). See Parameters section below for full list.
**Background Music**: The weather endpoint includes shuffled background music from the Weatherscan collection, with tracks automatically changing when one ends.
**Logo Control**: Use `hideLogo=true` to hide the ws4kp Logo3.png image from the stream.
### Consume with FFmpeg
```bash
# Play the stream
ffplay "http://localhost:3000/stream?url=http://example.com"
# Save to file
ffmpeg -i "http://localhost:3000/stream?url=http://example.com" -c copy output.mp4
# Re-stream to RTMP
ffmpeg -i "http://localhost:3000/stream?url=http://example.com" \
-c:v copy -f flv rtmp://your-server/live/stream
```
### Parameters
**Stream endpoint (`/stream`)**:
- `url` (required): The webpage URL to capture
- `width` (optional): Video width in pixels (default: 1920)
- `height` (optional): Video height in pixels (default: 1080)
- `fps` (optional): Frames per second (default: 30)
- `hideLogo` (optional): Set to `true` to hide Logo3.png image (default: false)
**Weather endpoint (`/weather`)**:
- `city` (optional): City name in `City,State,Country` format (default: Toronto,ON,Canada)
- Examples: `Toronto,ON,Canada`, `New+York,NY,USA`, `London,UK`
- Automatically geocoded via OpenStreetMap Nominatim
- `units` (optional): Unit system - `metric` or `imperial` (default: metric)
- `metric`: Celsius, kph, km, mb
- `imperial`: Fahrenheit, mph, miles, inHg
- `timeFormat` (optional): Time format - `12h` or `24h` (default: 24h)
- `width` (optional): Video width in pixels (default: 1920)
- `height` (optional): Video height in pixels (default: 1080)
- `fps` (optional): Frames per second (default: 30)
- `hideLogo` (optional): Set to `true` to hide ws4kp Logo3.png image (default: false)
- **Forecast Section Toggles** (all optional, use `true` or `false`):
- **Commonly used sections** (default: true):
- `showHazards`: Weather alerts and hazards
- `showCurrent`: Current weather conditions
- `showHourly`: Hourly forecast
- `showHourlyGraph`: Hourly forecast graph
- `showLocalForecast`: Local forecast
- `showExtendedForecast`: Extended forecast (7-day)
- `showRadar`: Weather radar
- `showAQI`: Air Quality Index
- `showAlmanac`: Weather almanac (historical data, records)
- `showLatestObservations`: Real-time data from nearby stations
- `showRegionalForecast`: Regional forecast for surrounding areas
- **Specialized sections** (default: false):
- `showTravel`: Travel forecast for major destinations
- `showMarineForecast`: Marine/coastal forecast
## How It Works
1. Puppeteer opens the webpage in headless Chrome/Chromium
2. Screenshots are captured at the specified FPS using a sequential loop with backpressure handling
3. FFmpeg encodes the screenshots into an HLS stream (H.264 video, AAC audio for weather)
4. For weather streams: background music is shuffled and played from the Weatherscan collection
5. The HLS stream is piped directly to the HTTP response
6. City names are automatically geocoded to coordinates via OpenStreetMap's Nominatim API
## Development
```bash
# Install dependencies
yarn install
# Run locally (requires Chrome and FFmpeg)
yarn start
```
## Health Check
```bash
curl http://localhost:3000/health
```
## Notes
- The stream continues indefinitely until the client disconnects
- Each request creates a new browser instance
- Memory usage scales with the number of concurrent streams
- Weather streams include automatic geocoding (adds ~200-500ms initial latency)
- Background music for weather streams is shuffled from a collection of Weatherscan tracks
- Multi-platform Docker image available for AMD64 and ARM64 architectures
## License
MIT

9
build-commands.txt Normal file
View File

@@ -0,0 +1,9 @@
# Docker Build Commands
## Build and push multi-platform image (AMD64 + ARM64)
docker buildx build --platform linux/amd64,linux/arm64 -t ghcr.io/sethwv/ws4kp-to-hls:latest --push .
## Build without pushing (for testing)
docker buildx build --platform linux/amd64,linux/arm64 -t ghcr.io/sethwv/ws4kp-to-hls:latest .

11
docker-compose.yml Normal file
View File

@@ -0,0 +1,11 @@
services:
app:
build: .
ports:
- "3000:3000"
- "8080:8080"
shm_size: 2gb
environment:
- PORT=3000
- WS4KP_PORT=8080
restart: unless-stopped

487
index.js Normal file
View File

@@ -0,0 +1,487 @@
const express = require('express');
const puppeteer = require('puppeteer');
const { spawn } = require('child_process');
const path = require('path');
const fs = require('fs');
const https = require('https');
const app = express();
const PORT = process.env.PORT || 3000;
const MUSIC_PATH = process.env.MUSIC_PATH || '/music';
// Geocode city to lat/lon using Nominatim (OpenStreetMap)
async function geocodeCity(cityQuery) {
return new Promise((resolve, reject) => {
const encodedQuery = encodeURIComponent(cityQuery);
const url = `https://nominatim.openstreetmap.org/search?q=${encodedQuery}&format=json&limit=1`;
const options = {
headers: {
'User-Agent': 'webpage-to-hls-streaming-app/1.0'
}
};
https.get(url, options, (res) => {
let data = '';
res.on('data', (chunk) => {
data += chunk;
});
res.on('end', () => {
try {
const results = JSON.parse(data);
if (results && results.length > 0) {
resolve({
lat: parseFloat(results[0].lat),
lon: parseFloat(results[0].lon),
displayName: results[0].display_name
});
} else {
reject(new Error('No results found'));
}
} catch (error) {
reject(error);
}
});
}).on('error', (error) => {
reject(error);
});
});
}
// Get random music file
function getRandomMusicFile() {
try {
if (!fs.existsSync(MUSIC_PATH)) {
return null;
}
const files = fs.readdirSync(MUSIC_PATH).filter(f => f.endsWith('.ogg'));
if (files.length === 0) {
return null;
}
const randomFile = files[Math.floor(Math.random() * files.length)];
return path.join(MUSIC_PATH, randomFile);
} catch (error) {
console.error('Error getting music file:', error);
return null;
}
}
// Get all music files for shuffling
function getAllMusicFiles() {
try {
if (!fs.existsSync(MUSIC_PATH)) {
return [];
}
const files = fs.readdirSync(MUSIC_PATH).filter(f => f.endsWith('.ogg'));
return files.map(f => path.join(MUSIC_PATH, f));
} catch (error) {
console.error('Error getting music files:', error);
return [];
}
}
// Shuffle array in place
function shuffleArray(array) {
for (let i = array.length - 1; i > 0; i--) {
const j = Math.floor(Math.random() * (i + 1));
[array[i], array[j]] = [array[j], array[i]];
}
return array;
}
// Main streaming handler
async function streamHandler(req, res, useMusic = false) {
const { url, width = 1920, height = 1080, fps = 30, hideLogo = 'false' } = req.query;
if (!url) {
return res.status(400).send('URL parameter is required');
}
// Validate URL
try {
new URL(url);
} catch (error) {
return res.status(400).send('Invalid URL');
}
let browser = null;
let ffmpegProcess = null;
let isCleaningUp = false;
try {
console.log(`Starting stream for: ${url}`);
// Set HLS headers
res.setHeader('Content-Type', 'application/vnd.apple.mpegurl');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
// Launch browser
browser = await puppeteer.launch({
headless: true,
executablePath: process.env.PUPPETEER_EXECUTABLE_PATH || undefined,
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-gpu',
`--window-size=${width},${height}`
],
defaultViewport: { width: parseInt(width), height: parseInt(height) }
});
const page = await browser.newPage();
await page.setViewport({ width: parseInt(width), height: parseInt(height) });
await page.goto(url, { waitUntil: 'networkidle2', timeout: 30000 });
// Hide logo if requested
if (hideLogo === 'true') {
await page.evaluate(() => {
const images = document.querySelectorAll('img');
images.forEach(img => {
if (img.src && img.src.includes('Logo3.png')) {
img.style.display = 'none';
}
});
});
console.log('Logo hidden');
}
console.log('Page loaded, starting FFmpeg...');
// Build FFmpeg command with optional music
const ffmpegArgs = [];
let playlistFile = null;
if (useMusic) {
// Get all music files and shuffle them
const allMusicFiles = getAllMusicFiles();
if (allMusicFiles.length > 0) {
const shuffledFiles = shuffleArray([...allMusicFiles]);
// Create a temporary concat playlist file
playlistFile = path.join('/tmp', `playlist-${Date.now()}.txt`);
// Build playlist content - repeat the shuffled list multiple times to ensure we don't run out
let playlistContent = '';
for (let repeat = 0; repeat < 100; repeat++) {
// Re-shuffle for each repeat to keep it truly random
const currentShuffle = shuffleArray([...allMusicFiles]);
currentShuffle.forEach(file => {
playlistContent += `file '${file}'\n`;
});
}
fs.writeFileSync(playlistFile, playlistContent);
console.log(`Created shuffled playlist with ${allMusicFiles.length} unique tracks`);
// Input 0: video frames
ffmpegArgs.push(
'-use_wallclock_as_timestamps', '1',
'-f', 'image2pipe',
'-framerate', fps.toString(),
'-i', 'pipe:0'
);
// Input 1: audio from concat playlist
ffmpegArgs.push(
'-f', 'concat',
'-safe', '0',
'-i', playlistFile
);
// Encoding
ffmpegArgs.push(
'-c:v', 'libx264',
'-preset', 'ultrafast',
'-tune', 'zerolatency',
'-pix_fmt', 'yuv420p',
'-g', (fps * 2).toString(),
'-c:a', 'aac',
'-b:a', '128k',
'-shortest',
'-f', 'hls',
'-hls_time', '2',
'-hls_list_size', '5',
'-hls_flags', 'delete_segments',
'pipe:1'
);
} else {
console.warn('No music files found, streaming without audio');
useMusic = false;
}
}
if (!useMusic) {
// Video only (no music)
ffmpegArgs.push(
'-use_wallclock_as_timestamps', '1',
'-f', 'image2pipe',
'-framerate', fps.toString(),
'-i', 'pipe:0',
'-c:v', 'libx264',
'-preset', 'ultrafast',
'-tune', 'zerolatency',
'-pix_fmt', 'yuv420p',
'-g', (fps * 2).toString(),
'-f', 'hls',
'-hls_time', '2',
'-hls_list_size', '5',
'-hls_flags', 'delete_segments',
'pipe:1'
);
}
// Start FFmpeg
ffmpegProcess = spawn('ffmpeg', ffmpegArgs);
// Pipe FFmpeg output to response
ffmpegProcess.stdout.pipe(res);
ffmpegProcess.stderr.on('data', (data) => {
// Suppress verbose FFmpeg output
// console.error(`FFmpeg: ${data}`);
});
ffmpegProcess.on('error', (error) => {
console.error('FFmpeg error:', error);
cleanup();
});
ffmpegProcess.stdin.on('error', (error) => {
// Ignore EPIPE errors when client disconnects
if (error.code !== 'EPIPE') {
console.error('FFmpeg stdin error:', error);
}
});
// Capture frames using a sequential loop (avoids overlapping screenshots)
// and handle backpressure on ffmpeg.stdin (wait for 'drain' with timeout).
const frameInterval = 1000 / fps;
let captureLoopActive = true;
const captureLoop = async () => {
while (!isCleaningUp && captureLoopActive && ffmpegProcess && !ffmpegProcess.killed) {
const start = Date.now();
try {
// Take a screenshot (lower quality a bit to reduce CPU/pipe pressure)
const screenshot = await page.screenshot({ type: 'jpeg', quality: 70 });
if (ffmpegProcess && ffmpegProcess.stdin.writable && !isCleaningUp) {
const canWrite = ffmpegProcess.stdin.write(screenshot);
if (!canWrite) {
// Backpressure — wait for drain but don't block forever
await new Promise((resolve) => {
let resolved = false;
const onDrain = () => { if (!resolved) { resolved = true; cleanupListeners(); resolve(); } };
const onError = () => { if (!resolved) { resolved = true; cleanupListeners(); resolve(); } };
const timeout = setTimeout(() => { if (!resolved) { resolved = true; cleanupListeners(); resolve(); } }, 800);
function cleanupListeners() {
clearTimeout(timeout);
ffmpegProcess.stdin.removeListener('drain', onDrain);
ffmpegProcess.stdin.removeListener('error', onError);
}
ffmpegProcess.stdin.once('drain', onDrain);
ffmpegProcess.stdin.once('error', onError);
});
}
}
} catch (error) {
if (!isCleaningUp) {
console.error('Capture error:', error.message || error);
try { await cleanup(); } catch (e) {}
break;
}
}
const elapsed = Date.now() - start;
const wait = Math.max(0, frameInterval - elapsed);
// Wait the remaining time before next frame
if (wait > 0) await new Promise(r => setTimeout(r, wait));
}
};
// Start the capture loop (no overlapping runs)
captureLoop();
// Cleanup function
const cleanup = async () => {
if (isCleaningUp) return;
isCleaningUp = true;
// stop capture loop
try { captureLoopActive = false; } catch (e) {}
if (ffmpegProcess && !ffmpegProcess.killed) {
try {
ffmpegProcess.stdin.end();
} catch (err) {
// Ignore errors during cleanup
}
ffmpegProcess.kill('SIGTERM');
ffmpegProcess = null;
}
// Clean up temporary playlist file
if (playlistFile) {
try {
fs.unlinkSync(playlistFile);
} catch (err) {
// Ignore errors during cleanup
}
}
if (browser) {
try {
await browser.close();
} catch (err) {
// Ignore errors during cleanup
}
browser = null;
}
if (!res.headersSent && !res.writableEnded) {
res.end();
}
};
// Handle client disconnect
req.on('close', () => {
console.log('Client disconnected');
cleanup();
});
res.on('error', (error) => {
console.error('Response error:', error);
cleanup();
});
} catch (error) {
console.error('Error:', error);
if (ffmpegProcess && !ffmpegProcess.killed) ffmpegProcess.kill();
if (browser) await browser.close();
if (!res.headersSent) {
res.status(500).send('Internal server error');
}
}
}
// Stream endpoint
app.get('/stream', (req, res) => streamHandler(req, res, false));
app.get('/weather', async (req, res) => {
const {
city = 'Toronto, ON, CAN',
width = 1920,
height = 1080,
fps = 30,
hideLogo = 'false',
units = 'metric', // 'metric' or 'imperial'
timeFormat = '24h', // '12h' or '24h'
// Forecast section toggles - commonly used sections (default: true)
showHazards = 'true',
showCurrent = 'true',
showHourly = 'true',
showHourlyGraph = 'true',
showLocalForecast = 'true',
showExtendedForecast = 'true',
showRadar = 'true',
showAQI = 'true',
showAlmanac = 'true',
showLatestObservations = 'true',
showRegionalForecast = 'true',
// Less common sections (default: false)
showTravel = 'false',
showMarineForecast = 'false'
} = req.query;
// Default coordinates (Toronto, ON, Canada)
let lat = 43.6532;
let lon = -79.3832;
// Try to geocode the city if provided
if (city) {
try {
console.log(`Geocoding city: ${city}`);
const geoResult = await geocodeCity(city);
lat = geoResult.lat;
lon = geoResult.lon;
console.log(`Geocoded to: ${geoResult.displayName} (${lat}, ${lon})`);
} catch (error) {
console.warn(`Geocoding failed for "${city}", using default coordinates:`, error.message);
// Fall back to defaults
}
}
// Unit conversions for ws4kp
// Temperature: 1.00=Celsius, 2.00=Fahrenheit
// Wind: 1.00=kph, 2.00=mph
// Distance: 1.00=km, 2.00=miles
// Pressure: 1.00=mb, 2.00=inHg
// Hours: 1.00=12h, 2.00=24h
const isMetric = units.toLowerCase() === 'metric';
const temperatureUnit = isMetric ? '1.00' : '2.00';
const windUnit = isMetric ? '1.00' : '2.00';
const distanceUnit = isMetric ? '1.00' : '2.00';
const pressureUnit = isMetric ? '1.00' : '2.00';
const hoursFormat = timeFormat === '12h' ? '1.00' : '2.00';
// Build the ws4kp URL with all the parameters
// ws4kp runs on port 8080 inside the container
const ws4kpBaseUrl = process.env.WS4KP_URL || 'http://localhost:8080';
const ws4kpParams = new URLSearchParams({
'hazards-checkbox': showHazards,
'current-weather-checkbox': showCurrent,
'latest-observations-checkbox': showLatestObservations,
'hourly-checkbox': showHourly,
'hourly-graph-checkbox': showHourlyGraph,
'travel-checkbox': showTravel,
'regional-forecast-checkbox': showRegionalForecast,
'local-forecast-checkbox': showLocalForecast,
'extended-forecast-checkbox': showExtendedForecast,
'almanac-checkbox': showAlmanac,
'radar-checkbox': showRadar,
'marine-forecast-checkbox': showMarineForecast,
'aqi-forecast-checkbox': showAQI,
'settings-experimentalFeatures-checkbox': 'false',
'settings-hideWebamp-checkbox': 'true',
'settings-kiosk-checkbox': 'false',
'settings-scanLines-checkbox': 'false',
'settings-wide-checkbox': 'true',
'chkAutoRefresh': 'true',
'settings-windUnits-select': windUnit,
'settings-marineWindUnits-select': '1.00',
'settings-marineWaveHeightUnits-select': '1.00',
'settings-temperatureUnits-select': temperatureUnit,
'settings-distanceUnits-select': distanceUnit,
'settings-pressureUnits-select': pressureUnit,
'settings-hoursFormat-select': hoursFormat,
'settings-speed-select': '1.00',
'latLonQuery': city,
'latLon': JSON.stringify({ lat: lat, lon: lon }),
'kiosk': 'true'
});
const weatherUrl = `${ws4kpBaseUrl}/?${ws4kpParams.toString()}`;
console.log(`Weather stream requested for: ${city} (${lat}, ${lon})`);
// Forward to the main stream endpoint WITH MUSIC
req.query.url = weatherUrl;
req.query.width = width;
req.query.height = height;
req.query.fps = fps;
req.query.hideLogo = hideLogo;
// Call the stream handler with music enabled
return streamHandler(req, res, true);
});
app.get('/health', (req, res) => {
res.send('OK');
});
app.listen(PORT, () => {
console.log(`Webpage to HLS server running on port ${PORT}`);
console.log(`Usage: http://localhost:${PORT}/stream?url=http://example.com`);
console.log(`Weather: http://localhost:${PORT}/weather?city=YourCity`);
});

16
package.json Normal file
View File

@@ -0,0 +1,16 @@
{
"name": "webpage-to-hls",
"version": "1.0.0",
"description": "Simple webpage to HLS streaming service",
"main": "index.js",
"scripts": {
"start": "node index.js"
},
"keywords": ["puppeteer", "ffmpeg", "hls", "streaming"],
"author": "",
"license": "MIT",
"dependencies": {
"express": "^4.18.2",
"puppeteer": "^24.15.0"
}
}

1166
yarn.lock Normal file
View File

File diff suppressed because it is too large Load Diff