* feat: add stream output feature to SSE job updates Adds stream_output field to JobUpdate struct that extracts log lines starting with '[wm_stream]:' from job logs. Regular logs now exclude stream lines, which are captured separately for specialized handling. - Added stream_output: Option<String> field to JobUpdate struct - Created extract_stream_output_from_logs() function to filter stream lines - Modified get_job_update_data() to use stream extraction logic - SSE clients now receive both new_logs and stream_output in job updates Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com> * feat: rename stream_output to stream and handle newlines as \n - Renamed `stream_output` field to `stream` in JobUpdate struct - Updated extract_stream_output_from_logs to extract_stream_from_logs - Changed stream output to join with literal \n instead of actual newlines - Stream lines are properly excluded from regular new_logs 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Ruben Fiszel <rubenfiszel@users.noreply.github.com> * decision tree nits * push ee ref * push ee ref * fix: fix id renaming in apps * remove duplicate caching (#6285) * feat: migrate audit log ids to bigints (blocking migration for EE) * fix(mcp): add proper check for mcp routes (#6282) * add proper check for mcp routes * cleaner * apply to flow * fix add checks scopes --------- Co-authored-by: dieriba <dieriba.pro@gmail.com> * chore(main): release 1.514.0 (#6283) * chore(main): release 1.514.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix: pin tokio to 1.46.1 and aws-sdks-ts * pin rustls to 0.23.29 + pin aws-sdk * chore(main): release 1.514.1 (#6288) * chore(main): release 1.514.1 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix: improve docker logs collection in docker mode * support $res: string in form inputs of arrays * fix import nit * fix: fix DynSelect * nits * fix: resource-type-ts-parser (#6289) * fix: resource types as arg in typescript handle imported defined types * Update nix flake (#6291) * merge * Small UI fixes (#6294) * fix step history not refreshing with staticInputs * fix array of obj not showing up in json editor in test this step * datatable scales correctly in DisplayResult and scrolling is much more usable * avoid next button disapearing and changing layout / hurting ux * nits * fix bug when renaming module A to B then module C to A, C takes the schema of A * fix bug with comments in sql repl * fix aggrid theme randomly not loading * bindable script * better delete button in db manager * property select doesnt exist * fix all warnings * delete $flowStateStore[id] on delete * feat(cli): generate cursor rules on init (#6270) * create cursor rules on init * change gen * add missing resource-type command * add resource type command in guidance * add schema option * revert * nit * nit * add flow guidance * nit * chore(main): release 1.515.0 (#6292) * chore(main): release 1.515.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix: improved logs for script * nits logs * chore(main): release 1.515.1 (#6295) * chore(main): release 1.515.1 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * merge * even more indexer tracings * add more tracing logs * feat: prevent too large results (>500Mb) from OOMing database * nit naming * feat: add CA certificate update at startup via environment variable (#6280) * feat: add CA certificate update at startup via environment variable Add support for running 'update-ca-certificates' at binary startup when RUN_UPDATE_CA_CERTIFICATE_AT_START environment variable is set to "true". - Check for RUN_UPDATE_CA_CERTIFICATE_AT_START env var on startup - Execute update-ca-certificates command if env var is set to "true" - Log success/failure appropriately with tracing - Continue startup even if CA certificate update fails - Non-blocking implementation with proper error handling Fixes #6279 Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com> * refactor: extract CA certificate update logic into separate function Extract the CA certificate update logic from windmill_main() into a dedicated update_ca_certificates_if_requested() function for better code organization and maintainability. Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com> * improvements --------- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com> Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com> Co-authored-by: Alexander Petric <alex@windmill.dev> * fix: indexer collection of job logs before indexing (#6300) * Add flume as dependecy for indexer * Update ee-repo-ref * Remove flags from cargo.toml * Update ee-repo-ref * Update ee-repo-ref * fix rust sdk build error (#6305) Signed-off-by: pyranota <pyra@duck.com> * fix broken audit logs filter (#6304) * rename to from to * goto fix * default to false if field not present operator settings (#6301) * git sync UI improvements (#6303) * ui improvements round 1 * modal cleanup * init * UI refactor * UI cleanup + refactor * legacy cleanup * success model -> github actions, non-ee warnings * sqlx * npm check * ee warning everywhere * last comments * formatting * no hardcoded theme * claude review improvemenets * fix: no process relative imports for scripts with codebase * fix: sqs oidc authentication disconnect #6307 * handle metadata for new scripts happen after commit * handle_deployment_metadata in a task * nits * chore: add windmill-utils-internal package (#6299) * add utils package * naming * cleaning * add docs * remove log * use autogenerated types * remove old * fix * cleaning * add docs * chore(main): release 1.516.0 (#6298) * chore(main): release 1.516.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * merge * indexer improvements * upgrade tantivy to 0.24.2 * use tantivy fork * nit warnings * fix oss build * improve indexer * chore: use windmill-utils-internal for cli (#6297) * add utils package * naming * cleaning * simplify assignPath * rename old files * same for locks * create on confirm * default true * use replaceinlinescripts from utils * use extractscriptfromflows * make it compile * cleaning * use argsigtojson * fix * fix missing await * cleaner * cleaning * cleaning * use in frontend * add docs * testing * remove log * use autogenerated types * remove old * fix * cleaning * adapt usage * draft * better build script * fix build * revert to default creation * add docs * remove and rename * make everything work * add await * only if not installed * add vs code setting * add to publish action * fix bc * safer use of sep * fix * do not rename on push * no publish on release * use published package on frontend * nit * Add dependencies to run sqlx prepare to nix flake (#6309) * feat(cli): wmill-lock.yaml v2 for easier git merge diffs * merge * merge * all * all * rm warnings * fix styling on aichatinput (#6312) * fix: use with_capacity back presusre for tantivy directory multipart writes (#6313) * use with capacity for tantivy directory multi part uploads * Update ee repo ref * Update ee-repo-ref * Update ee-repo-ref * chore(main): release 1.517.0 (#6310) * chore(main): release 1.517.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix typo on cli build (#6314) * cleanup * feat(utils): add flow.yaml validation function (#6316) * add validateflow function * cleaner code * preprocess json * cleaning * create specific package * cleaning * add tests * fix: cleanup concurrency_counter automatically + remove orphans keys automatically * fix: add disabled support to resource picker in schema forms * fix: add wm_labels to tracing spans * all * merge * all * fix: delete empty git connection (#6318) * fix checks * bun handling * all * all? * all * all * update * all * update * check * fix history * all * all * all * Remove leftover debug tracing statements - Remove commented debug trace in jobs.rs for stream output - Remove commented debug trace in result_stream.rs for stream processing Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com> * fix test * all * handle iter * fix --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: centdix <40307056+centdix@users.noreply.github.com> Co-authored-by: dieriba <dieriba.pro@gmail.com> Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> Co-authored-by: wendrul <53628737+wendrul@users.noreply.github.com> Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com> Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com> Co-authored-by: Alexander Petric <alex@windmill.dev> Co-authored-by: pyranota <92104930+pyranota@users.noreply.github.com>
925 lines
30 KiB
Rust
925 lines
30 KiB
Rust
use std::{collections::HashMap, path::PathBuf, process::Stdio, sync::Arc};
|
|
|
|
use anyhow::{anyhow, bail};
|
|
use async_recursion::async_recursion;
|
|
use itertools::Itertools;
|
|
use serde_json::value::RawValue;
|
|
use tokio::{
|
|
fs::{create_dir_all, metadata, remove_dir_all, File},
|
|
io::AsyncWriteExt,
|
|
process::Command,
|
|
};
|
|
use uuid::Uuid;
|
|
use windmill_common::{
|
|
error::{self, Error},
|
|
utils::calculate_hash,
|
|
worker::{copy_dir_recursively, save_cache, write_file, Connection},
|
|
};
|
|
use windmill_parser::Arg;
|
|
use windmill_parser_java::parse_java_sig_meta;
|
|
use windmill_queue::{append_logs, CanceledBy, MiniPulledJob};
|
|
|
|
use crate::{
|
|
common::{
|
|
create_args_and_out_file, get_reserved_variables, par_install_language_dependencies,
|
|
read_result, start_child_process, OccupancyMetrics, RequiredDependency,
|
|
},
|
|
handle_child, COURSIER_CACHE_DIR, DISABLE_NSJAIL, DISABLE_NUSER, JAVA_CACHE_DIR,
|
|
JAVA_REPOSITORY_DIR, MAVEN_REPOS, NO_DEFAULT_MAVEN, NSJAIL_PATH, PATH_ENV, PROXY_ENVS,
|
|
};
|
|
use windmill_common::client::AuthedClient;
|
|
|
|
lazy_static::lazy_static! {
|
|
static ref JAVA_CONCURRENT_DOWNLOADS: usize = std::env::var("JAVA_CONCURRENT_DOWNLOADS").ok().map(|flag| flag.parse().unwrap_or(20)).unwrap_or(20);
|
|
static ref JAVA_PATH: String = std::env::var("JAVA_PATH").unwrap_or_else(|_| "/usr/bin/java".to_string());
|
|
static ref JAVAC_PATH: String = std::env::var("JAVAC_PATH").unwrap_or_else(|_| "/usr/bin/javac".to_string());
|
|
static ref CS_PATH: String = std::env::var("COURSIER_PATH").unwrap_or_else(|_| "/usr/bin/coursier".to_string());
|
|
static ref STOREPASS: String = std::env::var("JAVA_STOREPASS").unwrap_or("123456".into());
|
|
static ref TRUST_STORE_PATH: String = std::env::var("JAVA_TRUST_STORE_PATH").unwrap_or("/usr/local/share/ca-certificates/truststore.jks".into());
|
|
}
|
|
|
|
const NSJAIL_CONFIG_RUN_JAVA_CONTENT: &str = include_str!("../nsjail/run.java.config.proto");
|
|
|
|
#[allow(dead_code)]
|
|
pub(crate) struct JobHandlerInput<'a> {
|
|
pub base_internal_url: &'a str,
|
|
pub canceled_by: &'a mut Option<CanceledBy>,
|
|
pub client: &'a AuthedClient,
|
|
pub parent_runnable_path: Option<String>,
|
|
pub conn: &'a Connection,
|
|
pub envs: HashMap<String, String>,
|
|
pub inner_content: &'a str,
|
|
pub job: &'a MiniPulledJob,
|
|
pub job_dir: &'a str,
|
|
pub mem_peak: &'a mut i32,
|
|
pub occupancy_metrics: &'a mut OccupancyMetrics,
|
|
pub requirements_o: Option<&'a String>,
|
|
pub shared_mount: &'a str,
|
|
pub worker_name: &'a str,
|
|
}
|
|
|
|
pub async fn handle_java_job<'a>(mut args: JobHandlerInput<'a>) -> Result<Box<RawValue>, Error> {
|
|
// --- Prepare ---
|
|
{
|
|
prepare(&mut args).await?;
|
|
}
|
|
// --- Generate Lockfile ---
|
|
|
|
let deps = resolve(
|
|
&args.job.id,
|
|
&args.inner_content,
|
|
&args.job_dir,
|
|
&args.conn,
|
|
&args.job.workspace_id,
|
|
)
|
|
.await?;
|
|
|
|
// --- Install ---
|
|
|
|
let classpath = install(&mut args, deps).await?;
|
|
|
|
// --- Build .java files ---
|
|
{
|
|
compile(&mut args, &classpath).await?;
|
|
}
|
|
// --- Run ---
|
|
{
|
|
run(&mut args, &classpath).await?;
|
|
}
|
|
// --- Retrieve results ---
|
|
{
|
|
read_result(&args.job_dir, None).await
|
|
}
|
|
}
|
|
|
|
async fn prepare<'a>(
|
|
JobHandlerInput { job, conn, job_dir, client, inner_content, .. }: &mut JobHandlerInput<'a>,
|
|
) -> Result<(), Error> {
|
|
// Create needed files
|
|
{
|
|
create_args_and_out_file(&client, job, job_dir, conn).await?;
|
|
let app_path = format!("{}/src/main/java/net/script/", job_dir);
|
|
create_dir_all(&app_path).await?;
|
|
File::create(format!("{app_path}/App.java"))
|
|
.await?
|
|
.write_all(&wrap(inner_content)?.into_bytes())
|
|
.await?;
|
|
File::create(format!("{app_path}/Main.java"))
|
|
.await?
|
|
.write_all(
|
|
&format!(
|
|
"package net.script;\n{MINI_CLIENT_IMPORTS}\n{}\n{MINI_CLIENT}",
|
|
inner_content
|
|
)
|
|
.into_bytes(),
|
|
)
|
|
.await?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
pub async fn resolve<'a>(
|
|
job_id: &Uuid,
|
|
code: &str,
|
|
job_dir: &str,
|
|
conn: &Connection,
|
|
w_id: &str,
|
|
) -> Result<String, Error> {
|
|
let deps = {
|
|
let find_requirements = code.lines().find_position(|x| {
|
|
x.starts_with("//requirements:") || x.starts_with("// requirements:")
|
|
});
|
|
|
|
let specified_deps = if let Some((pos, _)) = find_requirements {
|
|
code.lines()
|
|
.skip(pos + 1)
|
|
.map_while(|x| {
|
|
if x.starts_with("//") {
|
|
Some(x.replace("//", "").trim().to_owned())
|
|
} else {
|
|
None
|
|
}
|
|
})
|
|
.collect::<Vec<String>>()
|
|
} else {
|
|
Default::default()
|
|
};
|
|
|
|
let mut deps = vec![
|
|
// Default requirements
|
|
"com.fasterxml.jackson.core:jackson-databind:2.9.8".to_owned(),
|
|
];
|
|
deps.extend(specified_deps);
|
|
deps.join("\n")
|
|
};
|
|
|
|
let req_hash = format!("java-{}", calculate_hash(&deps));
|
|
if let Connection::Sql(db) = conn {
|
|
if let Some(cached) = sqlx::query_scalar!(
|
|
"SELECT lockfile FROM pip_resolution_cache WHERE hash = $1",
|
|
req_hash
|
|
)
|
|
.fetch_optional(db)
|
|
.await?
|
|
{
|
|
return Ok(cached);
|
|
}
|
|
}
|
|
let lock = {
|
|
append_logs(
|
|
job_id,
|
|
w_id,
|
|
format!("\n--- RESOLVING LOCKFILE ---\n"),
|
|
&conn,
|
|
)
|
|
.await;
|
|
|
|
let mut cmd = Command::new(if cfg!(windows) {
|
|
"java"
|
|
} else {
|
|
JAVA_PATH.as_str()
|
|
});
|
|
cmd.env_clear()
|
|
.current_dir(job_dir.to_owned())
|
|
.env("PATH", PATH_ENV.as_str())
|
|
.envs(PROXY_ENVS.clone());
|
|
|
|
// Configure proxies
|
|
{
|
|
let jps = parse_proxy()?;
|
|
if let Some(val) = jps.https_host {
|
|
cmd.arg(&format!("-Dhttps.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.https_port {
|
|
cmd.arg(&format!("-Dhttps.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.http_host {
|
|
cmd.arg(&format!("-Dhttp.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.http_port {
|
|
cmd.arg(&format!("-Dhttp.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.no_proxy {
|
|
cmd.arg(&format!("-Dhttp.nonProxyHosts=\"{}\"", val));
|
|
}
|
|
}
|
|
if metadata(TRUST_STORE_PATH.clone()).await.is_ok() {
|
|
cmd.args(&[
|
|
&format!("-Djavax.net.ssl.trustStore={}", *TRUST_STORE_PATH),
|
|
&format!("-Djavax.net.ssl.trustStorePassword={}", *STOREPASS),
|
|
]);
|
|
}
|
|
cmd.args(&[
|
|
"-jar",
|
|
&CS_PATH,
|
|
"resolve",
|
|
&get_no_default(),
|
|
"--parallel",
|
|
&format!("{}", *JAVA_CONCURRENT_DOWNLOADS),
|
|
"--cache",
|
|
COURSIER_CACHE_DIR,
|
|
])
|
|
.args(&get_repos().await)
|
|
.args(&deps.split("\n").collect_vec())
|
|
.stderr(Stdio::piped());
|
|
|
|
#[cfg(windows)]
|
|
{
|
|
cmd.env("SystemRoot", crate::SYSTEM_ROOT.as_str())
|
|
.env("USERPROFILE", crate::USERPROFILE_ENV.as_str())
|
|
.env(
|
|
"TMP",
|
|
std::env::var("TMP").unwrap_or_else(|_| String::from("/tmp")),
|
|
);
|
|
}
|
|
let output = cmd.output().await?;
|
|
// Check if the command was successful
|
|
if output.status.success() {
|
|
String::from_utf8(output.stdout).expect("Failed to convert output to String")
|
|
} else {
|
|
let stderr =
|
|
String::from_utf8(output.stderr).expect("Failed to convert error output to String");
|
|
return Err(error::Error::internal_err(stderr));
|
|
}
|
|
};
|
|
|
|
if let Connection::Sql(db) = conn {
|
|
sqlx::query!(
|
|
"INSERT INTO pip_resolution_cache (hash, lockfile, expiration) VALUES ($1, $2, now() + ('5 mins')::interval) ON CONFLICT (hash) DO UPDATE SET lockfile = $2",
|
|
req_hash,
|
|
lock.clone(),
|
|
)
|
|
.fetch_optional(db)
|
|
.await?;
|
|
}
|
|
|
|
append_logs(job_id, w_id, format!("\n{}", &lock), &conn).await;
|
|
Ok(lock)
|
|
}
|
|
|
|
async fn install<'a>(
|
|
JobHandlerInput { worker_name, job, conn, job_dir, .. }: &mut JobHandlerInput<'a>,
|
|
deps: String,
|
|
) -> Result<String, Error> {
|
|
let deps = deps
|
|
.lines()
|
|
.map(|line| {
|
|
let unparsed_dep = line.replace(":jar", "").replace(":lib", "");
|
|
let mut it = unparsed_dep.split(":");
|
|
|
|
match (it.next(), it.next(), it.next()) {
|
|
(Some(group_id), Some(artifact_id), Some(version)) => {
|
|
let path = format!(
|
|
"{JAVA_REPOSITORY_DIR}/{}/{artifact_id}/{version}",
|
|
group_id.replace(".", "/")
|
|
);
|
|
Ok(RequiredDependency {
|
|
path,
|
|
custom_name: Some(format!("{group_id}:{artifact_id}:{version}")),
|
|
short_name: Some(format!("{artifact_id}:{version}")),
|
|
})
|
|
}
|
|
_ => anyhow::bail!("{line} is not parsable"),
|
|
}
|
|
})
|
|
.collect::<anyhow::Result<Vec<RequiredDependency>>>()?;
|
|
|
|
let classpath = deps
|
|
.clone()
|
|
.into_iter()
|
|
.map(|RequiredDependency { path, .. }| path + "/*")
|
|
.collect_vec()
|
|
.join(":")
|
|
+ ":target";
|
|
|
|
#[cfg(windows)]
|
|
let classpath = classpath.replace(":", ";");
|
|
|
|
tracing::debug!(
|
|
workspace_id = %job.workspace_id,
|
|
"JAVA classpath: {}", &classpath
|
|
);
|
|
let (repos, no_default, trust_store_metadata) = (
|
|
get_repos().await,
|
|
get_no_default(),
|
|
metadata(TRUST_STORE_PATH.clone()).await,
|
|
);
|
|
let job_dir = job_dir.to_owned();
|
|
let fetch_dir = format!("{JAVA_CACHE_DIR}/tmp-fetch-{}", Uuid::new_v4());
|
|
let fetch_dir2 = fetch_dir.clone();
|
|
par_install_language_dependencies(
|
|
deps,
|
|
"java",
|
|
"java",
|
|
true,
|
|
*JAVA_CONCURRENT_DOWNLOADS,
|
|
true,
|
|
crate::common::InstallStrategy::AllAtOnce(Arc::new(move |dependencies| {
|
|
let mut cmd = Command::new(if cfg!(windows) {
|
|
"java"
|
|
} else {
|
|
JAVA_PATH.as_str()
|
|
});
|
|
let artifacts = dependencies
|
|
.into_iter()
|
|
.map(|e| {
|
|
e.custom_name.ok_or(anyhow::anyhow!(
|
|
"Internal Error: Artifact name should be Some!"
|
|
))
|
|
})
|
|
.collect::<anyhow::Result<Vec<String>>>()?;
|
|
cmd.env_clear()
|
|
.current_dir(&job_dir)
|
|
.env("PATH", PATH_ENV.as_str())
|
|
.envs(PROXY_ENVS.clone());
|
|
// Configure proxies
|
|
{
|
|
let jps = parse_proxy()?;
|
|
if let Some(val) = jps.https_host {
|
|
cmd.arg(&format!("-Dhttps.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.https_port {
|
|
cmd.arg(&format!("-Dhttps.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.http_host {
|
|
cmd.arg(&format!("-Dhttp.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.http_port {
|
|
cmd.arg(&format!("-Dhttp.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.no_proxy {
|
|
cmd.arg(&format!("-Dhttp.nonProxyHosts=\"{}\"", val));
|
|
}
|
|
}
|
|
|
|
if trust_store_metadata.is_ok() {
|
|
cmd.args(&[
|
|
&format!("-Djavax.net.ssl.trustStore={}", *TRUST_STORE_PATH),
|
|
&format!("-Djavax.net.ssl.trustStorePassword={}", *STOREPASS),
|
|
]);
|
|
}
|
|
cmd.args(&[
|
|
"-jar",
|
|
&CS_PATH,
|
|
"fetch",
|
|
&no_default,
|
|
"--quiet",
|
|
"--parallel",
|
|
&format!("{}", *JAVA_CONCURRENT_DOWNLOADS),
|
|
"--cache",
|
|
&fetch_dir,
|
|
])
|
|
.args(&repos)
|
|
.arg("--intransitive")
|
|
.args(artifacts)
|
|
.stdout(Stdio::piped())
|
|
.stderr(Stdio::piped());
|
|
#[cfg(windows)]
|
|
{
|
|
cmd.env("SystemRoot", crate::SYSTEM_ROOT.as_str())
|
|
.env("USERPROFILE", crate::USERPROFILE_ENV.as_str())
|
|
.env(
|
|
"TMP",
|
|
std::env::var("TMP").unwrap_or_else(|_| String::from("/tmp")),
|
|
);
|
|
}
|
|
|
|
Ok(cmd)
|
|
})),
|
|
async move |_| {
|
|
move_to_repository(&fetch_dir2, 0).await?;
|
|
remove_dir_all(&fetch_dir2).await?;
|
|
#[async_recursion]
|
|
async fn move_to_repository(path: &str, depth: u8) -> anyhow::Result<()> {
|
|
if depth == 3 {
|
|
copy_dir_recursively(
|
|
&PathBuf::from(path),
|
|
&PathBuf::from(JAVA_REPOSITORY_DIR),
|
|
)?;
|
|
|
|
return Ok(());
|
|
}
|
|
let mut entries = tokio::fs::read_dir(path).await?;
|
|
loop {
|
|
let Some(entry) = entries.next_entry().await? else {
|
|
break Ok(());
|
|
};
|
|
|
|
let path = entry
|
|
.path()
|
|
.to_str()
|
|
.ok_or(anyhow!("Internal Error: Cannot convert Path to Str"))?
|
|
.to_owned();
|
|
|
|
move_to_repository(&path, depth + 1).await?;
|
|
}
|
|
}
|
|
Ok(())
|
|
},
|
|
&job.id,
|
|
&job.workspace_id,
|
|
worker_name,
|
|
conn,
|
|
)
|
|
.await?;
|
|
Ok(classpath)
|
|
}
|
|
|
|
async fn compile<'a>(
|
|
JobHandlerInput {
|
|
occupancy_metrics,
|
|
mem_peak,
|
|
canceled_by,
|
|
worker_name,
|
|
job,
|
|
conn,
|
|
job_dir,
|
|
client,
|
|
envs,
|
|
base_internal_url,
|
|
inner_content,
|
|
requirements_o,
|
|
parent_runnable_path,
|
|
..
|
|
}: &mut JobHandlerInput<'a>,
|
|
classpath: &'a str,
|
|
// plugins: Vec<&'a str>,
|
|
) -> Result<(), Error> {
|
|
fn compute_hash(code: &str, requirements_o: Option<&String>) -> String {
|
|
calculate_hash(&format!(
|
|
"{}{}",
|
|
code,
|
|
requirements_o
|
|
.as_ref()
|
|
.map(|x| x.to_string())
|
|
.unwrap_or_default()
|
|
))
|
|
}
|
|
let reserved_variables =
|
|
get_reserved_variables(job, &client.token, conn, parent_runnable_path.clone()).await?;
|
|
let hash = compute_hash(inner_content, *requirements_o);
|
|
let bin_path = format!("{}/{hash}", JAVA_CACHE_DIR);
|
|
let remote_path = format!("java_jar/{hash}");
|
|
let (cache, ..) = windmill_common::worker::load_cache(&bin_path, &remote_path, true).await;
|
|
|
|
if cache {
|
|
let target = format!("{job_dir}/target");
|
|
|
|
#[cfg(unix)]
|
|
let symlink = std::os::unix::fs::symlink(&bin_path, &target);
|
|
#[cfg(windows)]
|
|
let symlink = copy_dir_recursively(&PathBuf::from(&bin_path), &PathBuf::from(&target));
|
|
|
|
symlink.map_err(|e| {
|
|
Error::ExecutionErr(format!(
|
|
"could not copy cached binary from {bin_path} to {job_dir}/main: {e:?}"
|
|
))
|
|
})?;
|
|
} else {
|
|
// let plugin_registry = format!("{job_dir}/plugin-registry");
|
|
let child = {
|
|
append_logs(
|
|
&job.id,
|
|
&job.workspace_id,
|
|
format!("\n--- COMPILING .JAVA FILES\n"),
|
|
&conn,
|
|
)
|
|
.await;
|
|
|
|
let mut cmd = Command::new(if cfg!(windows) {
|
|
"javac"
|
|
} else {
|
|
JAVAC_PATH.as_str()
|
|
});
|
|
cmd.env_clear()
|
|
.current_dir(job_dir.to_owned())
|
|
.env("PATH", PATH_ENV.as_str())
|
|
.env("BASE_INTERNAL_URL", base_internal_url)
|
|
.envs(envs)
|
|
.envs(reserved_variables)
|
|
.envs(PROXY_ENVS.clone())
|
|
.args(&[
|
|
"-classpath",
|
|
&classpath,
|
|
"src/main/java/net/script/Main.java",
|
|
"src/main/java/net/script/App.java",
|
|
"-d",
|
|
"./target",
|
|
])
|
|
.stdout(Stdio::piped())
|
|
.stderr(Stdio::piped());
|
|
|
|
#[cfg(windows)]
|
|
{
|
|
cmd.env("SystemRoot", crate::SYSTEM_ROOT.as_str())
|
|
.env("USERPROFILE", crate::USERPROFILE_ENV.as_str())
|
|
.env(
|
|
"TMP",
|
|
std::env::var("TMP").unwrap_or_else(|_| String::from("/tmp")),
|
|
);
|
|
}
|
|
start_child_process(cmd, "javac").await?
|
|
};
|
|
handle_child::handle_child(
|
|
&job.id,
|
|
conn,
|
|
mem_peak,
|
|
canceled_by,
|
|
child,
|
|
!*DISABLE_NSJAIL,
|
|
worker_name,
|
|
&job.workspace_id,
|
|
"javac",
|
|
job.timeout,
|
|
false,
|
|
&mut Some(occupancy_metrics),
|
|
None,
|
|
)
|
|
.await?;
|
|
|
|
match save_cache(
|
|
&bin_path,
|
|
&format!("java_jar/{hash}"),
|
|
&format!("{job_dir}/target"),
|
|
true,
|
|
)
|
|
.await
|
|
{
|
|
Err(e) => {
|
|
let em = format!(
|
|
"could not save {bin_path} to {} to java cache: {e:?}",
|
|
format!("{job_dir}/main"),
|
|
);
|
|
tracing::error!(em);
|
|
}
|
|
Ok(logs) => {
|
|
tracing::trace!(logs);
|
|
}
|
|
}
|
|
};
|
|
|
|
Ok(())
|
|
}
|
|
async fn run<'a>(
|
|
JobHandlerInput {
|
|
occupancy_metrics,
|
|
mem_peak,
|
|
canceled_by,
|
|
worker_name,
|
|
job,
|
|
conn,
|
|
job_dir,
|
|
shared_mount,
|
|
client,
|
|
envs,
|
|
base_internal_url,
|
|
parent_runnable_path,
|
|
..
|
|
}: &mut JobHandlerInput<'a>,
|
|
classpath: &'a str,
|
|
) -> Result<(), Error> {
|
|
let reserved_variables =
|
|
get_reserved_variables(job, &client.token, conn, parent_runnable_path.clone()).await?;
|
|
|
|
let child = if !cfg!(windows) && !*DISABLE_NSJAIL {
|
|
append_logs(
|
|
&job.id,
|
|
&job.workspace_id,
|
|
format!("\n--- ISOLATED JAVA CODE EXECUTION ---\n"),
|
|
&conn,
|
|
)
|
|
.await;
|
|
|
|
write_file(
|
|
job_dir,
|
|
"run.config.proto",
|
|
&NSJAIL_CONFIG_RUN_JAVA_CONTENT
|
|
.replace("{JOB_DIR}", job_dir)
|
|
.replace("{CACHE_DIR}", JAVA_CACHE_DIR)
|
|
.replace("{SHARED_MOUNT}", &shared_mount)
|
|
// .replace("{CACHED_TARGET}", &shared_mount)
|
|
.replace("{CLONE_NEWUSER}", &(!*DISABLE_NUSER).to_string()),
|
|
)?;
|
|
let mut cmd = Command::new(NSJAIL_PATH.as_str());
|
|
cmd.env_clear()
|
|
.current_dir(job_dir)
|
|
.env("PATH", PATH_ENV.as_str())
|
|
.env("BASE_INTERNAL_URL", base_internal_url)
|
|
.envs(envs)
|
|
.envs(reserved_variables)
|
|
.envs(PROXY_ENVS.clone())
|
|
.args(vec![
|
|
"--config",
|
|
"run.config.proto",
|
|
"--",
|
|
JAVA_PATH.as_str(),
|
|
]);
|
|
if metadata(TRUST_STORE_PATH.clone()).await.is_ok() {
|
|
cmd.args(&[
|
|
&format!("-Djavax.net.ssl.trustStore={}", *TRUST_STORE_PATH),
|
|
&format!("-Djavax.net.ssl.trustStorePassword={}", *STOREPASS),
|
|
]);
|
|
}
|
|
// Configure proxies
|
|
{
|
|
let jps = parse_proxy()?;
|
|
if let Some(val) = jps.https_host {
|
|
cmd.arg(&format!("-Dhttps.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.https_port {
|
|
cmd.arg(&format!("-Dhttps.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.http_host {
|
|
cmd.arg(&format!("-Dhttp.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.http_port {
|
|
cmd.arg(&format!("-Dhttp.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.no_proxy {
|
|
cmd.arg(&format!("-Dhttp.nonProxyHosts=\"{}\"", val));
|
|
}
|
|
}
|
|
cmd.args(vec!["-classpath", &classpath, "net.script.App"]);
|
|
cmd.stdout(Stdio::piped()).stderr(Stdio::piped());
|
|
|
|
start_child_process(cmd, NSJAIL_PATH.as_str()).await?
|
|
} else {
|
|
append_logs(
|
|
&job.id,
|
|
&job.workspace_id,
|
|
format!("\n--- JAVA CODE EXECUTION ---\n"),
|
|
&conn,
|
|
)
|
|
.await;
|
|
|
|
let mut cmd = Command::new(if cfg!(windows) {
|
|
"java"
|
|
} else {
|
|
JAVA_PATH.as_str()
|
|
});
|
|
cmd.env_clear()
|
|
.current_dir(job_dir.to_owned())
|
|
.env("PATH", PATH_ENV.as_str())
|
|
.env("BASE_INTERNAL_URL", base_internal_url)
|
|
.envs(envs)
|
|
.envs(reserved_variables);
|
|
if metadata(TRUST_STORE_PATH.clone()).await.is_ok() {
|
|
cmd.args(&[
|
|
&format!("-Djavax.net.ssl.trustStore={}", *TRUST_STORE_PATH),
|
|
&format!("-Djavax.net.ssl.trustStorePassword={}", *STOREPASS),
|
|
]);
|
|
}
|
|
// Configure proxies
|
|
{
|
|
let jps = parse_proxy()?;
|
|
if let Some(val) = jps.https_host {
|
|
cmd.arg(&format!("-Dhttps.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.https_port {
|
|
cmd.arg(&format!("-Dhttps.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.http_host {
|
|
cmd.arg(&format!("-Dhttp.proxyHost={}", val));
|
|
}
|
|
if let Some(val) = jps.http_port {
|
|
cmd.arg(&format!("-Dhttp.proxyPort={}", val));
|
|
}
|
|
if let Some(val) = jps.no_proxy {
|
|
cmd.arg(&format!("-Dhttp.nonProxyHosts=\"{}\"", val));
|
|
}
|
|
}
|
|
cmd.args(&["-classpath", &classpath, "net.script.App"])
|
|
.stdout(Stdio::piped())
|
|
.stderr(Stdio::piped());
|
|
|
|
#[cfg(windows)]
|
|
{
|
|
cmd.env("SystemRoot", crate::SYSTEM_ROOT.as_str())
|
|
.env("USERPROFILE", crate::USERPROFILE_ENV.as_str())
|
|
.env(
|
|
"TMP",
|
|
std::env::var("TMP").unwrap_or_else(|_| String::from("/tmp")),
|
|
);
|
|
}
|
|
start_child_process(cmd, "java").await?
|
|
};
|
|
handle_child::handle_child(
|
|
&job.id,
|
|
conn,
|
|
mem_peak,
|
|
canceled_by,
|
|
child,
|
|
!*DISABLE_NSJAIL,
|
|
worker_name,
|
|
&job.workspace_id,
|
|
"java",
|
|
job.timeout,
|
|
false,
|
|
&mut Some(occupancy_metrics),
|
|
None,
|
|
)
|
|
.await?;
|
|
Ok(())
|
|
}
|
|
|
|
#[derive(Default, Debug)]
|
|
struct JavaProxySettings {
|
|
http_host: Option<String>,
|
|
http_port: Option<String>,
|
|
https_host: Option<String>,
|
|
https_port: Option<String>,
|
|
no_proxy: Option<String>,
|
|
}
|
|
fn parse_proxy() -> anyhow::Result<JavaProxySettings> {
|
|
let mut jps = JavaProxySettings::default();
|
|
for (ident, mut val) in PROXY_ENVS.clone() {
|
|
match ident {
|
|
"HTTPS_PROXY" => {
|
|
if val.contains("http://") {
|
|
bail!("HTTPS_PROXY url cannot contain http scheme.");
|
|
}
|
|
if !val.contains("https://") {
|
|
val = format!("https://{val}");
|
|
}
|
|
let mut url = url::Url::parse(&val)?;
|
|
let port = url.port();
|
|
// Make sure port and schema is not included in final url
|
|
{
|
|
url.set_port(None).unwrap_or_default();
|
|
jps.https_host = Some(url.as_str().replace("https://", ""));
|
|
if let Some(port) = port {
|
|
jps.https_port = Some(format!("{}", port));
|
|
}
|
|
}
|
|
}
|
|
"HTTP_PROXY" => {
|
|
if val.contains("https://") {
|
|
bail!("HTTP_PROXY url cannot contain https scheme.");
|
|
}
|
|
if !val.contains("http://") {
|
|
val = format!("http://{val}");
|
|
}
|
|
let mut url = url::Url::parse(&val)?;
|
|
let port = url.port();
|
|
// Make sure port and schema is not included in final url
|
|
{
|
|
url.set_port(None).unwrap_or_default();
|
|
jps.http_host = Some(url.as_str().replace("http://", ""));
|
|
if let Some(port) = port {
|
|
jps.https_port = Some(format!("{}", port));
|
|
}
|
|
}
|
|
}
|
|
// Java uses | instead of ,
|
|
"NO_PROXY" => jps.no_proxy = Some(val.replace(",", "|")),
|
|
_ => {}
|
|
}
|
|
}
|
|
|
|
Ok(jps)
|
|
}
|
|
async fn get_repos() -> Vec<String> {
|
|
MAVEN_REPOS
|
|
.read()
|
|
.await
|
|
.as_ref()
|
|
.map(|repos| {
|
|
repos
|
|
.trim()
|
|
.split_whitespace()
|
|
.into_iter()
|
|
.map(|el| vec!["--repository".to_owned(), el.to_owned()])
|
|
.collect_vec()
|
|
})
|
|
.unwrap_or_default()
|
|
.concat()
|
|
}
|
|
|
|
fn get_no_default() -> String {
|
|
if NO_DEFAULT_MAVEN.load(std::sync::atomic::Ordering::Relaxed) {
|
|
"--no-default"
|
|
} else {
|
|
// Command does not take empty arguments
|
|
"-q"
|
|
}
|
|
.into()
|
|
}
|
|
|
|
/// Wraps content script
|
|
/// that upon execution reads args.json (which are piped and transformed from previous flow step or top level inputs)
|
|
/// Also wrapper takes output of program and serializes to result.json (Which windmill will know how to use later)
|
|
fn wrap(inner_content: &str) -> Result<String, Error> {
|
|
let sig = parse_java_sig_meta(inner_content)?;
|
|
let ret_void = sig.returns_void;
|
|
let spread = sig
|
|
.main_sig
|
|
.args
|
|
.clone()
|
|
.into_iter()
|
|
.map(|Arg { name, .. }| {
|
|
// Apply additional input transformation
|
|
format!(" parsedArgs.{name}")
|
|
})
|
|
.collect_vec()
|
|
.join(",");
|
|
let args = sig
|
|
.main_sig
|
|
.args
|
|
.clone()
|
|
.into_iter()
|
|
.map(|Arg { name, otyp, .. }| {
|
|
// Apply additional input transformation
|
|
format!("public {} {name};\n", otyp.unwrap())
|
|
})
|
|
.collect_vec()
|
|
.join(" ");
|
|
Ok(r#"
|
|
package net.script;
|
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
|
import java.io.FileInputStream;
|
|
import java.io.InputStream;
|
|
import java.io.FileOutputStream;
|
|
import net.script.Main;
|
|
|
|
public class App{
|
|
|
|
public static class Args {ARGS}
|
|
|
|
public static void main(String[] args) {
|
|
try {
|
|
InputStream fileInputStream = new FileInputStream("args.json");
|
|
ObjectMapper mapper = new ObjectMapper();
|
|
Args parsedArgs = mapper.readValue(fileInputStream, Args.class);
|
|
fileInputStream.close();
|
|
{MAIN_HANDLER}
|
|
FileOutputStream fileOutputStream = new FileOutputStream("result.json");
|
|
mapper.writeValue(fileOutputStream, res);
|
|
fileOutputStream.close();
|
|
|
|
} catch (Exception e) { // Catching general Exception
|
|
e.printStackTrace(); // Handle the exception
|
|
}
|
|
}
|
|
}
|
|
"#
|
|
.replace(
|
|
"{MAIN_HANDLER}",
|
|
if ret_void {
|
|
"
|
|
Main.main(SPREAD);
|
|
Object res = null;
|
|
"
|
|
} else {
|
|
"
|
|
Object res = Main.main(SPREAD);
|
|
"
|
|
},
|
|
)
|
|
.replace("SPREAD", &spread)
|
|
.replace("ARGS", &args))
|
|
}
|
|
const MINI_CLIENT_IMPORTS: &str = r#"
|
|
import java.net.URI;
|
|
import java.net.http.HttpClient;
|
|
import java.net.http.HttpRequest;
|
|
import java.net.http.HttpResponse;
|
|
"#;
|
|
const MINI_CLIENT: &str = r#"
|
|
class Wmill {
|
|
public static String getVariable(String path) {
|
|
var baseUrl = System.getenv("BASE_INTERNAL_URL");
|
|
var workspace = System.getenv("WM_WORKSPACE");
|
|
var uri = java.text.MessageFormat.format("{0}/api/w/{1}/variables/get_value/{2}", baseUrl, workspace, path);
|
|
|
|
// Create an HttpRequest
|
|
HttpRequest request = HttpRequest.newBuilder()
|
|
.uri(URI.create(uri))
|
|
.header("Authorization", "Bearer " + System.getenv("WM_TOKEN")) // Add the Authorization header
|
|
.GET() // Set the request method to GET
|
|
.build();
|
|
|
|
// Send the request and get the response
|
|
return HttpClient.newHttpClient().sendAsync(request, HttpResponse.BodyHandlers.ofString())
|
|
.thenApply(HttpResponse::body)
|
|
.join(); // Wait for the completion
|
|
}
|
|
public static String getResource(String path) {
|
|
var baseUrl = System.getenv("BASE_INTERNAL_URL");
|
|
var workspace = System.getenv("WM_WORKSPACE");
|
|
var uri = java.text.MessageFormat.format("{0}/api/w/{1}/resources/get_value_interpolated/{2}", baseUrl, workspace, path);
|
|
|
|
// Create an HttpRequest
|
|
HttpRequest request = HttpRequest.newBuilder()
|
|
.uri(URI.create(uri))
|
|
.header("Authorization", "Bearer " + System.getenv("WM_TOKEN")) // Add the Authorization header
|
|
.GET() // Set the request method to GET
|
|
.build();
|
|
|
|
// Send the request and get the response
|
|
return HttpClient.newHttpClient().sendAsync(request, HttpResponse.BodyHandlers.ofString())
|
|
.thenApply(HttpResponse::body)
|
|
.join(); // Wait for the completion
|
|
}
|
|
}
|
|
"#;
|