Editing environment templates and multi-tier config files during deployment automation often degrades into broken string substitutions. You write a fast regex to bump a version or inject an API token, and within minutes you have corrupted quotes, stripped trailing newlines, or mangled JSON arrays.
The failure mode with naive regex and stream edits
Consider a standard deployment scenario. You need to update an application JSON manifest to point to a new release artifact, add a service flag, and sync these values into an unquoted INI environment file.
The immediate trap is running sed against the JSON payload directly:
bashsed -i 's/"version": ".*"/"version": "1.8.4"/' config.json
If config.json contains nested objects or dependencies with their own version keys, this blind regex burns down your configuration:
text{
"name": "payment-worker",
"version": "1.8.4",
"dependencies": {
"auth-lib": "1.8.4",
"metrics-collector": "1.8.4"
}
}
Every nested "version" string got rewritten to 1.8.4 because standard line-oriented string replacements lack structural awareness.
The reverse issue hits INI and .env parsing with jq. Developers try converting raw text configs to JSON, processing them, and dumping them back out using standard Bash iteration:
bash# Typical brittle attempt
for line in $(jq -r '.settings | to_entries[] | "\(.key)=\(.value)"' manifest.json); do
echo "$line" >> app.env
done
If any value contains spaces, escapes, or punctuation, standard shell word-splitting breaks the string:
textDATABASE_URL=postgres://user:[email protected]:5432/core
LOG_LEVEL=info
FEATURE_FLAGS=audit_logging
beta_dashboard
The string "audit_logging beta_dashboard" split on the space, generating an invalid key on the next line.
Structural JSON manipulation with jq
When editing JSON, treat it strictly as a tree, never as text. Use jq to isolate specific branches without disturbing neighboring properties or formatting.
To update only the root-level version string while leaving dependencies untouched, target the root property specifically:
bashjq '.version = "1.8.4"' config.json > config.json.tmp && mv config.json.tmp config.json
Avoid the urge to redirect back into the same file (jq ... file.json > file.json). The shell opens the write descriptor and truncates the file to zero bytes before jq even reads it. Use a temporary file, or pipe through sponge from moreutils.
To conditionally update fields only if they exist, or append them if they are missing, use the addition assignment operator +=:
bashjq '.build += {"commit": "a1c8f3e", "dirty": false}' config.json > config.json.tmp && mv config.json.tmp config.json
If you are dealing with arrays of objects, say, updating the image tag for a single container inside a Nomad or Kubernetes task group:
bashjq '
.services[] |=
if .name == "web-api"
then .image = "registry.internal/api:v2.10.0"
else .
end
' deployments.json > deployments.json.tmp && mv deployments.json.tmp deployments.json
Here, |= acts as an update-assignment operator. It maps over the .services array, alters only the object matching .name == "web-api", and leaves the remaining elements intact.
Synchronizing JSON into INI and Shell Variables
Passing JSON properties into plain key-value environment files requires careful escaping. You cannot trust shell string concatenation when handling complex values.
Use jq with the @sh formatter to create safely escaped Shell/Bash variable declarations:
bashjq -r '
.environment |
to_entries[] |
"export \(.key)=\(.value | @sh)"
' service.json > service.env
Given this input:
json{
"environment": {
"APP_NAME": "Data Ingest",
"SPECIAL_CHARS": "foo'bar\"baz$qux",
"TIMEOUT": 30
}
}
The @sh filter outputs standard POSIX single-quoted strings with correctly escaped internal quotes:
bashexport APP_NAME='Data Ingest'
export SPECIAL_CHARS='foo'\''bar"baz$qux'
export TIMEOUT='30'
Now you can source service.env directly in your shell script without arbitrary code execution or variable expansion bugs.
In-Place INI Edits with sed and awk
When updating existing .env or .ini files that cannot be wiped or rewritten from scratch, use sed or awk with strict line-anchoring.
The standard POSIX sed syntax avoids dialect variations between GNU sed (Linux) and BSD sed (macOS). On Linux, sed -i takes an optional extension; on macOS, it requires one. To keep scripts portable across your local workstation and Linux CI runners, provide an empty backup suffix explicitly or use a temporary file:
bash# Portable in-place edit using sed
sed -i.bak -E 's/^([[:space:]]*PORT=).*/\18080/' .env && rm -f .env.bak
The ^([[:space:]]*PORT=) regex ensures two things:
- It matches only lines that actually begin with
PORT=, ignoring commented lines like# DEFAULT_PORT=3000. - It captures optional leading whitespace to preserve the original file indentation.
For section-aware INI edits (where keys are scoped under [headers]), plain sed becomes messy. Use awk to track context through state flags:
bashawk -v target_section="[database]" -v key="max_connections" -v val="50" '
$0 ~ /^\[.*\]/ { in_section = ($0 == target_section) }
in_section && $1 ~ ("^" key "[[:space:]]*=") {
print key " = " val
found = 1
next
}
{ print }
END {
if (!found) {
exit 1
}
}
' config.ini > config.ini.tmp && mv config.ini.tmp config.ini
This updates max_connections inside [database] while leaving the same key alone if it appears in [cache]. The exit 1 block ensures your deployment script halts if the key was never matched, preventing silent failures.
Transforming JSON Payloads for REST APIs
When stitching together data pipelines between CLI tools and REST endpoints, you frequently need to take text streams and map them to JSON payloads for curl.
Suppose you run lsof or a network probe and need to send open ports and host metrics as a JSON POST payload:
bash# Sample input stream (ss -tulpn)
ss -H -t -a -n state listening | awk '{print $4}'
Output:
text127.0.0.1:5432
0.0.0.0:80
0.0.0.0:22
To ingest that stream, split the IP and port, and construct a structured JSON object in a single pipeline without invoking external interpreters like Python or Node:
bashss -H -t -a -n state listening | awk '{print $4}' | jq -R -s '
split("\n") |
map(select(length > 0)) |
map(split(":") | {ip: .[0], port: (.[1] | tonumber)}) |
{timestamp: now | floor, listeners: .}
'
Output:
json{
"timestamp": 1711802314,
"listeners": [
{
"ip": "127.0.0.1",
"port": 5432
},
{
"ip": "0.0.0.0",
"port": 80
},
{
"ip": "0.0.0.0",
"port": 22
}
]
}
The -R flag treats the incoming stream as raw text instead of JSON. The -s flag reads all lines into a single string. From there, jq slices the lines, cleans out empty records, converts the numeric port from a string to an integer, and wraps everything into a single document ready for curl -d @-.
Gotchas and Edge Cases
Numbers Converted to Scientific Notation
By default, jq decodes numbers as IEEE 754 64-bit floating-point numbers. If your config file contains large 64-bit integer IDs (like Snowflake IDs or Twitter IDs), standard jq processing can silently truncate precision or convert them to scientific notation:
bashecho '{"id": 18446744073709551615}' | jq .
text{
"id": 18446744073709552000
}
To preserve exact 64-bit integers during pass-through operations, use the --unbuffered and --raw-output flags carefully, or keep those fields strictly quoted in the source files, as jq does not natively support arbitrary-precision integers or a --bigint flag, or keep those fields strictly quoted in the source files.
Shell Parameter Injection inside jq Queries
Passing variables into jq using raw string interpolation opens up syntax breaks when values contain quotes or backslashes:
bash# Brittle: breaks when $VAR has quotes
NAME="team\"alpha"
jq ".user.name = \"$NAME\"" payload.json
Use the --arg or --argjson flags instead. These import Bash variables directly into jq scope as clean internal variables, sidestepping all shell escape issues:
bashNAME="team\"alpha"
API_TIMEOUT=45
jq --arg n "$NAME" --argjson t "$API_TIMEOUT" '
.user.name = $n |
.network.timeout = $t
' payload.json > payload.json.tmp && mv payload.json.tmp payload.json
--arg treats the incoming value strictly as a string. --argjson parses the value as JSON, which handles booleans, numbers, arrays, and objects correctly without accidental type casting.
Verification
To verify that your target file contains the valid structure after automated edits, do not rely on exit codes alone. A command might exit 0 even if it produced an empty file.
Validate the result explicitly:
bash# Validate JSON syntax
jq empty config.json || { echo "Malformed JSON produced"; exit 1; }
# Validate INI key presence
awk -F'=' '
BEGIN { found=0 }
$1 ~ /^[[:space:]]*PORT[[:space:]]*$/ && $2 ~ /^[[:space:]]*8080[[:space:]]*$/ { found=1 }
END { if (!found) exit 1 }
' .env || { echo "PORT was not set to 8080"; exit 1; }
Running jq empty parses the entire document structure and returns an exit status of 0 on success, or a non-zero exit code with the exact line and column number of the syntax failure if parsing fails.