_assign and _modify create an allocator on every call, so the value they
are given is never one that allocator created, and updateObject and
updateArrayIndex always take the copying branch. Accumulating into an
object or an array with a reduce expression therefore copies the whole
accumulated value on every iteration, which is quadratic in the number
of elements. The query of #241,
[range(20000)] | reduce .[] as $item ({}; .[($item|tostring)] = {})
took between 10 and 21 seconds here over three runs, against 0.03
seconds for jq.
Create the allocator once per reduce expression and pass it to the
assignment as a third argument, so that the accumulated value is
allocated on the first iteration and modified in place afterwards. The
query above now takes 0.03 seconds, and the memory allocated by an
accumulating reduction grows linearly instead of quadratically: 2.1 MB
instead of 545 MB for 4000 elements.
This is only correct while the accumulated value, and the values the
allocator created inside it, are reachable through the reduction alone.
Query#inPlaceUpdate therefore requires the update query to be a single
assignment, and its right-hand side to neither observe the input value,
which would let the assigned value alias the value it is assigned into,
nor yield more than one value, which would run the assignment again
against a value it already modified. Everything else keeps copying. A
foreach expression is left alone, because it yields every intermediate
accumulated value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_assign and _modify create an allocator on every call, so the value they
are given is never one that allocator created, and updateObject and
updateArrayIndex always take the copying branch. Accumulating into an
object or an array with a reduce expression therefore copies the whole
accumulated value on every iteration, which is quadratic in the number
of elements. The query of #241,
[range(20000)] | reduce .[] as $item ({}; .[($item|tostring)] = {})
took between 10 and 21 seconds here over three runs, against 0.03
seconds for jq.
Create the allocator once per reduce expression and pass it to the
assignment as a third argument, so that the accumulated value is
allocated on the first iteration and modified in place afterwards. The
query above now takes 0.03 seconds, and the memory allocated by an
accumulating reduction grows linearly instead of quadratically: 2.1 MB
instead of 545 MB for 4000 elements.
This is only correct while the accumulated value, and the values the
allocator created inside it, are reachable through the reduction alone.
Query#inPlaceUpdate therefore requires the update query to be a single
assignment, and its right-hand side to neither observe the input value,
which would let the assigned value alias the value it is assigned into,
nor yield more than one value, which would run the assignment again
against a value it already modified. Everything else keeps copying. A
foreach expression is left alone, because it yields every intermediate
accumulated value.
_assign and _modify create an allocator on every call, so the value they
are given is never one that allocator created, and updateObject and
updateArrayIndex always take the copying branch. Accumulating into an
object or an array with a reduce expression therefore copies the whole
accumulated value on every iteration, which is quadratic in the number
of elements. The query of #241,
[range(20000)] | reduce .[] as $item ({}; .[($item|tostring)] = {})
took between 10 and 21 seconds here over three runs, against 0.03
seconds for jq.
Create the allocator once per reduce expression and pass it to the
assignment as a third argument, so that the accumulated value is
allocated on the first iteration and modified in place afterwards. The
query above now takes 0.03 seconds, and the memory allocated by an
accumulating reduction grows linearly instead of quadratically: 2.1 MB
instead of 545 MB for 4000 elements.
This is only correct while the accumulated value, and the values the
allocator created inside it, are reachable through the reduction alone.
Query#inPlaceUpdate therefore requires the update query to be a single
assignment, and its right-hand side to neither observe the input value,
which would let the assigned value alias the value it is assigned into,
nor yield more than one value, which would run the assignment again
against a value it already modified. Everything else keeps copying. A
foreach expression is left alone, because it yields every intermediate
accumulated value.
The allocator of _modify tracks the values it created so that a later
update can modify them in place instead of copying them. When a target
path is a proper prefix of an earlier one, getpath hands such a value to
the update function, which may store it in more than one place; updating
through one of those places then changes all of them.
$ gojq -cn '{"a":{"b":1}} | (.a.b, .a, .a.p.z) |= {p: ., q: .}'
.a.p and .a.q hold separate values in jq, so only .a.p gains the "z" key,
while gojq adds it to both. Release the allocator whenever a value it
created leaves the update, so that the values are copied on update from
then on. The same applies to an object used as a path element, which
stays reachable through the path after the update.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The allocator of _modify tracks the values it created so that a later
update can modify them in place instead of copying them. When a target
path is a proper prefix of an earlier one, getpath hands such a value to
the update function, which may store it in more than one place; updating
through one of those places then changes all of them.
$ gojq -cn '{"a":{"b":1}} | (.a.b, .a, .a.p.z) |= {p: ., q: .}'
.a.p and .a.q hold separate values in jq, so only .a.p gains the "z" key,
while gojq adds it to both. Release the allocator whenever a value it
created leaves the update, so that the values are copied on update from
then on. The same applies to an object used as a path element, which
stays reachable through the path after the update.
dualfroz
upstream