Precision generic fluid routing and splitting

This board is to show, discuss and archive useful combinator- and logic-creations.
Smart triggering, counters and sensors, useful circuitry, switching as an art :), computers.
Please provide if possible always a blueprint of your creation.
Post Reply
User avatar
Ivelieu
Burner Inserter
Burner Inserter
Posts: 9
Joined: Mon Apr 22, 2024 2:06 am
Contact:
Contact Ivelieu

Precision generic fluid routing and splitting

Post by Ivelieu »

I came across a previous thread recently: viewtopic.php?f=18&t=62440 - I was initially going to post there, but since they didn't discuss fluid splitting I decided instead to post in a separate thread. For very different reasons, I've independently had to struggle with precise fluid counts for a project. I didn't find any of the solutions presented satisfactory for me, because I want absolutely zero leniency - I want to transport an exact amount of fluid at high enough throughput and have it work 100% of the time, in order to keep pipes completely free of sloshing liquid so I can hotswap liquid types over the same pipeline. This is the kind of thing that would be easier to do reading the source code, but through some experimentation I found a relatively simple (albeit bulky/obtuse) technique to get precise fluid amounts into containers/assemblers/etc without using barrels.

Using this technique I can build a precision fluid splitter, which has multiple tanks of different fluid types, and on receiving a signal can distribute a fixed amount of fluid to each of the parallel tanks using either round robin or parallel push. Without mods, this fixed value is either 100 or 200, depending on whether the input side of the pump faces a full single pipe segment or another pipe/tank respectively. With just one mod, the crafting combinators mod (which I happen to be using anyway to utilise the precision fluid routing/splitting), I can get the better precision with the smallest unit of fluid I can transfer being twice of any single recipe's cost, with some caveats in speed due to crafting combinator delays. And if I use barrels with this crafting combinator technique, I can transport exactly 10 units of fluid at a time. When crafting combinators are added to the base game according to FFF 394, I think the performance caveats will mostly disappear, although I'm not sure if it will be as fully featured as xeraph's fork crafting combinators mod with a recipe combinator and the find uses/find machines and vary input count features, etc. So in that sense I'm "planning ahead" for this technique to be vanilla.

So, the idea. Occam's razor fails. Nixie tubes are used in order to see rounded down fluid counts which hovering over the container won't display.
399_fluid.png
399_fluid.png (231.77 KiB) Viewed 1000 times
It's not currently possible to measure fluid inside an assembler, but even if I put two assemblers next to each other I find this rounding error still happens sometimes. Not often, but often enough to cause problems with automation.

Why does it break? This works, why does it work?
bp1.png
bp1.png (177.17 KiB) Viewed 1000 times
There are three issues, none of them immediately obvious:

- The first is if a pump output is facing a tank (NOT an assembler/chemplant/etc, specifically just tanks) and its fluid storage is 40% or higher, the pump won't push a full 200 fluid each tick, usually around 199 but it varies by a few decimals.
40_percent_cutoff.png
40_percent_cutoff.png (271.93 KiB) Viewed 1000 times
- The second issue is that fluid sloshes in between long pipe segments. Even a pipe segment of two tiles gets this sloshing, and sometimes when small amounts of fluid slosh they are "forgotten" about by the fluid network until fluid is sent through that pipe again. While "forgotten", pumps won't pull this leftover sloshed liquid. I assume this is an intentional optimisation to make fluid not slosh endlessly, but the challenge is how to override this behavior for specialised setups.
fluid_2_long.png
fluid_2_long.png (122.55 KiB) Viewed 1000 times
- The third issue is if a pump is right next to two tanks, then factorio optimises the three components as a single fluid network, and this optimised behavior involves leaving over trace leftover amounts. The trick is to get factorio to recognise two separate fluid networks so this optimisation doesn't happen.

For the first issue, just make sure that all storage tanks never go above 40% (or 10k) fluid. Easy to set this condition on the adjacent pump with no combinators.

For the second issue, it's relatively simple - don't have any pipe segments of more than two tiles without pumps in between. Sounds annoying, but it's not as bad as it sounds in practice.

For the third issue, it was mostly trial and error, but I found four pumps and three fluid pipes was the minimum to trick factorio into treating the input and output tanks as two separate fluid networks. If you want more than 10k total fluid storage, I thought initially I'd need four pumps in between each 10k tank, but it turned out I could have just one pump in between the storage tanks and it worked fine. More specifically, three pipe-pump tiles after the pump with the circuit condition. Again, no way to really know for sure without seeing source code. (Don't forget to hook up all pumps with their output facing a tank, to the 10k storage cutoff.) Keep in mind, the first pump in this four pump-pipe chain needs to be active for exactly one tick only - for the precision fluid splitter, you need a "buffer" pump to store the 200 units being sent to it in that iteration, which is why there are five pumps per parallel slice in that blueprint.
bp2.png
bp2.png (222.75 KiB) Viewed 1000 times
For this to work, it's important that the fuel unloading is done in tick pulses. If you go too fast, then the intermediate pipes won't catch up and so not all pumps will work at exactly 200 fluid/tick. I found with a pulse former I was able to do 2 tick pulses 100% of the time to transfer 400 fluid on the one tank setup, but it broke the 10k fluid stopping condition. There may be a way around this, but I am finding 200 tick pulses satisfactory for my throughput needs for the time being.

The next important question, is the minimum number of ticks between pulses. In other words, how many ticks in the worst case may it take for one pipe plus one pump to shuffle all the liquid down. In the example without fluid splitting this is 6. The formula for number of ticks needed, signal U in my blueprints, is the number of pipes in the longest chain in the network, plus three. Based on this, I come to the conclusion it takes exactly four ticks for a pipe+pump segment to fully move 200 units of fluid along to the next segment. Of course, with multiple channels of different fluid types along the same row of pipes, the delay needs to be a bit longer due to the longer chain of pumps.
bp3 multi_channel_no_splitting.png
bp3 multi_channel_no_splitting.png (500.58 KiB) Viewed 1000 times
Splitting precise fluids
bp4.png
bp4.png (439.87 KiB) Viewed 1000 times
For the six channel splitter, the overall throughput with the tank on the same row as the main buffer, is 200/15 = 13.33 fluid/tick, or 200/(15*6) = 2.22 fluid/tick for each channel. The number of different fluids increases the maximum chain length, but you can use a separate combinator row per fluid type so that not all combinators are impacted by the increased tick delay due to higher chain length. While this may sound slow on paper, keep in mind that the buffer tank/s on the input side can have an unlimited throughput input rate for storage. Also, the highest liquid consuming rate in vanilla is a chemplant is sulfuric acid, with 100 water every second without beacons, or 100/60 = 1.67 fluid/tick, so this splitting method meets throughput even for the highest vanilla consumer.

Actually, in my design I made a modulo clock pulser that spends one clock cycle resetting, so it's a small bit slower in practice, I'm just not sure of a good combinator design to modulo pulse without the reset clock cycle. Increasing the parallel number and adding rows further down will both increase the chain length, but it's easy to figure out what the throughput would be with the equation known. Simple answer, keep water in the first row, followed by oil if you choose to use such a splitter setup for generic oil processing (which I plan to be posting about in a separate thread soon!).

In practice, the disadvantages are having to use the extra pumps, space for combinators to pulse the signal, and that you can only use each tank for 10k fluid or 40% of their normal capacity. Overall I find this an acceptable tradeoff, also pointing out that multiple setups can share the same modulo clock pulser.

For the splitter, if you don't want to share fluid types via the same pump, there is a more efficient setup runs with just a clock pulser carefully timed to refill all the buffer pumps before each pulse (as shown below). In this case, all pumps can push every 11 ticks, so total fluid distributed for the example six channel splitter is (200*6)/11=109.09 fluid/tick, or 18.18 fluid/tick for each channel. I have this in the blueprint, I don't think it is that useful though.
How Low Can You Go?
Just for the sake of it, what's the smallest precise fluid amount I can pull out? Here are two setups, both using the crafting combinator:
bp7.png
bp7.png (992.55 KiB) Viewed 1000 times
Getting 10 fluid in a tank was more of a technical challenge than serving any useful purpose - at least currently, but maybe in future if a mod makes some machines get overflow penalties, this could be useful. The trick to making this work is ensuring the chemplant never receives power, so it won't start the recipe, and when the crafting combinator unsets the recipe the leftover fluid is dumped back into the adjacent input tank. In other words, the chemplant for a lubricant recipe can only hold 20 fluid input, so all that's left to do is solve this simplified "water pouring puzzle" (https://en.wikipedia.org/wiki/Water_pouring_puzzle). I spent a while trying to do it with fluid detection, but I found that I'd need quite a few buffer tanks for the detection which makes the setup dramatically bigger. So I elected to go with manually calibrated delay lines specifically for the heavy oil -> lubricant recipe, the timings aren't optimised and it still takes up a decent amount of space, and the crafting combinator takes about 2 seconds to set or unset recipes which is the main time bottleneck. Due to relying on crafting combinator timing estimates, this is the only blueprint that's UPS sensitive.

The algorithm is:
- unbarrel 50 fluid and deposit into a tank facing a chemplant, deposit 20 into the chemplant.
- store the 30 leftover in an intermediate buffer (in my solution, a pump buffer rather than a tank buffer. Keep in mind you can't read pump internal storage with signals.)
- remove 20 deposited into chemplant back into the input buffer
- deposit 30 back into the chemplant facing tank, deposit 20 into the chemplant again
- send the 10 leftover to the output and the 20 in the chemplant back to the input buffer a second time.

It's definitely possible to build a "generic" version of this setup to minimise the fluid count of any recipe, but it'd require more combinators, doing some math on the selected recipe capacity and the minimum fluid stack size that can be obtained. With recipe combinators these calculations can all be done automatically. Given the poor performance and complexity of this design, I think it's easier to just calculate LCM(x, 100) or LCM(x, 200) of how much fluid a recipe needs and scale up its crafts with combinator logic rather than subdividing fluid amounts like this.
Curious determinism
I found that fluid sloshing was not as probabilistic as I'd expected. Repeating this setup I found there was very little variation in the split outputs, which I also found were affected by chunk alignment:
ugly_fluid.png
ugly_fluid.png (208.14 KiB) Viewed 1000 times
Nturally, all of these lead to the dreaded fluid rounding errors, meaning they can't be used for precise splitting even assuming they are fully deterministic 100% of the time. I don't think it's worth reverse engineering the fluid code any more...
Blueprint
The blueprints use the nixie tubes, blueprint sandboxes and pushbutton mods. All just for ease of use in testing, of course. When making it I used the map editor to do step through ticks.

Also, I should mention, this is my first post but I've been lurking on these forums for a few months, I'm just a bit shy :)

Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Mon Apr 22, 2024 6:02 am
 I found that fluid sloshing was not as probabilistic as I'd expected. Repeating this setup I found there was very little variation in the split outputs, which I also found were affected by chunk alignment:
Not sure what 'sloshing' means, and it's used a lot =) I would say the little variations in the splits outputs are affected by "build order" if there is a junction. If you have 100 fluid in a pipe and the game is paused. 2 pumps trying to push this fluid out, in 2 different direction, 1 tick later game will always give priority to 1 pump, reset setup, remove and rebuild the other pump, now the other pump has the priority, 1 tick later 100% fluid will go through the other pump. ( while keeping chunk aligment unchanged).

The formula from the Wiki for "fluid throughput" are "explained" :viewtopic.php?p=572106 maybe this is what you called sloshing, the way the quantity of fluid is updated in each pipe every tick that create back and forth waves ?

It is deterministic, and reproductible. Supposedly if you time control everything, you will end up with the same quantity after x or y number of ticks. The "build order" is what makes it very difficult to predict when one pump will have 100% and the other 0% because even if you copy your setup with the same chunk alignment, you'd also need to make sure robots build the pipes/pumps in the same order.... On straight lines of pipes though, the chunk aligment doesn't matter ( due to the sloshing ?) the way the updates works will go back and forth changing the values of fluid in the pipes everytime a little less and less until it reaches stable level. ( considering source in on side and consumer in the other) .

It was interesting to read the experiments, but realistically, why not have a "loop" of fluid similar to picture (6) with the lube water and crude on the left side. Instead of having vertical pump , having the chemplant there, with 1 input in a pipe and the other on the other pipe separated by pump. This way you could send 200 light oil 200 water, 200 light oil, 200 water alternating, Would that work for you requirement of 0 waste ?

Light oil craking is particular in that it require same amount of water and light oil. But it doesn't divide by 100 or 200, so there would be leftover, it could be dealt with by having the excess loop around i thought.

As long as the "excess" can loop around, there is also another "strat" i thought you can "offer" the chemplant/Assembly, with excess input fluid, but make sure when the receipe is switch, the excess in the chemplant/Assembly can be rejected into the inputs. ( 2 tanks in front of each input, each tank having a way "in" and a way "out" , tanks never filled fully, when receipe change, wait 2 ticks, and emty tanks using "way out". Then if fluid needed, fill tank using "way in". As a way to also get "the minimum" required fluid into a single entity.

Not sure if it would be less complex though x). Maybe having some filter on pumps that can be changed by circuits as announced in a recent FFF would help.
Top
FuryoftheStars
Smart Inserter
Smart Inserter
Posts: 2573
Joined: Tue Apr 25, 2017 2:01 pm
Contact:
Contact FuryoftheStars

Re: Precision generic fluid routing and splitting

Post by FuryoftheStars »

mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 Not sure what 'sloshing' means [...]
I think they're referring to the oscillating/wave effect back and forth through the system, though I could be wrong.
My Mods: Classic Factorio Basic Oil Processing | Sulfur Production from Oils | Wood to Oil Processing | Infinite Resources - Normal Yield | Tree Saplings (Redux) | Alien Biomes Tweaked | Restrictions on Artificial Tiles | New Gear Girl & HR Graphics
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

FuryoftheStars wrote: ↑
Tue Apr 23, 2024 11:49 pm
 I think they're referring to the oscillating/wave effect back and forth through the system, though I could be wrong.
I didn't know the word and the wikipedia page in english is refering to a notion i know in french but it's called "balottement", it only applies when the container of the liquid is moving itself, like water inside of a water truck or fluid wagon would undergo sloshing. We also say "splash" in french, but when seeing "balottement" for "slosh" i thought maybe it's not exactly the same concept, the content of the pages is somewhat different too. I'm fne with waves x) If the cause of them give them their name, those in factorio do not really have equivalent in real life i think.
Top
User avatar
Ivelieu
Burner Inserter
Burner Inserter
Posts: 9
Joined: Mon Apr 22, 2024 2:06 am
Contact:
Contact Ivelieu

Re: Precision generic fluid routing and splitting

Post by Ivelieu »

mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 Not sure what 'sloshing' means, and it's used a lot =)
Sorry, it's a term commonly used from modded minecraft that I have much more experience with. I use it to mean fluid moving back and forth iteratively on a pipe over multiple ticks when it's intended to only go one direction. In Gregtech in minecraft there are these "pipe wars" about which version of pipe is the best, but almost all of them either have teleporting fluids, or this "sloshing" with fluid moving back and forth passively in a busy loop to supposedly accurately simulate fluid dynamics (but whether it behaves properly when some pipes are vertically above others has always left things to be desired.) Not to go off topic, this image is just as a reference, you have to do much more micro management setting it up with valves to get fluids to naturally "slosh" to the right machine, and pumps don't know how to pull from more than one tile away in a fluid network. In factorio the issue seems to be the point where if the fluid is "balanced enough", then the sloshing stops happening, which tends to leave behind small fractions of a fluid left.
2023-10-02_03.34.24.png
2023-10-02_03.34.24.png (1.04 MiB) Viewed 846 times

mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 I would say the little variations in the splits outputs are affected by "build order" if there is a junction.
Yeah, seems that way.
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 If you have 100 fluid in a pipe and the game is paused. 2 pumps trying to push this fluid out, in 2 different direction, 1 tick later game will always give priority to 1 pump, reset setup, remove and rebuild the other pump, now the other pump has the priority, 1 tick later 100% fluid will go through the other pump. ( while keeping chunk aligment unchanged).
I didn't realise that x_x. I suppose it's an alternative to alternating the pumps with circuit conditions, it might be tricky to get the right delay to avoid the leftover fluid fractions though.

With robots and the build order I agree it's too hard to reproduce accurately (for now at least). Also, after posting I discovered this thread and FFF 260 which shed a little more light on the topic. I'm not exactly sure what to expect in future changes.
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 The formula from the Wiki for "fluid throughput" are "explained" :viewtopic.php?p=572106 maybe this is what you called sloshing, the way the quantity of fluid is updated in each pipe every tick that create back and forth waves ?
Thank you, I didn't know about that thread either. That's a nice experiment. The exact "sloshing" algorithm of fluid moving back and forth is different to what I'm used to, but the principle is the same.
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 It was interesting to read the experiments, but realistically, why not have a "loop" of fluid similar to picture (6) with the lube water and crude on the left side. Instead of having vertical pump , having the chemplant there, with 1 input in a pipe and the other on the other pipe separated by pump. This way you could send 200 light oil 200 water, 200 light oil, 200 water alternating, Would that work for you requirement of 0 waste ?
I'm not sure what that would look like, could you try making a prototype?
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 Light oil craking is particular in that it require same amount of water and light oil. But it doesn't divide by 100 or 200, so there would be leftover, it could be dealt with by having the excess loop around i thought.
Yes, that's where I think you could choose to do LCM (least common multiple) arithmetic optionally with combinators. It takes 30 of each fluid, so you'd have to craft in batches of lcm(30,100) = 300 (batches of 10 crafts at a time) or lcm(30, 200) = 600 (batches of 20 crafts at a time). There is a third option I think you are suggesting, is craft for 60 units of fluid (the max that the machine will take in at once), and loop back the leftover 40 or 140 fluid, that's an option too. Not sure either which design would be "best" in terms of compactness/complexity, I'd be interested to see and might try myself.

On a separate note, would you happen to know of a nice combinator setup to calculate lcm(x,y)? Trying to implement the Euclidean algorithm with combinators in a neat way is giving me a headache...
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 Maybe having some filter on pumps that can be changed by circuits as announced in a recent FFF would help.
I did see that and I'm not sure how much the filter would help, if it could improve overall throughput by switching faster or not. it'll probably be easier to see when it's released I can just try it.


Also, after writing this out I had an idea for a challenge mode this morning. Items become itemfluids, inserters are disabled, you have to do precision fluid management even for basic crafting tasks with two components in assemblers. I have a good idea of the implementation details and I think this would lead to some cool strategies so I've started working on a mod to implement this idea. (Although it's my first Factorio mod it might take me some time to learn the conventions.)
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
 Sorry, it's a term commonly used from modded minecraft that I have much more experience with. I use it to mean fluid moving back and forth iteratively on a pipe over multiple ticks when it's intended to only go one direction. In Gregtech in minecraft there are these "pipe wars" about which version of pipe is the best, but almost all of them either have teleporting fluids, or this "sloshing" with fluid moving back and forth passively in a busy loop to supposedly accurately simulate fluid dynamics (but whether it behaves properly when some pipes are vertically above others has always left things to be desired.) Not to go off topic, this image is just as a reference, you have to do much more micro management setting it up with valves to get fluids to naturally "slosh" to the right machine, and pumps don't know how to pull from more than one tile away in a fluid network. In factorio the issue seems to be the point where if the fluid is "balanced enough", then the sloshing stops happening, which tends to leave behind small fractions of a fluid left.
Thanks for the clarifications. I had done some (game-related) research on the problem of modelising fluid, what you describe make sense to me. I don't know how it is refered to in the non-french-speaking world though. I learned about the vulgarisation under the name of "Dirichlet magic squares", but the only english ressources that mention those i found refers to the video and explanations in french : https://marajo.xyz/dirichlet/, it's a "magic square", but not one where the sum of each row and column is equal to same number, instead, each tile of the square is the average of the neighbouring tile, most of what i found online were about the other magic squares.

This also apply to heat in factorio, maybe it's more intuitive to see the parralel with heat, but really water in pipes will follow the same rules. In factorio the way it is programmed makes it so that fluid will try to have a "final stable state" that is in theory reached after infinite amount of time.

If you take a straight line of pipes, with the source at one end, and a consumer at the other end, the motion of the fluid will be seen by the different level of fluid in each pipe if you were to take a screenshot. Not sure if that make sense, the pump will have a buffer of "max" fluid, then the next pipe almost as much, maybe a little less, and the next same, little less, and the next same , with little less water and so on.

This would be like the "idealized steady end state". The fluid would be moving but the level would stay the same in each pipe because each pipe would have a quantity of water that is equal to the average of its 2 neighbour.

In factorio pipes can have from 0 to 100 water so if you have only 2 pipes in the lane it will be 100 and 0 simple and boring, if you have 3 pipes it will be 100 50 and 0, and it extend roughly, if you have 10 pipes, and the consumer is consuming everything the source is sending, the level of fluid on those pipe will end up looking like 100 90 80 70 60 50 40 30 20 10 0. This is shown in the video refered to in the link but in french before they add more rows to make a square from several of those lanes :/. In such situation each number is the average of the 2 neighbouring numbers. ( temperature in grid of heat pipes is similar to quantity of fluid in grids of pipes ).

What you call sloshing to me is the process that leads to this "end result", from a starting moment where fluid quantity are 100 0 0 0 0 0 0 0 0 0 0 0 0. The way fluid propagate in the game will act similarly as an algorithm that would approximate iteratively the solution of those squares. ( by looking only at neighbouring squares ! ). Since it is looking only at neighbouring squares, there is only a limited correction that can be applied every tick.

This will indeed "disappear" when the level is "flat enough" since the variations that would still occur would only touch the far hidden digits behind 0.

This process is "visible" on this page https://mathsmagiques.fr/pages/truc_mat ... chlet.html, the script stops looking for more significant digits at some points, but it could be very long to go through a true stable state with all the digits. This mathematical approach to the problem is prooven to have a unique solution that can be found / approximated by different methods. Part of the video is about the demonstration of such facts.

If no pumps are involved, the quantity of fluid during the second tick will not be 100 100 0 0 0 0 0 0 0 0 0 0 0 0 0, but more something like 100 50 0 0 0 0 0 0 0 0 0 0, then 100 75 50 25 0 0 0 0 0 0 0 0 , untill it reaches 100 90 80 70 60 50 ... The exact process is the source code of the game :D

Some of the formulas and the order in which they are processed is "explained" in the post linked in the wiki but the code was linked from this one i think viewtopic.php?p=476422#p476422

That's for the sloshing only ... fluids are complex things and if i understand well your setup you removed entirely this behavior by having the pipe system behaving as separated boxes, that would fully transfer their content when allowed by circuit, with timers , allowing each pipe separated by pumps to host a different fluid, which would not be allowed to "slosh" or settle with waves of updates, whatever the name to the complicated process, it's all removed in your experiments :)
Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 I would say the little variations in the splits outputs are affected by "build order" if there is a junction.
Yeah, seems that way.
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 If you have 100 fluid in a pipe and the game is paused. 2 pumps trying to push this fluid out, in 2 different direction, 1 tick later game will always give priority to 1 pump, reset setup, remove and rebuild the other pump, now the other pump has the priority, 1 tick later 100% fluid will go through the other pump. ( while keeping chunk aligment unchanged).
I didn't realise that x_x. I suppose it's an alternative to alternating the pumps with circuit conditions, it might be tricky to get the right delay to avoid the leftover fluid fractions though.

With robots and the build order I agree it's too hard to reproduce accurately (for now at least). Also, after posting I discovered this thread and FFF 260 which shed a little more light on the topic. I'm not exactly sure what to expect in future changes.
I meant it only as an illustration of the mechanics, it's not really an alternative as you cannot really automate the process of removing and rebuilding a pump ( in vanilla ). It just shows the problem you tried to overcome i thought, in that its not going to "split" the fluid between the 2 pumps, the "first one" to update will steal all the fluid, if it can transfer all the fluid present which will leave the "2nd" pump dry. Chunk boundaries AND build orders are thing to look for when trying to do some very accurate prediction on such small scales.
Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
 Thank you, I didn't know about that thread either. That's a nice experiment. The exact "sloshing" algorithm of fluid moving back and forth is different to what I'm used to, but the principle is the same.
The principles got different names given to them depending on how many years of math or programming or physics one has as background it seems to me x). "Sloshing" is a new one to me, it describe the behavior of the water as the object of interest, this is like the physic way of seeing but given your experimentations for me personnaly, it feels more intuitive to consider the "the fluid system" as an "information system" with stream of "data" in "packet". That's how your experiment is pictured in my mind, it is similar to things i already knew in some way so it make sense, but it may be off topic, not sure if it make sense that things are organised this way in my mind x).
Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
mmmPI wrote: ↑
Tue Apr 23, 2024 11:18 pm
 It was interesting to read the experiments, but realistically, why not have a "loop" of fluid similar to picture (6) with the lube water and crude on the left side. Instead of having vertical pump , having the chemplant there, with 1 input in a pipe and the other on the other pipe separated by pump. This way you could send 200 light oil 200 water, 200 light oil, 200 water alternating, Would that work for you requirement of 0 waste ?
I'm not sure what that would look like, could you try making a prototype?
I would have to think a bit more about it, i was asking the question why not do this ? because i'm not sure it's possible, and thought maybe you thought of it already or had reasons to discard it i couldn't see, it was a thought from seeing your setup, it would be this i think :
Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
 I think you are suggesting, is craft for 60 units of fluid (the max that the machine will take in at once), and loop back the leftover 40 or 140 fluid, that's an option too. Not sure either which design would be "best" in terms of compactness/complexity, I'd be interested to see and might try myself.
But I had not thought about giving the machine the max that it will take at once, in my mind it was a rough idea, that the receipe could be changed while the "inputs" are still connected, so that when when chaning receipe, the machine would expell the excess input fluid it had stored back where it came from. Then logic would happen to loop the fluids around to present to machines with new receipe their newly required inputs. They would process the receipe while the logic for fluid bus is stopped, and the "fluid loop" would be active again after 1 receipe by default, to avoid having to count how many cycle and the LCM math.

Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
 Yes, that's where I think you could choose to do LCM (least common multiple) arithmetic optionally with combinators. It takes 30 of each fluid, so you'd have to craft in batches of lcm(30,100) = 300 (batches of 10 crafts at a time) or lcm(30, 200) = 600 (batches of 20 crafts at a time). There is a third option I think you are suggesting, is craft for 60 units of fluid (the max that the machine will take in at once), and loop back the leftover 40 or 140 fluid, that's an option too. Not sure either which design would be "best" in terms of compactness/complexity, I'd be interested to see and might try myself.

On a separate note, would you happen to know of a nice combinator setup to calculate lcm(x,y)? Trying to implement the Euclidean algorithm with combinators in a neat way is giving me a headache...
No i don't happen to know of a nice combinator setup for this ,i would be interested to have one for music creation x). Euclidean (algo)rythm are something i learned about recently LCM are used for mathematical notation of rythm, 5/16 or 3/16 meaning 5 or 3 beat during the 16 times of the music. I'd be more trying to do something about this that the water idea, i have the time for it, but it doesn't mean it will be a sucess :/ That's also part of why i'm not going to attempt to show the setup mentionned ealier, i'm not sure it can be done, and, i'd be busy with this other task :)
Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
 I did see that and I'm not sure how much the filter would help, if it could improve overall throughput by switching faster or not. it'll probably be easier to see when it's released I can just try it.
I thought it could make the "logic" easier in that instead of "enabling the pump" while "predicting which fluid will be pushed". It would allow to "enable the pump only for such fluid" by giving it the filter to expell excess fluid, it would push out fluid and automatically stop when the fluid has changed instead of forcing players to manage the timing and count/predict all the quantities that would be present in all pipes at all times, for such system to have no leftovers.

I saw your experiments as way of trying to achieve similar result before the feature is available x).
Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
 Also, after writing this out I had an idea for a challenge mode this morning. Items become itemfluids, inserters are disabled, you have to do precision fluid management even for basic crafting tasks with two components in assemblers. I have a good idea of the implementation details and I think this would lead to some cool strategies so I've started working on a mod to implement this idea. (Although it's my first Factorio mod it might take me some time to learn the conventions.)
Made me think of : https://mods.factorio.com/mod/LiquifyScienceandLab https://mods.factorio.com/mod/LiquifyRawMaterials and https://mods.factorio.com/mod/LiquifyIntermediates, but with inserters disabled ? That sound like a nightmare mod if precision is required x) In factorio all fluids are somewhat infinite, sulfuric acid being the only one requiring "iron". Else they are from water and crude, which are infinite making it forgiving.

There are little quirks in the game to smoothen the general experience that may get in the way of such precise setup, as you mentionned earlier, the logic to group entities containing the same fluids into a group, as can be interacted and "flushed" manually, this one can lead to some tiny amount of fluids being removed in the roundings to avoid some 0.000001 fluid blocking an entire setup with little way for players to understand. I'm not exactly sure when it can occur, more than it's a general principle in the game to not rely too much on fluid being hypothetically integer number in pipes. It would be the case in wagon or barrel, or in ratios over long time but not in pipes, in pipes it's complicated. It made me curious anyway, a mod that would force players to deal with that complexity would be very difficult x)
Top
mrvn
Smart Inserter
Smart Inserter
Posts: 5710
Joined: Mon Sep 05, 2016 9:10 am
Contact:
Contact mrvn

Re: Precision generic fluid routing and splitting

Post by mrvn »

A few thoughts on this:

1) Don't forget that the fluid system uses floating point arithmatic. It isn't precise. So it is totally possible that you sometimes loose a bit of fluid or even gain a bit of fluid. This might be 0.000001 fluid but if that gets left in the pipe somewhere then the next fluid won't go through.

2) Do you really need precise amounts of fluids in the tanks? Why not fill the tank up to 1000 and when you change recipes and need a different fluid then pump it empty and return the fluid to where it came from.

3) Which raises my 3rd issue: You only showed how to pump multiple fluids into a pipe. How do you split fluids from a mixed pipe into different tanks again? I imagine you need to have multiple pumps going out of a tank, fill each pump with one fluid type and then always stop the pipe when the tank is empty but the pump still has some fluid in it.

4) I would like to run a pipe (full) speed with some fluid. Then stop the flow, empty the pipe and push a final wave of fluid through to flush it. All the fluid in the pipe should be removed. If you can make that work reliable you can forget about imprecise fluid counts and work with a "best guess and return the overflow" system.
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Wed Apr 24, 2024 1:45 am
 Trying to implement the Euclidean algorithm with combinators in a neat way is giving me a headache...
So i did some researchs, and i made a thing to math out the GCD, which seem to be what the Euclidean algorithm is about:

viewtopic.php?f=193&t=113159

It's not doing the LCM though , just the GCD, not sure if that is what would be required ? It's still some more research x)
Top
User avatar
Ivelieu
Burner Inserter
Burner Inserter
Posts: 9
Joined: Mon Apr 22, 2024 2:06 am
Contact:
Contact Ivelieu

Re: Precision generic fluid routing and splitting

Post by Ivelieu »

mmmPI wrote: ↑
Wed Apr 24, 2024 1:10 pm
 this one can lead to some tiny amount of fluids being removed in the roundings to avoid some 0.000001 fluid blocking an entire setup with little way for players to understand. I'm not exactly sure when it can occur, more than it's a general principle in the game to not rely too much on fluid being hypothetically integer number in pipes.
Well, actually, I figured out the exact situations in which it occurs, which I put in my first post:
- If you have a segment of two or more pipe tiles in a row
- If you have a pump facing a tank that doesn't have three pipe-pump segments separate behind it
- If you push into a tank over 40% (10000) fluid full
mrvn wrote: ↑
Wed Apr 24, 2024 7:54 pm
 A few thoughts on this:

1) Don't forget that the fluid system uses floating point arithmatic. It isn't precise. So it is totally possible that you sometimes loose a bit of fluid or even gain a bit of fluid. This might be 0.000001 fluid but if that gets left in the pipe somewhere then the next fluid won't go through.

2) Do you really need precise amounts of fluids in the tanks? Why not fill the tank up to 1000 and when you change recipes and need a different fluid then pump it empty and return the fluid to where it came from.

3) Which raises my 3rd issue: You only showed how to pump multiple fluids into a pipe. How do you split fluids from a mixed pipe into different tanks again? I imagine you need to have multiple pumps going out of a tank, fill each pump with one fluid type and then always stop the pipe when the tank is empty but the pump still has some fluid in it.

4) I would like to run a pipe (full) speed with some fluid. Then stop the flow, empty the pipe and push a final wave of fluid through to flush it. All the fluid in the pipe should be removed. If you can make that work reliable you can forget about imprecise fluid counts and work with a "best guess and return the overflow" system.
1) That's the whole point of me posting though! I was able to figure out a way to get fluids to work with 100% precision, 0 leftover fluid, 100% of the time - something that I don't think anyone has done before in this game in vanilla. This is also why I used nixie tubes, because they round down fluid counts when directly connected to a tank, so if it has 999.99 fluid, it displays as 999 rather than factorio hover gui which would display 1000.

2) Yes, for the project I'm working on, I really do need precise fluid counts split across multiple machines, fully filling tanks up to capacity won't be an option. Although I'm filling tanks in this example, keep in mind that those tanks are placeholders for any machine which takes fluid also - the only reason I use tanks in my prototypes is because I can read their fluid contents with the circuit network, but currently I can't read the contents of an assembler's fluid buffer.

3) I hope that this below image demonstration makes more clear what I was able to accomplish. It's an example of how I can extend one of the blueprints I gave in the first post to automatically split three fluid types into six tanks. Note that all the pumps and pipes are completely devoid of fluid; alt-mode would display if there was even a small fraction left over. By all means, test it yourself too.

4) Not sure what you mean. I can't return the overflow if it gets stuck in small fractions in the pipeline, which my design prevents. The reason I want to fully clear a pipeline is so I can hotswap fluids on it, if I have a pipe always running full speed moving fluids I can't do that, not unless I do some very creative fluid sushi belt inside a pipe. Mind you, being able to hotswap fluids does mean I could make a fluid sushi belt... And with filters in pumps in the future that'll be possible (I think the FFF post even speculated to it).
screenshot_15491897_832x1088.png
screenshot_15491897_832x1088.png (470.04 KiB) Viewed 701 times


mmmPI wrote: ↑
Wed Apr 24, 2024 1:10 pm
 Thanks for the clarifications. I had done some (game-related) research on the problem of modelising fluid, what you describe make sense to me [...]
Yeah. The concept/algorithm goes by many names in 1d, 2d, 3d, even more dimensions, even theoretical infinite dimensional systems, and it's a concept that shows up in many places. For me I like the term sloshing, but balancing data throughput/packets is also reasonable. Dirichlet squares/magic squares is also a term. You can even describe the update rule with cellula automata which I've previously done academic research on. Very similar is the square lattice Ising model too... the list is endless.
mmmPI wrote: ↑
Wed Apr 24, 2024 1:10 pm
 I meant it only as an illustration of the mechanics, it's not really an alternative as you cannot really automate the process of removing and rebuilding a pump ( in vanilla ).
Well, if you want to split 100 fluid into two pumps the exact tick order doesn't matter that much, so the fact that the pumps alternate priority on their own is nice. Although I suppose you're right that mechanic can't be used for splitting precision fluids without either circuit conditions or automatically deconstructing them to force the order to change.
mmmPI wrote: ↑
Wed Apr 24, 2024 1:10 pm
 I would have to think a bit more about it, i was asking the question why not do this ?
Well, yes, you could alternate fluids on the input for the chemplant input buffer. My post here is just showing a proof of concept, I'm not claiming that it's the most efficient in terms of throughput, I didn't try super hard to optimise it. There are certainly a few different ways to arrange pipes, pumps and tanks to make the actual fluid hotswapping faster if you happen to be playing modded and have 20 different fluid types to hotswap between for example.
Thanks, I didn't know that existed. Those mods look like only using the itemfluids for storage not crafting, but it's a start for me to work from.
mmmPI wrote: I thought it could make the "logic" easier in that instead of "enabling the pump" while "predicting which fluid will be pushed". It would allow to "enable the pump only for such fluid" by giving it the filter to expell excess fluid, it would push out fluid and automatically stop when the fluid has changed instead of forcing players to manage the timing and count/predict all the quantities that would be present in all pipes at all times, for such system to have no leftovers.
Oh I see what you mean. Yes, using the fluid filter on a circuit condition instead of toggling the pump on/off with a circuit condition would work, but I think you still need circuit logic to handle the predicted fluid timing in order to guarantee that a filtered pump has a full 200/100 fluid available to push. All the combinators I used in my design are just for setting the right timings, no fluid filters are needed since you can encode the fluid type to a signal value instead, so I don't think you could save many combinators with that. Overall this fluid splitting design I'm proposing doesn't use many combinators to begin with.
mmmPI wrote: It's not doing the LCM though , just the GCD
Nice work. I'll have a look at it myself later and respond in that thread. Also, it's okay because
gcd.PNG
gcd.PNG (16.12 KiB) Viewed 701 times
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
Well, actually, I figured out the exact situations in which it occurs, which I put in my first post:
- If you have a segment of two or more pipe tiles in a row
- If you have a pump facing a tank that doesn't have three pipe-pump segments separate behind it
- If you push into a tank over 40% (10000) fluid full
I'm not convinced by those, i'm convinced by the blueprints that works, but those rules seems to apply too often compared to another logic that i was mentionning. Maybe there is a setup of the blueprint book that is meant to illustrate the statement and i didn't understand it. To me :

roughly in two distinct groups :
1) error due to rounding because of floating point math
2) error due to the game logic having special cases of fluid removal to prevent 0.00001 fluid or so blocking a system.

I don't think the second 2) triggers as often as "if you have a segment of more than 2 pipes in a row". It's only specific situations like exceptions to my understanding. Whereas the rounding due to floating points would occur more often, those could occur as often as when liquid is allowed to slosh, which would be the condition you describe, or at least the same magnitude of frequency, but in such case it would be sometimes rounded up, sometimes rounded down, not always the same direction of rounding down, which to my understanding is a special case that may make precise calculation of fluid quantity even more difficults than just floating point approximations.

Floating point approximation of 0.0000001 fluid missing for an integer count is also problematic for circuit detection, but there is a special case for circuits and for nixies tube i suppose that the fluid quantity is rounded UP instead of truncated when it is between 0 and 1, so that 0.5 fluid in pipes will be represented by 1 fluid, and 1.5 fluid in pipes will also be represented by 1 fluid in circuits, useful to know for such precision, and maybe counterintuitive.

Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Yeah. The concept/algorithm goes by many names in 1d, 2d, 3d, even more dimensions, even theoretical infinite dimensional systems, and it's a concept that shows up in many places. For me I like the term sloshing, but balancing data throughput/packets is also reasonable. Dirichlet squares/magic squares is also a term. You can even describe the update rule with cellula automata which I've previously done academic research on. Very similar is the square lattice Ising model too... the list is endless.
cellular automata is something i heard about hehe i can understand how it would work but i'm hobbyist, i don't always/often understand academic research, and i have no idea what square lattice ising model was before looking it up. I think i heard of it when searching how to model trains, which somehow could be seen as a fluid, or maybe more as data/packet, i suppose also as molecule of gas in a room, that will spread around and have average distance between them only thanks to feedback from train to train and their braking distance, it is endless and very interesting to me but i'll leave the higher dimensions for specialists x).
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Well, if you want to split 100 fluid into two pumps the exact tick order doesn't matter that much, so the fact that the pumps alternate priority on their own is nice. Although I suppose you're right that mechanic can't be used for splitting precision fluids without either circuit conditions or automatically deconstructing them to force the order to change.
I don't know if there is a misunderstanding, i think this is more the later part of the experiment, not the cool fluid sushi, but the crafting combinator-related "only pick up 10 fluid" from a tank. If you consider the later task, you can use circuit condition to trigger 2 pumps at the same time but it will not split the fluid only 1 will be getting it all. The pumps won't alternate priority on their own, unlike inserters outputting from an assembly.
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Well, yes, you could alternate fluids on the input for the chemplant input buffer. My post here is just showing a proof of concept, I'm not claiming that it's the most efficient in terms of throughput, I didn't try super hard to optimise it. There are certainly a few different ways to arrange pipes, pumps and tanks to make the actual fluid hotswapping faster if you happen to be playing modded and have 20 different fluid types to hotswap between for example.
Part of my suggestion came from overlooking the spacing , i realized when tried to make a mockup of what i meant : it only apply to certain things but you can't provide a chemplant with 2 different fluid input with the "easy/lazy" option :
easyonlywhensingleinputfluid.jpg
easyonlywhensingleinputfluid.jpg (229.4 KiB) Viewed 614 times
With such "loop" there wouldn't need to be precise count of how much fluid is sent i thought, you could have "some receipe" being made for say 30 seconds, and when they are "done", assembly and chemplant are turned off, pump activate to flush the leftovers , the fluids would loop around, and another combo of fluid could be provided to fit other set of receipe for "30 seconds". But that's not quite a finished product.
The fact that some of the fluid goes "off" the loop in your setup makes the precision "required" in how much fluid is going to be supplied, otherwide some leftover in the "branches" would cause the next fluid incoming to be blocked. Lazy setup i though would only makes player manage timing, not quantity of fluid branching out, i thought to make it "easier" given this is already not "easy" in general x).

The number of combinators i found is low already for what it does it wouldn't be to reduce it even further but rather to make it easier (for me) to use without having to use ratio calculators or implement such logic in a square loop it could be as "simple" as defining the fluids for each of the 4 side every 30 second with 1 second for the pumps to do the transfer. It would already mean quite some logic to handle with combinators to set the fluid for each side based on some threshold storage of the factory :)
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Nice work. I'll have a look at it myself later and respond in that thread. Also, it's okay because
Well thank you it make research easier x) i made a version that does the upper one, then the lower one , it's just adding 2 combinators, it doesn't take the absolute value, and both are not design to prevent overflow if the numbers are "too big" it will fail :

upper version that overflow more often :


better version, the last of the bottom lane that overflow less often :


I tested quickly the result with online calculator but it is possible i have made a mistake that i didn't see. It wasn't tested for a real purpose i mean.
Top
User avatar
Ivelieu
Burner Inserter
Burner Inserter
Posts: 9
Joined: Mon Apr 22, 2024 2:06 am
Contact:
Contact Ivelieu

Re: Precision generic fluid routing and splitting

Post by Ivelieu »

mmmPI wrote: ↑
Thu Apr 25, 2024 6:24 pm
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
Well, actually, I figured out the exact situations in which it occurs, which I put in my first post:
- If you have a segment of two or more pipe tiles in a row
- If you have a pump facing a tank that doesn't have three pipe-pump segments separate behind it
- If you push into a tank over 40% (10000) fluid full
I'm not convinced by those, i'm convinced by the blueprints that works, but those rules seems to apply too often compared to another logic that i was mentionning. Maybe there is a setup of the blueprint book that is meant to illustrate the statement and i didn't understand it. To me :

roughly in two distinct groups :
1) error due to rounding because of floating point math
2) error due to the game logic having special cases of fluid removal to prevent 0.00001 fluid or so blocking a system.
I can't blame you, I only found the "what" not the "why" in my head at first. I wasn't 100% sure myself, but after some testing I'm more confident on the reason, and I DONT think it's due to fluid floating point errors.

My first intuition was, I found sometimes fluid got stuck with 199.9 on a transfer, but without exception there was always a fluid leftover piece on another network component - the 0.1 fluid didn't vanish, it simply wasn't transferred. And if I do a few more transfer pulses, eventually that 0.1 leftover fluid will get swept up - in other words, factorio is dumb enough to forget about trace fluid amounts in active pumps and pipes, but smart enough to make sure no fluid is ever lost.
leftover_fluids.png
leftover_fluids.png (226.31 KiB) Viewed 527 times
Even in theory - assuming that fluid is encoded in the widely used IEEE 754 float format ( see https://www.h-schmidt.net/FloatConverter/IEEE754.html), the precision loss for even for the maximum vanilla storable 25k fluid is never going to exceed 0.001, and float formats are painstakingly designed to round correctly in arithmetic operations. In other words I don't think it is possible to even intentionally induce a rounding error to remove or create fluid - there are likely additional checks the Factorio developers added to ensure this in their dirichlet-esque fluid balancing algorithm.

So, what DO I think causes it? Well, I think it can be explained fully deterministically with two principles.
The first is "lazy loading" - when a pump activates for one tick, and it's behind another pump with 200 fluid, it doesn't actually move a full 200 units of fluid from the thing behind it to the thing in front of it. It has a buffer inventory. What happens if the pump is empty and it activates for one tick is, it moves some % of the fluid into its buffer, and a reciprocal % into the pipe in front - in just one tick of activation. What are these percentages? It is impossible to know the precise value without source code or inordinate testing, but with this basic experiment it seems to be around 73% to the buffer in front, and 27% to the pump's internal buffer.
measure_fluid.png
measure_fluid.png (474.69 KiB) Viewed 527 times
Well, that's interesting, if I have the pipe at the end it still has 0.3 fluid in it. Why is that, well, in one tick all the fluid goes into the tank, then in the next tick some of the fluid leaks out. Tanks naturally leak fluid this way, but how much exactly do they leak? If we extend the pipe length of this 0.3 pipe, it shows something a little interesting. After a few momements it leaks out another 0.3 fluid or so for each new pipe segment. Actually, sometimes it leaks out 0.4 or 0.2 on the GUI display for the tank instead. Hmm...
measure_fluid_2.png
measure_fluid_2.png (625.99 KiB) Viewed 527 times
If I use the tank storage value at read time and the amount leaked each iteration, I can more closely approximate the percentage tank fluid leaked from the tank per tick.
tank leak measurements.PNG
tank leak measurements.PNG (116.27 KiB) Viewed 527 times
And as expected the amount leaked decreases as I make the pipe length longer, because the tank itself leaks less fluid. It is around 0.4% to 0.38% fluid leaked overall, and if the tank has higher capacity or the equalisation cutoff is reached sooner, you might encounter a slightly higher or lower value of this threshold in practice -- putting one pipe next to a tank filled exactly 10k full leaves the tank with 9059.?? fluid and the pipe with 40.6?? fluid, suggesting a 0.406% empty rate, and doing so with a 25k (completely full tank) leaves 24900.3?? and 99.6?? fluid respectively, or in other words 0.3984%. In the code it is almost certainly set to 0.4% and the variation is due to equalisation cutoff.

What do I mean by "equalisation cutoff"? That's the second relevant principle of fluids in factorio. Suppose I have a row of pipes and their fluid values are like this:
0.0 0.5
When we update the next tick, depending on specific dirichlet/sloshing parameters, some of the fluid will migrate based on a percentage. If we assume a "slosh-rate" of 50%, let's say each pipe distributes 50% to its neighbors each tick. In factorio this value is fluid-dependent cause each fluid has different pressure values, at least in theory.
0.25 0.25

In this case the two pipes are perfectly distributed - there is a difference of exactly 0 between their values. But if we use this 50% parameter with three pipes the story is different:

0.0 0.0 0.5
0.0 0.25 0.25
0.125 0.25 0.125

The specific implementation will usually have a "damping factor" that slows the rate of transfer as they get closer to equilibrium. But the end result after a few ticks might end up looking like this:
0.16 0.19 0.16
Instead of what we want (assuming we hide small decimal points):
0.17 0.17 0.17
Even with a damping factor it can take a long time to reach a true equilibrium where the difference in fluid in each pipe is exactly 0. Depending on the values with IEEE 754 floats, it might never reach true equilibrium or even a stable state. And if it doesn't it has to continue doing very small fluid updates every tick.
So virtually every implementation instead has what I call an equalisation cutoff; if the difference in the two values is small enough, then stop sloshing the fluid. With any equalisation cutoff at all, sometimes fluid will get left in pipes and not pulled out. The exact equalisation cutoff in factorio can either be implemented like a fixed cutoff value (e.g. 0.1) or a relative percentage between the fluids, e.g. if the fluid holders have 99.9% similar fluid stored.

And remember, this is all without turbulence and all fixed to a 2d grid. If you remove those two constraints you get a something virtually identical to the world famous Navier-Stokes equations. Also in factorio I'm not sure if pressure changes as a function of temperature but that is also a real world parameter to consider.

Anyway, to me the most plausible explanation of leftover fluids is by using these two properties of the factorio implementation which I already know to be true, which do explain the problem on their own. On the other hand, I have not yet seen a demonstration of a convincing example of losing or gaining fluid due to a so-called floating point error, so it lacks plausibility to me as a different theory.
mmmPI wrote: ↑
Thu Apr 25, 2024 6:24 pm
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Well, if you want to split 100 fluid into two pumps the exact tick order doesn't matter that much, so the fact that the pumps alternate priority on their own is nice. Although I suppose you're right that mechanic can't be used for splitting precision fluids without either circuit conditions or automatically deconstructing them to force the order to change.
I don't know if there is a misunderstanding, i think this is more the later part of the experiment, not the cool fluid sushi, but the crafting combinator-related "only pick up 10 fluid" from a tank. If you consider the later task, you can use circuit condition to trigger 2 pumps at the same time but it will not split the fluid only 1 will be getting it all. The pumps won't alternate priority on their own, unlike inserters outputting from an assembly.
Ah... I should of tested it before realising that... you meant the exact opposite of what I thought you meant. Testing now I see the fluid in fact does not alternate, whoops. Well, the circuit logic to alternate pulses isn't that hard.
mmmPI wrote: ↑
Thu Apr 25, 2024 6:24 pm
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Well, yes, you could alternate fluids on the input for the chemplant input buffer. My post here is just showing a proof of concept, I'm not claiming that it's the most efficient in terms of throughput, I didn't try super hard to optimise it. There are certainly a few different ways to arrange pipes, pumps and tanks to make the actual fluid hotswapping faster if you happen to be playing modded and have 20 different fluid types to hotswap between for example.
Part of my suggestion came from overlooking the spacing , i realized when tried to make a mockup of what i meant : it only apply to certain things but you can't provide a chemplant with 2 different fluid input with the "easy/lazy" option :
Well, it's not that pretty, but this works:
2_fluid_1_chemplant.png
2_fluid_1_chemplant.png (167.4 KiB) Viewed 527 times
It occurs to me you can actually use these crafting combinator machines as diodes/valves. They always return the overflow fluid from unsetting a recipe to only one of the input sides. Incredibly slow to use as a valve though given the long set/unset times.
crafting_combinator_acts_as_valve.png
crafting_combinator_acts_as_valve.png (195.63 KiB) Viewed 527 times
mmmPI wrote: ↑
Thu Apr 25, 2024 6:24 pm
 The number of combinators i found is low already for what it does it wouldn't be to reduce it even further but rather to make it easier (for me) to use without having to use ratio calculators or implement such logic in a square loop it could be as "simple" as defining the fluids for each of the 4 side every 30 second with 1 second for the pumps to do the transfer. It would already mean quite some logic to handle with combinators to set the fluid for each side based on some threshold storage of the factory :)
I see what you mean. It's hard calculating the timings both in theory and in practice due to circuit calculation delays.

I took some time to fully design a sushi fluid prototype, this one works on 7 fluids and a sushi loop of 16 tanks. It auto refills, does not break if sushi fluids are fully consumed, and has a configurable clock cycle and I have tested that it works 100% fine with a clock cycle of both 60 and 250 ticks (the blueprint has 250 ticks but it's easy to change). I also made it expandable being able to set the amount of different fluid inputs, their order is essentially set to the order of the input tank row but it could be fluid signal encoded with some extra work.


Also, two recommendations for practical use:
- Don't change the clock cycle time or other params during operation. Best to empty it out first.
- I tested it with precise fluid extractors, "dummy" machines which pull 200 fluid out at a time into a void, and it refills. But if you pull out amounts from the sushi tanks that aren't precise, I don't know if it would jam or not. Keep in mind that you might not spot a jam straight away since tanks, pipes and pumps can all contain fluid inside.
fluid_sushi_overall.png
fluid_sushi_overall.png (1.28 MiB) Viewed 527 times
fluid_sushi_guide.png
fluid_sushi_guide.png (869.73 KiB) Viewed 527 times
EDIT: copied wrong blueprint, correct one below.
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 - If you push into a tank over 40% (10000) fluid full
When attempting to redo the setup shown with the 5.2 and 10.4 leftover fluids with various starting of initial fluid in the left tank and using the feature of the editor to "tick once", i realized sometimes there will be some leftovers in the pump even when the tank contain only 9.4K fluid. But sometimes you can go a little over 10K without having leftovers. :?

I am still not convinced by the following explanations :
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 My first intuition was, I found sometimes fluid got stuck with 199.9 on a transfer, but without exception there was always a fluid leftover piece on another network component - the 0.1 fluid didn't vanish, it simply wasn't transferred. And if I do a few more transfer pulses, eventually that 0.1 leftover fluid will get swept up - in other words, factorio is dumb enough to forget about trace fluid amounts in active pumps and pipes, but smart enough to make sure no fluid is ever lost.
Even in theory - assuming that fluid is encoded in the widely used IEEE 754 float format ( see https://www.h-schmidt.net/FloatConverter/IEEE754.html), the precision loss for even for the maximum vanilla storable 25k fluid is never going to exceed 0.001, and float formats are painstakingly designed to round correctly in arithmetic operations. In other words I don't think it is possible to even intentionally induce a rounding error to remove or create fluid - there are likely additional checks the Factorio developers added to ensure this in their dirichlet-esque fluid balancing algorithm.
I understand you rule out floating point roundings. That make sense as it seem unrelated with the previous observation of fluid leftover in the pump, it's none of the 1) or 2) i proposed that would justify the leftover fluid. That's a another thing more related to the fluid mechanic itself, but then (imo) you seem to mix up different things :
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 So, what DO I think causes it? Well, I think it can be explained fully deterministically with two principles.
The first is "lazy loading" - when a pump activates for one tick, and it's behind another pump with 200 fluid, it doesn't actually move a full 200 units of fluid from the thing behind it to the thing in front of it. It has a buffer inventory. What happens if the pump is empty and it activates for one tick is, it moves some % of the fluid into its buffer, and a reciprocal % into the pipe in front - in just one tick of activation. What are these percentages? It is impossible to know the precise value without source code or inordinate testing, but with this basic experiment it seems to be around 73% to the buffer in front, and 27% to the pump's internal buffer.
To me this is related to the update order,after considering the action of "entity" the game consider each connexion of that entity, so when dealing with a "pump", it will first update one of its connexion, then the other. Then the pipe just after it, its first connexion, and its second connexion. It could also be the pipe that update first, and then the pump depending on which entity was last built. The build order doesn't have to follow the flow of fluid.

My "impression" is when decomposing tick per tick, the result of quantity are determined by the build order, which is not the case when considering the throughput of a full lane over time. For the quantity of fluid to "settle" in a hypthetic final state of steady flow, the system would need to have the "information", the "propagation of the dirichlet algo", the "sloshing" time to reach the "end of the lane" for the information to start converging into the final step from successive iteration of small corrections.

This i'm not sure i can explain well the relation of the intuive approach for the whole system and how it influence per tick and per pipe/entity measurement but i am under the impression that the % you are attempting to math out are related to this phenomenon. Or that at least the % approach is not going to be enoug to explain.

I think from a very abstract point of view it can be linked with the way you have fluids circling around the loop with an empty tank alternating. If there was no empty tanks, then fluids couldn't move from one tank to another because they can't mix, they can only move in an empty tank so there needs to be "gaps" existing for fluid to move in a discrete world as in your setup with tanks.

In the regular game though, this is not a requirement onto players. If you have a situation of "steady flow" the fluid amount in each pipe will stabilize over time. It will not create "moving gaps" that are visible, the "update wave" is becoming tinyer and tinyer in a variation of each pipe quantity from one tick to another that is only touching less and less significant digit over time.

The systems are a little different when considering what you call "lazy loading", this is the case in YOUR setup imo (not a critic just the name of the thing), only transfer fluid when the next tank is 100% empty. This means only half of the time a tank is getting input, and half of the time a tank is outputting toward the next one. In factorio pipe network though it's not the case, every tick each entity does both input and output.

This i think this can be made visible with this :
buildorder quantity relation.jpg
buildorder quantity relation.jpg (108.07 KiB) Viewed 483 times


20 fluid need to be transfered per tick for 1200 fluid per second which is the result of the formulas for fluid flow over long time. The last pipe before the pump will always contain around 20 fluid in a situation of steady flow when the sloshing is over. But the quantity in the pipe just before it depend weirdly enough on the build order of the pipes that do not have a quantity attached to them, closer to the left most tank. Deleting and rebuilding those will have no effect on some pipes further down the chain, but impact 1 in particular.

The last pipe of the lane having a steady "20 fluid" means that every tick roughly 20 fluid enters in, and leave the pipe which is visible on the pump placed later.

But how and why is the previous pipe showing sometimes 5 or sometimes 25 fluids ? that's a discrependacy of exactly the throughput, on a single pipe, this is related to the build order i think as illustrated when deleting and rebuilding some pipes earlier in the lane not even directly connected to the pipe they influence the steady quantity at the end.

Your explanation to me is bound to be incompletly describing what can happen in those setups if it doesn't propose explanations that would reflect how the build order yield different results. It doesn't mean the blueprints aren't working or wouldn't work 100% of time, it can happen that you have isolated some situations where there is no influence by the build order. Which is what i suspect, is done when using pumps like you do. But i do not think the % of water transfer are the proper approach for this problem, i think it would maybe unfortunatly require to dig into the game code to have more accurate generalist predictions AND have knowledge of the build order for precise setups that do not remove its impact.

I think your setup works because it removes those hard to predict situation where the build order would cause discrepancies.

I think this is also a different phenomenon than what you mention here :
noerroheretomemeasure_fluid.jpg
noerroheretomemeasure_fluid.jpg (358.3 KiB) Viewed 483 times
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 Well, that's interesting, if I have the pipe at the end it still has 0.3 fluid in it. Why is that, well, in one tick all the fluid goes into the tank, then in the next tick some of the fluid leaks out. Tanks naturally leak fluid this way, but how much exactly do they leak? If we extend the pipe length of this 0.3 pipe, it shows something a little interesting. After a few momements it leaks out another 0.3 fluid or so for each new pipe segment. Actually, sometimes it leaks out 0.4 or 0.2 on the GUI display for the tank instead. Hmm...
That's no leak to me, fluid is pushed into a pipe in this particular setup, not in the tank, fluid would not be leftover in pipe only if tanks had a lower depth than pipe regarding those pictures => viewtopic.php?t=19851Tons

If you try with long lane of boilers, that do have a lower depth than the pipe, then the pipe after the pump and before the long lane of boiler will have 0 fluid left in it as long as the boiler do not have more than 100 in their buffer. When all boilers have 100 water in their buffer, then the "additionnal" depth they have compared to the pipe is filled, and thus the pipe start to have some leftover fluids, in a setup like this :
boilerdeeperthantank.jpg
boilerdeeperthantank.jpg (122.26 KiB) Viewed 483 times
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 If I use the tank storage value at read time and the amount leaked each iteration, I can more closely approximate the percentage tank fluid leaked from the tank per tick.
tank leak measurements.PNG
And as expected the amount leaked decreases as I make the pipe length longer, because the tank itself leaks less fluid. It is around 0.4% to 0.38% fluid leaked overall, and if the tank has higher capacity or the equalisation cutoff is reached sooner, you might encounter a slightly higher or lower value of this threshold in practice -- putting one pipe next to a tank filled exactly 10k full leaves the tank with 9059.?? fluid and the pipe with 40.6?? fluid, suggesting a 0.406% empty rate, and doing so with a 25k (completely full tank) leaves 24900.3?? and 99.6?? fluid respectively, or in other words 0.3984%. In the code it is almost certainly set to 0.4% and the variation is due to equalisation cutoff.
I think you'd be interested by the previous link x) There is indeed a 4/10 factor in one of the equation for the flow of fluid. This specific part :
Flow

Okay, now we know and understand basic concepts of every entity and it is time to figure out how they interacts between themselves.
Each pair of adjacent entities balances their pressure every tick, this happens in the following way: entity with bigger pressure (here I means current pressure, there was a formula in the beginning) transfer its liquid to the entity with smaller pressure. This direct transfer is equal to (Pressure_A - Pressure_B)*0.4 (this coefficient is unique for every type of liquid, but as far as I know it is always 0.4). Also there are exist an inertial transfer, it can be calculated as part of previous tick flow: Previous_Flow * 0.59 (also predefined coefficient). Inertial component is limited with 10% of the max capacity of the destination entity.
So, total formula is (Pressure_A - Pressure_B) * 0.4 + Limited[Previous_Flow * 0.59, Target_Capacity * 0.1].
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 Even with a damping factor it can take a long time to reach a true equilibrium where the difference in fluid in each pipe is exactly 0. Depending on the values with IEEE 754 floats, it might never reach true equilibrium or even a stable state. And if it doesn't it has to continue doing very small fluid updates every tick.
So virtually every implementation instead has what I call an equalisation cutoff; if the difference in the two values is small enough, then stop sloshing the fluid. With any equalisation cutoff at all, sometimes fluid will get left in pipes and not pulled out. The exact equalisation cutoff in factorio can either be implemented like a fixed cutoff value (e.g. 0.1) or a relative percentage between the fluids, e.g. if the fluid holders have 99.9% similar fluid stored.
I don't know if the previous formulas would help you tell what is used as mechanism to stop the everending sloshing or if there is one. It's not like i fully understand them but it seem like complementary to your approach, adding things on top of the 40% factor that you identified already.

I don't know how to "use the tank storage value at read time " to have ability to read more digits than the circuits give. If the circuits or hovering shows 34.6 fluid in all the boilers after it ''settle", i understand it may continue stabilizing/sloshing , but i don't know if it is necessary to prevent that from happening. As fluid are calculated each tick anyway in the game no more no less. There may be some needless updates when quantities didn't change, situation where a damping factor would cut off on the calculations but to me that would only concern pipes that have no throughput for "some"time, so "idle" factories/buildings/production assemblies. Considering a well interconnected network of pipes, i thought it becomes increasingly less and less worth it to try and do something to cut off on calculations only when fluid have enough time to settle given the increasing number of entites consuming/producting fluids making them chaotic. The factorio game to me can be seen as an algorithm attempting to solve dirichlet magic square, but with a twist that machines/players are regularly updating the numbers outside of the squares x). In such case, is it really necessary to try and tell when the algo should stop updating its value in some part ? i explained why i think not, but it is speculative/hypothesis i'm asking the question to be more informed if there are things to say the other way that i didn't consider.

I don't have much knowledge on floating points arithmetic i mostly learned about those things in factorio circuits, not in factorio game code. I couldn't tell if a damping factor high enough would prevent the game to ever have floating point approximations. If the damping factor would act like the minimum difference between 2 fluid quantity in adjacent pipe, which would then means the "end result" would not be "flat" but like a staircase, [100] , [100-(1*damping factor)] , [100-(2*damping factor)] , [100-(3*damping factor)].I think something like this is more visible with the heat in adjacent heat pipes, where it would be 1°C as a flat damping factor. It is not exactly the same though , as heat never "slosh" backward. Maybe it's an equalization cutoff ?

Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 And remember, this is all without turbulence and all fixed to a 2d grid. If you remove those two constraints you get a something virtually identical to the world famous Navier-Stokes equations. Also in factorio I'm not sure if pressure changes as a function of temperature but that is also a real world parameter to consider.
I had heard somewhere that it was a particular case of the Navier-Stokes equations, or that it had a relation at least but i wasn't sure if it was said about the factorio game fluid system, or the dirichlet magic square/sloshing algo. Which in a way made sense to me, but it's not like i can understand why by myself.

I would say pressure do not change as a function of temperature. But in game water can have different temperature when in editor mode, and thus the game also has a part that will math out the temperature of the fluids, not only their quantity, both propagate, there could be even worse sushi if you had some cooling and heating need for certain fluids x) like side loop to re-introduce fluids in the main loop when their temperature is monitored to be correct or some other nightmarish-difficulty setup. That sound like 2 or 3 level forward.
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Anyway, to me the most plausible explanation of leftover fluids is by using these two properties of the factorio implementation which I already know to be true, which do explain the problem on their own. On the other hand, I have not yet seen a demonstration of a convincing example of losing or gaining fluid due to a so-called floating point error, so it lacks plausibility to me as a different theory.
Floating point error magnitude is much lower than what is being made visible when considering the fluid leftovers. I agree that other explanations from fluid mechanic implementation are source of what's visible in game by hovering or in circuits,one or the other, i'm not sure which one are the 2 properties, or if we are making the same things into categories at this point. I think floating points errors detection would require either exporting many (all) significant digits in spreadsheet, or make a setup that takes a lot of time for the error to accumulate in a specific way that would make it visible in game so can be ruled out for now.

I'm not sure your explanation was complete for the fluid mechanic in general, maybe i don't understand the context of your reasonning :"why you did not consider some other things" maybe is just me not seeing. Although given the working blueprint, it is clear to me that your explanations/understanding are enough to overcome the problem with a functionnal and very cool fluid sushi without considering nor build order nor floating point errors.


Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Well, it's not that pretty, but this works:
I think there are other ways maybe with underground pipes. Maybe it's against theory, but your setup actually still works with default settings if some pumps are removed :



At least for the cycling fluid and keeping "27K" of each in the system at all time not creating too much or jamming. Maybe it would be possible to extend the 3 straight pipes in a row into an underground pipes and open different shape, using 2 of such loop in parrallel for input 1 and input 2 ?
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 It occurs to me you can actually use these crafting combinator machines as diodes/valves. They always return the overflow fluid from unsetting a recipe to only one of the input sides. Incredibly slow to use as a valve though given the long set/unset times.
That's nice but what if there are 2 fluid inputs though ? :D
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 I see what you mean. It's hard calculating the timings both in theory and in practice due to circuit calculation delays.

I took some time to fully design a sushi fluid prototype, this one works on 7 fluids and a sushi loop of 16 tanks. It auto refills, does not break if sushi fluids are fully consumed, and has a configurable clock cycle and I have tested that it works 100% fine with a clock cycle of both 60 and 250 ticks (the blueprint has 250 ticks but it's easy to change). I also made it expandable being able to set the amount of different fluid inputs, their order is essentially set to the order of the input tank row but it could be fluid signal encoded with some extra work.
Indeed, it gets harder and harder to keep everything in sync the more you add x). Although i'm not convinced by all the explanations, i can see the blueprint works very well.

I have attempted to click a tank and delete "27K" of a water, and it will replenish after several cycle, also to remove some random quantity and it worked.

It is more robust than you think it is maybe ? given that it also work when removing 1 pump in every section provided it's not the last or first one ? :D

I'm not sure it's absolutely necessary to give a definite number of pulse , rather than maybe a continuous signal that would last the same amount of time as the pulses combined. Provided each section containing a fluid is separated by an empty section, and pumps opening and closing a section can be triggered alternatively "some time" with little margin of safety, to make sure all fluid contained in a "section" will be transfered to the next without leftovers. That's is fairly well demonstrated by your setup to me and seem an easier path to implement functionnalities around the concept. Allowing some imprecision, considering a section will never have more fluid in it than its max capacity and they would all have the same. A section being not only a tank or a pipe but some entities between 2 pumps. That make me think of a lock gate or floodgate ? in a canal system for boats not sure the proper word.
Still with the idea of even and odd clock cycle, but considering tansfer is allowed during the first 15 ticks a (odd or even) clock cycle that last 20 ticks. To make it easier to synchronize, may be it will be off by 1 or 2 after some reworking but won't matter if the first 2 tick of a cycle are missed due to combinator delay it will still be 15 ticks of transfer every 40 ticks (15/20 and 0/20 odd/even).

The relation between quantity of fluid and number of pulse of activation maybe can be "broken" if the amount of time of "transfer" allowed is always enough to refill entirely the section. It may require "loading" on the loop, in parralel rather than in serie, so that there is only the pumps inside the loop that may contain different fluid and not the those from the single branch used for loading. The refill counter wouldn't be necessary anymore. That's when i try to make a lazy version that i'm thinking about that. It would be less impressive but with the aim of making it easier to adjust/ configure.

It is inspiring :)
Top
User avatar
Ivelieu
Burner Inserter
Burner Inserter
Posts: 9
Joined: Mon Apr 22, 2024 2:06 am
Contact:
Contact Ivelieu

Re: Precision generic fluid routing and splitting

Post by Ivelieu »

First I want to say I'm glad you like my maniac fluid sushi :D
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 - If you push into a tank over 40% (10000) fluid full
When attempting to redo the setup shown with the 5.2 and 10.4 leftover fluids with various starting of initial fluid in the left tank and using the feature of the editor to "tick once", i realized sometimes there will be some leftovers in the pump even when the tank contain only 9.4K fluid. But sometimes you can go a little over 10K without having leftovers. :?
Ahh.. yes, I should of been more precise with my wording. If it is over 10k the leftover issue *might* happen, it's not on all ticks after 10k. For the same reason what I call "lazy loading" doesn't always happen on pump activation either.
When you say you used the editor to tick once, I'm confused. If I run it for as many ticks as it needs then I never seemed to get overflow. So it wouldn't be realistic to do only one tick step through - we care about the fluid permanently getting stuck in the pump/pipe, not just for one tick.
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
 I am still not convinced by the following explanations :
To me this is related to the update order,after considering the action of "entity" the game consider each connexion of that entity, so when dealing with a "pump", it will first update one of its connexion, then the other. Then the pipe just after it, its first connexion, and its second connexion. It could also be the pipe that update first, and then the pump depending on which entity was last built. The build order doesn't have to follow the flow of fluid. [...]
This is getting very long and technical so I'm going to start from the beginning and give a summary. Here are some properties we know about fluid in factorio:
- (1) Under some circumstances a pump will forget about its contents and not push some fluid (what I call "lazy loading")
- (2) Pipe chunk order, and pipe chunk alignment, both affect the way fluid propogates through a chain of pipes.
- (3) It would seem implausible (though not 100% ruled out) that rounding errors occur which can add or remove small amounts of fluid.
- (4) Pipes sometimes get stuck with small amounts of fluid inside.
- (5) Tanks next to an empty pipe will leak some amount of fluid to the adjacent pipe. (I claim this is coded to be 0.4%, but after reading the code and the document I understand it is equivalently 1/250, and the value is different for pipes/pumps/etc.)
- (6) In the certain setup I showed earlier (measure_fluid.png), with a one tick pulse a pump will transfer 72.9 fluid to the pipe in front and 27.0 to its internal inventory. Because fluid is allowed to flow in only one direction, it is unaffected by build order. But this precise ratio is more likely derived from other properties rather than an intrinsic property of the pump.

Here are some things I was personally not 100% sure of:
- The exact heuristics used by factorio with fluids. Specifically:
- (6) Whether the equalisation cutoff exists and how large the is if it does exist. The equalisation cutoff is defined to be the smallest (relative or absolute) value of fluid, and if the difference in fluid between two pipes is less than or equal to this value, dirichlet fluid updates stop occurring.
- (7) Whether there is a damping factor for fluid updates and how large it is if it exists. What I mean by a damping factor is different to an equalisation cutoff; the damping factor is defined to be a value which additionally slows or accelerates the rate of fluid transfer as smaller amounts are transferred, in order to converge to a stable state faster (so you don't "overshoot" stabilisation with oscilations). It can also be thought of as "momentum"; with a negative damping factor fluid builds momentum faster than velocity, with a damping factor of 0 they are equal, and with a positive damping factor momentum builds slower (i.e. decays faster) than velocity.
- (8) Whether the source code has meaningfully changed since that 2020 forum post about 0.17 factorio disassembly. Based on some test I've done now, I think it has not changed. Actually I read this disassembled function before but I didn't understand it at first, rereading it makes a little more sense to me now. The code indicates two types of fluid movement, "pressure-flow" and "momentum-flow" - the former is dirichlet updates/sloshing, and the latter is a damping factor.

What I found originally was that four pumps and three pipes in a row acted such that in a 7 tick window, all the fluid from one end is transferred to the other. This is likely due to the deterministic "momentum-flow" which always completes the transfer with the same calculations. On the other hand, your modification replacing one pump with two pipes creates some indeterminism based on the pipe placement order. What it seems like, is that it so happens that any pipe placement order with three pipes in a row also happens to satisfy "momentum-flow" evaluations to fully transfer all fluid with no leftover. In total, (40/100)% of fluid movement is due to dirichlet updates, and (59/100)% of fluid movement is due to fluid damping - putting the damping factor at (59/100)/(40/100), or 147.5% of the sloshing rate.

Well, I went ahead and read https://www.reddit.com/r/factorio/comme ... cts_myths/ written 6 years ago and last updated 2 years ago by haibane tenshi. While the document is probably technically correct, the equations rely on ambiguous subscript and superscript making it hard to read. For example, it took me some time to figure out μ is 0.4, the "pressure-to-speed-ratio" and v is 0.59, the "flow-to-energy-ratio", it's not defined anywhere. Considering the author does not make comparisons to Navier-Stokes I suspect they haven't studied it, also. The equations match the Lisp code which is easier to read and helps verification.

Important takeaways from that document:
- Flow rate through a row or "line" of pipes is unaffected by build order
- Flow rate through a junction of pipes is affected by build order
- Flow rate through a row is affected if the start and end of the row have different minimum pressure values; for highest flow, having a 0 minimum pressure entity (pipes or storage tanks) at one end and a -100 pressure entity at the other end (boilers, steam engines, steam turbines, heat
exchangers, mining drills) will increase throughput slightly.
- Underground pipes only count as 2 "fluidboxes", for example if you build a row of 8 pipes then by comparison underground pipes have four times less fluidboxes to propogate the fluid through.
- Pumps now have 400 max "pressure" compared to 200:
haibane_tenshi wrote: ↑
Fri Aug 19, 2022 11:40 am
 I looked into game files and believe this is due increase in maximum pressure on pump (200 -> 400). I'm kinda impressed that someone managed to fit the formula to this data.
It overall has limited usefulness in this context because it doesn't examine fluid throughput under precision fluid manipulation. I also disagree about the author's stable state conclusions. They mention it first in section 2.2.2 as an assumption in their words saying the fluid values converage to equal values over time. My experiements show this is false even for rows of fluid, they don't converge to an equilibrium, and it's highly unlikely any fluid network converges to a stable state in general.

Now the problem of solving the "why" is: explain property (4) using all the other information we know or suspect. I think while all the properties are interesting, I believed only (1) and (6) together can explain (4). Now, after understanding this source code function properly, and reading that document with flow rate calculations, I realise that there is no equalisation cutoff, so (6) is irrelevant.

If I try to explain (4) with (2), which is what I think you're suggesting, we run into a problem - no matter what the order of the pipe updates, they should still always fully clear. On its own, the build order changing the update rule can't explain why fluid gets stuck in pipes sometimes. Now, I am more convinced it is due to the damping factor being higher than the sloshing factor, which creates situations where the fluid busy-loops and moves back and forth.
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
 The systems are a little different when considering what you call "lazy loading", this is the case in YOUR setup imo (not a critic just the name of the thing), only transfer fluid when the next tank is 100% empty. This means only half of the time a tank is getting input, and half of the time a tank is outputting toward the next one. In factorio pipe network though it's not the case, every tick each entity does both input and output.

This i think this can be made visible with this :
20 fluid need to be transfered per tick for 1200 fluid per second which is the result of the formulas for fluid flow over long time. The last pipe before the pump will always contain around 20 fluid in a situation of steady flow when the sloshing is over. But the quantity in the pipe just before it depend weirdly enough on the build order of the pipes that do not have a quantity attached to them, closer to the left most tank. Deleting and rebuilding those will have no effect on some pipes further down the chain, but impact 1 in particular.

The last pipe of the lane having a steady "20 fluid" means that every tick roughly 20 fluid enters in, and leave the pipe which is visible on the pump placed later.
But how and why is the previous pipe showing sometimes 5 or sometimes 25 fluids ? that's a discrependacy of exactly the throughput, on a single pipe, this is related to the build order i think as illustrated when deleting and rebuilding some pipes earlier in the lane not even directly connected to the pipe they influence the steady quantity at the end.
Yeah, I agree. Which pipe has which amount of fluid is affected by build order. However, if the source and sink both consume and provide as much as they can (i.e. what you call a "steady flow" situation), the flow rate ends up still being the same regardless of build order (at least according to the haibane tenshi document). I am glad that I didn't have to do the complicated math and I can point to this haibane tenshi document and say, "I think this is correct". I didn't verify it line by line. But it is strong evidence against the theory that build order creates leftover fluid.
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 Even with a damping factor it can take a long time to reach a true equilibrium where the difference in fluid in each pipe is exactly 0. Depending on the values with IEEE 754 floats, it might never reach true equilibrium or even a stable state. And if it doesn't it has to continue doing very small fluid updates every tick.
So virtually every implementation instead has what I call an equalisation cutoff; if the difference in the two values is small enough, then stop sloshing the fluid. With any equalisation cutoff at all, sometimes fluid will get left in pipes and not pulled out. The exact equalisation cutoff in factorio can either be implemented like a fixed cutoff value (e.g. 0.1) or a relative percentage between the fluids, e.g. if the fluid holders have 99.9% similar fluid stored.
I don't know if the previous formulas would help you tell what is used as mechanism to stop the everending sloshing or if there is one. It's not like i fully understand them but it seem like complementary to your approach, adding things on top of the 40% factor that you identified already.
Yes. So the "big thing" I realised going over the code was regarding this experiment. I have two tanks and one has 100.2 fluid and one has 99.8 fluid. And the fluid amount isn't changing per tick. There are two explanations: either it is busy looping, or there is an equalisation cutoff. But I reproduced this experiment with different fluid amounts - in this case it is 200 fluid across both, but I tried with 400, 100, even 10, I tried with boilers, and I realised that the difference is proportional to the fluid amount - the difference was always at 0.4% for the tanks.
two_possibilities.png
two_possibilities.png (393.75 KiB) Viewed 398 times
If the difference in fluid after apparent stabilisation was the same irrespective of the total fluid in the system, it would imply that an equalisation cutoff is being used. But I found the opposite result, which (supported by the lack of such an equalisation cutoff in the source code presented earlier), strongly suggests this fluid system is actually busy looping and updating fluid twice per tick. It never reaches a stable state, it just reaches a state where the fluid values continue to oscilate internally, and this particular setup makes it clear that oscilation never converged to similar fluid values.

This now seems to be the most plausible theory to me, to explain why fluid gets stuck in pipes - and this is affected by build order slightly. If in the process of updating fluid sloshing, it ends up with a small amount of fluid in a pipe that fully transfers in one tick, in that one tick it can transfer both forward and backwards so it gets stuck in one spot.
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
 I don't know how to "use the tank storage value at read time " to have ability to read more digits than the circuits give.
So, in that above experiment I did, I realised that the hover gui display follows some rules. Fluid containers display 3 significant figures and the visual display always rounds down to those 3 s.f. If fluid is between 100 and 101, it'll always display 100 but if it's between 99 and 100, it'll display between 99.0 and 100. And it will also never display more than 2 decimal places - 5.55 fluid displays as 5.5, so if the value is 10 or less it displays only 2 s.f. This example has three tanks filled with a total of 800 fluid, I would expect them to have 266.67 fluid but the gui display only shows 266, meaning that I know it rounded down from __.67 to __.0 in the display Likewise, if I do the same test with 80 fluid instead, I indeed see the 26.6 readout - the last digit is always rounded down in the gui display.
displayed_vs_actual_2.png
displayed_vs_actual_2.png (109.65 KiB) Viewed 398 times
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
 I would say pressure do not change as a function of temperature. But in game water can have different temperature when in editor mode, and thus the game also has a part that will math out the temperature of the fluids, not only their quantity, both propagate, there could be even worse sushi if you had some cooling and heating need for certain fluids x) like side loop to re-introduce fluids in the main loop when their temperature is monitored to be correct or some other nightmarish-difficulty setup. That sound like 2 or 3 level forward.
I like your idea on temperature controlled sushi :) Can finally fulfill my video game dream of being a food safety inspector in factorio! This gives me new ideas for mods... Fluid crafting by mixing them in tanks, if two different types of fluid touch they might mix into something new based on the ratio of what was provided...
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 Well, it's not that pretty, but this works:
I think there are other ways maybe with underground pipes. Maybe it's against theory, but your setup actually still works with default settings if some pumps are removed :
At least for the cycling fluid and keeping "27K" of each in the system at all time not creating too much or jamming. Maybe it would be possible to extend the 3 straight pipes in a row into an underground pipes and open different shape, using 2 of such loop in parrallel for input 1 and input 2 ?
Hmm... I suppose that pump wasn't necessary then. I only devised 4 pumps 3 pipes through trial and error, I didn't think to try that specific combination. Interesting.
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
Ivelieu wrote: ↑
Wed Apr 24, 2024 11:08 pm
 I see what you mean. It's hard calculating the timings both in theory and in practice due to circuit calculation delays.

I took some time to fully design a sushi fluid prototype, this one works on 7 fluids and a sushi loop of 16 tanks. It auto refills, does not break if sushi fluids are fully consumed, and has a configurable clock cycle and I have tested that it works 100% fine with a clock cycle of both 60 and 250 ticks (the blueprint has 250 ticks but it's easy to change). I also made it expandable being able to set the amount of different fluid inputs, their order is essentially set to the order of the input tank row but it could be fluid signal encoded with some extra work.
Indeed, it gets harder and harder to keep everything in sync the more you add x). Although i'm not convinced by all the explanations, i can see the blueprint works very well.

I have attempted to click a tank and delete "27K" of a water, and it will replenish after several cycle, also to remove some random quantity and it worked.

It is more robust than you think it is maybe ? given that it also work when removing 1 pump in every section provided it's not the last or first one ? :D
Yes the blueprint is definitely more robust than it needs to be. I am certain some timings can be cut down, I just did conservative overestimates - my design philosophy is, first make something that works, then try to optimise it. The fact you were able to remove that pump and replace with pipes and it never jams still, I don't have a good explanation for that.
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
 I'm not sure it's absolutely necessary to give a definite number of pulse , rather than maybe a continuous signal that would last the same amount of time as the pulses combined. Provided each section containing a fluid is separated by an empty section, and pumps opening and closing a section can be triggered alternatively "some time" with little margin of safety, to make sure all fluid contained in a "section" will be transfered to the next without leftovers. That's is fairly well demonstrated by your setup to me and seem an easier path to implement functionnalities around the concept. Allowing some imprecision, considering a section will never have more fluid in it than its max capacity and they would all have the same. A section being not only a tank or a pipe but some entities between 2 pumps. That make me think of a lock gate or floodgate ? in a canal system for boats not sure the proper word.
Still with the idea of even and odd clock cycle, but considering tansfer is allowed during the first 15 ticks a (odd or even) clock cycle that last 20 ticks. To make it easier to synchronize, may be it will be off by 1 or 2 after some reworking but won't matter if the first 2 tick of a cycle are missed due to combinator delay it will still be 15 ticks of transfer every 40 ticks (15/20 and 0/20 odd/even).
Yes... I made sure that the pump transfers stop towards the last few ticks of a clock cycle. This is in order to keep things running smoothly, but if all the combinators are synchronised correctly, it shouldn't strictly be necessary to have that delay, and the other delays can likely be reduced a little bit further too. The tricky bit is I only really trust the manual tests, but manually testing takes a while.
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 First I want to say I'm glad you like my maniac fluid sushi :D
mmmPI wrote: ↑
Sat Apr 27, 2024 2:58 pm
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 - If you push into a tank over 40% (10000) fluid full
When attempting to redo the setup shown with the 5.2 and 10.4 leftover fluids with various starting of initial fluid in the left tank and using the feature of the editor to "tick once", i realized sometimes there will be some leftovers in the pump even when the tank contain only 9.4K fluid. But sometimes you can go a little over 10K without having leftovers. :?
Ahh.. yes, I should of been more precise with my wording. If it is over 10k the leftover issue *might* happen, it's not on all ticks after 10k. For the same reason what I call "lazy loading" doesn't always happen on pump activation either.
When you say you used the editor to tick once, I'm confused. If I run it for as many ticks as it needs then I never seemed to get overflow. So it wouldn't be realistic to do only one tick step through - we care about the fluid permanently getting stuck in the pump/pipe, not just for one tick.
Yeah and the "issue" might also happen before the tank has 10K too, this is why i will not rely on such sentences.

tick once allow you to read the buffer in the pump in a tick, then the next tick, then the next tick,to make sure it is 0, but doing so you can see that some ticks the buffer in the pump is not 0 even before the tank reach 10K.

It doesn't mean your system will "overflow" such as trying to put more fluid in the loop than it can handle due to reading less fluid some ticks because some would be stuck in a pump. There is no such things to me, when attempting to make a simpler setup that naively pretend this doesn't exist, then it doesnt manifest, therefore i am tempted to think that this is overcomplication due to some number being "displayed" by the game maybe misleading, but the fluid themselves aren't behaving in such misleading way.

To me it would be more akin to : when a pipe as 5 fluid, and each tick 20 enter and 20 left, the game could happen to show a quantity of 5 or 25, yet those 2 quantities would represent the same situation, it is just a "different moment" to stop time and count what's in the pipes for "displaying it to player", but the algorithm repeating, the fluid behavior is the same wheter it is shown 5 or 25. To me that's bound to happen when fluid is moving, that's the "wave update pattern" emerging from dirichlet-sloshing.
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 This is getting very long and technical so I'm going to start from the beginning and give a summary. Here are some properties we know about fluid in factorio:
Yes, and such summary to me make it worse because there's like half of the points i disagree x), it's not narrowing things down, rather the opposite, i'm not going to do it on all of them, but for illustration of the phenomenon :
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 - (1) Under some circumstances a pump will forget about its contents and not push some fluid (what I call "lazy loading")
Pump sometimes have fluid inside them when you expected them to push everything in the next entity, but that is a misconception in my mind that pump will always push 100% of their internal buffer in the next entity if it can handle it. Part of the equation for "100%" fluid transfer can only be achieved if the 59% part of the equation that represent fluid momentum is maxed out and not only the 40% part that depend on pressure differential. Therefore pumps "do not forget" and "lazy load". it was unclear to me what you called lazy load, you could remove the "leftover" fluid visible in pumps by making sure the tank doesn't reach a quantity that would cause them to be visible "on display while hovering". It represent in the game the fact that you need more energy/time to push 100 more fluid in a tank that contain already 20K than in tank that contain nothing.

The way the different "height" and "depth" of fluid box are coded makes this " a thing" in the game. Pumps push less fluid per tick in partially filled tank than in empty tanks.
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 - (2) Pipe chunk order, and pipe chunk alignment, both affect the way fluid propogates through a chain of pipes.
Yes and no to me x). "Affect" can be split into 2 different impact:
1) change the expected throughput in long term (solution of dirichlet algo is different ?),
2) change tick-to-tick behavior of fluid (update steps of the dirichlet algo takes the same value ?).

Pipe "build order" and "chunk alignment" affect the way fluid propagate through junctions not "chain of entity". regarding the 1)

Pipe "build order" and "chunk alignment" affect the way fluid propagate through junctions AND "chain of entity". regarding the 2).

Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 - (3) It would seem implausible (though not 100% ruled out) that rounding errors occur which can add or remove small amounts of fluid.
No, it is very possible, even bound to occur, although the magnitude of it would be much smaller than some already difficult to understand mechanic and it is very unlikely that it is visible with the level of precision currently discussed. ( 0.000000000000001 fluid versus 0.001 order of magnitude with an approximation for the number of 0s solely based on the lenght of the number )
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 - (4) Pipes sometimes get stuck with small amounts of fluid inside.
This is hardly a point x). That's seem the contrary to me, the game will automatically remove some fluid to unstuck some network if a very tiny quantity would be the cause of a block, more likely to occur after manual player intervention such as changing receipe, removing pipes, rotating entity and so on... This is not something that was demonstrated at all in this thread thought, when i'm refering to this, i'm not sure what you are refering to, but to me this is the logic that was introduced since 0.17 that prevent fluid mixing in most cases. It is "possible in theory" that it occurs". But on this thread i have seen no fluid stuck, nor such logic operate. So you must be refering to something else i think.
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 - (5) Tanks next to an empty pipe will leak some amount of fluid to the adjacent pipe. (I claim this is coded to be 0.4%, but after reading the code and the document I understand it is equivalently 1/250, and the value is different for pipes/pumps/etc.)
It's not just that, the equation is in 2 parts, one that is the pressure differential, the other being the momentum part, based on previous flow. Tanks or any entity next to any other entity that has a fluidbox will behave according to the hardcoded "height" and "depth" of the fluidbox that is defined in its prototype. tank next to pipes is no different than pipes next to other pipes in this regard, as both have a a similar "depth" of fluid box, unlike the boilers that is "deeper".
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 - (6) In the certain setup I showed earlier (measure_fluid.png), with a one tick pulse a pump will transfer 72.9 fluid to the pipe in front and 27.0 to its internal inventory. Because fluid is allowed to flow in only one direction, it is unaffected by build order. But this precise ratio is more likely derived from other properties rather than an intrinsic property of the pump.
Fluid allowed to flow in only one direction is caused by the pump, but i think it will still be affected by build order because of the " 2) tick to tick behavior".

Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 I also disagree about the author's stable state conclusions. They mention it first in section 2.2.2 as an assumption in their words saying the fluid values converage to equal values over time. My experiements show this is false even for rows of fluid, they don't converge to an equilibrium, and it's highly unlikely any fluid network converges to a stable state in general.
Beware! some of it is prooven mathematically to be true, regarding the "dirichlet magic squares" : There WILL be 1 unique solution, and IT IS POSSIBLE for an algorithm to converge toward it. It is even possible to calculate the "end result" through various methods. I have seen at least 4 x). 2 of which i can understand : 1) involve a lot of exponentiation, it is the finite difference method i think for the name, it calculate the end result more and more precisely as you repeat the calculations, higher and higher exponent, not super good for video game i thought.

Another i can understand is by making a system of equations and solving it, either manually or using matrices math that i'm not too familiar with it would require roughly redoing the whole thing everytime a pipe is built, and it would allow to reach the "end result" directly without any intermediate steps, so the fluid wouldn't be "moving" in my understanding.

Thos are mathematical proof, that may not be applied in the game, but i do not share your experiments, on the contrary, the two setup i provided shows the stable stable being reached, in particular the one with increasing quantity of water in pipes when going left. The quantity is 35 30 25 20 15 10 (5or25), really 5 or 25 could be used to describe the same behavior to me, the state is stable in both cases, the flow is the same, the displayed quantity may differ because of "build order". But the fluid throughput is the same.
Ivelieu wrote: ↑
Sat Apr 27, 2024 12:57 am
 Yeah, I agree. Which pipe has which amount of fluid is affected by build order. However, if the source and sink both consume and provide as much as they can (i.e. what you call a "steady flow" situation), the flow rate ends up still being the same regardless of build order (at least according to the haibane tenshi document). I am glad that I didn't have to do the complicated math and I can point to this haibane tenshi document and say, "I think this is correct". I didn't verify it line by line. But it is strong evidence against the theory that build order creates leftover fluid.
Build order is responsible for the quantity of fluid in a pipe being displayed differently, (5 or 25) as i showed in my setup, but yes THIS DOESN'T impact fluid throughput though. Visible "leftovers" is a result of a tick-to-tick behavior. The throughput over long period of time when system get more and more stable predicted by haibane tenshi were corroborated by Bilka's test in the previously linked topic. I have done my own experiment too, and yes this is correct, THERE IS A STABLE STATE. You have stated the opposite just before :
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 I also disagree about the author's stable state conclusions. They mention it first in section 2.2.2 as an assumption in their words saying the fluid values converage to equal values over time.
The flow rate ends up : stable , the system converge to the proper value, and as a consequence the quantity in pipes stabilize in a gradient as shown in the dirichlet magic squares. Not a flat level like still water. Rather a staircase. 100 90 80 70 ... as visible in game in previous setup.
Ivelieu wrote: ↑
Sun Apr 28, 2024 12:57 pm
 Yes the blueprint is definitely more robust than it needs to be. I am certain some timings can be cut down, I just did conservative overestimates - my design philosophy is, first make something that works, then try to optimise it. The fact you were able to remove that pump and replace with pipes and it never jams still, I don't have a good explanation for that.
Well my philosophy to understand things sometimes goes as "remove a little piece it and see what breaks to know more about the reason there is the little piece :D". Sometimes i also learn how to put it back, or that it can't be put back, but sometimes it does give good results :).

To me this has to do with the fact that the cutoff formula/part is preventing fluid to be added when not at least 200 is missing compared to desired threshold, that's a "simple" thing that prevent overflowing. Wether some "fluid" is not accounted for doesn't matter, because it can never be more than 200 fluid. BY DESIGN, we are considering "is some fluid leftover in a pump (from the main loop) could amount to more than the amount of fluid a pump (from the feeding branch) would transfer in a tick ?"

Since those are the same "pump" it cannot happen that the cutoff formula triggers for a "refill" of "200" which would cause an overflow, since it cannot be missing more than 200 fluid "stuck" from a pump buffer. not sure if that is a clear explanations though x). I think it's more intuitive to consider lock/floodgate in canals. You refill the canals of say 1000 water, when it misses at least 1000, some of it stays in the refilling pipe, but no more than a 1000. The canal can't overflow. At worst an very inpredictle flow in and out the refilling pipe and the canal would amount to an amplitude of change that is less than the refilling amount.

Your system only detect when at least 200 fluid are missing to trigger a refill of exactly 200, if 120 fluid are missing, no refill. The buffer in a single pump would need be to be 100% filled and stuck for a refill to be accidentaly triggered which can only occur if the tank is 100% full = bad threshold set up by human to me.
Top
User avatar
Ivelieu
Burner Inserter
Burner Inserter
Posts: 9
Joined: Mon Apr 22, 2024 2:06 am
Contact:
Contact Ivelieu

Re: Precision generic fluid routing and splitting

Post by Ivelieu »

mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 tick once allow you to read the buffer in the pump in a tick, then the next tick, then the next tick,to make sure it is 0, but doing so you can see that some ticks the buffer in the pump is not 0 even before the tank reach 10K.
Yes. I agree that in general it takes more than one tick for a pump to fully deposit into an adjacent tank. My current theory based on testing is that one pipe-pump segment takes four ticks to fully move fluid from a source to a sink, and this is based on the conclusion that I can fully move 200 units of fluid in 7 ticks with a pump-pipe-pump-pipe-pump-pipe-pump (4 pump 3 pipe) segment, into a tank containing at most 10k fluid, as I described in the first post. If the maximum capacity of the sink tank is set lower, then the total number of ticks required might be slightly less. For precision fluid it isn't necessary that the sink pump acts in only one tick, it's only essential that the source pump pushes 100% of the measured fluid in one tick only.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 Yes, and such summary to me make it worse because there's like half of the points i disagree x)
At risk of stating the obvious I think there is a bit of a language barrier. It's not like there are any common language words for most of these technical concepts, which naturally means I come up with term A and you use term B. At first it might seem like A is different to B, but I really think the terms are different but the concepts are the same.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 Pump sometimes have fluid inside them when you expected them to push everything in the next entity, but that is a misconception in my mind that pump will always push 100% of their internal buffer in the next entity if it can handle it. Part of the equation for "100%" fluid transfer can only be achieved if the 59% part of the equation that represent fluid momentum is maxed out and not only the 40% part that depend on pressure differential. Therefore pumps "do not forget" and "lazy load". it was unclear to me what you called lazy load, you could remove the "leftover" fluid visible in pumps by making sure the tank doesn't reach a quantity that would cause them to be visible "on display while hovering". It represent in the game the fact that you need more energy/time to push 100 more fluid in a tank that contain already 20K than in tank that contain nothing.

The way the different "height" and "depth" of fluid box are coded makes this " a thing" in the game. Pumps push less fluid per tick in partially filled tank than in empty tanks.
Yes yes that's all correct and what I mean by "lazy loading", i.e. due to the tank being more filled, it has lower relative pressure, so the tank "forgets" to push the 40% pressure differential part when it crosses a relative pressure threshold. And different components have different maximum and minimum pressure values which affects this differential. Whether I'm inaccurate in calling this forgetting versus some other term, it's like sloshing vs dirichlet etc, I just make up terms as I go because there's nothing else to work with. In this sense what I call lazy loading is not a fundamental principle but more accurately, a derived principle from the source code that I think is significant based on actual experiments.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 "Affect" can be split into 2 different impact:
1) change the expected throughput in long term (solution of dirichlet algo is different ?),
2) change tick-to-tick behavior of fluid (update steps of the dirichlet algo takes the same value ?).
Yes, propogation includes converged flow rate (i.e. (1)) and initial flow rates (i.e. (2)). I refer to both with the term "propagation". Of course the pdf goes in more detail about the converged flow rates and I only tried to give one line summary, I didn't mean to make confusion.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 No, it is very possible, even bound to occur, although the magnitude of it would be much smaller than some already difficult to understand mechanic and it is very unlikely that it is visible with the level of precision currently discussed. ( 0.000000000000001 fluid versus 0.001 order of magnitude with an approximation for the number of 0s solely based on the lenght of the number )
Well even if a floating point error occurs in a very small amount it should be enough in some cases to jam rows of fluid pipes. If a pipe contains 10e-20 fluid, then it still contains X fluid, and wouldn't accept Y fluid later. The reason I say it is implausible, is because I am able to run designs which never have this occur, they never have trace amounts of fluid blocking some of the pipes. I say it's implausible given that my blueprints work fine, and if there really was rounding errors, then my blueprints would get stuck on trace fluid amounts. That's not to say it doesn't occur in other situations, but without a clear experiment to demonstrate it occuring (which is hard to design in itself), it seems unlikely.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 This is hardly a point x). That's seem the contrary to me, the game will automatically remove some fluid to unstuck some network if a very tiny quantity would be the cause of a block, more likely to occur after manual player intervention such as changing receipe, removing pipes, rotating entity and so on... This is not something that was demonstrated at all in this thread thought, when i'm refering to this, i'm not sure what you are refering to, but to me this is the logic that was introduced since 0.17 that prevent fluid mixing in most cases. It is "possible in theory" that it occurs". But on this thread i have seen no fluid stuck, nor such logic operate. So you must be refering to something else i think.
I am referring to when this happens:
trace_fluid_stuck.png
trace_fluid_stuck.png (518.24 KiB) Viewed 252 times
I didn't think I needed to demonstrate it since it is such a widespread occurrence in regular fluid networks, and for precision fluid handling this can never be allowed to occur. This occurs in both fully automated systems and after manual player intervention.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 It's not just that, the equation is in 2 parts, one that is the pressure differential, the other being the momentum part, based on previous flow. Tanks or any entity next to any other entity that has a fluidbox will behave according to the hardcoded "height" and "depth" of the fluidbox that is defined in its prototype. tank next to pipes is no different than pipes next to other pipes in this regard, as both have a a similar "depth" of fluid box, unlike the boilers that is "deeper".
Yes, the 2 parts of the equation. As two adjacent tanks become more similar in fluid, the momentum part will oscillate and decrease in size, and eventually the pressure differential term will dominate the flow. In the case of the image I sent previously [two_possibilities.png] the momentum term is 0 and the pressure differential term is 100% of the effective flow, but the ratio of momentum to pressure differential flow in practice can change significantly before convergence to a steady flow.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 Beware! some of it is prooven mathematically to be true, regarding the "dirichlet magic squares" : There WILL be 1 unique solution, and IT IS POSSIBLE for an algorithm to converge toward it. It is even possible to calculate the "end result" through various methods. I have seen at least 4 x). 2 of which i can understand : 1) involve a lot of exponentiation, it is the finite difference method i think for the name, it calculate the end result more and more precisely as you repeat the calculations, higher and higher exponent, not super good for video game i thought.
I do understand, there are three relevant kinds of convergence to equilibrium:
1: All fluidboxes in a row contain the same amount of fluid. I call this "fluid equlibrium".
2: All the fluidboxes have stopped transferring fluid to one another. I call this a "stable state".
3: All fluidboxes converge to produce an overall stable flow rate from source to sink. I call this a "steady flow".

I am saying that stable state convergence does not occur in practice, in factorio. I am also saying fluid equlibrium convergence basically never occurs. I gave an example of both fluid equilibrium and stable state convergence not occuring in [two_possibilities.png] but in general it never occurs in practice, because we can only see 3 significant figures we just don't always see the error displayed through the gui.In theory, with perfect precision calculations and no damping, fluid equilibrium is in general not guaranteed. But a stable state can be guaranteed in theory with a damping factor less between 0 and 1 (and factorio's damping factor is 1.475, i.e. the ratio between the pressure differential factor of 0.4 and the momentum factor of 0.59). And I agree with you that steady flow convergence happens 100% of the time if you allow the pipes to operate for long enough. These are three different kinds of convergence, I care about the first kind the most in this topic because it's an obstacle to ensuring precise fluid amounts are handled. But knowing the rules of stable state convergence can give some explanation to when fluid equilibrium convergence occurs. Both are basically unrelated to the steady flow convergence, which we agree is guaranteed in both theory and practice. But it seems you think I am talking about the third type when I am talking about the first type, the third type isn't really relevant to ensuring precision fluid movement.
mmmPI wrote: ↑
Mon Apr 29, 2024 10:31 am
 Your system only detect when at least 200 fluid are missing to trigger a refill of exactly 200, if 120 fluid are missing, no refill. The buffer in a single pump would need be to be 100% filled and stuck for a refill to be accidentaly triggered which can only occur if the tank is 100% full = bad threshold set up by human to me.
Well yes, that would be too high a threshold. Originally I intended the fluid sushi to only be used with consuming pumps pulling 200 fluid exactly at a time. If you do that, and all the fluid precision is correct, then we don't get the canal/floodgate issue you describe. But if we do decide to just pull fluid without precision, then that's right they won't refill exactly to the threshold. Even as it is now, some sushi tanks might fill to slightly different amounts because of the delays where fluid is traveling between tanks and can't be measured by the circuit network. This is why making conservative overestimates helped my design, because it's hard to predict something like that. The max tank value can be thought of as a rough guide for the automation so it doesn't break itself, rather than a precise value to meet 100% of the time.

Also I tested the modification you proposed with a different fluid cutoff and found it worked only when the maximum capacity was low, if I raised it then it wouldn't pull precise amounts any more.
1k_vs_10k_threshold.png
1k_vs_10k_threshold.png (353.74 KiB) Viewed 252 times
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Mon Apr 29, 2024 11:59 pm
 I am referring to when this happens: trace_fluid_stuck.png
I didn't think I needed to demonstrate it since it is such a widespread occurrence in regular fluid networks, and for precision fluid handling this can never be allowed to occur. This occurs in both fully automated systems and after manual player intervention.

This never occured to me since the update making fluid not allowed to mix easily x), i was able to reproduce the situation only by removing manually many item to make sure i'd spread 1 fluid into 20 pipes or so, then removing some in the middle and adding them back.
Ivelieu wrote: ↑
Mon Apr 29, 2024 11:59 pm
 2: All the fluidboxes have stopped transferring fluid to one another. I call this a "stable state".
I am saying that stable state convergence does not occur in practice, in factorio.

For the most part i think i understand the 3 differents things you mention, but i also think your previous point is an example of the 2) where the fluid boxes have stopped transfering to one another, i wouldn't know how to make it happen in automated setup though :ugeek:
Ivelieu wrote: ↑
Mon Apr 29, 2024 11:59 pm
 Also I tested the modification you proposed with a different fluid cutoff and found it worked only when the maximum capacity was low, if I raised it then it wouldn't pull precise amounts any more.
I am going to say that it doesn't matter if the amount is not "precise" anymore :
I let this one design run for some time when i built another one :
tested for hours.jpg
tested for hours.jpg (538.85 KiB) Viewed 217 times
Yes it doesn't show nice round value, but it doesn't jam, and run four hours, keeping everything the same as original design as far as parameter and values goes, only removing some pumps and replacing them with underground pipes.

I have made a design to show what yours inspired me with the floodgate system :
Fish Driven Fluid Sushi.jpg
Fish Driven Fluid Sushi.jpg (913.23 KiB) Viewed 217 times

It's kinda silly, i had already made such design based on loop of belt instead of combinators, but never thought it could be used to make a fluid sushi.

It should be possible to remove the 2 combinators per tank, and instead use 2 pump in a row with the same conditions provided there is nothing in between them for even less combinators :)

2 fish are required ( placing them in the chest will do) because there are 2 different loops, on has some green and red combinator next to it, those are just cosmetic, it shows when the pump connected to a red wire or green wire are active, never both at the same time, ( https://en.wikipedia.org/wiki/Lock_(water_navigation) )

The other other loop has 12 belts that only let the 2nd fish through when the first loop has completed a lap.

This means 12 times red pump activate, and 12 time green pump activate. This means the fluids are back in their starting position.

This is when the refill of the looping fluid occur. At this point there is a special delay added by the inserter that steal the fish from the loop 1 and put it on the side belt that will reload it after the fish has traveled back to loop 1.

There is a single constant combinator that is used to transmit the threshold quantity to all refilling pumps under the "signal Q". It could be hard coded on every pump but that's easier to change this way and also to configure the pumps.

By default it tries to load "15K" fluids, and i don't think it will be less precise than your setup, it makes them loop around and refill it periodically with the amount missing to reach 15K by a pump that stop the tick just after 15K is reached. So at most there will be an excess lower than what a pump can push in a tick in an empty tank. Since at worst the pump triggers 1 tick when the fluid in tank is 14999 but in such case it will not transfer all its buffer since the tank has more than 10K.

If you hover over the pump, you could see some leftover forgotten fluid ready to be lazy loaded. Not sure why all those terms seems like something is wrong , nothing is x).
Top
User avatar
Ivelieu
Burner Inserter
Burner Inserter
Posts: 9
Joined: Mon Apr 22, 2024 2:06 am
Contact:
Contact Ivelieu

Re: Precision generic fluid routing and splitting

Post by Ivelieu »

mmmPI wrote: ↑
Tue Apr 30, 2024 4:16 am
 For the most part i think i understand the 3 differents things you mention, but i also think your previous point is an example of the 2) where the fluid boxes have stopped transfering to one another, i wouldn't know how to make it happen in automated setup though :ugeek:
My point is illustrated in the image I showed earlier, two_possibilities.png. There are two possibilities - either fluid has stopped moving entirely, or it is busy looping (and as a result consuming CPU time) between the two tanks. I was able to show that in fact they are busy looping even though it is not visible directly in the UI.

To me that is meaningfully different than the fluids not moving at all, because the pairs which pipes may oscillate fluid is affected by build order, which in turn also may affect the likelihood or possibility of a trace fluid jam. And as I alluded to, I thought that an equalisation cutoff was implemented but it was not, which makes me think there is still room to optimise fluid performance on the factorio source code. I imagine they spent much more effort optimising belts by comparison, but if I want a large base to run efficiently on fluiditems, then it is worth considering.
mmmPI wrote: ↑
Tue Apr 30, 2024 4:16 am
 I am going to say that it doesn't matter if the amount is not "precise" anymore :
I let this one design run for some time when i built another one :
tested for hours.jpg
Yes it doesn't show nice round value, but it doesn't jam, and run four hours, keeping everything the same as original design as far as parameter and values goes, only removing some pumps and replacing them with underground pipes.
Yes. It's not that important the value circulated is precise even if the value extracted needs to be, so I think using undergrounds here is a good choice where circulation precision isn't required.
mmmPI wrote: ↑
Tue Apr 30, 2024 4:16 am
 I have made a design to show what yours inspired me with the floodgate system :
Fish Driven Fluid Sushi.jpg
It's kinda silly, i had already made such design based on loop of belt instead of combinators, but never thought it could be used to make a fluid sushi.
Nice work.
Spreading out the refill points to be aligned already, does save a lot of work with the logic. Although it can't really a custom clock cycle time (a limitation of conveyor circuit designs), it is a much simpler design with less points of failure with combinators doing tick timing.
mmmPI wrote: ↑
Tue Apr 30, 2024 4:16 am
 If you hover over the pump, you could see some leftover forgotten fluid ready to be lazy loaded. Not sure why all those terms seems like something is wrong , nothing is x).
Nah, lazy = good!
Top
mmmPI
Smart Inserter
Smart Inserter
Posts: 2771
Joined: Mon Jun 20, 2016 6:10 pm
Contact:
Contact mmmPI

Re: Precision generic fluid routing and splitting

Post by mmmPI »

Ivelieu wrote: ↑
Wed May 01, 2024 1:49 am
mmmPI wrote: ↑
Tue Apr 30, 2024 4:16 am
 For the most part i think i understand the 3 differents things you mention, but i also think your previous point is an example of the 2) where the fluid boxes have stopped transfering to one another, i wouldn't know how to make it happen in automated setup though :ugeek:
My point is illustrated in the image I showed earlier, two_possibilities.png. There are two possibilities - either fluid has stopped moving entirely, or it is busy looping (and as a result consuming CPU time) between the two tanks. I was able to show that in fact they are busy looping even though it is not visible directly in the UI.
I meant those 3 :
Ivelieu wrote: ↑
Mon Apr 29, 2024 11:59 pm
 I do understand, there are three relevant kinds of convergence to equilibrium:
1: All fluidboxes in a row contain the same amount of fluid. I call this "fluid equlibrium".
2: All the fluidboxes have stopped transferring fluid to one another. I call this a "stable state".
3: All fluidboxes converge to produce an overall stable flow rate from source to sink. I call this a "steady flow".
Those are precise definition to me, and i like them, they make sense to me, they allow me to classify what i see in game as either fulfilling those criterions or not objectively.

Illustration video for case number 2 in my opinion disprooving your claim that fluid are busy looping between 2 tanks :



My logic would then goes to say that It also illustrate case 3), where the steady flow has a value of 0, and in that special case it is also an illustration case of how the game approximate number 1) (not fully flat ):D.
Ivelieu wrote: ↑
Mon Apr 29, 2024 11:59 pm
 Nah, lazy = good!
Ah i'm using "minimalist" for the "good lazy", like music or code ? when you do a lot of effort so that there is less at the end but it's the "best", and lazy to me is bad, like when not doing much effort and thus not having much at the end as consequence ^^

I attempted a minimalist answer for me to be able to follow the discussion. It's not adressing all the points, but many of them i feel may be narrowed down to precise definition earlier in the reasonning not being considered the same way, which only creates more and more diverging interpretations as steps of reasonning are added.
Attachments
Fluid-stable-between-2-tanks.mp4
(477.44 KiB) Downloaded 10 times
Top
Post Reply

Return to “Combinator Creations”