Page 1 of 1

( 12.22 ) Headless server crash.

Posted: Tue Feb 16, 2016 3:56 am
by NoPantsMcDance
My vanilla server had a hard crash so I figured I'd post the log here as it's interesting looking. Posted the log to my server should be readable though. It's the best I can do with my cellphone. http://www.sugaming.us/downloads/factorio-current.log

Re: ( 12.22 ) Headless server crash.

Posted: Tue Feb 16, 2016 12:45 pm
by kovarex
This looks like a desync, I can't do anything until I get the desync report, or save and reproducible way how to cause it.

Re: ( 12.22 ) Headless server crash.

Posted: Tue Feb 16, 2016 5:21 pm
by NoPantsMcDance
http://www.sugaming.us/downloads/desync.zip here is the desync report it's quite large. I'm not sure who was online before it crashed so idk what they were doing. I'll check and have them reply here to reproduce it if possible.

EDIT: This is the first hard crash on this server since I've been running it. Hopefully it's nothing serious and a one time thing.

Re: ( 12.22 ) Headless server crash.

Posted: Tue Feb 16, 2016 6:03 pm
by kovarex
NoPantsMcDance wrote:http://www.sugaming.us/downloads/desync.zip here is the desync report it's quite large. I'm not sure who was online before it crashed so idk what they were doing. I'll check and have them reply here to reproduce it if possible.

EDIT: This is the first hard crash on this server since I've been running it. Hopefully it's nothing serious and a one time thing.
We have made 2 fixes related to desync for 0.12.23, so lets see if it happens again in that version.

The 0.12.23 will be released in two days.

Re: ( 12.22 ) Headless server crash.

Posted: Tue Feb 16, 2016 6:04 pm
by NoPantsMcDance
kovarex wrote:
NoPantsMcDance wrote:http://www.sugaming.us/downloads/desync.zip here is the desync report it's quite large. I'm not sure who was online before it crashed so idk what they were doing. I'll check and have them reply here to reproduce it if possible.

EDIT: This is the first hard crash on this server since I've been running it. Hopefully it's nothing serious and a one time thing.
We have made 2 fixes related to desync for 0.12.23, so lets see if it happens again in that version.

The 0.12.23 will be released in two days.
Sweet. Looking forward to the new version.

Re: ( 12.24 ) Headless server crash.

Posted: Tue Mar 01, 2016 6:09 pm
by NoPantsMcDance
Had another crash on my vanilla server. Included the log and desync report. If you need any other info lemme know.

Re: ( 12.24 ) Headless server crash.

Posted: Thu Mar 03, 2016 2:47 pm
by kovarex
NoPantsMcDance wrote:Had another crash on my vanilla server. Included the log and desync report. If you need any other info lemme know.
This looks, like someone is sending lot of packets not related to factorio to your ip.

Re: ( 12.24 ) Headless server crash.

Posted: Fri Mar 04, 2016 8:34 am
by NoPantsMcDance
kovarex wrote:
NoPantsMcDance wrote:Had another crash on my vanilla server. Included the log and desync report. If you need any other info lemme know.
This looks, like someone is sending lot of packets not related to factorio to your ip.
Why would that cause the server to crash?

Re: ( 12.22 ) Headless server crash.

Posted: Fri Mar 04, 2016 10:48 am
by kovarex
Because we have just 2 bytes of authorization that the packet really relates to Factorio. Once it passes that, it is deserialised, and different kind of problems can happen.

In other words, sending random packets to Factorio port while the server is running can eventually lead to crash of the server.

I fixed the 1. frontier, that sending wrong action type will make the server ignore the packet instead of closing itself, but basically all the deserialization of packet data will have to be checked and sanitised. This is planned for 0.13

But anyway, the fact that you were receiving a lot of packets on the port that don't belong to Factorio is strange.

Re: ( 12.22 ) Headless server crash.

Posted: Mon Mar 07, 2016 9:26 pm
by ofca
I suggest banning source IP after 10 such packets/similar offenses -- otherwise headless servers will be useless now that you've opened the Internet's most famous kidiots storage (a.k.a. Steam)
Another quick idea is green-lighting correct IPs, and process only initial handshakes from other IPs.

Re: ( 12.22 ) Headless server crash.

Posted: Fri Mar 11, 2016 3:17 pm
by cube
This error was actually caused by something else than kovarex said. His fix will still help with the symptoms, though.
ofca wrote:I suggest banning source IP after 10 such packets/similar offenses -- otherwise headless servers will be useless now that you've opened the Internet's most famous kidiots storage (a.k.a. Steam)
Another quick idea is green-lighting correct IPs, and process only initial handshakes from other IPs.
This looks like good ideas and I'd like to have filters like that in 0.13.

Re: ( 12.22 ) Headless server crash.

Posted: Mon Mar 14, 2016 12:04 am
by NoPantsMcDance
Looks like someone read this post and decided to abuse it. You guys have a quick solution or should I just add a check in my script for x amount of connection requests to block the IP. What would be a safe number for the amount of failed connection requests so this doesn't affect legit players?

Re: ( 12.22 ) Headless server crash.

Posted: Wed Mar 16, 2016 1:14 pm
by cube
It took just three seconds ... we really need to work on this. And that IP is from Prague :-)

There is no quick solution for that on our side yet, sorry. For a safe attempt number I would say that 5 per minute and IP address is enough. Generally the first one should succeed (I don't think that I've seen any connection attempt to fail), but five should work even for weird connection problems and multiple people connecting from the same NATed address.

As I already said, we will be adding this kind of protection to 0.13. This is the bug report thread for it: 17720