You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Empty body on POST request not compatible with some legacy servers (411 Length Required) #117
When sending a HTTP request with a method which has a request body (e.g. POST or PUT) but with an empty body (size=0), then Buzz does not apply a Content-Length header, despite there being no body and as such it violates the HTTP RFC which requires a Content-Length header (or an actual body with chunked transfer).
Looking at the code the body seems to be handled as if it were a real stream, even if the size is known and 0. According to PSR-7 getSize() returns null if the size is unknown, otherwise an integer if the size is known.
As such this implementation is non-compliant with PSR-7. Is there any reason for this check to be there?
I'm not familiar whether all streams, or only a subset of streams, return 0 if the size is unknown. But PSR-7 compliant streams[1] always return null for unknown sizes, so wouldn't it make sense to only respect 0 as unknown size if the stream is not a PSR-7 stream?
When sending a HTTP request with a method which has a request body (e.g. POST or PUT) but with an empty body (size=0), then Buzz does not apply a Content-Length header, despite there being no body and as such it violates the HTTP RFC which requires a Content-Length header (or an actual body with chunked transfer).
Thanks for reporting! I'm not sure I follow the exact issue you're seeing, so perhaps you can provide a link to what specs are being violated here?
It's my understanding that this library properly does not include a ˋContent-Lengthˋ header in this case and as such indicates that the request does not contain a request body in accordance to to https://tools.ietf.org/html/rfc7230#section-3.3.3.
ghost changed the title [-]Empty body on requests with body results in HTTP non-compliance[/-][+]Empty body on requests with body results in non-compliance[/+]on Dec 1, 2018
I'm not sure I follow the exact issue you're seeing, so perhaps this may boil down to a misunderstanding or lack of documentation. You can gist the problem you're seeing to help clear things up a bit?
You can explicitly pass a Content-Length request header if you know the outgoing request size. Accordingly, you can also assign a Content-Length: 0 request header if you know the request body is empty.
Similarly, this library automatically adds a Content-Length request header with the appropriate value if you pass in a string as a request body. If you pass in a ReadableStreamInterface as a request body, it is unable to determine the request size and as such uses Transfer-Encoding: chunked instead. This all happens according to HTTP specs, but many servers are known to reject requests without an explicit length (which again is valid behavior according to HTTP specs).
Similarly, this library automatically adds a Content-Length request header with the appropriate value if you pass in a string as a request body
That's exactly were Buzz goes wrong. An empty string gets treated as a stream with no known size.
@CharlotteDunois I agree that the comment above this statement may be misleading, but it's my understanding that its behavior is actually correct. If the request body is known to be empty, the request will not include an outgoing Content-Length header. Omitting this header as the same effect as explicitly assigning a 0 value, both will indicate that the request does not include a request body. Unless I'm missing something, this is all done in accordance to the HTTP specs already linked above.
Again, what you're saying seems to imply that there might in fact be an issue in this library and I would love to help fix this. So far, I'm unable to reproduce the problem you're seeing, so perhaps you can provide a small gist that highlights this issue? Thank you.
Thank you for providing this test case to reproduce the problem you're seeing! With this piece of code, I can reproduce the problem you're seeing for some servers (Google) and can not reproduce this for others (httpbin.org and many others).
The way I read the newer RFC 7230 and the legacy RFC 2616 there is indeed a slight discrepancy in how one can read this requirement.
If this is a request message and none of the above are true, then
the message body length is zero (no message body is present).
The newer RFC 7230 clearly states that the server side can determine that the request body is empty when no Content-Length (and no Transfer-Encoding: chunked) header is present. According to the RFC, this should have the exact same effect has explicitly specifying a Content-Length: 0 header. For this reason, this library does not currently send this arguably superfluous header and I do not currently consider this to be in violation of either the HTTP specs or the PSR-7 definition.
For compatibility with HTTP/1.0 applications, HTTP/1.1 requests
containing a message-body MUST include a valid Content-Length header
field unless the server is known to be HTTP/1.1 compliant. If a
request contains a message-body and a Content-Length is not given,
the server SHOULD respond with 400 (bad request) if it cannot
determine the length of the message, or with 411 (length required) if
it wishes to insist on receiving a valid Content-Length.
A user agent SHOULD send a Content-Length in a request message when
no Transfer-Encoding is sent and the request method defines a meaning
for an enclosed payload body. For example, a Content-Length header
field is normally sent in a POST request even when the value is 0
(indicating an empty payload body). A user agent SHOULD NOT send a
Content-Length header field when the request message does not contain
a payload body and the method semantics do not anticipate such a
body.
The newer RFC 7230 is clear in that the client SHOULD send the Content-Length: 0 header for some requests. This library honors any explicit headers, so you can currently specify this like this:
With the above definition in mind, I agree that it makes sense to add this to the library as a sane default behavior for the future. Given that this library supports any (custom) request method and the definition mentions this behavior should trigger "when […] the request method defines a meaning for an enclosed payload body" I'm currently undecided if it makes more sense to go with a blacklist or whitelist for certain request methods. Perhaps anybody else has some insights into which request methods could be affected by this? (POST/PUT/PATCH likely are, GET/HEAD/TRACE likely not à la aio-libs/aiohttp#2167)
@CharlotteDunois Thank you for reporting! I've just filed #120 which should fix this for affected HTTP servers that seem to follow the older RFC 2616 semantics. Perhaps you can give it a try and see if fixes the issues you were seeing?
In case anybody else stumbles over this in the future: The above behavior only triggers for an empty request body and only for POST, PUT and PATCH requests. If you are affected by this, keep in mind that this library honors any explicit headers, so you can specify this like this:
changed the title [-]Empty body on requests with body results in non-compliance[/-][+]Empty body on POST request not compatible with some legacy servers (411 Length Required)[/+]on Dec 17, 2018
Activity
clue commentedon Nov 30, 2018
Thanks for reporting! I'm not sure I follow the exact issue you're seeing, so perhaps you can provide a link to what specs are being violated here?
It's my understanding that this library properly does not include a ˋContent-Lengthˋ header in this case and as such indicates that the request does not contain a request body in accordance to to https://tools.ietf.org/html/rfc7230#section-3.3.3.
[-]Empty body on requests with body results in HTTP non-compliance[/-][+]Empty body on requests with body results in non-compliance[/+]clue commentedon Dec 1, 2018
I'm not sure I follow the exact issue you're seeing, so perhaps this may boil down to a misunderstanding or lack of documentation. You can gist the problem you're seeing to help clear things up a bit?
You can explicitly pass a
Content-Lengthrequest header if you know the outgoing request size. Accordingly, you can also assign aContent-Length: 0request header if you know the request body is empty.Similarly, this library automatically adds a
Content-Lengthrequest header with the appropriate value if you pass in a string as a request body. If you pass in aReadableStreamInterfaceas a request body, it is unable to determine the request size and as such usesTransfer-Encoding: chunkedinstead. This all happens according to HTTP specs, but many servers are known to reject requests without an explicit length (which again is valid behavior according to HTTP specs).clue commentedon Dec 2, 2018
@CharlotteDunois I agree that the comment above this statement may be misleading, but it's my understanding that its behavior is actually correct. If the request body is known to be empty, the request will not include an outgoing
Content-Lengthheader. Omitting this header as the same effect as explicitly assigning a0value, both will indicate that the request does not include a request body. Unless I'm missing something, this is all done in accordance to the HTTP specs already linked above.Again, what you're saying seems to imply that there might in fact be an issue in this library and I would love to help fix this. So far, I'm unable to reproduce the problem you're seeing, so perhaps you can provide a small gist that highlights this issue? Thank you.
WyriHaximus commentedon Dec 10, 2018
@CharlotteDunois && @clue anyway I can help fix/resolve/prove this problem?
clue commentedon Dec 14, 2018
Thank you for providing this test case to reproduce the problem you're seeing! With this piece of code, I can reproduce the problem you're seeing for some servers (Google) and can not reproduce this for others (httpbin.org and many others).
The way I read the newer RFC 7230 and the legacy RFC 2616 there is indeed a slight discrepancy in how one can read this requirement.
The newer RFC 7230 clearly states that the server side can determine that the request body is empty when no
Content-Length(and noTransfer-Encoding: chunked) header is present. According to the RFC, this should have the exact same effect has explicitly specifying aContent-Length: 0header. For this reason, this library does not currently send this arguably superfluous header and I do not currently consider this to be in violation of either the HTTP specs or the PSR-7 definition.The legacy RFC 2616 does indeed not have this definition and leaves some room for speculation here. This seems to be the source of confusion and is likely the cause for this incompatibility you're seeing. (https://stackoverflow.com/a/18352457, https://stackoverflow.com/a/15619681, https://stackoverflow.com/a/1233569, https://serverfault.com/questions/164220/is-there-a-way-to-avoid-nginx-411-content-length-required-errors and https://stackoverflow.com/questions/8540931/what-is-the-correct-content-length-to-set-for-a-get-request)
The newer RFC 7230 is clear in that the client SHOULD send the
Content-Length: 0header for some requests. This library honors any explicit headers, so you can currently specify this like this:With the above definition in mind, I agree that it makes sense to add this to the library as a sane default behavior for the future. Given that this library supports any (custom) request method and the definition mentions this behavior should trigger "when […] the request method defines a meaning for an enclosed payload body" I'm currently undecided if it makes more sense to go with a blacklist or whitelist for certain request methods. Perhaps anybody else has some insights into which request methods could be affected by this? (POST/PUT/PATCH likely are, GET/HEAD/TRACE likely not à la aio-libs/aiohttp#2167)
Any input would be much appreciated 👍
clue commentedon Dec 16, 2018
@CharlotteDunois Thank you for reporting! I've just filed #120 which should fix this for affected HTTP servers that seem to follow the older RFC 2616 semantics. Perhaps you can give it a try and see if fixes the issues you were seeing?
In case anybody else stumbles over this in the future: The above behavior only triggers for an empty request body and only for
POST,PUTandPATCHrequests. If you are affected by this, keep in mind that this library honors any explicit headers, so you can specify this like this:[-]Empty body on requests with body results in non-compliance[/-][+]Empty body on POST request not compatible with some legacy servers (411 Length Required)[/+]WyriHaximus commentedon Dec 17, 2018
🎉 !