Skip to content

Empty body on POST request not compatible with some legacy servers (411 Length Required) #117

Closed

Description

When sending a HTTP request with a method which has a request body (e.g. POST or PUT) but with an empty body (size=0), then Buzz does not apply a Content-Length header, despite there being no body and as such it violates the HTTP RFC which requires a Content-Length header (or an actual body with chunked transfer).

Looking at the code the body seems to be handled as if it were a real stream, even if the size is known and 0. According to PSR-7 getSize() returns null if the size is unknown, otherwise an integer if the size is known.
grafik

As such this implementation is non-compliant with PSR-7. Is there any reason for this check to be there?
I'm not familiar whether all streams, or only a subset of streams, return 0 if the size is unknown. But PSR-7 compliant streams[1] always return null for unknown sizes, so wouldn't it make sense to only respect 0 as unknown size if the stream is not a PSR-7 stream?

[1] e.g. https://github.com/ringcentral/psr7/blob/360faaec4b563958b673fb52bbe94e37f14bc686/src/Stream.php#L135-L157

Activity

  1. clue commented on Nov 30, 2018

    @clue
    Owner

    When sending a HTTP request with a method which has a request body (e.g. POST or PUT) but with an empty body (size=0), then Buzz does not apply a Content-Length header, despite there being no body and as such it violates the HTTP RFC which requires a Content-Length header (or an actual body with chunked transfer).

    Thanks for reporting! I'm not sure I follow the exact issue you're seeing, so perhaps you can provide a link to what specs are being violated here?

    It's my understanding that this library properly does not include a ˋContent-Lengthˋ header in this case and as such indicates that the request does not contain a request body in accordance to to https://tools.ietf.org/html/rfc7230#section-3.3.3.

  2. ghost changed the title [-]Empty body on requests with body results in HTTP non-compliance[/-] [+]Empty body on requests with body results in non-compliance[/+] on Dec 1, 2018
  3. clue commented on Dec 1, 2018

    @clue
    Owner

    I'm not sure I follow the exact issue you're seeing, so perhaps this may boil down to a misunderstanding or lack of documentation. You can gist the problem you're seeing to help clear things up a bit?

    You can explicitly pass a Content-Length request header if you know the outgoing request size. Accordingly, you can also assign a Content-Length: 0 request header if you know the request body is empty.

    Similarly, this library automatically adds a Content-Length request header with the appropriate value if you pass in a string as a request body. If you pass in a ReadableStreamInterface as a request body, it is unable to determine the request size and as such uses Transfer-Encoding: chunked instead. This all happens according to HTTP specs, but many servers are known to reject requests without an explicit length (which again is valid behavior according to HTTP specs).

  4. clue commented on Dec 2, 2018

    @clue
    Owner

    Similarly, this library automatically adds a Content-Length request header with the appropriate value if you pass in a string as a request body

    That's exactly were Buzz goes wrong. An empty string gets treated as a stream with no known size.

    @CharlotteDunois I agree that the comment above this statement may be misleading, but it's my understanding that its behavior is actually correct. If the request body is known to be empty, the request will not include an outgoing Content-Length header. Omitting this header as the same effect as explicitly assigning a 0 value, both will indicate that the request does not include a request body. Unless I'm missing something, this is all done in accordance to the HTTP specs already linked above.

    Again, what you're saying seems to imply that there might in fact be an issue in this library and I would love to help fix this. So far, I'm unable to reproduce the problem you're seeing, so perhaps you can provide a small gist that highlights this issue? Thank you.

  5. WyriHaximus commented on Dec 10, 2018

    @WyriHaximus
    Contributor

    @CharlotteDunois && @clue anyway I can help fix/resolve/prove this problem?

  6. clue commented on Dec 14, 2018

    @clue
    Owner

    @CharlotteDunois This is what I'm doing at all (dynamically):

    $request = new Request('PUT', $url, array(
        'Authorization' => $auth
    ));
    $browser->send($request);

    Thank you for providing this test case to reproduce the problem you're seeing! With this piece of code, I can reproduce the problem you're seeing for some servers (Google) and can not reproduce this for others (httpbin.org and many others).

    The way I read the newer RFC 7230 and the legacy RFC 2616 there is indeed a slight discrepancy in how one can read this requirement.

    1. If this is a request message and none of the above are true, then
      the message body length is zero (no message body is present).

    The newer RFC 7230 clearly states that the server side can determine that the request body is empty when no Content-Length (and no Transfer-Encoding: chunked) header is present. According to the RFC, this should have the exact same effect has explicitly specifying a Content-Length: 0 header. For this reason, this library does not currently send this arguably superfluous header and I do not currently consider this to be in violation of either the HTTP specs or the PSR-7 definition.

    For compatibility with HTTP/1.0 applications, HTTP/1.1 requests
    containing a message-body MUST include a valid Content-Length header
    field unless the server is known to be HTTP/1.1 compliant. If a
    request contains a message-body and a Content-Length is not given,
    the server SHOULD respond with 400 (bad request) if it cannot
    determine the length of the message, or with 411 (length required) if
    it wishes to insist on receiving a valid Content-Length.

    The legacy RFC 2616 does indeed not have this definition and leaves some room for speculation here. This seems to be the source of confusion and is likely the cause for this incompatibility you're seeing. (https://stackoverflow.com/a/18352457, https://stackoverflow.com/a/15619681, https://stackoverflow.com/a/1233569, https://serverfault.com/questions/164220/is-there-a-way-to-avoid-nginx-411-content-length-required-errors and https://stackoverflow.com/questions/8540931/what-is-the-correct-content-length-to-set-for-a-get-request)

    A user agent SHOULD send a Content-Length in a request message when
    no Transfer-Encoding is sent and the request method defines a meaning
    for an enclosed payload body. For example, a Content-Length header
    field is normally sent in a POST request even when the value is 0
    (indicating an empty payload body). A user agent SHOULD NOT send a
    Content-Length header field when the request message does not contain
    a payload body and the method semantics do not anticipate such a
    body.

    The newer RFC 7230 is clear in that the client SHOULD send the Content-Length: 0 header for some requests. This library honors any explicit headers, so you can currently specify this like this:

    $body = '';
    $request = new Request('PUT', $url, array(
        'Authorization' => $auth,
        'Content-Length' => strlen($body)
    ), $body);
    $browser->send($request);

    With the above definition in mind, I agree that it makes sense to add this to the library as a sane default behavior for the future. Given that this library supports any (custom) request method and the definition mentions this behavior should trigger "when […] the request method defines a meaning for an enclosed payload body" I'm currently undecided if it makes more sense to go with a blacklist or whitelist for certain request methods. Perhaps anybody else has some insights into which request methods could be affected by this? (POST/PUT/PATCH likely are, GET/HEAD/TRACE likely not à la aio-libs/aiohttp#2167)

    Any input would be much appreciated 👍

  7. clue commented on Dec 16, 2018

    @clue
    Owner

    @CharlotteDunois Thank you for reporting! I've just filed #120 which should fix this for affected HTTP servers that seem to follow the older RFC 2616 semantics. Perhaps you can give it a try and see if fixes the issues you were seeing?

    In case anybody else stumbles over this in the future: The above behavior only triggers for an empty request body and only for POST, PUT and PATCH requests. If you are affected by this, keep in mind that this library honors any explicit headers, so you can specify this like this:

    $body = '';
    $request = new Request(
        'POST',
        $url,
        array(
            'Content-Length' => strlen($body)
        ),
        $body
    );
    $browser->send($request);
  8. changed the title [-]Empty body on requests with body results in non-compliance[/-] [+]Empty body on POST request not compatible with some legacy servers (411 Length Required)[/+] on Dec 17, 2018
  9. WyriHaximus commented on Dec 17, 2018

    @WyriHaximus
    Contributor

    🎉 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions